Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: BFloat16 error during sample image generation #1275

Closed
1 task done
ashleykleynhans opened this issue Jun 10, 2023 · 5 comments
Closed
1 task done

[Bug]: BFloat16 error during sample image generation #1275

ashleykleynhans opened this issue Jun 10, 2023 · 5 comments
Labels
Stale

Comments

@ashleykleynhans
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16' error is returned during training which prevents sample image from being generated. The training continues, but no sample images are generated throughout.

Steps to reproduce the problem

  1. Create a new model.
  2. Configure the Dreambooth settings for the new model.
  3. Ensure that sample preview images is set.
  4. Start training.
  5. Observe BFloat16 error when sample images are being generated, resulting in no sample images.

Commit and libraries

Initializing Dreambooth
Dreambooth revision: dc413a1
[+] xformers version 0.0.20 installed.
[+] torch version 2.0.1+cu118 installed.
[+] torchvision version 0.15.2+cu118 installed.
[+] accelerate version 0.19.0 installed.
[+] diffusers version 0.16.1 installed.
[+] transformers version 4.29.2 installed.
[+] bitsandbytes version 0.35.4 installed.

Command Line Arguments

--listen \
  --no-hashing \
  --skip-python-version-check \
  --skip-torch-cuda-test \
  --enable-insecure-extension-access \
  --api \
  --skip-version-check \
  --skip-install

Console logs

Traceback (most recent call last): | 0/4 [00:00<?, ?it/s]
  File "/workspace/stable-diffusion-webui/extensions/sd_dreambooth_extension/dreambooth/train_dreambooth.py", line 1033, in save_weights
    s_image = s_pipeline(
  File "/workspace/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/workspace/venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 645, in __call__
    prompt_embeds = self._encode_prompt(
  File "/workspace/venv/lib/python3.10/site-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 357, in _encode_prompt
    prompt_embeds = self.text_encoder(
  File "/workspace/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workspace/venv/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/workspace/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 816, in forward
    return self.text_model(
  File "/workspace/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workspace/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 717, in forward
    causal_attention_mask = self._build_causal_attention_mask(
  File "/workspace/venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 760, in _build_causal_attention_mask
    mask.triu_(1)  # zero out the lower diagonal
RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'

Additional information

No response
@ashleykleynhans ashleykleynhans mentioned this issue Jun 10, 2023
Merged
4 tasks
@soyatlas
Copy link

Hi,
(running win10pro Powershell administrator)
I hope it helps, I get the same error but also the registration continues and ends with an error:

...
RuntimeError: "triu_tril_cuda_template" not implemented for 'BFloat16'
Traceback (most recent call last):
File "I:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\ui_functions.py", line 729, in start_training
result = main(class_gen_method=class_gen_method)
File "I:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1548, in main
return inner_loop()
File "I:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\memory.py", line 119, in decorator
return function(batch_size, grad_size, depth, *args, **kwargs)
File "I:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1502, in inner_loop
check_save(True)
File "I:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 794, in check_save
save_weights(
File "I:\stable-diffusion-webui\extensions\sd_dreambooth_extension\dreambooth\train_dreambooth.py", line 1069, in save_weights
shutil.rmtree(tmp_dir)
File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 749, in rmtree
return _rmtree_unsafe(path, onerror)
File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 614, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 619, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\xxxxx\AppData\Local\Programs\Python\Python310\lib\shutil.py", line 617, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 5] Access Denied: 'C:\Users\xxxxx\AppData\Local\Temp\tmprwspzx0b\unet\diffusion_pytorch_model.safetensors'

THANKS !

@github-actions
Copy link

This issue is stale because it has been open 5 days with no activity. Remove stale label or comment or this will be closed in 5 days

@github-actions github-actions bot added the Stale label Jun 30, 2023
@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Jul 5, 2023
@ashleykleynhans
Copy link
Contributor Author

This still needs to be merged into the main branch.

@FurkanGozukara
Copy link
Contributor

This still needs to be merged into the main branch.

please merge people having issues

@ashleykleynhans
Copy link
Contributor Author

Its merged into dev but not main.

@ashleykleynhans ashleykleynhans mentioned this issue Jul 8, 2023
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ashleykleynhans @FurkanGozukara @soyatlas