fix: error on device for lpw_stable_diffusion_xl pipeline if pipe.enable_sequential_cpu_offload() enabled

[VicGrygorchyk]

What does this PR do?

This PR fixes the bug NotImplementedError: Cannot copy out of meta tensor; no data!, which occurs if enable_sequential_cpu_offload() method is invoked on the pipeline

Traceback (most recent call last):
  ...in get_weighted_text_embeddings_sdxl
    prompt_embeds_1 = pipe.text_encoder(token_tensor.to(device), output_hidden_states=True)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 822, in forward
    return self.text_model(
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 730, in forward
    hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "venv/lib/python3.10/site-packages/accelerate/hooks.py", line 160, in new_forward
    args, kwargs = module._hf_hook.pre_forward(module, *args, **kwargs)
  File "venv/lib/python3.10/site-packages/accelerate/hooks.py", line 290, in pre_forward
    return send_to_device(args, self.execution_device), send_to_device(
  File "venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 160, in send_to_device
    {
  File "venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 161, in <dictcomp>
    k: t if k in skip_keys else send_to_device(t, device, non_blocking=non_blocking, skip_keys=skip_keys)
  File "venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 167, in send_to_device
    return tensor.to(device, non_blocking=non_blocking)
NotImplementedError: Cannot copy out of meta tensor; no data!

What does cause the bug?

The get_weighted_text_embeddings_sdxl method gets device from pipe.device instead of pipe._execution_device.

How to recreate the issue

pipe: DiffusionPipeline = DiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0",
   custom_pipeline="lpw_stable_diffusion_xl"
    torch_dtype=torch.float16,
    use_safetensors=True,
)

pipe.enable_sequential_cpu_offload()

pipe.enable_vae_slicing()
pipe.enable_vae_tiling()

prompt = "photo of a cute (white) cat running on the grass"*20
prompt2 = "chasing (birds:1.5)"*20
prompt = f"{prompt},{prompt2}"
neg_prompt = "blur, low quality, carton, animate"

images = pipe(
    prompt=prompt, 
    negative_prompt=neg_prompt 
).images[0]

Before submitting

• Did you read the contributor guideline?
• Did you read our philosophy doc (important for complex PRs)?

[VicGrygorchyk]

@xhinker Please, review

[xhinkerzhu]

@xhinker Please, review

The code is good, thank you!

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.