Work plan and enhancement / 工作计划和用户诉求

[zRzRzRzRzRzRzR]

Tasks that have been identified and scheduled:

• Fine-tuning support for Diffusers version models
• Adaptation for CPU / NPU inference frameworks (e.g., Huawei, Intel devices)
• ComfyUI adaptation work and plugin support

已经明确并排期的任务：

• CogVideoX1.5 Diffusers版本适配
• CogVideoX-Factory等生态工具的更新

如果你有更多诉求，欢迎在这里提出

[zRzRzRzRzRzRzR]

#182 #191 #47 #84 have similar content, all looking forward to the open-source CogVideoX I2V model. We are conducting research and evaluation.

#111 #186 are similar, both expecting to provide fine-tuning work for VAE. We will try to place it in the fine-tuning version, and there is a probability that it can be adapted in diffusers fine-tuning, but it will consume relatively high resources

[rookiemann]

5b image to video please! I2V would be lovely!

[PR-Ryan]

The 3D VAE model consumes significantly more memory compared to diffusion models, which is severely limiting the batch size for fine-tuning. Any suggestions or optimizations to reduce memory usage would be greatly appreciated.

[zRzRzRzRzRzRzR]

The 3D VAE model consumes significantly more memory compared to diffusion models, which is severely limiting the batch size for fine-tuning. Any suggestions or optimizations to reduce memory usage would be greatly appreciated.

You make a very good point. We will work together with the Diffusers team to modify the fake quantization (fakecp) process in the VAE section to optimize it for lower memory usage. Please give us some time, as we will collaborate with the Diffusers team to develop a version of the model that is fine-tuned specifically for Diffusers, which is expected to save a significant amount of memory.

[KihongK]

First of all, thank you for your excellent work!

The dataset format used SAT way for fine tuning & full training be the same as the format that will be used for fine-tuning Diffusers version models?

+ wrong discord link

[zRzRzRzRzRzRzR]

We are currently completing several tasks

1. Adaptation work for the I2V model, expected to be open-sourced in September
   Detailed tutorial on model fine-tuning, expected to be completed in September

Work that has been completed

1. The model fine-tuned with SAT can be converted to a diffusers model and mounted directly. For specific usage, see here
2. The new Discord invitation link will be merged into the main branch today

[sincerity711]

When will vertical video generation be supported?

[zRzRzRzRzRzRzR]

When will vertical video generation be supported?

The current model cannot generate vertical videos, such as 480x720 resolution. We are working on fine-tuning to reach this capability, but it’s still in progress. Once we have any updates, we will share them as soon as possible.

[zRzRzRzRzRzRzR]

Two related issues working now:

1. Diffusers supports CogVideoX-I2V. This PR has been merged, but the patch has not yet been released.

2. Fine-tuning the Diffusers version of CogVideoX-2B T2V without using the SAT model, running directly under the Diffusers framework. It can run on a single A100 GPU.
   This PR is still under debugging. I am working with members of the Diffusers team to attempt fine-tuning the CogVideoX-5B and I2V models. We will provide a small dataset (a few dozen samples) for this PR, which is sufficient for LoRA fine-tuning CogVideoX.

Many thanks to @a-r-r-o-w for the help with these two tasks!