3D VAE finetune

[Simona0212]

我们的科研项目想要在3D VAE的基础上进行改动和微调。我们尝试在原有模块的训练代码上进行修改，发现有点困难。
请问能否开源3D VAE模块的训练代码和配置文件？
感谢CogVideoX团队！

[Simona0212]

I am going to modify and finetune the 3D VAE for our research project. We have tried to modify the training code of the original module and found it a little difficult. I would highly appreciate if you could provide the code and configuration files required to train and finetune the 3D VAE module (for video encoding and video decoding). We look forward to receiving feedback from your team. Thank you very much!

[zRzRzRzRzRzRzR]

Fine-tuning the VAE alone doesn’t seem very meaningful. If your goal is to fine-tune in conjunction with Transformers while reducing memory usage in the VAE encoder, we will continue working on this optimization. You can check for updates here.
#194 huggingface/diffusers#9302

[serend1p1ty]

@Simona0212 I recommend you use Open-Sora-Plan's code. I have trained a stronger VAE than CogVideoX using Open-Sora-Plan's codebase, along with many tricks.

[vanche]

@serend1p1ty I'm really interested in the tricks you've used and how much you've improved the VAE's performance. Could you share more details about the techniques you applied and the results you achieved?

[serend1p1ty]

@vanche Sorry I cannot share more tech detail. Here is my results.

There is still a lot of room for improvement (loss has not converged).

[vanche]

@serend1p1ty Thank you for sharing the performance results. Do you plan to publish a paper or technical report?

[serend1p1ty]

@vanche Basically, it's just some increment improvements, and unable to support a paper.

[derekcheng08]

@serend1p1ty - very interesting, thanks for sharing. Did you then need to fine-tune CogVideoX to work with your VAE?