-
Notifications
You must be signed in to change notification settings - Fork 237
Add multimodal flow matching example #83
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
Thanks @amorehead! Can you please give a more detailed explanation on the multimodal model? |
Hi, @itaigat. The multimodal model consists of similar MLPs as used for the discrete and continuous (checkerboard) flow matching examples. However, now they are used for generating both input and output embeddings of each respective modality, and they each share a common |
Improve performance of multimodal model by integrating all modalities into a single model and solver
@itaigat, I've added a new |
This reverts commit bed9ec6.
…for discrete sampling more robust; initialize solver once; and assert for all continuous floating-point types
flow_matching
package's components.flow_matching/utils/multimodal.py
module andflow_matching/solver/multimodal_solver.py
are introduced to construct instances of the new multimodalFlow
class and sample from it.