EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs

Xiangyu Zhao, Bo Liu, Qijiong Liu, Guangyuan Shi, Xiao-Ming Wu
10/13/2023
cs.AIcs.CLcs.CV

No official code found. Best implementation: xiangyu-mm/EasyGen (73 stars, last commit 2024-11-21).

Abstract

We present EasyGen, an efficient model designed to enhance multimodal understanding and generation by harnessing the capabilities of diffusion models and large language models (LLMs), Unlike existing multimodal models that predominately depend on encoders like CLIP or ImageBind and need ample amounts of training data to bridge modalities,EasyGen leverages BiDiffuser,a bidirectional conditional diffusion model, to foster more efficient modality interactions. Easygen achieves text generation by training a projection layer linking BiDiffuser and an LLM, and facilities image generation by training an adapter to align the LLM's text space with the BiDiffuser's image space, Comprehensive quantitative and qualitative experiments show that EasyGen excels in data-efficient training, high-quality image generation, and extendibility, effectively addressing the challenges in multimodal generation. The source code is available at https://github.com/zxy556677/EasyGen.

View on arXivView PDF

Code Implementations(1)

xiangyu-mm/EasyGenCommunity100%

The official code for paper "EasyGen: Easing Multimodal Generation with a Bidirectional Conditional Diffusion Model and LLMs"

731Shell, PythonOct 8, 20231 years ago

Similar Papers

UNIMO-G: Unified Image Generation through Multimodal Conditional Diffusion

90% match2024

UniAlignment: Semantic Alignment for Unified Image Generation, Understanding, Manipulation and Perception

88% match1 repo62025

Efficient Multimodal Diffusion Models Using Joint Data Infilling with Partially Shared U-Net

88% match2023

LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Model

88% match1 repo7622026

LDGen: Enhancing Text-to-Image Synthesis via Large Language Model-Driven Language Representation

88% match1 repo382025

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

88% match4 repos1.4k2024

Cite this paper

@article{zhao2023easygen,
  title  = {EasyGen: Easing Multimodal Generation with BiDiffuser and LLMs},
  author = {Xiangyu Zhao and Bo Liu and Qijiong Liu and Guangyuan Shi and Xiao-Ming Wu},
  year   = {2023},
  eprint = {2310.08949},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2310.08949v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.08949.svg)](https://codeofpaper.com/paper/2310.08949)

Discussion