Kosmos-G: Generating Images in Context with Multimodal Large Language Models

Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen +1 more
10/1/2023

No official code found. Best implementation: microsoft/unilm (22,168 stars, MIT, last commit 2026-01-23). 2 more below.

Abstract

Recent advancements in subject-driven image generation have made significant strides. However, current methods still fall short in diverse application scenarios, as they require test-time tuning and cannot accept interleaved multi-image and text input. These limitations keep them far from the ultimate goal of"image as a foreign language in image generation."This paper presents Kosmos-G, a model that leverages the advanced multimodal perception capabilities of Multimodal Large Language Models (MLLMs) to tackle the aforementioned challenge. Our approach aligns the output space of MLLM with CLIP using the textual modality as an anchor and performs compositional instruction tuning on curated data. Kosmos-G demonstrates an impressive capability of zero-shot subject-driven generation with interleaved multi-image and text input. Notably, the score distillation instruction tuning requires no modifications to the image decoder. This allows for a seamless substitution of CLIP and effortless integration with a myriad of U-Net techniques ranging from fine-grained controls to personalized image decoder variants. We posit Kosmos-G as an initial attempt towards the goal of"image as a foreign language in image generation."The code can be found at https://aka.ms/Kosmos-G

View on arXivView PDF

Code Implementations(3)

microsoft/unilmCommunity100%
MIT

Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities

22,1682,703C, C++Jul 23, 20199 months agoMIT
beitbeit-3bitnetdeepnetdocument-ai+15 more
kyegomez/KosmosGCommunity68%
MIT

My implementation of the model KosmosG from "KOSMOS-G: Generating Images in Context with Multimodal Large Language Models"

131Python, MakefileOct 8, 20231 years agoMIT
attention-is-all-you-needattention-mechanismattention-mechanismscomputer-visionmultimodal+1 more
xichenpan/Kosmos-GCommunity60%

Official Implementation of ICLR'24: Kosmos-G: Generating Images in Context with Multimodal Large Language Models

744C, C++Apr 26, 20242 years ago

Similar Papers

Lavida-O: Elastic Large Masked Diffusion Models for Unified Multimodal Understanding and Generation

85% match2025

MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data

84% match2024

Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning

84% match1 repo3652023

Harmonizing Visual Text Comprehension and Generation

84% match1 repo1272024

Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining

84% match1 repo6482024

MMCORE: MultiModal COnnection with Representation Aligned Latent Embeddings

84% match1 repo2026

Cite this paper

@article{pan2023kosmosg,
  title  = {Kosmos-G: Generating Images in Context with Multimodal Large Language Models},
  author = {Xichen Pan and Li Dong and Shaohan Huang and Zhiliang Peng and Wenhu Chen and Furu Wei},
  year   = {2023},
  eprint = {2310.02992},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/pdf/2310.02992},
  journal = {ICLR 2023 2023}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2310.02992.svg)](https://codeofpaper.com/paper/2310.02992)

Discussion