An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano +2 more
8/2/2022
cs.CVcs.CLcs.GR

No official code found. Best implementation: JoePenna/Dreambooth-Stable-Diffusion (3,211 stars, MIT, last commit 2024-01-08). 1 more below.

Abstract

Text-to-image models offer unprecedented freedom to guide creation through natural language. Yet, it is unclear how such freedom can be exercised to generate images of specific unique concepts, modify their appearance, or compose them in new roles and novel scenes. In other words, we ask: how can we use language-guided models to turn our cat into a painting, or imagine a new product based on our favorite toy? Here we present a simple approach that allows such creative freedom. Using only 3-5 images of a user-provided concept, like an object or a style, we learn to represent it through new "words" in the embedding space of a frozen text-to-image model. These "words" can be composed into natural language sentences, guiding personalized creation in an intuitive way. Notably, we find evidence that a single word embedding is sufficient for capturing unique and varied concepts. We compare our approach to a wide range of baselines, and demonstrate that it can more faithfully portray the concepts across a range of applications and tasks. Our code, data and new words will be available at: https://textual-inversion.github.io

View on arXivView PDF

Code Implementations(2)

rinongal/textual_inversionCommunity100%
MIT
3,055284Shell, PythonAug 2, 20223 years agoMIT
JoePenna/Dreambooth-Stable-DiffusionCommunity64%
MIT

Implementation of Dreambooth (https://arxiv.org/abs/2208.12242) by way of Textual Inversion (https://arxiv.org/abs/2208.01618) for Stable Diffusion (https://arxiv.org/abs/2112.10752). Tweaks focused on training faces, objects, and styles.

3,211527Shell, PythonSep 17, 20222 years agoMIT
aiartificial-intelligenceimage-generationimg2imglatent-diffusion+4 more

Similar Papers

ConceptLab: Creative Concept Generation using VLM-Guided Diffusion Prior Constraints

87% match1 repo2562023

Learning to Imagine: Visually-Augmented Natural Language Generation

86% match1 repo52023

Trash to Treasure: Using text-to-image models to inform the design of physical artefacts

85% match2023

VLM-Guided Adaptive Negative Prompting for Creative Generation

85% match2025

MyVLM: Personalizing VLMs for User-Specific Queries

85% match1 repo1882024

A Creative Agent is Worth a 64-Token Template

84% match2026

Cite this paper

@article{gal2022image,
  title  = {An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion},
  author = {Rinon Gal and Yuval Alaluf and Yuval Atzmon and Or Patashnik and Amit H. Bermano and Gal Chechik and Daniel Cohen-Or},
  year   = {2022},
  eprint = {2208.01618},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2208.01618v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2208.01618.svg)](https://codeofpaper.com/paper/2208.01618)

Discussion