Cycle Text-To-Image GAN with BERT

Trevor Tsue, Samir Sen, Jason Li
3/26/2020
cs.CV

Official code: suetAndTie/cycle-image-gan (28 stars, MIT, last commit 2020-03-30).

Abstract

We explore novel approaches to the task of image generation from their respective captions, building on state-of-the-art GAN architectures. Particularly, we baseline our models with the Attention-based GANs that learn attention mappings from words to image features. To better capture the features of the descriptions, we then built a novel cyclic design that learns an inverse function to maps the image back to original caption. Additionally, we incorporated recently developed BERT pretrained word embeddings as our initial text featurizer and observe a noticeable improvement in qualitative and quantitative performance compared to the Attention GAN baseline.

View on arXivView PDF

Code Implementations(1)

suetAndTie/cycle-image-ganOfficial100%
MIT
2812PythonMay 27, 20196 years agoMIT

Similar Papers

Retrieval-Augmented Transformer for Image Captioning

88% match2022

Synth$^2$: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings

87% match2024

With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning

87% match1 repo192023

CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

86% match1 repo402024

ClipCap: CLIP Prefix for Image Captioning

86% match1 repo1.4k2021

VAR-CLIP: Text-to-Image Generator with Visual Auto-Regressive Modeling

85% match1 repo1462024

Cite this paper

@article{tsue2020cycle,
  title  = {Cycle Text-To-Image GAN with BERT},
  author = {Trevor Tsue and Samir Sen and Jason Li},
  year   = {2020},
  eprint = {2003.12137},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2003.12137}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2003.12137.svg)](https://codeofpaper.com/paper/2003.12137)

Discussion