Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu +7 more
4/13/2020
cs.CVcs.CLcs.IR

Official code: microsoft/Oscar (1,054 stars, MIT, last commit 2023-08-28).

Abstract

Large-scale pre-training methods of learning cross-modal representations on image-text pairs are becoming popular for vision-language tasks. While existing methods simply concatenate image region features and text features as input to the model to be pre-trained and use self-attention to learn image-text semantic alignments in a brute force manner, in this paper, we propose a new learning method Oscar (Object-Semantics Aligned Pre-training), which uses object tags detected in images as anchor points to significantly ease the learning of alignments. Our method is motivated by the observation that the salient objects in an image can be accurately detected, and are often mentioned in the paired text. We pre-train an Oscar model on the public corpus of 6.5 million text-image pairs, and fine-tune it on downstream tasks, creating new state-of-the-arts on six well-established vision-language understanding and generation tasks.

View on arXivView PDF

Code Implementations(1)

microsoft/OscarOfficial100%
MIT

Oscar and VinVL

1,054249PythonMay 14, 20203 years agoMIT
image-captioningimage-text-searchoscarpre-trainingvinvl+2 more

Similar Papers

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

93% match1 repo2082021

Revisiting Multimodal Representation in Contrastive Learning: From Patch and Token Embeddings to Finite Discrete Tokens

91% match1 repo602023

Enhancing Vision-Language Model with Unmasked Token Alignment

91% match1 repo112024

CMAL: A Novel Cross-Modal Associative Learning Framework for Vision-Language Pre-Training

90% match2024

Weakly Supervised Vision-and-Language Pre-training with Relative Representations

90% match2023

Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training

90% match2024

Cite this paper

@article{li2020oscar,
  title  = {Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks},
  author = {Xiujun Li and Xi Yin and Chunyuan Li and Pengchuan Zhang and Xiaowei Hu and Lei Zhang and Lijuan Wang and Houdong Hu and Li Dong and Furu Wei and Yejin Choi and Jianfeng Gao},
  year   = {2020},
  eprint = {2004.06165},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2004.06165}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2004.06165.svg)](https://codeofpaper.com/paper/2004.06165)

Discussion