TVLT: Textless Vision-Language Transformer

Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal
9/28/2022
cs.CVcs.AIcs.CL

Official code: zinengtang/TVLT (126 stars, MIT, last commit 2023-02-24).

Abstract

In this work, we present the Textless Vision-Language Transformer (TVLT), where homogeneous transformer blocks take raw visual and audio inputs for vision-and-language representation learning with minimal modality-specific design, and do not use text-specific modules such as tokenization or automatic speech recognition (ASR). TVLT is trained by reconstructing masked patches of continuous video frames and audio spectrograms (masked autoencoding) and contrastive modeling to align video and audio. TVLT attains performance comparable to its text-based counterpart on various multimodal tasks, such as visual question answering, image retrieval, video retrieval, and multimodal sentiment analysis, with 28x faster inference speed and only 1/3 of the parameters. Our findings suggest the possibility of learning compact and efficient visual-linguistic representations from low-level visual and audio signals without assuming the prior existence of text. Our code and checkpoints are available at: https://github.com/zinengtang/TVLT

View on arXivView PDF

Code Implementations(1)

zinengtang/TVLTOfficial100%
MIT

PyTorch code for “TVLT: Textless Vision-Language Transformer” (NeurIPS 2022 Oral)

12612Shell, PythonSep 28, 20223 years agoMIT
audiopretrainingtextlesstransformerstvlt+2 more

Similar Papers

Zorro: the masked multimodal transformer

88% match2023

MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers

87% match1 repo82024

Semantic Noise Reduction via Teacher-Guided Dual-Path Audio-Visual Representation Learning

85% match1 repo2026

FLAP: Fast Language-Audio Pre-training

85% match2023

AV-MaskEnhancer: Enhancing Video Representations through Audio-Visual Masked Autoencoder

85% match2023

VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation

85% match1 repo862023

Cite this paper

@article{tang2022tvlt,
  title  = {TVLT: Textless Vision-Language Transformer},
  author = {Zineng Tang and Jaemin Cho and Yixin Nie and Mohit Bansal},
  year   = {2022},
  eprint = {2209.14156},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2209.14156}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2209.14156.svg)](https://codeofpaper.com/paper/2209.14156)

Discussion