ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning

Xiao Xu, Bei Li, Chenfei Wu, Shao-Yen Tseng, Anahita Bhiwandiwalla +4 more
5/31/2023
cs.CVcs.CLcs.LG

Official code: LooperXX/ManagerTower (12 stars, MIT, last commit 2025-08-23).

Abstract

Two-Tower Vision-Language (VL) models have shown promising improvements on various downstream VL tasks. Although the most advanced work improves performance by building bridges between encoders, it suffers from ineffective layer-by-layer utilization of uni-modal representations and cannot flexibly exploit different levels of uni-modal semantic knowledge. In this work, we propose ManagerTower, a novel VL model architecture that gathers and combines the insights of pre-trained uni-modal experts at different levels. The managers introduced in each cross-modal layer can adaptively aggregate uni-modal semantic knowledge to facilitate more comprehensive cross-modal alignment and fusion. ManagerTower outperforms previous strong baselines both with and without Vision-Language Pre-training (VLP). With only 4M VLP data, ManagerTower achieves superior performances on various downstream VL tasks, especially 79.15% accuracy on VQAv2 Test-Std, 86.56% IR@1 and 95.64% TR@1 on Flickr30K. Code and checkpoints are available at https://github.com/LooperXX/ManagerTower.

View on arXivView PDF

Code Implementations(1)

LooperXX/ManagerTowerOfficial100%
MIT

Code for ACL 2023 Oral Paper: ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning

123Shell, PythonMay 28, 20231 years agoMIT
multi-modal-learningvision-languagevision-language-learningvision-language-pretraining

Similar Papers

Manager: Aggregating Insights from Unimodal Experts in Two-Tower VLMs and MLLMs

89% match1 repo122025

MMRL: Multi-Modal Representation Learning for Vision-Language Models

85% match1 repo1152025

UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling

84% match2 repos772023

TowerVision: Understanding and Improving Multilinguality in Vision-Language Models

83% match2025

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

83% match1 repo2.0k2024

MaPLe: Multi-modal Prompt Learning

83% match1 repo8242022

Cite this paper

@article{xu2023managertower,
  title  = {ManagerTower: Aggregating the Insights of Uni-Modal Experts for Vision-Language Representation Learning},
  author = {Xiao Xu and Bei Li and Chenfei Wu and Shao-Yen Tseng and Anahita Bhiwandiwalla and Shachar Rosenman and Vasudev Lal and Wanxiang Che and Nan Duan},
  year   = {2023},
  eprint = {2306.00103},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2306.00103v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2306.00103.svg)](https://codeofpaper.com/paper/2306.00103)

Discussion