Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs

Dingjie Song, Wenjun Wang, Shunian Chen, Xidong Wang, Michael Guan +1 more
9/17/2024
cs.CLcs.AIcs.CV

No official code found. Best implementation: FreedomIntelligence/TRIM (22 stars, Apache-2.0, last commit 2026-01-11).

Abstract

The rapid advancement of Multimodal Large Language Models (MLLMs) has led to remarkable performances across various domains. However, this progress is accompanied by a substantial surge in the resource consumption of these models. We address this pressing issue by introducing a new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance. Inspired by human attention patterns in Visual Question Answering (VQA) tasks, TRIM presents a fresh perspective on the selection and reduction of image tokens. The TRIM method has been extensively tested across 12 datasets, and the results demonstrate a significant reduction in computational overhead while maintaining a consistent level of performance. This research marks a critical stride in efficient MLLM development, promoting greater accessibility and sustainability of high-performing models.

View on arXivView PDF

Code Implementations(1)

FreedomIntelligence/TRIMCommunity100%
Apache-2.0

We introduce new approach, Token Reduction using CLIP Metric (TRIM), aimed at improving the efficiency of MLLMs without sacrificing their performance.

221CSS, HTMLSep 23, 20249 months agoApache-2.0
llmmllmmultimodalvision-and-languagevision-language-model+1 more

Similar Papers

PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models

89% match2024

LFTR: Learning-Free Token Reduction for Multimodal Large Language Models

88% match2025

LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models

88% match1 repo1772024

LEO-MINI: An Efficient Multimodal Large Language Model using Conditional Token Reduction and Mixture of Multi-Modal Experts

87% match2025

Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models

86% match1 repo702024

Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs

86% match1 repo1082025

Cite this paper

@article{song2024less,
  title  = {Less is More: A Simple yet Effective Token Reduction Method for Efficient Multi-modal LLMs},
  author = {Dingjie Song and Wenjun Wang and Shunian Chen and Xidong Wang and Michael Guan and Benyou Wang},
  year   = {2024},
  eprint = {2409.10994},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.10994v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.10994.svg)](https://codeofpaper.com/paper/2409.10994)

Discussion