LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression

Shimon Murai, Heming Sun, Jiro Katto
11/20/2024
eess.IVcs.CV

Official code: tokkiwa/ImageTextCoding (26 stars, last commit 2025-03-12).

Abstract

Supported by powerful generative models, low-bitrate learned image compression (LIC) models utilizing perceptual metrics have become feasible. Some of the most advanced models achieve high compression rates and superior perceptual quality by using image captions as sub-information. This paper demonstrates that using a large multi-modal model (LMM), it is possible to generate captions and compress them within a single model. We also propose a novel semantic-perceptual-oriented fine-tuning method applicable to any LIC network, resulting in a 41.58\% improvement in LPIPS BD-rate compared to existing methods. Our implementation and pre-trained weights are available at https://github.com/tokkiwa/ImageTextCoding.

View on arXivView PDF

Code Implementations(1)

tokkiwa/ImageTextCodingOfficial100%

Official Repository for VCIP'24 paper "LMM-driven Image-Text Coding for Ultra Low-bitrate Learned Image Compression"

262Shell, PythonSep 28, 20241 years ago

Similar Papers

Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior

86% match1 repo142025

SLIM: Semantic-based Low-bitrate Image compression for Machines by leveraging diffusion

85% match2025

Beyond Hallucinations: A Multimodal-Guided Task-Aware Generative Image Compression for Ultra-Low Bitrate

85% match2025

Extremely low-bitrate Image Compression Semantically Disentangled by LMMs from a Human Perception Perspective

85% match2025

Decouple Distortion from Perception: Region Adaptive Diffusion for Extreme-low Bitrate Perception Image Compression

85% match2026

Compact Latent Representation for Image Compression (CLRIC)

85% match2025

Cite this paper

@article{murai2024lmmdriven,
  title  = {LMM-driven Semantic Image-Text Coding for Ultra Low-bitrate Learned Image Compression},
  author = {Shimon Murai and Heming Sun and Jiro Katto},
  year   = {2024},
  eprint = {2411.13033},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2411.13033v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2411.13033.svg)](https://codeofpaper.com/paper/2411.13033)

Discussion