From Simple to Professional: A Combinatorial Controllable Image Captioning Agent

Xinran Wang, Muxi Diao, Baoteng Li, Haiwen Zhang, Kongming Liang +1 more
12/15/2024
cs.CVcs.AI

Official code: xin-ran-w/CapAgent (7 stars, Apache-2.0, last commit 2024-12-22).

Abstract

The Controllable Image Captioning Agent (CapAgent) is an innovative system designed to bridge the gap between user simplicity and professional-level outputs in image captioning tasks. CapAgent automatically transforms user-provided simple instructions into detailed, professional instructions, enabling precise and context-aware caption generation. By leveraging multimodal large language models (MLLMs) and external tools such as object detection tool and search engines, the system ensures that captions adhere to specified guidelines, including sentiment, keywords, focus, and formatting. CapAgent transparently controls each step of the captioning process, and showcases its reasoning and tool usage at every step, fostering user trust and engagement. The project code is available at https://github.com/xin-ran-w/CapAgent.

View on arXivView PDF

Code Implementations(1)

xin-ran-w/CapAgentOfficial100%
Apache-2.0

From Simple to Professional - A Combinatorial Controllable Image Captioning Agent

72Shell, PythonNov 15, 20241 years agoApache-2.0
agentimage-captioninginstruction

Similar Papers

ViTOC: Vision Transformer and Object-aware Captioner

87% match2024

SCRA-VQA: Summarized Caption-Rerank for Augmented Large Language Models in Visual Question Answering

86% match1 repo112025

Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation

86% match2026

MoColl: Agent-Based Specific and General Model Collaboration for Image Captioning

86% match2025

Explainable Image Captioning using CNN- CNN architecture and Hierarchical Attention

85% match2024

ITIScore: An Image-to-Text-to-Image Rating Framework for the Image Captioning Ability of MLLMs

85% match2026

Cite this paper

@article{wang2024from,
  title  = {From Simple to Professional: A Combinatorial Controllable Image Captioning Agent},
  author = {Xinran Wang and Muxi Diao and Baoteng Li and Haiwen Zhang and Kongming Liang and Zhanyu Ma},
  year   = {2024},
  eprint = {2412.11025},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.11025v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.11025.svg)](https://codeofpaper.com/paper/2412.11025)

Discussion