LLM-Guided Agentic Object Detection for Open-World Understanding

Furkan Mumcu, Michael J. Jones, Anoop Cherian, Yasin Yilmaz
7/14/2025
cs.CV

No official code found. Best implementation: furkanmumcu/LAOD (5 stars, last commit 2025-07-30).

Abstract

Object detection traditionally relies on fixed category sets, requiring costly re-training to handle novel objects. While Open-World and Open-Vocabulary Object Detection (OWOD and OVOD) improve flexibility, OWOD lacks semantic labels for unknowns, and OVOD depends on user prompts, limiting autonomy. We propose an LLM-guided agentic object detection (LAOD) framework that enables fully label-free, zero-shot detection by prompting a Large Language Model (LLM) to generate scene-specific object names. These are passed to an open-vocabulary detector for localization, allowing the system to adapt its goals dynamically. We introduce two new metrics, Class-Agnostic Average Precision (CAAP) and Semantic Naming Average Precision (SNAP), to separately evaluate localization and naming. Experiments on LVIS, COCO, and COCO-OOD validate our approach, showing strong performance in detecting and naming novel objects. Our method offers enhanced autonomy and adaptability for open-world understanding.

View on arXivView PDF

Code Implementations(1)

furkanmumcu/LAODCommunity66%

LLM-Guided Agentic Object Detection for Open-World Understanding

50PythonJul 8, 20251 years ago

Similar Papers

Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection

86% match1 repo2242024

Training an Open-Vocabulary Monocular 3D Object Detection Model without 3D Data

85% match2024

From Open Vocabulary to Open World: Teaching Vision Language Models to Detect Novel Objects

85% match1 repo392024

Training-free Boost for Open-Vocabulary Object Detection with Confidence Aggregation

84% match1 repo132024

Hierarchical Cross-Modal Alignment for Open-Vocabulary 3D Object Detection

84% match1 repo52025

Generative Region-Language Pretraining for Open-Ended Object Detection

84% match1 repo1962026

Cite this paper

@article{mumcu2025llmguided,
  title  = {LLM-Guided Agentic Object Detection for Open-World Understanding},
  author = {Furkan Mumcu and Michael J. Jones and Anoop Cherian and Yasin Yilmaz},
  year   = {2025},
  eprint = {2507.10844},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2507.10844v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2507.10844.svg)](https://codeofpaper.com/paper/2507.10844)

Discussion