Towards Unconstrained Human-Object Interaction

Francesco Tonini, Alessandro Conti, Lorenzo Vaquero, Cigdem Beyan, Elisa Ricci
4/15/2026
cs.CV

No official code found. Best implementation: francescotonini/anyhoi (2 stars, MIT, last commit 2026-06-20).

Abstract

Human-Object Interaction (HOI) detection is a longstanding computer vision problem concerned with predicting the interaction between humans and objects. Current HOI models rely on a vocabulary of interactions at training and inference time, limiting their applicability to static environments. With the advent of Multimodal Large Language Models (MLLMs), it has become feasible to explore more flexible paradigms for interaction recognition. In this work, we revisit HOI detection through the lens of MLLMs and apply them to in-the-wild HOI detection. We define the Unconstrained HOI (U-HOI) task, a novel HOI domain that removes the requirement for a predefined list of interactions at both training and inference. We evaluate a range of MLLMs on this setting and introduce a pipeline that includes test-time inference and language-to-graph conversion to extract structured interactions from free-form text. Our findings highlight the limitations of current HOI detectors and the value of MLLMs for U-HOI. Code will be available at https://github.com/francescotonini/anyhoi

View on arXivView PDF

Code Implementations(1)

francescotonini/anyhoiCommunity25%
MIT

Official repo of the paper "Towards Unconstrained Human-Object Interaction" (FG 2026)

20Shell, PythonApr 14, 20264 months agoMIT

Similar Papers

Weakly-Supervised HOI Detection from Interaction Labels Only and Language/Vision-Language Priors

89% match2023

Harnessing Multimodal Large Language Models for Training-Free Human-Object Interaction Detection

88% match2026

A Review of Human-Object Interaction Detection

88% match2024

HOI-R1: Exploring the Potential of Multimodal Large Language Models for Human-Object Interaction Detection

88% match1 repo62025

Detecting Any Human-Object Interaction Relationship: Universal HOI Detector with Spatial Prompt Learning on Foundation Models

87% match1 repo282026

Enhancing HOI Detection with Contextual Cues from Large Vision-Language Models

87% match2023

Cite this paper

@article{tonini2026towards,
  title  = {Towards Unconstrained Human-Object Interaction},
  author = {Francesco Tonini and Alessandro Conti and Lorenzo Vaquero and Cigdem Beyan and Elisa Ricci},
  year   = {2026},
  eprint = {2604.14069},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2604.14069v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2604.14069.svg)](https://codeofpaper.com/paper/2604.14069)

Discussion