Enabling Performant and Flexible Model-Internal Observability for LLM Inference

Nengneng Yu, Sixian Xiong, Yibo Zhao, Wei Wang, Zaoxing Liu
5/11/2026
cs.LGcs.AIcs.PF

Official code: ProjectDMX/DMI (21 stars, Apache-2.0, last commit 2026-05-13).

Abstract

Today's inference-time workloads increasingly depend on timely access to a model's internal states. We present DMI-Lib, a high-speed deep model inspector that treats internal observability as a first-class systems primitive, decoupling it from the inference hot path via an asynchronous observability substrate built from Ring^2, a GPU-CPU memory abstraction for capturing and staging tensors, and a policy-controlled host backend that exports them. DMI-Lib enables the placement of observation points across a rich space of internal signals and diverse inference backends while preserving serving optimizations and adhering to tight GPU memory budgets. Our experiments demonstrate that DMI-Lib incurs only 0.4%--6.8% overhead in offline batch inference and an average of 6% in moderate online serving, reducing latency overhead by 2x-15x compared to existing baselines with similar observability features. DMI-Lib is open-sourced at https://github.com/ProjectDMX/DMI.

View on arXivView PDF

Code Implementations(1)

ProjectDMX/DMIOfficial100%
Apache-2.0

DMI: A decoupled, asynchronous observation substrate for high-speed LLM inference.

213C, C++Sep 8, 20255 months agoApache-2.0

Similar Papers

Dooly: Configuration-Agnostic, Redundancy-Aware Profiling for LLM Inference Simulation

81% match2026

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

80% match2 repos82026

DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference

80% match1 repo1602020

TPI-LLM: Serving 70B-scale LLMs Efficiently on Low-resource Edge Devices

79% match2024

BRUNO: A Deep Recurrent Model for Exchangeable Data

79% match1 repo352018

STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining

79% match2022

Cite this paper

@article{yu2026enabling,
  title  = {Enabling Performant and Flexible Model-Internal Observability for LLM Inference},
  author = {Nengneng Yu and Sixian Xiong and Yibo Zhao and Wei Wang and Zaoxing Liu},
  year   = {2026},
  eprint = {2605.11093},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2605.11093v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2605.11093.svg)](https://codeofpaper.com/paper/2605.11093)

Discussion