MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference

Anurita Das
4/22/2026
cs.LG

No confident code implementation found for this paper.

Abstract

Deploying large language models to heterogeneous hardware is often constrained by memory, not compute. We introduce MCAP (Monte Carlo Activation Profiling), a load-time per-layer importance estimator that enables dynamic precision and memory placement decisions on the target device. MCAP produces a lightweight per-layer signal that drives both precision dispatch (W4A8 vs. W4A16) and residency tier (GPU, RAM, SSD), allowing a single set of weights to operate across diverse memory budgets. Our system, NVE, achieves 1.5-1.8x higher decode throughput than llama.cpp Q4_0 on NVIDIA T4 and enables models to run in memory regimes previously infeasible without modifying weights.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

GUIDE: A Global Unified Inference Engine for Deploying Large Language Models in Heterogeneous Environments

88% match2024

MobileLLM-Flash: Latency-Guided On-Device LLM Design for Industry Scale Deployment

86% match2026

HBQ: Hierarchical Scaling Block Quantization with Hardware-Efficiency-Aware Design for Accurate LLM Inference

85% match2026

Replay the Curvature: Accurate and Scalable NVFP4 Quantization for Large Language Model Inference

85% match2026

Characterizing High Bandwidth Flash for LLM Serving

85% match2026

MoE-Lens: Towards the Hardware Limit of High-Throughput MoE LLM Serving Under Resource Constraints

85% match2025

Cite this paper

@article{das2026mcap,
  title  = {MCAP: Deployment-Time Layer Profiling for Memory-Constrained LLM Inference},
  author = {Anurita Das},
  year   = {2026},
  eprint = {2604.21026},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2604.21026v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2604.21026.svg)](https://codeofpaper.com/paper/2604.21026)

Discussion