Parser Transitions Already Reveal Structural Responsibility: KV Persistence for Structured Generation

Abstract

LLM agents use structured generation for JSON, SQL, and function calls. Constrained decoding ensures valid formats, but KV compression can discard history needed for correct function and argument selection. Structural responsibility can diverge from model saliency: importance scores can overlook generated tokens' structural roles. Parser transitions already computed for legality encode these roles, including required arguments and structural boundaries, providing a free retention signal. We introduce PASK (Parser-Aware Structural KV Persistence), which reuses this signal for KV persistence by layer group without online importance estimation. Task-error sensitivity sets protection floors, while attention-output distortion ranks upgrades under the remaining budget. Offline calibration produces a frozen table for structural lookup and budget accounting. PASK outperforms budget-controlled baselines on function calling and text-to-SQL, improving BFCL accuracy by up to 8.67 pp over the strongest comparator. Against Full KV, it cuts peak GPU memory by 47.5 percent and achieves up to 3.3 times TPOT speedup.