Serving Agentic Workflows with a Physical-Plan Compiler and Adaptive Runtime

Abstract

Efficient serving of agentic workflows requires selecting each LLM node's model, verification policy, and backend to balance output quality, latency, and throughput. These assignments must also adapt to changes in serving load. Existing approaches address parts of this problem through model routing, verifier placement, and backend scheduling. However, independent optimization overlooks their dependencies: model and backend choices determine verification cost, while verification changes the quality-cost trade-off among models. Ignoring these interactions can waste serving resources and degrade workflow performance. To address this problem, we propose \textbf{Dyserve}, which provides the missing workflow physical-planning layer between orchestration and model serving through compiler-runtime co-design. Our design is guided by three observations: planning headroom is request-dependent; node vulnerability, the impact of local errors on final correctness, depends on position and task type; and serving load changes the cost of a plan during execution. For each request, its profile-guided compiler jointly selects node implementations and prepares pressure-specialized variants for the materialized workflow. The runtime selects variants using live backend pressure and updates only undispatched assignments, without invoking the optimizer on the load-change path. Across four agentic workloads, Dyserve improves accuracy by \textbf{3-9} percentage points with \textbf{1.1-6.8}$\times$ mean-latency speedups over the highest-accuracy evaluated baseline for each workload. On a burst trace, variant switching raises the fraction of correct, on-time completions from \textbf{18.1\%} to \textbf{67.2\%} relative to admission-only execution.