AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation

Miriam Horovicz
12/14/2025
cs.AIcs.CL

Official code: GenAISHAP/TokenSHAP (83 stars, MIT, last commit 2026-01-13).

Abstract

LLM agents that use external tools can solve complex tasks, but understanding which tools actually contributed to a response remains a blind spot. No existing XAI methods address tool-level explanations. We introduce AgentSHAP, the first framework for explaining tool importance in LLM agents. AgentSHAP is model-agnostic: it treats the agent as a black box and works with any LLM (GPT, Claude, Llama, etc.) without needing access to internal weights or gradients. Using Monte Carlo Shapley values, AgentSHAP tests how an agent responds with different tool subsets and computes fair importance scores based on game theory. Our contributions are: (1) the first explainability method for agent tool attribution, grounded in Shapley values from game theory; (2) Monte Carlo sampling that reduces cost from O(2n) to practical levels; and (3) comprehensive experiments on API-Bank showing that AgentSHAP produces consistent scores across runs, correctly identifies which tools matter, and distinguishes relevant from irrelevant tools. AgentSHAP joins TokenSHAP (for tokens) and PixelSHAP (for image regions) to complete a family of Shapley-based XAI tools for modern generative AI. Code: https://github.com/GenAISHAP/TokenSHAP.

View on arXivView PDF

Code Implementations(1)

GenAISHAP/TokenSHAPOfficial100%
MIT

A framework for interpreting modern AI systems using Monte Carlo Shapley value estimation. Model-agnostic explainability across language models, vision-language models, video understanding, and autonomous agents.

839TeX, ShellJun 20, 20249 months agoMIT

Similar Papers

TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation

84% match1 repo832024

Tool Preferences in Agentic LLMs are Unreliable

80% match1 repo42025

MCPVerse: An Expansive, Real-World Benchmark for Agentic Tool Use

80% match2025

Toward Scalable Verifiable Reward: Proxy State-Based Evaluation for Multi-turn Tool-Calling LLM Agents

80% match2026

AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition

80% match2026

Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol

79% match2026

Cite this paper

@article{horovicz2025agentshap,
  title  = {AgentSHAP: Interpreting LLM Agent Tool Importance with Monte Carlo Shapley Value Estimation},
  author = {Miriam Horovicz},
  year   = {2025},
  eprint = {2512.12597},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2512.12597v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2512.12597.svg)](https://codeofpaper.com/paper/2512.12597)

Discussion