Large Language Model Routing with Benchmark Datasets

Tal Shnitzer, Anthony Ou, Mírian Silva, Kate Soule, Yuekai Sun +3 more
9/27/2023
cs.CLcs.LG

No official code found. Best implementation: UMich-FATML/llm-routing (0 stars, last commit 2025-06-04).

Abstract

There is a rapidly growing number of open-source Large Language Models (LLMs) and benchmark datasets to compare them. While some models dominate these benchmarks, no single model typically achieves the best accuracy in all tasks and use cases. In this work, we address the challenge of selecting the best LLM out of a collection of models for new tasks. We propose a new formulation for the problem, in which benchmark datasets are repurposed to learn a "router" model for this LLM selection, and we show that this problem can be reduced to a collection of binary classification tasks. We demonstrate the utility and limitations of learning model routers from various benchmark datasets, where we consistently improve performance upon using any single model for all tasks.

View on arXivView PDF

Code Implementations(1)

UMich-FATML/llm-routingCommunity63%

code for "Large Language Model Routing with Benchmark Datasets" by Shnitzer et al.

00PythonJun 4, 20251 years ago

Similar Papers

RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs

89% match1 repo1212025

RouteLLM: Learning to Route LLMs with Preference Data

88% match1 repo5.2k2024

LLMRank: Understanding LLM Strengths for Model Routing

88% match2025

RouterBench: A Benchmark for Multi-LLM Routing System

88% match1 repo1792024

Lookahead Routing for Large Language Models

87% match1 repo92025

Meta-Router: Bridging Gold-standard and Preference-based Evaluations in Large Language Model Routing

86% match2025

Cite this paper

@article{shnitzer2023large,
  title  = {Large Language Model Routing with Benchmark Datasets},
  author = {Tal Shnitzer and Anthony Ou and Mírian Silva and Kate Soule and Yuekai Sun and Justin Solomon and Neil Thompson and Mikhail Yurochkin},
  year   = {2023},
  eprint = {2309.15789},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.15789v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.15789.svg)](https://codeofpaper.com/paper/2309.15789)

Discussion