AgentSims: An Open-Source Sandbox for Large Language Model Evaluation

Jiaju Lin, Haoran Zhao, Aochi Zhang, Yiting Wu, Huqiuyue Ping +1 more
8/8/2023
cs.AI

No official code found. Best implementation: py499372727/AgentSims (960 stars, MIT, last commit 2023-11-18).

Abstract

With ChatGPT-like large language models (LLM) prevailing in the community, how to evaluate the ability of LLMs is an open question. Existing evaluation methods suffer from following shortcomings: (1) constrained evaluation abilities, (2) vulnerable benchmarks, (3) unobjective metrics. We suggest that task-based evaluation, where LLM agents complete tasks in a simulated environment, is a one-for-all solution to solve above problems. We present AgentSims, an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in. Researchers can build their evaluation tasks by adding agents and buildings on an interactive GUI or deploy and test new support mechanisms, i.e. memory, planning and tool-use systems, by a few lines of codes. Our demo is available at https://agentsims.com .

View on arXivView PDF

Code Implementations(1)

py499372727/AgentSimsCommunity100%
MIT

AgentSims is an easy-to-use infrastructure for researchers from all disciplines to test the specific capacities they are interested in.

960120CSS, HTMLAug 7, 20232 years agoMIT

Similar Papers

State of What Art? A Call for Multi-Prompt LLM Evaluation

89% match1 repo152023

Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization

88% match1 repo192025

SEAL: Suite for Evaluating API-use of LLMs

88% match2024

AgentGym2: Benchmarking Large Language Model Agents in De-Idealized Real-World Environments

88% match2026

Reliable and Developer-Aligned Evaluation of Agents for Software Engineering

88% match2026

T1-Bench: Benchmarking Multi-Scenario Agents in Real-World Domains

88% match1 repo1.9k2026

Cite this paper

@article{lin2023agentsims,
  title  = {AgentSims: An Open-Source Sandbox for Large Language Model Evaluation},
  author = {Jiaju Lin and Haoran Zhao and Aochi Zhang and Yiting Wu and Huqiuyue Ping and Qin Chen},
  year   = {2023},
  eprint = {2308.04026},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2308.04026v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2308.04026.svg)](https://codeofpaper.com/paper/2308.04026)

Discussion