How well do Large Language Models perform in Arithmetic tasks?

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang
3/16/2023
cs.CLcs.AI

Official code: GanjinZero/math401-llm (57 stars, last commit 2023-04-17).

Abstract

Large language models have emerged abilities including chain-of-thought to answer math word problems step by step. Solving math word problems not only requires abilities to disassemble problems via chain-of-thought but also needs to calculate arithmetic expressions correctly for each step. To the best of our knowledge, there is no work to focus on evaluating the arithmetic ability of large language models. In this work, we propose an arithmetic dataset MATH 401 to test the latest large language models including GPT-4, ChatGPT, InstrctGPT, Galactica, and LLaMA with various arithmetic expressions and provide a detailed analysis of the ability of large language models. MATH 401 and evaluation codes are released at \url{https://github.com/GanjinZero/math401-llm}.

View on arXivView PDF

Code Implementations(1)

GanjinZero/math401-llmOfficial100%

Source codes and datasets for How well do Large Language Models perform in Arithmetic tasks?

572Mar 10, 20233 years ago

Similar Papers

Calc-X and Calcformers: Empowering Arithmetical Chain-of-Thought through Interaction with Symbolic Systems

89% match1 repo92023

ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoning

87% match2024

MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark

87% match1 repo1162024

Mathematical Reasoning in Large Language Models: Assessing Logical and Arithmetic Errors across Wide Numerical Ranges

87% match1 repo32025

Mathify: Evaluating Large Language Models on Mathematical Problem Solving Tasks

87% match1 repo82024

Evaluating Mathematical Reasoning of Large Language Models: A Focus on Error Identification and Correction

87% match1 repo92024

Cite this paper

@article{yuan2023well,
  title  = {How well do Large Language Models perform in Arithmetic tasks?},
  author = {Zheng Yuan and Hongyi Yuan and Chuanqi Tan and Wei Wang and Songfang Huang},
  year   = {2023},
  eprint = {2304.02015},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2304.02015v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2304.02015.svg)](https://codeofpaper.com/paper/2304.02015)

Discussion