Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists

Michał Pietruszka, Łukasz Borchmann, Aleksander Jędrosz, Paweł Morawiecki
10/30/2024
cs.CL

No official code found. Best implementation: FeatEng/FeatEng (22 stars, Apache-2.0, last commit 2024-10-28).

Abstract

We present a benchmark for large language models designed to tackle one of the most knowledge-intensive tasks in data science: writing feature engineering code, which requires domain knowledge in addition to a deep understanding of the underlying problem and data structure. The model is provided with a dataset description in a prompt and asked to generate code transforming it. The evaluation score is derived from the improvement achieved by an XGBoost model fit on the modified dataset compared to the original data. By an extensive evaluation of state-of-the-art models and comparison to well-established benchmarks, we demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs, in contrast to the existing methods.

View on arXivView PDF

Code Implementations(1)

FeatEng/FeatEngCommunity100%
Apache-2.0

The benchmark for LLMs designed to tackle one of the most knowledge-intensive tasks in data science: writing feature engineering code, which requires domain knowledge in addition to a deep understanding of the underlying problem and data structure.

223PythonOct 21, 20241 years agoApache-2.0

Similar Papers

ELF-Gym: Evaluating Large Language Models Generated Features for Tabular Prediction

86% match1 repo62024

Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks

86% match2025

Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration

86% match1 repo62024

DataSciBench: An LLM Agent Benchmark for Data Science

85% match2 repos662025

The LLM Data Auditor: A Metric-oriented Survey on Quality and Trustworthiness in Evaluating Synthetic Data

85% match2026

Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering

85% match2026

Cite this paper

@article{pietruszka2024models,
  title  = {Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists},
  author = {Michał Pietruszka and Łukasz Borchmann and Aleksander Jędrosz and Paweł Morawiecki},
  year   = {2024},
  eprint = {2410.23331},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.23331v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.23331.svg)](https://codeofpaper.com/paper/2410.23331)

Discussion