Themisto: Jupyter-Based Runtime Benchmark

Konstantin Grotov, Sergey Titov
4/16/2025
cs.SEcs.AIcs.LG

No confident code implementation found for this paper.

Abstract

In this work, we present a benchmark that consists of Jupyter notebooks development trajectories and allows measuring how large language models (LLMs) can leverage runtime information for predicting code output and code generation. We demonstrate that the current generation of LLMs performs poorly on these tasks and argue that there exists a significantly understudied domain in the development of code-based models, which involves incorporating the runtime context.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Suggesting Code Edits in Interactive Machine Learning Notebooks Using Large Language Models

82% match2025

LLM Benchmarking with LLaMA2: Evaluating Code Development Performance Across Multiple Programming Languages

82% match1 repo12025

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

81% match1 repo432024

Evaluating the Limitations of Local LLMs in Solving Complex Programming Challenges

81% match2025

EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems

81% match2026

Evaluating Quantized Large Language Models for Code Generation on Low-Resource Language Benchmarks

81% match1 repo12024

Cite this paper

@article{grotov2025themisto,
  title  = {Themisto: Jupyter-Based Runtime Benchmark},
  author = {Konstantin Grotov and Sergey Titov},
  year   = {2025},
  eprint = {2504.12365},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2504.12365v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2504.12365.svg)](https://codeofpaper.com/paper/2504.12365)

Discussion