ProcessTBench: An LLM Plan Generation Dataset for Process Mining

Andrei Cosmin Redis, Mohammadreza Fani Sani, Bahram Zarrin, Andrea Burattin
9/13/2024
cs.LGcs.AIcs.ET

No official code found. Best implementation: microsoft/ProcessTBench (3 stars, MIT, last commit 2024-09-19).

Abstract

Large Language Models (LLMs) have shown significant promise in plan generation. Yet, existing datasets often lack the complexity needed for advanced tool use scenarios - such as handling paraphrased query statements, supporting multiple languages, and managing actions that can be done in parallel. These scenarios are crucial for evaluating the evolving capabilities of LLMs in real-world applications. Moreover, current datasets don't enable the study of LLMs from a process perspective, particularly in scenarios where understanding typical behaviors and challenges in executing the same process under different conditions or formulations is crucial. To address these gaps, we present the ProcessTBench synthetic dataset, an extension of the TaskBench dataset specifically designed to evaluate LLMs within a process mining framework.

View on arXivView PDF

Code Implementations(1)

microsoft/ProcessTBenchCommunity100%
MIT

Improving Plan Generation in Large Language Models with Process Mining

30Python, Jupyter NotebookAug 18, 20242 years agoMIT

Similar Papers

Evaluating the Ability of LLMs to Solve Semantics-Aware Process Mining Tasks

88% match1 repo32024

Skill Learning Using Process Mining for Large Language Model Plan Generation

87% match1 repo2024

PM-LLM-Benchmark: Evaluating Large Language Models on Process Mining Tasks

87% match1 repo222024

Domain Adaptation of LLMs for Process Data

86% match1 repo2025

LLMs that Understand Processes: Instruction-tuning for Semantics-Aware Process Mining

85% match2025

Re-Thinking Process Mining in the AI-Based Agents Era

85% match2024

Cite this paper

@article{redis2024processtbench,
  title  = {ProcessTBench: An LLM Plan Generation Dataset for Process Mining},
  author = {Andrei Cosmin Redis and Mohammadreza Fani Sani and Bahram Zarrin and Andrea Burattin},
  year   = {2024},
  eprint = {2409.09191},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2409.09191v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2409.09191.svg)](https://codeofpaper.com/paper/2409.09191)

Discussion