Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition

Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Henley +2 more
7/2/2024
cs.HCcs.AI

No confident code implementation found for this paper.

Abstract

LLM-powered tools like ChatGPT Data Analysis, have the potential to help users tackle the challenging task of data analysis programming, which requires expertise in data processing, programming, and statistics. However, our formative study (n=15) uncovered serious challenges in verifying AI-generated results and steering the AI (i.e., guiding the AI system to produce the desired output). We developed two contrasting approaches to address these challenges. The first (Stepwise) decomposes the problem into step-by-step subgoals with pairs of editable assumptions and code until task completion, while the second (Phasewise) decomposes the entire problem into three editable, logical phases: structured input/output assumptions, execution plan, and code. A controlled, within-subjects experiment (n=18) compared these systems against a conversational baseline. Users reported significantly greater control with the Stepwise and Phasewise systems, and found intervention, correction, and verification easier, compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

LLM4DS: Evaluating Large Language Models for Data Science Code Generation

85% match2024

Performance Evaluation of Large Language Models in Statistical Programming

83% match1 repo72025

Data Analysis in the Era of Generative AI

83% match2024

From Prompt Engineering to Prompt Science With Human in the Loop

82% match2024

A Tutorial on Teaching Data Analytics with Generative AI

82% match2024

From Inductive to Deductive: LLMs-Based Qualitative Data Analysis in Requirements Engineering

81% match1 repo32025

Cite this paper

@article{kazemitabaar2024improving,
  title  = {Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition},
  author = {Majeed Kazemitabaar and Jack Williams and Ian Drosos and Tovi Grossman and Austin Henley and Carina Negreanu and Advait Sarkar},
  year   = {2024},
  eprint = {2407.02651},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2407.02651v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2407.02651.svg)](https://codeofpaper.com/paper/2407.02651)

Discussion