AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries

Irina Saparina, Mirella Lapata
6/1/2024

No official code found. Best implementation: saparina/ambrosia (16 stars, last commit 2024-10-31).

Abstract

Practical semantic parsers are expected to understand user utterances and map them to executable programs, even when these are ambiguous. We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-SQL parsers capable of recognizing and interpreting ambiguous requests. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness), their interpretations, and corresponding SQL queries. In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch. We benchmark various LLMs on AMBROSIA, revealing that even the most advanced models struggle to identify and interpret ambiguity in questions.

View on arXivView PDF

Code Implementations(1)

saparina/ambrosiaCommunity100%

π”Έπ•„π”Ήβ„π•†π•Šπ•€π”Έ: A Benchmark for Parsing Ambiguous Questions into Database Queries

161CSS, HTMLJun 25, 20241 years ago

Similar Papers

AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQL

89% match1 repo212025

PRACTIQ: A Practical Conversational Text-to-SQL dataset with Ambiguous and Unanswerable Queries

83% match2024

Disambiguate First, Parse Later: Generating Interpretations for Ambiguity Resolution in Semantic Parsing

82% match1 repo52025

CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems

80% match2026

Aligning Language Models to Explicitly Handle Ambiguity

80% match1 repo52024

PAQA: Toward ProActive Open-Retrieval Question Answering

79% match2024

Cite this paper

@article{saparina2024ambrosia,
  title  = {AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries},
  author = {Irina Saparina and Mirella Lapata},
  year   = {2024},
  eprint = {2406.19073},
  archivePrefix = {arXiv},
  doi    = {10.48550/arXiv.2406.19073},
  url    = {https://arxiv.org/abs/2406.19073},
  journal = {NEURIPS 2024 2024}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2406.19073.svg)](https://codeofpaper.com/paper/2406.19073)

Discussion