SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints

Rocky Klopfenstein, Yang He, Andrew Tremante, Yuepeng Wang, Nina Narodytska +1 more
3/4/2026
cs.DBcs.AIcs.LO

No confident code implementation found for this paper.

Abstract

We present SpotIt+, an open-source tool for evaluating Text-to-SQL systems via bounded equivalence verification. Given a generated SQL query and the ground truth, SpotIt+ actively searches for database instances that differentiate the two queries. To ensure that the generated counterexamples reflect practically relevant discrepancies, we introduce a constraint-mining pipeline that combines rule-based specification mining over example databases with LLM-based validation. Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases, while preserving its ability to efficiently uncover numerous discrepancies between generated and gold SQL queries that are missed by standard test-based evaluation.

View on arXivView PDF

Code Implementations

No confident code match yet

We couldn't find an author-owned or strongly-evidenced community implementation for this paper. Any repos shown below are weak matches — verify before relying on them.

No code implementations found yet.

Know of an implementation? Let us know in the comments below!

Similar Papers

Can the Rookies Cut the Tough Cookie? Exploring the Use of LLMs for SQL Equivalence Checking

80% match1 repo12024

PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency

78% match1 repo1182024

Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration

78% match2025

FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark

78% match1 repo122024

SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps

78% match2025

Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQL

77% match2025

Cite this paper

@article{klopfenstein2026spotit,
  title  = {SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints},
  author = {Rocky Klopfenstein and Yang He and Andrew Tremante and Yuepeng Wang and Nina Narodytska and Haoze Wu},
  year   = {2026},
  eprint = {2603.04334},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2603.04334v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2603.04334.svg)](https://codeofpaper.com/paper/2603.04334)

Discussion