BIS: NL2SQL Service Evaluation Benchmark for Business Intelligence Scenarios

Bora Caglayan, Mingxue Wang, John D. Kelleher, Shen Fei, Gui Tong +2 more
10/30/2024
cs.AI

No official code found. Best implementation: boracaglayan/bis-nl2sql (3 stars, last commit 2024-10-10).

Abstract

NL2SQL (Natural Language to Structured Query Language) transformation has seen wide adoption in Business Intelligence (BI) applications in recent years. However, existing NL2SQL benchmarks are not suitable for production BI scenarios, as they are not designed for common business intelligence questions. To address this gap, we have developed a new benchmark focused on typical NL questions in industrial BI scenarios. We discuss the challenges of constructing a BI-focused benchmark and the shortcomings of existing benchmarks. Additionally, we introduce question categories in our benchmark that reflect common BI inquiries. Lastly, we propose two novel semantic similarity evaluation metrics for assessing NL2SQL capabilities in BI applications and services.

View on arXivView PDF

Code Implementations(1)

boracaglayan/bis-nl2sqlCommunity100%

Repository for "BIS: NL2SQL Service Evaluation Benchmark for Business Intelligence Scenarios" paper

30PythonOct 9, 20242 years ago

Similar Papers

From Search to Reasoning: A Five-Level RAG Capability Framework for Enterprise Data

81% match2025

CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering

79% match1 repo152024

CodeAssistBench (CAB): Dataset & Benchmarking for Multi-turn Chat-Based Code Assistance

78% match1 repo112025

Dynamic Intelligence Assessment: Benchmarking LLMs on the Road to AGI with a Focus on Model Confidence

77% match2024

Opus: A Prompt Intention Framework for Complex Workflow Generation

77% match2025

MTFinEval:A Multi-domain Chinese Financial Benchmark with Eurypalynous questions

77% match2024

Cite this paper

@article{caglayan2024bis,
  title  = {BIS: NL2SQL Service Evaluation Benchmark for Business Intelligence Scenarios},
  author = {Bora Caglayan and Mingxue Wang and John D. Kelleher and Shen Fei and Gui Tong and Jiandong Ding and Puchao Zhang},
  year   = {2024},
  eprint = {2410.22925},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2410.22925v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2410.22925.svg)](https://codeofpaper.com/paper/2410.22925)

Discussion