A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

Aiwei Liu, Xuming Hu, Lijie Wen, Philip S. Yu
3/12/2023
cs.CLcs.AI

Official code: THU-BPM/chatgpt-sql (107 stars, last commit 2023-06-14).

Abstract

This paper presents the first comprehensive analysis of ChatGPT's Text-to-SQL ability. Given the recent emergence of large-scale conversational language model ChatGPT and its impressive capabilities in both conversational abilities and code generation, we sought to evaluate its Text-to-SQL performance. We conducted experiments on 12 benchmark datasets with different languages, settings, or scenarios, and the results demonstrate that ChatGPT has strong text-to-SQL abilities. Although there is still a gap from the current state-of-the-art (SOTA) model performance, considering that the experiment was conducted in a zero-shot scenario, ChatGPT's performance is still impressive. Notably, in the ADVETA (RPL) scenario, the zero-shot ChatGPT even outperforms the SOTA model that requires fine-tuning on the Spider dataset by 4.1\%, demonstrating its potential for use in practical applications. To support further research in related fields, we have made the data generated by ChatGPT publicly available at https://github.com/THU-BPM/chatgpt-sql.

View on arXivView PDF

Code Implementations(1)

THU-BPM/chatgpt-sqlOfficial100%

The prediction results of ChatGPT on various datasets of Text-to-SQL.

10712PythonMar 12, 20233 years ago

Similar Papers

ChatGPT for Zero-shot Dialogue State Tracking: A Solution or an Opportunity?

87% match1 repo9632023

VicunaNER: Zero/Few-shot Named Entity Recognition using Vicuna

85% match1 repo4012023

Instances Need More Care: Rewriting Prompts for Instances with LLMs in the Loop Yields Better Zero-Shot Performance

85% match1 repo32023

ChatGPT may Pass the Bar Exam soon, but has a Long Way to Go for the LexGLUE benchmark

85% match1 repo292023

Is ChatGPT a General-Purpose Natural Language Processing Task Solver?

85% match2023

Narrowing the Gap between Zero- and Few-shot Machine Translation by Matching Styles

85% match2023

Cite this paper

@article{liu2023comprehensive,
  title  = {A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability},
  author = {Aiwei Liu and Xuming Hu and Lijie Wen and Philip S. Yu},
  year   = {2023},
  eprint = {2303.13547},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2303.13547v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2303.13547.svg)](https://codeofpaper.com/paper/2303.13547)

Discussion