VerilogEval: Evaluating Large Language Models for Verilog Code Generation

Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, Haoxing Ren
9/14/2023
cs.LGcs.SE

Official code: NVlabs/verilog-eval (454 stars, last commit 2025-07-14).

Abstract

The increasing popularity of large language models (LLMs) has paved the way for their application in diverse domains. This paper proposes a benchmarking framework tailored specifically for evaluating LLM performance in the context of Verilog code generation for hardware design and verification. We present a comprehensive evaluation dataset consisting of 156 problems from the Verilog instructional website HDLBits. The evaluation set consists of a diverse set of Verilog code generation tasks, ranging from simple combinational circuits to complex finite state machines. The Verilog code completions can be automatically tested for functional correctness by comparing the transient simulation outputs of the generated design with a golden solution. We also demonstrate that the Verilog code generation capability of pretrained language models could be improved with supervised fine-tuning by bootstrapping with LLM generated synthetic problem-code pairs.

View on arXivView PDF

Code Implementations(1)

NVlabs/verilog-evalOfficial100%

Verilog evaluation benchmark for large language model

45491M4, PythonAug 23, 20231 years agoNOASSERTION

Similar Papers

VeriGen: A Large Language Model for Verilog Code Generation

92% match2023

A Multi-Expert Large Language Model Architecture for Verilog Code Generation

90% match2024

Advanced Large Language Model (LLM)-Driven Verilog Development: Enhancing Power, Performance, and Area Optimization in Code Synthesis

89% match2023

VerilogDB: The Largest, Highest-Quality Dataset with a Preprocessing Framework for LLM-based RTL Generation

88% match2025

LLM-VeriPPA: Power, Performance, and Area Optimization aware Verilog Code Generation with Large Language Models

88% match2025

Pluto: A Benchmark for Evaluating Efficiency of LLM-generated Hardware Code

87% match2025

Cite this paper

@article{liu2023verilogeval,
  title  = {VerilogEval: Evaluating Large Language Models for Verilog Code Generation},
  author = {Mingjie Liu and Nathaniel Pinckney and Brucek Khailany and Haoxing Ren},
  year   = {2023},
  eprint = {2309.07544},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2309.07544v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2309.07544.svg)](https://codeofpaper.com/paper/2309.07544)

Discussion