Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities

Hongseok Oh, Wonseok Hwang, Kyoung-Woon On
12/31/2025
cs.CL

Official code: lbox-kr/kcl (7 stars, last commit 2026-01-23).

Abstract

We introduce the Korean Canonical Legal Benchmark (KCL), a benchmark designed to assess language models' legal reasoning capabilities independently of domain-specific knowledge. KCL provides question-level supporting precedents, enabling a more faithful disentanglement of reasoning ability from parameterized knowledge. KCL consists of two components: (1) KCL-MCQA, multiple-choice problems of 283 questions with 1,103 aligned precedents, and (2) KCL-Essay, open-ended generation problems of 169 questions with 550 aligned precedents and 2,739 instance-level rubrics for automated evaluation. Our systematic evaluation of 30+ models shows large remaining gaps, particularly in KCL-Essay, and that reasoning-specialized models consistently outperform their general-purpose counterparts. We release all resources, including the benchmark dataset and evaluation code, at https://github.com/lbox-kr/kcl.

View on arXivView PDF

Code Implementations(1)

lbox-kr/kclOfficial100%
70Shell, PythonJan 22, 20269 months agoNOASSERTION

Similar Papers

Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models

89% match1 repo192024

CALRK-Bench: Evaluating Context-Aware Legal Reasoning in Korean Law

86% match2026

LawBench: Benchmarking Legal Knowledge of Large Language Models

85% match1 repo4432023

PLawBench: A Rubric-Based Benchmark for Evaluating LLMs in Real-World Legal Practice

84% match1 repo332026

LegalEval-Q: A New Benchmark for The Quality Evaluation of LLM-Generated Legal Text

84% match1 repo32025

VLegal-Bench: Cognitively Grounded Benchmark for Vietnamese Legal Reasoning of Large Language Models

84% match2025

Cite this paper

@article{oh2025korean,
  title  = {Korean Canonical Legal Benchmark: Toward Knowledge-Independent Evaluation of LLMs' Legal Reasoning Capabilities},
  author = {Hongseok Oh and Wonseok Hwang and Kyoung-Woon On},
  year   = {2025},
  eprint = {2512.24572},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2512.24572v2}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2512.24572.svg)](https://codeofpaper.com/paper/2512.24572)

Discussion