Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

Clemencia Siro, Pourya Aliannejadi, Mohammad Aliannejadi
2/9/2026
cs.CLcs.LG

No official code found. Best implementation: Clemenciah/llm-generated-rubrics (3 stars, MIT, last commit 2026-02-03).

Abstract

Large language models (LLMs) are increasingly used as evaluators for natural language generation, applying human-defined rubrics to assess system outputs. However, human rubrics are often static and misaligned with how models internally represent language quality. We introduce GER-Eval (Generating Evaluation Rubrics for Evaluation) to investigate whether LLMs can design and apply their own evaluation rubrics. We evaluate the semantic coherence and scoring reliability of LLM-defined criteria and their alignment with human criteria. LLMs reliably generate interpretable and task-aware evaluation dimensions and apply them consistently within models, but their scoring reliability degrades in factual and knowledge-intensive settings. Closed-source models such as GPT-4o achieve higher agreement and cross-model generalization than open-weight models such as Llama. Our findings position evaluation as a learned linguistic capability of LLMs, consistent within models but fragmented across them, and call for new methods that jointly model human and LLM evaluative language to improve reliability and interpretability.

View on arXivView PDF

Code Implementations(1)

Clemenciah/llm-generated-rubricsCommunity59%
MIT

This repository contains reusable implementations of the evaluation metrics introduced in Learning to Judge: LLMs Designing and Applying Evaluation Rubrics. The metrics support the analysis of LLM-defined evaluation criteria, their alignment with human rubrics, and the reliability of LLM-based scoring.

30Python, Jupyter NotebookJan 27, 20268 months agoMIT

Similar Papers

An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability

90% match2025

Training an LLM-as-a-Judge Model: Pipeline, Insights, and Practical Lessons

89% match2025

Neither Valid nor Reliable? Investigating the Use of LLMs as Judges

89% match2025

LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods

89% match1 repo6182024

A Survey on LLM-as-a-Judge

89% match1 repo1752024

ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models

88% match2024

Cite this paper

@article{siro2026learning,
  title  = {Learning to Judge: LLMs Designing and Applying Evaluation Rubrics},
  author = {Clemencia Siro and Pourya Aliannejadi and Mohammad Aliannejadi},
  year   = {2026},
  eprint = {2602.08672},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2602.08672v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2602.08672.svg)](https://codeofpaper.com/paper/2602.08672)

Discussion