Refining and Reusing Annotation Guidelines for LLM Annotation

Kon Woo Kim, Jin-Dong Kim, Akiko Aizawa
5/20/2026
cs.CL

No official code found. Best implementation: KonWooKim/llm-guideline-moderation (1 star, last commit 2026-04-20).

Abstract

While Large Language Models (LLMs) demonstrate remarkable performance on zero-shot annotation tasks, they often struggle with the specialized conventions of gold-standard benchmarks. We propose the systematic reuse and refinement of annotation guidelines as an alignment mechanism, introducing an iterative moderation framework that simulates the early phases of annotation projects. We evaluate three hypotheses: (1) the efficacy of guideline integration, (2) the advantage of reasoning optimized models, and (3) the viability of moderation under minimal supervision. Testing across biomedical NER tasks (NCBI Disease, BC5CDR, BioRED) with three LLM families (GPT, Gemini, DeepSeek), our results empirically confirm all three hypotheses. While the iterative moderation framework shows good potential in effectively refining guidelines, our analysis also reveals substantial room for improvement.

View on arXivView PDF

Code Implementations(1)

KonWooKim/llm-guideline-moderationCommunity68%

Resources for refining and reusing annotation guidelines for LLM annotation, including prompts, moderation outputs, and evaluation artifacts.

10Apr 19, 20266 months ago

Similar Papers

CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

86% match1 repo242023

Synthesized Annotation Guidelines are Knowledge-Lite Boosters for Clinical Information Extraction

86% match2025

DiZiNER: Disagreement-guided Instruction Refinement via Pilot Annotation Simulation for Zero-shot Named Entity Recognition

85% match2026

Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study

84% match2025

GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction

84% match1 repo4412026

GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction

84% match1 repo4412023

Cite this paper

@article{kim2026refining,
  title  = {Refining and Reusing Annotation Guidelines for LLM Annotation},
  author = {Kon Woo Kim and Jin-Dong Kim and Akiko Aizawa},
  year   = {2026},
  eprint = {2605.20809},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2605.20809v1}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2605.20809.svg)](https://codeofpaper.com/paper/2605.20809)

Discussion