OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages

Chester Palen-Michel, Maxwell Pickering, Maya Kruse, Jonne Sälevä, Constantine Lignos
12/12/2024
cs.CL

Official code: bltlab/open-ner (4 stars, last commit 2025-12-22).

Abstract

We present OpenNER 1.0, a standardized collection of openly-available named entity recognition (NER) datasets. OpenNER contains 36 NER corpora that span 52 languages, human-annotated in varying named entity ontologies. We correct annotation format issues, standardize the original datasets into a uniform representation with consistent entity type names across corpora, and provide the collection in a structure that enables research in multilingual and multi-ontology NER. We provide baseline results using three pretrained multilingual language models and two large language models to compare the performance of recent models and facilitate future research in NER. We find that no single model is best in all languages and that significant work remains to obtain high performance from LLMs on the NER task. OpenNER is released at https://github.com/bltlab/open-ner.

View on arXivView PDF

Code Implementations(1)

bltlab/open-nerOfficial100%

OpenNER

40Dec 18, 202410 months ago

Similar Papers

MSNER: A Multilingual Speech Dataset for Named Entity Recognition

86% match2024

DynamicNER: A Dynamic, Multilingual, and Fine-Grained Dataset for LLM-based Named Entity Recognition

85% match1 repo52024

Assessment of Generative Named Entity Recognition in the Era of Large Language Models

84% match1 repo42026

What Matters When Building Universal Multilingual Named Entity Recognition Models?

84% match2026

Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset

84% match2026

AWED-FiNER: Agents, Web applications, and Expert Detectors for Fine-grained Named Entity Recognition across 36 Languages for 6.6 Billion Speakers

83% match1 repo12026

Cite this paper

@article{palenmichel2024openner,
  title  = {OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets in 50+ Languages},
  author = {Chester Palen-Michel and Maxwell Pickering and Maya Kruse and Jonne Sälevä and Constantine Lignos},
  year   = {2024},
  eprint = {2412.09587},
  archivePrefix = {arXiv},
  url    = {http://arxiv.org/abs/2412.09587v3}
}

Add a badge to your README

Shows whether official code is linked for this paper.

Code of Paper badge
[![Code of Paper](https://codeofpaper.com/badge/2412.09587.svg)](https://codeofpaper.com/paper/2412.09587)

Discussion