entity-recognition-datasets
A curated collection of annotated NER corpora across multiple domains and languages.
Access a comprehensive collection of annotated corpora for named entity recognition tasks.
This repository provides a structured directory and conversion code for named entity recognition datasets spanning news, medical, social media, and specialized domains. It suits NLP engineers and researchers hunting for standardized training data.
What it is
It aggregates references, licensing details, and conversion scripts for transforming diverse entity recognition corpora into CoNLL 2003 format. The collection covers English datasets alongside multilingual resources and code-switching text.
Who it's for
It suits NLP researchers and machine learning engineers building custom entity extraction models. It does not suit practitioners seeking a ready-to-deploy NER inference API or pre-trained models.
Key features
CoNLL Conversion Code
Includes scripts to convert various source formats into the standard CoNLL 2003 structure.
Multi-Domain Coverage
Contains corpora spanning news, medical records, social media, financial filings, and defense.
Multilingual Resources
Lists specialized entity recognition datasets for German, code-switching text, and other languages.
Access Routing
Provides explicit availability paths and references for datasets restricted by strict licenses.
Up and running in 3 steps
Our honest verdict
What's great
- Saves hours of manual searching for specialized NER datasets
- Includes conversion utilities for format standardization
- Covers niche domains like medical, malware, and finance
- Catalogs hard-to-find historical and social media corpora
Where it falls short
- Dataset maintenance is inactive since 2020, meaning newer corpora are missing
- Many listed datasets require separate LDC access or restrictive user agreements
- Does not contain all dataset files directly due to distribution licenses
Pricing & license
The repository is free under its MIT licence, but the true cost depends on acquiring restricted corpora, some of which require paid LDC subscriptions.
Open remote AI-training roles
873 open remote AI-training roles, updated November 15, 2026; most pay $20–$170 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
These reviewed AI tools do the same job instantly — from our scored directory.
Use it, or have us set it up
FAQ
Are all datasets included directly in the repository?
No. While some permissively licensed datasets are included, others require separate acquisition via LDC or external links due to distribution restrictions.
What format does the conversion code target?
The repository provides code to convert supported corpora into the CoNLL 2003 format.
Is the repository actively updated with new datasets?
No, active dataset additions stopped around 2020, though pull requests and issues are welcome.
