sumeval
Evaluate text summarization models with ROUGE and BLEU in Python.
Multi-language evaluation framework for ROUGE and BLEU metrics.
sumeval is a pure Python evaluation framework designed for text summarization models. It provides well-tested implementations of ROUGE and BLEU metrics, making it suitable for data scientists and developers building language models in insurance.
What it is
The tool calculates ROUGE-1, ROUGE-2, ROUGE-L, ROUGE-BE, and BLEU scores, matching official scripts like Perl ROUGE-1.5.5.pl and SacréBLEU. It supports English, Japanese, and Chinese out of the box with language-specific tokenizers and dependencies.
Who it's for
It suits Python developers and ML engineers building or auditing text summarization systems for policy documents or claims. It does not suit teams looking for a managed SaaS platform or non-programmers.
Key features
Multi-language Support
Calculate scores for English, Japanese, and Chinese text with language-specific tokenization.
Official Metric Parity
ROUGE scores are tested against the original Perl script and BLEU uses SacréBLEU.
Command Line Interface
Run evaluations directly from the command line with JSON output formatting.
ROUGE-BE Support
Calculate ROUGE-BE scores utilizing spaCy for deeper semantic evaluation.
Up and running in 3 steps
Our honest verdict
What's great
- Pure Python implementation that is easy to integrate into existing pipelines
- Command line interface allows quick evaluations without writing custom scripts
- Built-in support for multiple languages including Japanese and Chinese
- Outputs structured JSON results for automated testing and CI integration
Where it falls short
- Requires external NLP libraries like spaCy, janome, or MeCab depending on features and languages used
- Requires technical skill to install, configure dependencies, and integrate into evaluation pipelines
- No graphical user interface; purely a command line and programmatic library
Pricing & license
It is free and open-source under its license, with no paid hosted option provided. The only cost is running it yourself and managing any required Python dependencies.
Open remote AI-training roles
880 open remote AI-training roles, updated November 15, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
Use it, or have us set it up
FAQ
Does sumeval support languages other than English?
Yes, it supports Japanese and Chinese with additional dependencies, and the base class allows extending to other languages.
How is BLEU calculated?
BLEU is calculated using SacréBLEU to ensure consistency with official WMT evaluation scripts.
