AI tools, scored for your job
Get paid to train AI
🤝 You're in the Free & Open Source library — community-built tools you can use without paying. A different concept from our scored AI directory.
Home › Free & Open Source › Medical › keystone-bench
✦ Free & open sourceMedical

keystone-bench

A clinical decision-evidence benchmark for evaluating chat assistants.

Evaluate medical LLMs with evidence perturbation

Test whether clinical chat assistants adapt their actions when decisive facts change.

Free forever · open source · MIT
✓ Light setup✓ Browser-based✓ Real-time collaboration
7.3
ZEKAI
Essential clinical AI benchmark
Free & open source · Medical
Ease of setup4
Usefulness9
Maintenance9.5
Community6.5
★ 511
GITHUB
Sep 2026
UPDATED
MIT
LICENSE
🛠 Don't want to self-host? We can set it up →
⚡ Quick answer

Keystone-bench evaluates medical AI chat assistants by modifying core facts in HealthBench source conversations and observing if model actions follow the evidence. It is built for medical AI developers and researchers testing safety and reasoning reliability.

Runs on
Cross-platform
Hosting
Self-host
Languages
Python
License
MIT

What it is

The tool takes HealthBench conversations and applies eight distinct evidence families—such as missing, conflicting, or alternative evidence—along with paraphrase controls. It runs models against these perturbations and grades whether the assistant's action appropriately shifts with the updated facts.

Who it's for

Suits AI engineers and medical researchers building clinical chat assistants who need rigorous evaluation beyond basic rubric scoring. Does not suit casual users or those without API access for running model inferences.

Key features

Eight Evidence Families

Tests models against missing evidence, conflicting facts, demographic shifts, and negative controls.

Decision-Evidence Layer

Links each source to a decision frame and annotates acceptable or forbidden actions for each state.

Multi-Turn Support

Includes over 2,800 multi-turn conversations with byte-identical earlier turns for complex testing.

Automated Regression Checks

Compares two benchmark runs item by item to pinpoint exact regressions and fixes in new model versions.

How to start

Up and running in 3 steps

1
Install the Python package from PyPI using pip install keystone-bench.
2
Run keystone build to fetch HealthBench and rebuild the release data locally.
3
Execute keystone run with your API keys to evaluate your model against the benchmark.
Real talk

Our honest verdict

What's great

  • Tests actual action changes rather than relying solely on static physician rubrics
  • Includes negative control families to detect stubborn model persistence
  • Ships with reference results and classifications for five major assistants
  • Provides built-in validation checks for judges and degenerate strategies

Where it falls short

  • Requires rebuilding release data locally using OpenAI's public copy of HealthBench
  • Demands external API keys and token spend for model runs and judge evaluation
  • Requires a second vendor model to act as an independent action judge
Cost

Pricing & license

The code is free under the MIT licence. The only ongoing cost is your own infrastructure and API token spend for running model inferences and multi-vendor judge evaluations.

Paid work

Open remote AI-training roles

880 open remote AI-training roles, updated November 16, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.

See all open roles →
Compare

Alternatives

🎁 Free & open-source alternatives

These reviewed AI tools do the same job instantly — from our scored directory.

Get more

Use it, or have us set it up

📄 Free PDF: keystone-bench for Medical
A plain-language guide to getting your first result. Sent to your inbox.
🔔 64 people want a full video tutorial — add your vote
🛠 Don't want to set it up yourself?
We'll install, host & configure keystone-bench for you — and train your team.
Get it done for you ↗
Questions

FAQ

What data sources does keystone-bench use?

It builds upon OpenAI's public copy of HealthBench, augmenting conversations with decision frames and evidence state twins.

Why must the judge come from a different vendor?

To avoid self-evaluation bias, the tool requires an independent model vendor to map replies to the annotations.

How do I handle API costs before running?

You can use the keystone estimate command to check anticipated calls and token usage before spending anything.

Radar

Latest AI news

✉️ The weekly free toolkit
The best new free & open-source tools for Medical — one short email a week, in your language.
Free · 1 email/week · unsubscribe anytime
Get paid to train AI →

See Zekai first in Google