A clinical decision-evidence benchmark for evaluating chat assistants.
Test whether clinical chat assistants adapt their actions when decisive facts change.
Keystone-bench evaluates medical AI chat assistants by modifying core facts in HealthBench source conversations and observing if model actions follow the evidence. It is built for medical AI developers and researchers testing safety and reasoning reliability.
The tool takes HealthBench conversations and applies eight distinct evidence families—such as missing, conflicting, or alternative evidence—along with paraphrase controls. It runs models against these perturbations and grades whether the assistant's action appropriately shifts with the updated facts.
Suits AI engineers and medical researchers building clinical chat assistants who need rigorous evaluation beyond basic rubric scoring. Does not suit casual users or those without API access for running model inferences.
Tests models against missing evidence, conflicting facts, demographic shifts, and negative controls.
Links each source to a decision frame and annotates acceptable or forbidden actions for each state.
Includes over 2,800 multi-turn conversations with byte-identical earlier turns for complex testing.
Compares two benchmark runs item by item to pinpoint exact regressions and fixes in new model versions.
The code is free under the MIT licence. The only ongoing cost is your own infrastructure and API token spend for running model inferences and multi-vendor judge evaluations.
880 open remote AI-training roles, updated November 16, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
These reviewed AI tools do the same job instantly — from our scored directory.
It builds upon OpenAI's public copy of HealthBench, augmenting conversations with decision frames and evidence state twins.
To avoid self-evaluation bias, the tool requires an independent model vendor to map replies to the annotations.
You can use the keystone estimate command to check anticipated calls and token usage before spending anything.
See Zekai first in Google