Why this benchmark exists and what it tests
AI-assisted blood test interpretation is increasingly used in consumer and clinical workflows, yet reproducible evaluation frameworks tailored to laboratory medicine remain uncommon. The questions that matter most in this setting are not the ones covered by general medical question-answering benchmarks: can an engine separate iron deficiency from thalassaemia trait when the mean corpuscular volume is identical, does it over-diagnose Gilbert's syndrome as hepatitis, and does it manufacture pathology in a fully normal screening panel?
A single blood test panel typically contains enough signal to support several competing interpretations, and the job of the interpreting clinician is to weigh those interpretations against each other rather than to retrieve a textbook answer. An engine that does well on textbook cases can still fail on the cases that matter most: the differential-diagnosis pitfalls, the benign variants that look alarming in isolation, and the fully normal panels that tempt confident assistants into manufacturing pathology.
This benchmark was built around exactly those failure modes. Each of the fifteen cases was selected for a specific diagnostic property: an iron-deficient microcytosis that must be kept distinct from a beta-thalassaemia trait with identical mean corpuscular volume, a Gilbert's syndrome presentation where the only abnormality is an isolated indirect hyperbilirubinaemia, and a fifteen-parameter screening panel in which every analyte sits inside its reference range. The rubric rewards engines that read each case on its own terms and penalises engines that reach for a confident diagnosis where no such diagnosis is warranted.
As Thomas Klein, MD, I selected the case panel because these are the patterns I see laboratory-medicine assistants get wrong most often. The expensive failure mode is not "missing a rare disease" — it is fabricating routine pathology in patients who do not have it. Our Medical Validation hub describes the broader framework; this page describes the V11 initial proof-of-concept and the V11 Second Update that scaled it to 100,000 synthetic cases drawn from a synthetic case set spanning 127 country labels — using the same scoring rubric, byte-identical, with no post-hoc tuning permitted.
Latest reference run — V11 Second Update (April 26, 2026)
The V11 Second Update reference run of 26 April 2026 produced a composite score of 99.80% on the same pre-registered rubric used in the V11 initial release, evaluated on 100,000 synthetic cases drawn from the Kantesti synthetic case set and spanning 127 country labels and 75+ languages. Every case completed on the engine's primary path; trap-case hyperdiagnosis flag activations remained at 0 / 87,412. The original V11 run on 23 April 2026 covered 15 hand-curated cases (composite 99.12%) and validated the rubric; the Second Update keeps that rubric byte-identical and extends evaluation to a population-scale cohort.
The composite formula combines three components: structural conformance with the seven mandatory report sections and sixteen mandatory subsections, content accuracy measured as keyword recall plus scoring-system recall plus a probability-distribution validity check, and response latency against the primary-path service-level target. The exact decomposition is shown in the rubric formula below — none of these weights or sub-rubrics were altered for the Second Update.
The remaining 0.20 percentage points of headroom decompose almost entirely into the clinical sub-score — a small fraction of cases (predominantly in Hepatology and Rheumatology) had one expected scoring-system keyword absent from the engine's interpretation despite the diagnostic content being correct. No case in the 100,000-case Second-Update cohort missed the diagnosis itself. Latency improved from a mean of 20.17 s in the V11 initial release to 13.26 s in the Second Update, reflecting production engine optimisations between the two runs; the rubric, the scoring code, and the API endpoint are unchanged.
Per-label composite scores ranged from 0.9971 to 0.9985 across the 30 most-represented country labels. The long tail of 97 additional labels (≈7,300 cases combined) showed no systematic degradation. The most frequent labels by case count were the United States (10,500), Brazil (9,500), Spain (9,000), Italy (8,000), Germany (7,800), France (7,400), Portugal (5,800), Türkiye (3,400), the United Kingdom (2,900), and Mexico (2,500).
From 15 cases to 100,000: cohort evolution across 127 country labels
The original V11 case panel covered seven specialties — hematology, endocrinology, metabolic medicine, hepatology, nephrology, cardiology, rheumatology — plus two dedicated hyperdiagnosis trap cases, with each case a synthetically generated blood-test panel. The V11 Second Update extends evaluation to 100,000 synthetic cases across 127 country labels, distributed across eight specialties (the original seven plus a dedicated internal-medicine bucket that absorbs the trap subset). The same scoring rubric is applied byte-identically across both runs.
Because all cases are synthetically generated, there are no real identifiers to remove and no personal data is involved. Each synthetic case carries a benchmark-internal case code (BT-NNN-LABEL in the V11 initial set, a stable case_uid in the Second Update). No personal data appears anywhere in the published harness, the technical report, or the released datasets.
V11 initial release — 15 hand-curated cases
The original V11 case panel was hand-curated by Dr. Thomas Klein to exercise the diagnostic patterns that laboratory-medicine assistants get wrong most often. Each of the fifteen cases was selected for a specific diagnostic property, listed below.
Why this particular distribution
Hematology gets three cases because microcytic differentials and macrocytic differentials are the highest-volume traps in real-world laboratory practice. Endocrinology gets three because the Hashimoto's, PCOS, and vitamin D deficiency presentations exercise different diagnostic shapes (autoantibody-driven, hormone-ratio-driven, single-marker-driven). The single-case specialties are still meaningful because each of CKD, ASCVD risk, and SLE has its own scoring system that the engine should invoke (KDIGO staging, ASCVD 10-year risk, 2019 EULAR/ACR SLE criteria respectively).
V11 Second Update — 100,000 synthetic cases across 127 country labels
The Second Update replaces the original V11 hard-coded 15-case Python literal with a larger, programmatically generated synthetic case set. The case set is loaded at the start of every run and the configuration is logged for transparency. The cohort distribution by content area is shown below.
Synthetic country-label distribution — top 10 labels
The 100,000 synthetic cases carry 127 country labels (ISO 3166-1 alpha-2) to exercise locale handling. Label assignment: Europe 57.7%, the Americas 25.4%, Asia-Pacific 6.2%, named Middle-East/Africa labels 3.4%, and a long tail of 97 additional labels roughly 7.3% combined. The ten most frequent labels by case count are the United States (10,500), Brazil (9,500), Spain (9,000), Italy (8,000), Germany (7,800), France (7,400), Portugal (5,800), Türkiye (3,400), the United Kingdom (2,900), and Mexico (2,500). Per-label composite scores ranged from 0.9971 to 0.9985. These label counts are properties of the generated cases used to exercise locale handling — they are not real users and not real-world geographic coverage.
The pre-registered rubric, explained
Pre-registration is the single most important methodological choice in this benchmark. Every expected diagnosis, every clinical scoring system, and every report section was committed to source code before the engine was invoked. Post-hoc tuning of the rubric to flatter the engine is therefore impossible.
Three components make up the composite score. The structural component contributes 35 percent and measures whether the engine returned the seven mandatory report sections (header, summary, key findings, differential, scoring systems, recommendations, follow-up) and the sixteen mandatory subsections within them. Section presence weighs 40 percent and subsection presence weighs 60 percent within the structural calculation.
The clinical component contributes 55 percent and combines three things: diagnosis-keyword recall (70 percent of the clinical sub-score), scoring-system recall (20 percent — does the engine compute Mentzer, FIB-4, HOMA-IR, ASCVD risk, KDIGO staging, EULAR/ACR criteria where relevant), and a probability-sum validity check (10 percent — the differential probabilities should sum to within the [90, 110] interval). For trap cases, an explicit hyperdiagnosis penalty of up to 0.30 is subtracted, calculated as 0.10 per fabricated pathology flag, capped at three flags.
The latency component contributes 10 percent. A response under 20 seconds earns the full 0.10, a response under 40 seconds earns 0.05, and anything slower earns zero. The 20-second target reflects the production primary-path service-level objective; the 40-second ceiling reflects the Phase 2 fallback budget for heavy-engine invocations.
What pre-registration prevents
First-party benchmarks are notorious for inflating their own numbers through post-hoc rubric tuning. The pattern is almost always the same: the team runs the engine, sees where it underperforms, then quietly adjusts the rubric so the underperforming areas count for less. By committing the rubric to source code before the first engine call and publishing the harness under MIT licence, that adjustment becomes visible in version control. Anyone can clone the repository, check the rubric author dates, and verify the engine results were not used to shape the scoring.
Hyperdiagnosis trap cases — why over-calling is the real failure mode
Aggressive over-calling of pathology on normal screens is a documented failure mode of consumer-facing medical assistants. Its downstream costs include unnecessary investigation, patient anxiety, and iatrogenic workup. The two trap cases in this benchmark are designed to make that failure mode visible and scoreable.
🟡 Trap 1 — BT-014-GILBERT
Presentation. A 24-year-old male with a total bilirubin of 2.4 mg/dL. The direct fraction is normal, transaminases and alkaline phosphatase sit inside their reference ranges, reticulocytes are unremarkable, and haptoglobin and LDH rule out haemolysis.
Correct interpretation. Gilbert's syndrome — a benign UGT1A1 polymorphism. The interpretation should not invoke hepatitis, cirrhosis, haemolytic anaemia, or biliary obstruction.
V11 result. Composite 1.000. None of the six monitored over-diagnosis flags appeared as active diagnoses.
🟡 Trap 2 — BT-015-HEALTHY
Presentation. A 35-year-old female with a fifteen-parameter routine screening panel. Every analyte sits comfortably inside its reference range.
Correct interpretation. Reassurance and lifestyle maintenance. The interpretation should not manufacture borderline pathology to sound clinically useful.
V11 result. Composite 1.000. None of the seven monitored over-diagnosis flags — diabetes, anaemia, hypothyroidism, dyslipidaemia, hepatitis, kidney disease, deficiency — appeared as active diagnoses.
Across both traps, thirteen monitored hyperdiagnosis flags were checked. Zero were triggered. This is the result that matters most for any clinician considering using an AI engine as a triage or pre-consultation tool: the system did not invent disease where none existed.
Mentzer index: separating iron deficiency from thalassaemia trait
A second high-value finding concerns the pairing of case BT-001 (iron deficiency anaemia) with case BT-007 (beta-thalassaemia minor). Both present with microcytosis and are a well-known stumbling block for naive classifiers. The Mentzer index, calculated as MCV divided by RBC count, exceeds 13 in iron deficiency and falls below 13 in thalassaemia trait.
In BT-001, the patient was a 34-year-old female with hemoglobin 10.4 g/dL, MCV 72.4 fL, RBC 4.1 × 10¹²/L, ferritin 6 ng/mL, and elevated TIBC. The Mentzer index of approximately 17.7 supports absolute iron deficiency. In BT-007, the patient was a 28-year-old male with microcytosis (MCV 65.8 fL) but a high RBC count of 6.2, a normal RDW, normal ferritin, and HbA2 of 5.6 percent. The Mentzer index of approximately 10.6 points to thalassaemia trait, and the elevated HbA2 confirms beta-thalassaemia minor.
Both cases scored 1.000. The engine invoked the Mentzer index explicitly in both interpretations and returned the correct diagnosis in each instance. This is the single most clinically reassuring result in the entire benchmark, because misclassifying thalassaemia trait as iron deficiency leads to inappropriate iron supplementation and missed family-screening opportunities, and misclassifying iron deficiency as thalassaemia delays straightforward replacement therapy. Our ferritin range guide explains the broader differential context.
Per-case results from the V11 initial reference run (April 23, 2026)
The original V11 reference run on the 15-case proof-of-concept cohort serves as the methodological foundation of the Second Update: every per-case detail below illustrates how the rubric handles a real engine response. Twelve of fifteen cases achieved the ceiling composite score of 1.000 on the primary path; three cases were served via the Phase 2 fallback, losing the 0.05 latency bonus while preserving all clinical and structural content. One case was missing a single mandatory subsection; one returned a marginally reduced probability distribution sum.
The PCOS case (BT-008) lost a single mandatory subsection in the response structure — fifteen of sixteen instead of sixteen of sixteen — which shaved structural score from 1.000 to 0.963. The SLE case (BT-011) returned a marginally reduced probability-distribution sum that dropped clinical score to 0.965 while preserving every diagnostic keyword and scoring system. Neither sub-perfect case missed a correct diagnosis.
V11 Second Update aggregate — 100,000 cases
At population scale, individual case rows are not human-readable, so the Second Update reports aggregated metrics rather than a 100,000-row table. The headline aggregate is shown below; per-specialty and per-country-label breakdowns are published in the technical report and the Figshare deposit. A stratified random sample of n = 201 raw engine responses (deterministic seed 20260426) is published in the GitHub results/ directory for inspection.
What the headline score does not tell us
A composite score of 99.80 percent under this particular pre-registered rubric, on a 100,000-case synthetic cohort spanning 127 country labels, represents near-ceiling performance — but it deserves careful framing. The result describes the engine's behaviour against the rubric we committed to source code in V11; it is not a universal claim about the engine's correctness on every blood test panel that exists in the wild.
The score says the engine handled the diagnostic patterns selected for this evaluation correctly across a population-scale cohort, on a methodology that is published and reproducible. It does not say the engine is correct on every blood test panel that exists in the wild. It does not say the engine should replace clinician judgment. And it does not say the engine outperforms alternative AI systems — comparative analyses against other engines were deliberately out of scope for this report.
What the score does establish is a baseline. With the rubric and harness public, future versions of the engine can be evaluated against the same rubric — applied to the V11 initial 15 cases, the Second Update 100,000-case cohort, or any subsequent expansion — and the gap between the published score and any subsequent run is itself measurable. This is the value of pre-registration: it converts performance claims into testable claims.
How to reproduce this benchmark in 10 minutes
Reproduction requires only a Kantesti API credential pair and a Python 3.10 or later environment with the requests and reportlab libraries installed. The full harness is a single self-contained Python module released under the MIT licence.
Four steps to a fresh run
One. Clone the repository: git clone https://github.com/emirhanai/kantesti-blood-test-benchmark.git. Two. Install dependencies with pip install -r requirements.txt (Second Update adds mysql-connector-python ≥ 8.0 for the SQL case loader). Three. Set KANTESTI_USERNAME and KANTESTI_PASSWORD as environment variables for the engine API. For the Second Update SQL case loader, also set KANTESTI_DB_HOST, KANTESTI_DB_PORT, KANTESTI_DB_NAME, KANTESTI_DB_USER, and KANTESTI_DB_PASSWORD — the loader connects through a read-only role (bench_reader) that has no privileges on identifying tables. Four. Run python benchmark_bloodtest.py --limit 100000 for the full Second-Update run, or python benchmark_bloodtest.py --limit 1000 for quick iteration. Outputs land in ./benchmark_results/: a CSV scorecard with per-country-label and per-specialty columns, a JSON aggregate, a stratified-random raw-response sample, and a Markdown report.
The reference runs from 23 April 2026 (V11 initial, 15 cases) and 26 April 2026 (V11 Second Update, 100,000 cases) are preserved in the results/ directory of the repository. A fresh run will produce a new timestamped scorecard while leaving the reference runs untouched. If your run produces a meaningfully different result, please open a GitHub issue with the run timestamp and the engine version returned in the response metadata.
Limitations and future work
Even at 100,000 cases across 127 country labels, four limitations deserve explicit acknowledgement: long-tail label undersampling, single-shot evaluation, single-engine scope, and single-source data origin. Each is being addressed in active follow-up work.
Long-tail label coverage. The Second Update spans 127 country labels, but the distribution is unbalanced — the top 10 labels account for ≈66.4% of cases, and the long tail of 97 additional labels together contributes ≈7.3% (roughly 7,300 cases combined, ~75 cases per label on average). Per-label composites in this long tail are therefore noisier than headline figures suggest. Future runs will rebalance label assignment to firm up per-label estimates.
Single-shot evaluation. Each case in the cohort was evaluated once. Large language models exhibit non-trivial output variance even at low sampling temperature, so a multi-run protocol with five evaluations per case and reported variance is a natural next step — particularly on the trap-case subset, where consistency under sampling jitter is part of the safety claim.
Single-engine scope. This report characterises one engine. Comparative analyses against alternative AI systems are out of scope here; we may pursue them as a separate independent study with appropriate methodology, against the same MIT-licensed harness.
Synthetic data. The 100,000 cases are synthetically generated, not synthetic cases, and results do not transfer to real-world clinical performance. Evaluation on real, consented, externally-sourced data would require appropriate ethical oversight and is out of scope for this synthetic benchmark.
Beyond these four, the most impactful planned extension is multi-language parity per jurisdiction. The Kantesti AI Engine serves users in 75+ languages, and running language-stratified Second-Update sub-cohorts (Turkish, German, Spanish, French, Italian, Portuguese, Arabic, Mandarin) will quantify output quality across the engine's supported languages. Each language-stratified analysis will be published with its own DOI and harness branch.