MamaBench: Benchmarking LLM Robustness in Maternal
and Child Health Diagnosis through Counterfactual
Clinical Perturbation
July 15, 2026
Abstract
Large language models achieve strong scores on medical benchmarks, yet these benchmarks evaluate each question in isolation, providing no measure of whether a system can distinguish clinically similar presentations requiring different interventions. We introduce MamaBench, the first counterfactual benchmark for maternal and paediatric AI: 434 expert-authored clinical narratives in 217 pairs across 371 pathologies, evaluated via the Bias Trap Rate (BTR), the conditional probability that a model fails the counterfactual given success on the base case. We propose Evidence-Anchored RAG (EA-RAG), a three-stage retrieval method that replaces aggregate similarity with an evidence coverage objective through clinical parameter extraction, coverage auditing, and contrastive sub-queries. Across eight configurations of four frontier LLMs, base accuracy overstates robust accuracy by 16-28 percentage points in every model. EA-RAG achieves 20.3% BTR and 65.0% robust accuracy on Claude Sonnet 4.6, a 5.5 percentage point BTR reduction without degrading base accuracy. The residual 20% BTR confirms that counterfactual robustness in clinical AI remains an open challenge.
Keywords: counterfactual evaluation, clinical AI, maternal healthcare, retrieval-augmented generation, diagnostic robustness
1 Introduction↩︎
LLMs now pass medical licensing exams [1], [2] and outperform physician baselines on clinical QA [3], [4], accelerating deployment via retrieval-augmented generation for domain-grounded decision support [5], [6]. In maternal healthcare where a diagnostic error threatens two lives, such systems carry both exceptional promise and exceptional risk [7].
Yet current benchmarks (MedQA, PubMedQA, MedMCQA) score each question independently, hiding a dangerous failure mode: a system may answer correctly when the presentation matches its prior, yet fail when a single discriminative parameter shifts the correct diagnosis. Contrast sets [8] and counterfactual augmentation [9] exposed this fragility in general NLP; MedEinst [10] extended it to medical LLMs via cases derived from DDXPlus [11]. No counterfactual benchmark yet exists for maternal and paediatric care the domain where clinical stakes are highest.
We close this gap with two contributions:
MamaBench: 434 expert-authored clinical narratives in 217 counterfactual pairs across 371 pathologies, written in first-person patient-reported format by a three-member clinical team. We formalise Diagnostic Fixation (a model gets the base case right but rigidly persists when discriminative parameters shift) and adopt the Bias Trap Rate (BTR) [10] to quantify it.
Evidence-Anchored RAG (EA-RAG): A three-stage inference-time pipeline that replaces aggregate similarity retrieval with an evidence coverage objective: extracting typed clinical parameters, auditing retrieval coverage, and filling gaps via contrastive sub-queries, guided by a taxonomy-grounded reasoning scaffold.
Across eight configurations of four frontier LLMs we find: (i) base accuracy overstates robust accuracy by 16–28 percentage point in every model; (ii) vanilla RAG provides no counterfactual benefit; (iii) EA-RAG reduces BTR by 5.5 percentage point on the strongest model without degrading base accuracy; (iv) even the best system still fails one in five counterfactual pairs.
2 Related Work↩︎
2.0.0.1 Medical QA benchmarks.
MedQA [3], PubMedQA [12], and MedMCQA [4] evaluate isolated questions, enabling landmark results from Med-PaLM [1] and GPT-4 [2] but providing no pair-level discrimination signal.
2.0.0.2 Counterfactual and robustness evaluation.
Contrast sets [8] and counterfactually augmented data [9] showed that high-accuracy models exploit surface cues; CheckList [13] formalised behavioural testing via structured perturbations. MedEinst [10] benchmarks the Einstellung effect [14] through counterfactual diagnosis on DDXPlus [11], while [15] evaluated clinical RAG robustness using a specialized counterfactual retrieval testbed. MamaBench differs in being (a) entirely expert-authored, (b) focused on maternal/paediatric care, and (c) paired with a retrieval-level intervention. Complementary robustness studies address paraphrasing and demographic perturbations [16] but not diagnostic fixation.
2.0.0.3 Retrieval-augmented generation.
RAG [5] has been refined by dense retrieval [17], adaptive strategies (FLARE, [18]; Self-RAG, [19]; IRCoT, [20]), clinical example-selection frameworks like MMRAG [21], and clinical deployment [6]. None addresses retrieval stagnation: near-identical embeddings for base–counterfactual pairs yield the same retrieved context [22]. EA-RAG fills this gap through evidence coverage, drawing on query decomposition [23], [24] while operating entirely at inference time.
2.0.0.4 Structured reasoning.
CoT [25] and Tree of Thoughts [26] improve reasoning but are task-agnostic. EA-RAG’s scaffold derives from empirical failure taxonomy, targeting specific failure categories (missed critical info, underthinking, factual inaccuracy, overthinking) rather than providing generic instructions.
3 MamaBench Benchmark↩︎
3.1 Problem Formulation↩︎
We formalise diagnosis as \(f\!:\!\mathcal{X}\!\to\!\mathcal{Y}\), where \(\mathcal{X}\) is the space of patient-reported narratives and \(\mathcal{Y}\) the label space of 371 pathologies. A Counterfactual Pair \((x_b,x_c)\) comprises a base case \(x_b\) with ground truth \(y_b\) and a counterfactual \(x_c\)—a minimal perturbation shifting the ground truth to \(y_c\!\neq\!y_b\) while preserving shared context.
Definition 1 (Diagnostic Fixation). Model \(f\) exhibits Diagnostic Fixation on \((x_b,x_c)\) iff: \[f(x_b)=y_b \;\wedge\; f(x_c)=y_b \;\wedge\; y_c\neq y_b\]
The model succeeds in the base case, but persists rigidly when discriminative parameters change analogously to the Einstellung effect [14]. Diagnostic Fixation is architecture-agnostic: it can arise from memorised priors, retrieval stagnation, or heuristic shortcuts.
3.2 Construction and Statistics↩︎
A three-member medical team (obstetrics, gynaecology, paediatrics) authored each narrative in first-person patient-reported format: e.g., “My baby was delivered at 37 weeks by a traditional birth attendant. He is 10 days old and refusing to eat…” introducing naturalistic ambiguity absent from structured vignettes [3]. Unlike benchmarks derived from existing datasets [11] or LLM pipelines [10], every case is individually expert-authored.
For each base case, the team constructs a counterfactual by perturbing the fewest parameters necessary to shift the diagnosis, following five categories: symptom substitution, severity escalation, risk factor modification, temporal shift, and comorbidity introduction.
MamaBench totals 217 pairs (434 cases): paediatrics 254 (58.5%), obstetrics/gynaecology 178 (41.0%), unclassified 2 (0.5%), covering 371 unique pathologies. All cases passed multi-stage clinical review for diagnostic sufficiency, counterfactual validity, and minimal edit properties.
3.3 Evaluation Metrics↩︎
Each pair is classified as PP (both correct), PF (base correct, counterfactual wrong), FP, or FF. We report three metrics:
Base Accuracy: \(\text{Acc}_{\text{base}}= (|\text{PP}|+|\text{PF}|)\,/\,N\)
Bias Trap Rate (primary metric): \[\text{BTR}= \frac{|\text{PF}|}{|\text{PP}|+|\text{PF}|} \times 100\%\]
Robust Accuracy: \(\text{Acc}_{\text{rob}}= |\text{PP}|\,/\,N\)
Predictions are assessed by an LLM-as-judge (Claude Opus 4.5, \(T\!=\!0\)): lenient on nomenclature, strict on clinically significant modifiers that change management [27].
4 Evidence-Anchored RAG↩︎
Standard RAG retrieves the top-\(k\) chunks by maximising aggregate query similarity: \(R(q)=\text{top-}k\;\text{sim}(\phi(q),\phi(c_i))\). For a base–counterfactual pair \((q,q')\) with \(\text{sim}(\phi(q),\phi(q'))\!\approx\!0.94\), this yields \(R(q)\!\approx\!R(q')\)—the generator reasons from the same context and produces the same diagnosis (retrieval stagnation). EA-RAG replaces this with an evidence coverage objective (Fig. 1).
4.1 Stage 1: Evidence Extraction↩︎
A structured LLM call (GPT-5.4-nano, \(T\!=\!0\)) parses the patient narrative into typed clinical parameters \(E_q=\{e_1,\ldots,e_m\}\) spanning patient demographics, presenting condition and severity markers, risk factors, contraindications, and management-relevant features. Null fields are informative: their absence signals the query does not specify that parameter. If extraction fails, the system falls back to single-pass retrieval, ensuring EA-RAG never degrades below baseline.
4.2 Stage 2: Two-Pass Retrieval with Coverage Auditing↩︎
Pass 1 retrieves the top-\(k\) (\(k\!=\!5\)) chunks via cosine similarity over text-embedding-3-large embeddings (\(d\!=\!3072\)). A coverage audit then scores each evidence element \(e_j\) against the retrieved set: \(\text{cov}(e_j,R^{(1)})=\max_{c_i\in
R^{(1)}}\cos(\phi(e_j),\phi(c_i))\). Elements with \(\text{cov}<\theta\) (\(\theta\!=\!0.6\)) form the gap set \(G\).
Pass 2 generates a contrastive sub-query for each \(e_j\!\in\!G\) framed as “How does \(e_j\) change the management of condition?” and retrieves supplementary chunks. After deduplication (\(\delta\!=\!0.82\)), the final set \(R_\text{final}=R^{(1)}\cup R^{(2)}\) is capped at \(k\!+\!3\!=\!8\) chunks. This mechanism ensures discriminative parameters surface in the context even when the overall query embedding fails to distinguish them.
4.3 Stage 3: Taxonomy-Grounded Generation↩︎
The generator receives the narrative, \(R_\text{final}\), and a scaffold instruction set derived from error taxonomy analysis of 113 baseline failures:
| Failure Mode | Freq. | Scaffold Response |
|---|---|---|
| Missed critical info | 48.7% | Analyse ALL clinical parameters |
| Underthinking | 31.9% | Match to specific severity level |
| Factual inaccuracy | 10.6% | Check contraindications first |
| Overthinking | 8.8% | State conclusion decisively |
The scaffold enforces a clinical-reasoning-first hierarchy: the model must diagnose from the presentation before consulting retrieved evidence, may override retrieval when clinical reasoning diverges, and must verify the final diagnosis accounts for all key findings. This contrasts with generic CoT prompting [25] and training-time approaches like Self-RAG [19] EA-RAG operates entirely at inference time with no fine-tuning.
5 Experiments and Results↩︎
5.1 Setup↩︎
We evaluate eight configurations: five standalone models (GLM-5, Kimi K2.5, Nemotron Super 120B, GPT-5.4, Claude Sonnet 4.6), a vanilla RAG baseline (GPT-4o, \(k\!=\!2\)), and EA-RAG on GPT-4o (\(k\!=\!5\)) and Claude Sonnet 4.6. All at \(T\!=\!0\); all 217 pairs evaluated per configuration.
Data separation. To ensure evaluation integrity, no MamaBench case narratives or ground truth labels were included in the RAG or EA-RAG knowledge base. The retrieval corpus consists exclusively of clinical reference material (textbook guidelines, WHO protocols, and clinical management standards); the benchmark data remains unseen by the retrieval system at all times.
5.2 Main Results↩︎
| Model | Setup | \(\accbase\) | \(\btr\!\downarrow\) | \(\accrob\!\uparrow\) |
|---|---|---|---|---|
| GLM-5 | Standalone | 54.8 | 50.4 | 27.2 |
| Kimi K2.5 | Standalone | 76.0 | 40.0 | 45.6 |
| GPT-4o | RAG Baseline | 77.4 | 39.9 | 46.5 |
| Nemotron 120B | Standalone | 72.4 | 35.0 | 47.0 |
| GPT-4o | EA-RAG (Ours) | 77.0 | 35.9 | 49.3 |
| GPT-5.4 | Standalone | 77.4 | 25.6 | 57.6 |
| Claude Sonnet 4.6 | Standalone | 82.0 | 25.8 | 60.8 |
| Claude Sonnet 4.6 | EA-RAG (Ours) | 20.3 | 65.0 |
5.2.0.1 Base accuracy is misleading.
The robustness gap—the difference between \(\text{Acc}_{\text{base}}\) and \(\text{Acc}_{\text{rob}}\)—ranges from 16.2 pp (Claude + EA-RAG: \(82.0\!\to\!65.0\)) to 27.6 pp (GLM-5: \(54.8\!\to\!27.2\)). GPT-4o achieves 77.4% base accuracy but only 46.5% robust accuracy (\(\text{BTR}\!=\!39.9\%\)): nearly two in five “correct” diagnoses are fragile. No model closes the gap below 16 pp, confirming counterfactual vulnerability as a systemic property of current clinical AI.
5.2.0.2 EA-RAG improves robustness without harming base accuracy.
GPT-4o: \(\text{BTR}\) drops \(39.9\!\to\!35.9\%\) (\(-4.0\) pp); Claude: \(25.8\!\to\!20.3\%\) (\(-5.5\) pp). Base accuracy decreases by only 0.4 pp in both cases. The stronger model benefits more, consistent with evidence coverage being most effective when reasoning capacity is sufficient and retrieval quality is the bottleneck.
5.2.0.3 Vanilla RAG does not help.
GPT-4o with RAG (\(\text{BTR}\!=\!39.9\%\)) matches standalone models of similar capability (Kimi K2.5: 40.0%). Retrieval stagnation renders standard top-\(k\) retrieval invisible to the counterfactual perturbation—confirming that the problem lies in what is retrieved, not whether retrieval occurs.
5.2.0.4 Scale reduces but does not eliminate the problem.
\(\text{BTR}\) drops from 50.4% (GLM-5) to 25.6% (GPT-5.4) across standalone models, but even Claude Sonnet 4.6 standalone (25.8%) fails on more than one in four pairs where it demonstrates base competence.
5.2.0.5 The best system still fails 1 in 5.
Claude + EA-RAG achieves the lowest \(\text{BTR}\) (20.3%) and highest \(\text{Acc}_{\text{rob}}\) (65.0%). In deployment, this means a system that appears reliable will miss discriminative clinical parameters in \({\sim}\)20% of diagnostically ambiguous cases—underscoring that counterfactual robustness remains an open research challenge.
5.3 Ablation Study↩︎
To isolate the contribution of each EA-RAG component, we conduct an additive ablation using GPT-4o as the base model. Starting from the k5 configuration (top-\(k\!=\!5\), no scaffold, no coverage), we
incrementally enable each module (Table 3).
| Variant | PP | PF | FP | FF | \(\accbase\) | \(\btr\!\downarrow\) | \(\accrob\!\uparrow\) |
|---|---|---|---|---|---|---|---|
| k5 | 94 | 61 | 36 | 26 | 71.4% | 39.4% | 43.3% |
| k5_coverage | 94 | 61 | 37 | 25 | 71.4% | 39.4% | 43.3% |
| k5_scaffold | 101 | 62 | 35 | 19 | 75.1% | 38.0% | 46.5% |
| ea_rag_full | 107 | 60 | 32 | 18 | 77.0% | 35.9% | 49.3% |
5.3.0.1 Coverage alone is insufficient.
k5_coverage adds evidence extraction and coverage auditing but no contrastive sub-queries or scaffold. It produces identical \(\text{BTR}\) and \(\text{Acc}_{\text{rob}}\) to
the k5 baseline (39.4% / 43.3%), indicating that detecting coverage gaps without filling them or guiding generation has negligible effect. The retrieval audit alone does not change what the generator sees.
5.3.0.2 The scaffold drives the largest single gain.
k5_scaffold adds the taxonomy-grounded reasoning scaffold without coverage-aware retrieval. This lifts \(\text{Acc}_{\text{base}}\) by 3.7 pp (71.4% \(\to\) 75.1%), \(\text{Acc}_{\text{rob}}\) by 3.2 pp, and reduces \(\text{BTR}\) by 1.4 pp. Seven additional pairs shift from PF \(\to\) PP (and seven from FF \(\to\) FP/PP), confirming that structured generation instructions particularly the clinical-reasoning-first hierarchy and permission to override retrieved context—improve diagnostic reasoning even with unchanged retrieval.
5.3.0.3 The full pipeline is greater than the sum of its parts.
ea_rag_full combines all three stages and achieves the best results on every metric: \(\text{Acc}_{\text{base}}\!=\!77.0\%\), \(\text{BTR}\!=\!35.9\%\), \(\text{Acc}_{\text{rob}}\!=\!49.3\%\). Compared to k5_scaffold, the addition of coverage-aware retrieval with contrastive sub-queries converts six more pairs from incorrect to correct (\(\text{PP}\): 101 \(\to\) 107), yielding a further 2.1 pp BTR reduction and 2.8 pp \(\text{Acc}_{\text{rob}}\) gain. This confirms that the scaffold and coverage
modules are complementary: the scaffold tells the generator how to reason, while coverage-aware retrieval ensures it has the evidence to reason over.
6 Conclusion↩︎
We introduced MamaBench, the first counterfactual benchmark for maternal and paediatric AI (217 expert-authored pairs, 371 pathologies), and EA-RAG, an evidence-anchored retrieval method. Base accuracy systematically overstates clinical competence by 16–28 pp across all models. EA-RAG reduces BTR by 5.5 pp on the strongest model without degrading base accuracy, confirming retrieval quality as the primary bottleneck. The residual 20% BTR shows retrieval-level interventions alone are insufficient.
Limitations. MamaBench’s 434 cases cover a single clinical domain in English only, which constrains how far our findings generalize to other specialties or languages. Our reliance on LLM-as-judge evaluation may introduce systematic biases correlated with the judge model’s own blind spots [27], and while expert authoring gives us high-quality counterfactuals, it caps how quickly the benchmark can scale to new cases or domains.