1 Introduction↩︎

Clinical documentation consumes a disproportionate share of clinician time, and there is growing interest in systems that can automatically convert patient–clinician conversations into structured notes [1]–[3]. SOAP note generation is particularly appealing because the four-section format (Subjective, Objective, Assessment, Plan) is both clinically familiar and directly usable in downstream documentation workflows. The task is nevertheless demanding: a high-quality note must be concise, complete, and strictly grounded in dialogue evidence—plausible-sounding content that is not supported by the conversation is a patient-safety risk, not merely an annotation error.

Frontier models now expose explicit reasoning modes, and recent medical reasoning benchmarks report encouraging gains on diagnostic and treatment tasks [4]. For clinical text tasks, however, the picture is more nuanced. Chain-of-thought prompting has been shown to degrade performance on real-world clinical text understanding [5], and evaluation-oriented frameworks such as MEDIVAL examine reasoning-enhanced judging of documentation rather than reasoning-enhanced generation of it [6]. These observations raise two linked questions that we set out to answer: (1) when the task is structured SOAP note generation, does provider-native reasoning actually help, and (2) do reference-aware LLM judges support the same conclusions as automatic overlap metrics?

To answer both questions, we construct a source-aware benchmark from three complementary datasets and evaluate three frontier models under a fully factorial \(2{\times}2\) design that independently toggles reasoning and RAG. Using per-source reporting throughout prevents the large synthetic OMI Health corpus from obscuring the smaller, clinically closer ACI-Bench and PriMock57 datasets.

Our contributions are:

• A source-aware evaluation benchmark for SOAP note generation that preserves dataset identity and reports source-macro averages as the primary criterion.

• A controlled \(2{\times}2\) (reasoning \(\times\) RAG) comparison of three frontier models across 326 test examples.

• A dual LLM-as-a-judge evaluation that scores candidate notes for faithfulness, clinical coverage, safety, and SOAP structure using the dialogue, reference note, and candidate note jointly.

• Empirical evidence—consistent across seven automatic metrics and two independent judges—that reasoning capability does not reliably improve SOAP note generation and can substantially harm it.

2 Related Work↩︎

Work on clinical note generation from dialogue has focused primarily on benchmark creation and generation pipelines. ACI-Bench introduced a benchmark of role-played clinical encounters with structured visit notes [1], while PriMock57 released mock primary-care consultations with transcripts and notes to enable documentation research without PHI constraints [7]. The MEDIQA-Chat 2023 shared task standardized evaluation across note generation subtasks and established few-shot GPT-4 as a strong baseline [8]. More recent systems move toward modular pipelines: ClinicSum combines retrieval-based filtering with fine-tuned generation to reduce hallucination [2], and SpecialtyScribe introduces a three-component architecture for oncology that matches frontier model performance at a fraction of the cost [3]. These studies establish that LLMs and retrieval-augmented pipelines can be useful for clinical documentation, but they do not isolate whether explicit reasoning modes improve the generation of structured SOAP notes. Our work differs from all of these in that it is not a new generation architecture; it is a controlled evaluation of whether reasoning and retrieval independently and jointly improve SOAP note quality.

Clinical RAG systems have shown that retrieval can improve structured documentation, especially when evidence is fragmented. ClinicSum uses retrieval-based filtering before generation [2], MedRAG benchmarks multiple corpora against a medical QA framework and reports up to 18% improvement over chain-of-thought prompting [9], and CLI-RAG proposes hierarchical retrieval over heterogeneous EHR note types for progress-note synthesis [10]. These approaches motivate our retrieval condition but treat RAG as a standalone intervention without isolating reasoning mode as an experimental variable. Thus, while prior work has studied RAG for clinical note and EHR generation, the specific impact of provider-native reasoning on generated SOAP-note quality remains underexplored.

Recent medical reasoning benchmarks report encouraging results on diagnostic and treatment tasks [4], and o1-style models show gains on reasoning-intensive medical QA [11], [12]. However, the evidence for clinical text tasks is less straightforward: a large-scale study across 95 LLMs and 87 clinical text understanding tasks found that 86.3% of models suffered consistent performance degradation with chain-of-thought, attributing the failure to the noisy and fragmented nature of clinical text [5]. Evaluation-oriented work such as MEDIVAL examines reasoning-enhanced judging of documentation rather than reasoning-enhanced generation of it [6], leaving the generation direction open. The closest work to ours integrates CoT and RAG for rare disease diagnosis from clinical notes, finding that architecture order matters depending on note quality [13], but does not examine the interaction for structured SOAP note generation, where the output is fixed-format and fidelity-sensitive. A scoping review of 67 RAG studies in medical domains confirms that only 26 included explicit reasoning support, with most treating RAG and reasoning as independently deployed rather than jointly evaluated [14].

Finally, automatic metrics are known to correlate weakly with expert judgment for clinical text [15], motivating the use of LLM judges as a complementary evaluation signal. Rubric-based LLM evaluation achieves clinician-level inter-rater reliability at substantially lower cost [16], and multi-judge setups help surface calibration differences across judge models [17]. Our dual-judge design—using GPT-5.4 mini and Gemma-4-E4B with a shared rubric emphasizing faithfulness, clinical coverage, safety, and SOAP structure—follows this paradigm and lets us assess whether qualitative rankings hold across judge families.

3 Method↩︎

3.1 Source-aware benchmark↩︎

We normalise three publicly available datasets into a unified, source-aware benchmark while retaining a source_dataset field for every example. OMI Health contributes 10,000 synthetic dialogue–SOAP pairs based on NoteChat-style clinical-note-conditioned conversations, ACI-Bench contributes 112 role-played clinical encounters, and PriMock57 contributes 57 mock primary-care consultations [1], [7], [18], [19]. The combined corpus contains 10,169 examples split into 9,324 train, 519 validation, and 326 test items.

Because OMI Health contributes 97% of examples, pooled means would mask performance differences on the two smaller, clinically closer datasets. We therefore adopt source-macro averages—the unweighted mean of the three per-source means—as our primary reporting criterion, with pooled means provided as secondary context.

3.2 Experimental design↩︎

We evaluate each test example under a \(2 \times 2\) factorial design over reasoning and RAG (each in \(\{\text{off}, \text{on}\}\)).

3.2.0.1 Reasoning.

Reasoning is toggled exclusively through provider-native mechanisms; no “let’s think step by step” prompt variants are used, ensuring that any observed effect is attributable to the model’s internal reasoning process rather than prompt phrasing. This strict definition restricts the primary comparison to models with verifiable reasoning-off support: GPT-5.4, DeepSeek-V4-Flash, and Gemma-4-E4B.

3.2.0.2 RAG.

The retrieval setup is deliberately minimal. For each test example, we retrieve exactly one training example from the same source dataset using Chroma with text-embedding-3-small embeddings [20], [21]. The retrieved example is presented as a stylistic and structural reference, not as factual content to be copied verbatim. This conservative design isolates the structural guidance that in-context examples can provide without introducing retrieval noise from cross-dataset or off-topic documents.

3.2.0.3 Evaluation pipeline.

OMI Health, ACI-Bench, and PriMock57 are normalised into a source-aware benchmark. Each test dialogue is passed to GPT-5.4, DeepSeek-V4-Flash, and Gemma-4-E4B under all four reasoning \(\times\) RAG conditions. All models are used at their provider-default temperature settings. RAG retrieves one same-source training example per query. Each generated note is scored with seven automatic metrics and two reference-aware LLM judges. Final comparisons use source-macro means so OMI Health does not dominate ACI-Bench and PriMock57.

3.3 Automatic metrics↩︎

We score each predicted note with ROUGE-1, ROUGE-2, ROUGE-L, METEOR, chrF++, BERTScore F1, and ClinicalBERT F1 [22]–[26]. For example \(i\), model \(m\), and condition \(c\), the composite score is: \[s_{i,m,c} = \frac{1}{7}\sum_{k=1}^{7} \mathrm{metric}_{k}(i,m,c).\] The composite aggregates lexical and semantic similarity signals, reducing the influence of any single metric.

3.4 LLM-as-a-judge evaluation↩︎

To complement overlap metrics, GPT-5.4 mini and Gemma-4-E4B judge each saved generation against the dialogue and gold SOAP note on a 1–5 rubric covering faithfulness, clinical coverage, hallucination, safety, and SOAP organisation. Using two judge families helps separate robust findings from judge-specific calibration.

The 326-example test split contains 250 OMI Health, 66 ACI-Bench, and 10 PriMock57 examples. All four conditions use the same examples, so reasoning and RAG effects are paired within-example comparisons.

4 Results↩︎

4.1 Automatic metrics↩︎

Table 1 reports source-macro composite scores for all model–condition pairs.

4.1.0.1 Without reasoning, GPT-5.4 leads.

GPT-5.4 achieves the highest source-macro composite without reasoning: 0.515 direct and 0.527 with RAG, the best overall configuration.

4.1.0.2 With reasoning, DeepSeek-V4-Flash leads.

Once reasoning is enabled, DeepSeek-V4-Flash is strongest, reaching 0.500 direct and 0.512 with RAG.

4.1.0.3 Reasoning imposes a consistent penalty on GPT-5.4.

Enabling reasoning reduces GPT-5.4’s composite score by 0.063 on ACI-Bench, 0.033 on OMI Health, and 0.088 on PriMock57 in the direct setting. These degradations replicate across all three sources and corroborate evidence that explicit reasoning can hurt clinical text tasks [5].

4.1.0.4 RAG is helpful but secondary and uneven.

RAG yields positive but smaller and source-dependent gains; for example, on OMI Health without reasoning it adds +0.026 for DeepSeek-V4-Flash and +0.033 for Gemma-4-E4B. Table 2 summarises representative deltas.

Full per-metric tables and source decompositions are provided in Tables 3–6.

4.2 LLM-as-a-judge results↩︎

Table 7 gives full judge scores. Despite scale differences, both judges rank a non-reasoning GPT-5.4 variant first overall and DeepSeek-V4-Flash first among reasoning-enabled configurations. They also agree that reasoning strongly penalises GPT-5.4: by 0.824 (direct) and 0.488 (RAG) under GPT-5.4 mini Judge, and by 1.082 and 0.656 under Gemma-4-E4B Judge. RAG effects are smaller and judge-dependent, leaving the central conclusion unchanged.

5 Discussion↩︎

The empirical pattern across both automatic metrics and LLM judges is consistent and interpretable. Structured SOAP note generation appears to reward grounded compression and format adherence over open-ended deliberation: a model that “reasons well” on diagnostic benchmarks does not automatically produce better SOAP notes from dialogue. This gap is not attributable to any single dataset or evaluation metric; it replicates across three heterogeneous sources and two independent judge families.

6 Conclusion↩︎

We presented a source-aware evaluation of frontier reasoning-capable models for SOAP note generation under a controlled \(2{\times}2\) reasoning \(\times\) RAG design. Across three datasets, seven automatic metrics, and two independent LLM judges, provider-native reasoning did not consistently improve note quality and often reduced it substantially—especially for GPT-5.4. Same-source RAG provided modest, model-dependent gains that were consistently smaller than the quality gap introduced by enabling reasoning. DeepSeek-V4-Flash was the most robust model once reasoning was turned on, but even its best reasoning configuration did not match the top non-reasoning GPT-5.4 result.

The practical implication is direct: reasoning-enabled LLMs should be evaluated on clinical documentation tasks in their target configuration, not assumed to transfer gains from general or diagnostic reasoning benchmarks. More broadly, structured, fidelity-sensitive generation tasks may constitute a qualitatively different challenge from the open-ended reasoning tasks where thinking modes were developed—and they deserve purpose-built evaluation to reflect that distinction.

7 Limitations↩︎

Several limitations constrain the conclusions. First, OMI Health dominates the corpus by volume, so source-macro reporting is essential; pooled means would overweight the synthetic distribution. Second, ACI-Bench and PriMock57 are small, making per-source estimates noisier and per-example effects hard to generalise. Third, LLM judges are not clinician raters. They inspect the dialogue, reference note, and candidate note jointly, which provides a richer signal than overlap metrics alone, but they may encode model-family preferences and their calibration clearly differs (as the scale gap between the two judges illustrates). Fourth, the reference-aware rubric may partly reward paraphrastic similarity to the gold note’s phrasing and structure rather than independent clinical quality. Fifth, provider-native reasoning controls are not fully standardised across vendors, which is why strict reasoning-off support was required as an inclusion criterion. This also limited the model set: API budgets and the cost of running all reasoning \(\times\) RAG conditions restricted the study to three frontier models and two LLM judges, so broader evaluation across additional open, commercial, smaller, and specialty-tuned clinical models is needed. Sixth, our analysis is note-level rather than section-level. We have not yet decomposed performance by SOAP section, so it remains unclear whether reasoning harms or helps Subjective, Objective, Assessment, and Plan content in the same way.

These limitations point to productive directions for follow-up work: larger clinically realistic test sets, expert physician evaluation, expanded model coverage under larger API budgets, section-wise SOAP analysis, and model-agnostic reasoning interventions that allow broader comparison.

8 Additional Figures↩︎

9 Additional Tables↩︎

4pt

4pt

4pt

4pt

10 Generation and Judge Prompts↩︎

References↩︎