1 Introduction↩︎

LLMs are increasingly used by patients as a source of medical information, offering unprecedented accessibility to medical guidance outside traditional clinical settings [1]. Surveys indicate that a substantial and growing fraction of patients now seek health advice from LLMs, particularly where access to clinical care is limited [2]. In these interactions, patients often ask questions that embed incorrect assumptions or misconceptions about their condition, symptoms, or treatment [3]. These misconceptions are often implicit and shaped by incomplete knowledge or prior beliefs. Safe medical communication, therefore, requires not only answering the question but also identifying and correcting the underlying false belief rather than accepting the premise as stated.

Moreover, in clinical practice, patient–physician interactions are inherently conversational, unfolding over multiple turns as patients refine their concerns, introduce new information, or reinterpret prior guidance [11]. This pattern is increasingly reflected in interactions with LLMs, where users engage in iterative dialogue rather than isolated queries, and misconceptions may arise at any point in the conversation (Figure 1). However, existing evaluation frameworks do not assess model behavior in these settings. Early medical QA datasets primarily emphasize factual recall and exam-style reasoning. MedQA [4], MedMCQA [6], and PubMedQA [5] are constructed from licensing examinations or biomedical literature, offering limited insight into real-world patient interaction [12], [13]. While later datasets incorporate consumer-facing questions, including presuppositions [14], [15], they continue to evaluate responses in isolation [8]–[10]. More recent efforts probe multi-turn robustness through synthetic or adversarial follow-ups, yet they do not capture how misconceptions arise and evolve in authentic multi-turn interactions that real patients have when asking medical questions around their conditions [7], [16]. Consequently, a critical question remains unanswered: Can LLMs reliably identify and correct patient misconceptions as conversations unfold over multiple turns of patient-authored interactions?

To address this question, we introduce ThReadMed-QA, a multi-turn medical dialogue dataset derived from real patient–physician interactions on r/AskDocs. The dataset includes 2,437 conversation threads, comprising a total of 8,204 question–answer pairs. Unlike prior resources, ThReadMed-QAcaptures how real patients iteratively seek medical guidance, preserving the natural flow of follow-up questions, clarifications, and evolving concerns across turns. As shown in Table 1, it is the first dataset to combine patient-authored questions, multi-turn interactions, and physician-verified responses within a single evaluation setting. This structure enables systematic evaluation of whether LLMs can identify and correct misconceptions not only at initial contact, but as they emerge, persist, or evolve throughout a conversation.

We evaluate five state-of-the-art LLMs on their ability to identify and correct patient misconceptions in multi-turn conversations. Specifically, we assess whether models detect false assumptions embedded in patient questions and appropriately challenge or correct them as they arise throughout the interaction. To enable scalable evaluation, we employ a rubric-based LLM-as-a-Judge framework, validated against expert evaluations, to score how effectively models handle misconceptions. Our results reveal a consistent failure mode: models do not reliably sustain this behavior over time, with performance deteriorating as conversations unfold. This instability indicates that misconception correction by LLMs is not a persistent capability, but one that breaks down under realistic multi-turn interactions.

Generalizable Insights about Machine Learning in the Context of Healthcare↩︎

Our study highlights an important gap in how patient-facing medical LLMs are currently evaluated. We show that the ability to identify and correct patient misconceptions is not a stable capability, but one that degrades as interactions progress across turns. This highlights the need to evaluate how models handle misconceptions as they emerge and evolve across interactions. By enabling evaluation on real patient-authored questions and follow-ups with physician responses, ThReadMed-QAprovides a foundation for studying model behavior under realistic conditions. Beyond misconception handling, our proposed setup can support the evaluation of other safety-critical behaviors that emerge during interaction. Our findings suggest that future benchmarks and model development must explicitly account for interaction dynamics when evaluating patient-facing systems. To facilitate future research in this direction, we release the code ¹ and the dataset ².

2 Related Work↩︎

2.0.0.1 Evolution of medical QA datasets

Early benchmarks for medical question answering primarily emphasize factual recall and exam-style reasoning. Datasets such as MedQA [4], MedMCQA [6], and PubMedQA [5] are constructed from medical licensing examinations or biomedical literature, enabling controlled evaluation of medical knowledge but offering limited insight into real-world patient interaction [12], [13]. Subsequent efforts sought to better reflect consumer-facing use cases by incorporating health-related search queries and community-authored questions, including Medication QA [9], and HealthSearchQA [10]. There are several datasets constructed from online health forums, search logs, and tele-health consultations [8], [17]. These datasets improve coverage of lay language and consumer concerns, yet they continue to evaluate models largely through single-turn settings. To our knowledge, ThReadMed-QAis the first multi-turn medical dataset with patient-authored questions and follow-ups.

2.0.0.2 Medical misconceptions and LLM Sycophancy

Questions that contain misconceptions have long been studied in linguistics [18], [19], where the appropriate response is often to challenge or negate the incorrect presupposition. However, LLMs are known to align with users even when they are factually incorrect [20], termed as sycophancy. This tendency is particularly concerning in medical contexts, where incorrect assumptions may involve diagnoses, medication safety, or symptom severity. Recent work introduced datasets to explore misconception handling in both general domain [21] and specific medical settings, including pregnancy and maternal health [3] and cancer care [15]. However, most prior work focuses on single-turn settings [14] or relies on synthetically generated questions [15], which lack patient-specific context and underestimate the difficulty of real interactions where misconceptions emerge implicitly and evolve over time. Our dataset addresses this gap by enabling evaluation on authentic multi-turn patient interactions.

2.0.0.3 Multi-turn LLM Interaction in Medicine

Recent work highlights a disconnect between how medical LLMs are evaluated and how they are used in practice [13], [22]. Systematic reviews show that the majority of medical LLM evaluations rely on synthetic or exam-style data, with minimal use of real clinical or patient-generated text [23]. On the other hand, research reports LLMs having near-clinician accuracy in single-turn settings [24] in some tasks, their performance degrades sharply when moving to multi-turn interactions [16]. Error propagation is also a pattern observed in prior work on sequential and dialogue systems, showing that earlier model outputs can influence later predictions, producing compounding failures across an interaction [25], [26]. In medical settings this brittleness is especially concerning: models that appear competent on static questions may fail to maintain consistency, challenge incorrect premises, or preserve appropriate safety guidance as a conversation unfolds. However, [7] experimented with adversarial set-ups in examination-style MCQs and does not model real patient information-seeking trajectories. Our work addresses this gap by explicitly evaluating multi-turn patient–LLM interactions grounded in authentic patient behavior.

3 Dataset↩︎

We introduce ThReadMed-QA, a multi-turn medical dialogue dataset derived from real patient–physician interactions on r/AskDocs, a Reddit community where patients seek medical advice from healthcare professionals. Unlike prior QA datasets, ThReadMed-QAcaptures extended patient–physician dialogues in which patients ask follow-up questions to clarify, refine, or challenge initial answers.

3.1 r/AskDocs↩︎

We collect data from the subreddit r/AskDocs³, an online forum where lay users seek medical advice from a community of physicians and healthcare professionals. In this setting, users post detailed descriptions of their symptoms, medical history, and concerns, and may engage in follow-up questions as the conversation evolves. Responses are often provided by community-verified medical professionals, whose credentials are reviewed by subreddit moderators and indicated through user flairs or tags (e.g., “physician”, “doctor”). This structure enables multi-turn, patient-physician dialogues grounded in realistic clinical communication, where initial questions are frequently refined or expanded through subsequent interaction. The forum enforces structured posting guidelines that encourage users to include relevant demographic and clinical context, leading to detailed and consistent patient queries. As a result, r/AskDocs provides a natural source of longitudinal patient–physician exchanges suitable for studying how misconceptions arise and are addressed by humans over the course of a conversation.

3.2 Dataset Construction↩︎

Similar to prior work [8], we collected posts from January 2015 through June 2023 from r/AskDocs. We then apply standard pre-processing, following [8] to remove low-quality or incomplete content, including deleted posts and bot-generated responses. We also collected all comments from each post to reconstruct patient–physician exchanges. The same cleaning pipeline is applied to comments. An overview of the construction process is outlined in Figure 2 and more details about pre-processing and dataset construction are in Appendix 7.

3.3 Multi-turn Construction↩︎

We construct linear patient–physician conversation threads by retaining interactions between the original poster (OP) and verified medical professionals, identified via r/AskDocs flair. Follow-up questions are restricted to the OP to preserve coherent patient narratives. For posts with multiple comment branches, we select the branch most relevant to the original question using a word-overlap heuristic [27], favoring on-topic discussions. We require at least two follow-up questions per thread to ensure meaningful multi-turn structure and filter out threads with incomplete or low-quality responses.

3.3.0.1 Orphan Handling:

Some threads contain unanswered follow-up questions, which we refer to as orphans. To enable evaluation with complete ground-truth responses, we construct a fully answered subset by retaining only threads in which every question has a physician response. After filtering, we obtain our primary dataset of 2,437 conversation threads comprising 8,204 question–answer pairs.

Table 2 summarizes key statistics of the dataset. The complete collection contains 9,741 conversation threads (37,191 question–answer pairs), while the fully answered subset used for evaluation contains 2,437 threads (8,204 pairs). The data span 2015-2023, covering a period of increasing reliance on online medical advice. Conversations often involve patients providing detailed context (mean of 129 tokens) and refining their concerns through follow-up questions, while physician responses tend to be more concise (mean of 75 tokens) and targeted. This asymmetry reflects realistic patient–physician communication and creates a setting where misconceptions by patients may be introduced, clarified, or persist across turns.

4 Methods↩︎

4.1 Misconception Identification↩︎

We first identify questions that contain medical misconceptions. Each question is labeled using an LLM-based classifier as containing a misconception if it embeds an incorrect assumption or false presupposition about a medical condition, treatment, or outcome. LLM-as-a-Judge was validated on a subset of 310 questions against physician judgment with an agreement rate of 90% and Cohen’s \(\kappa\) = 0.72 (substantial agreement, following [15], [28] in the clinical domain). We provide detailed information on validation in Appendix 8. We retain all conversation threads containing at least one such question, yielding a subset of 505 conversation threads used for evaluation. We provide the full prompt in the Appendix 11.

4.2 Multi-Turn Inference↩︎

For each thread, models generate responses sequentially with full conversational context. At turn \(t\), the model receives all prior patient questions and its own previous responses:

[User]: \(\langle q_0 \rangle\)
[Assistant]: \(\langle a_0 \rangle\)
[User]: \(\langle q_1 \rangle\)
[Assistant]: \(\langle a_1 \rangle\)
[User]: \(\langle q_2 \rangle\)
\(\vdots\)

Each response conditions on all prior model outputs, such that the model’s own previous answers form part of the context for subsequent turns.

4.3 Models↩︎

We evaluate five state-of-the-art LLMs spanning the proprietary frontier and open-source categories: GPT-5 [29] (snapshot: gpt-5-2025-08-07), GPT-4o [30], Claude Haiku 4.5 [31] (snapshot: claude-haiku-4-5-20251001), Gemini 2.5 Flash [32], and Llama 3.3–70B-Instruct [33]. All models are evaluated zero-shot with no specific system prompt, no few-shot examples, or medical-domain specialization are applied, assessing out-of-the-box capability. Temperature is set to 0 for all models that support it. We only experiment with general-domain state-of-the-art models, as numerous studies, including [34] and [35], have shown that domain-specific models often underperform and are rarely deployed in patient-facing interfaces.

4.4 Evaluation Framework↩︎

We evaluate model responses using a rubric-based LLM-as-a-judge framework, following [15]. Each response is scored on a three-point scale (full prompt in Appendix 11):

• Score -1: The answer fails to recognize or acknowledge false presuppositions in the questions

• Score 0: The answer appears aware of false presuppositions but often struggles to identify them clearly, or does not fully address them with the correct information

• Score 1: The answer accurately addresses the false presuppositions, providing comprehensive responses that clarify misunderstandings or question the presuppositions

We validate our LLM-as-a-Judge with physician judgment, with an agreement rate of 87.14% and Cohen’s \(\kappa\) = 0.68 (substantial agreement). We provide detailed information on validation in Appendix 8. We also adapt multiple models as LLM judges and aggregate the scores to de-bias the judgments (more information in Appendix 8) following prior work [36].

4.5 Metrics↩︎

4.5.1 Single-turn Metrics:↩︎

We adopt single-turn metrics from [15]. PCR (Presupposition Correction Rate) measures the proportion of responses that correctly address the misconception (score \(=1\)), while PCS (Presupposition Correction Score) is the average score over \(\{-1, 0, 1\}\), capturing partial and incorrect responses.

4.5.2 Multi-turn Metrics:↩︎

To analyze model behavior over the course of a conversation, we extend single-turn metrics to the multi-turn setting.

4.5.2.1 Per-turn Performance:

We compute the Presupposition Correction Rate (PCR) at each turn \(t\): \[\text{PCR}_t = \frac{1}{N_t} \sum_{i=1}^{N_t} \mathbb{I}[s_{i,t} = 1]\] where \(s_{i,t} \in \{-1, 0, 1\}\) is the score for example \(i\) at turn \(t\), and \(N_t\) is the number of evaluated instances at that turn and \(\mathbb{I}(\cdot)\) is the indicator function, equal to 1 if the condition holds and 0 otherwise.

4.5.2.2 Recovery Rate:

We measure the ability of models to recover from prior failures in misconception correction across turns. Specifically, we compute the conditional probability that a model correctly resolves a misconception at turn \(t+1\) given that it failed to do so at turn \(t\): \[\text{RR} = P(s_{t+1} = 1 \mid s_t \neq 1)\] where \(s_t \in \{-1, 0, 1\}\) denotes the score at turn \(t\). Higher values indicate that a model is more likely to recover and correctly address a misconception after previously failing. We estimate this probability over all consecutive turn pairs \((t, t+1)\) where both turns are labeled as containing misconceptions.

4.6 Oracle Physician Ablation↩︎

To separate the effect of question difficulty and embedded false presuppositions from errors introduced by prior model-generated context, we conduct an oracle-style ablation in which all preceding turns use verified physician responses from the r/AskDocs thread, rather than the model’s own prior outputs.

For each multi-turn thread, let patient turns be indexed as \(t = 0, 1, \ldots, T-1\). For each target turn \(t \geq 1\), we construct the input by concatenating the dialogue history up to that turn. Specifically, for each \(i < t\), we include the patient question \(Q_i\) and the corresponding physician response \(A^{\mathrm{phys}}_i\). For the target turn, we append only the patient question \(Q_t\), which the model must answer. Under this setup, the model conditions exclusively on the oracle (physician) context and never on its own prior generations. Since turn \(t = 0\) has no preceding in-thread physician responses, oracle prompts are defined only for \(t \geq 1\). We report PCR (%) and Uncorrected (%) in this setting to assess the impact of oracle context on model behavior. We illustrate this setup in Figure 3.

5 Results↩︎

Figure 4 summarizes overall misconception correction performance aggregated across all turns. As shown in Figure 4 (a), frontier models achieve higher correction rates, with Claude Haiku and GPT-5 exceeding 60% PCR, while Gemini 2.5 Flash performs moderately (43.0%). In contrast, GPT-4o and Llama 3.3–70B show substantially lower correction rates. Figure 4 (b) provides a complementary view by reporting the proportion of responses that fail to fully correct misconceptions. These uncorrected rates remain high across all models, exceeding 35% even for the strongest systems and rising above 65% for weaker models. This indicates that even when evaluated across all turns, models frequently provide incomplete or incorrect responses to misconception-laden questions.

5.1 Observed Correction Rates Decline at Later Turns↩︎

We next analyze model performance across turns within a conversation by measuring PCR at each turn. As shown in Figure 5, performance decreases consistently as conversations progress. Even frontier models that achieve strong initial performance fail to sustain this behavior over time. GPT-5 starts at 88.1% PCR at the first turn but drops to 48.0% by turn 2, a reduction of over 40 percentage points. Similarly, Claude Haiku declines from 84.4% to 53.5% over the same span, indicating that initial success does not translate into stable multi-turn performance. This effect is more pronounced for weaker models. GPT-4o drops from 65.1% at the first turn to 21.2% by turn 2, while Llama 3.3–70B falls from 55.0% to 17.7%. At these levels, most misconception-laden queries are no longer correctly addressed. We report detailed results in Appendix 9.

Across all models, the largest drop occurs within the first few turns, after which performance stabilizes at substantially lower levels. Overall, these results show that misconception correction is not a stable capability, and models struggle to maintain it as conversations unfold.

5.2 Limited Recovery Across Turns↩︎

While the previous analysis shows that performance declines across turns, we next examine whether models can recover after an initial failure. Table 3 reports recovery rates (RR) with 95% confidence intervals estimated via bootstrap resampling (1,000 iterations). Because RR is conditioned on prior failure, the evaluated instances differ across models; consequently, RR should not be interpreted as a strict ranking of model capability, but rather as a diagnostic measure of how difficult it is for models to recover once an incorrect conversational trajectory has been established.

Recovery remains limited across all systems, indicating that once a model fails to correct a misconception, it is unlikely to recover in subsequent turns. Even stronger models show only partial recovery. Claude Haiku achieves the highest recovery rate at 51.9% (33.3–69.6), while GPT-5 recovers in only 25.9% (9.5–45) of cases following a failure. Recovery is substantially lower for weaker models, with GPT-4o and Llama 3.3–70B below 15%, and confidence intervals reflecting high uncertainty due to the limited number of failure-start pairs.

Taken together, these results suggest that once models fail to identify or correct a misconception, recovery is uncommon. Although RR is conditioned on model-specific failures and is not intended for direct cross-model comparison, it highlights a shared vulnerability where models struggle to recover from erroneous conversational trajectories.

5.3 Oracle Physician Context Substantially Improves Performance↩︎

To isolate the effect of conversational context, we evaluate models in an oracle setting where prior turns (LLMs’ responses) are replaced with physician-provided responses. This removes error accumulation from earlier model outputs and allows us to assess performance when conditioned on the correct context. Figure 6 shows that all models improve substantially under the oracle context. Correction rates increase across the board, indicating that a significant portion of multi-turn degradation arises from error propagation. For example, GPT-5 achieves 87.4% PCR, up from 62.4%, while Claude Haiku reaches 75.8% PCR, up from 60.6%. Even weaker models show notable gains, with GPT-4o and Llama 3.3–70B approaching 50% PCR under oracle context.

Overall, models continue to exhibit high rates of uncorrected responses, particularly for weaker systems, suggesting that correct context alone is insufficient to fully resolve misconception handling. Additionally, these results indicate that error propagation is a major contributor to multi-turn degradation, but intrinsic limitations in identifying and correcting misconceptions remain an important source of failure.

5.3.0.1 Turn Difficulty & Oracle Style Shifts:

To fully isolate error propagation from both question difficulty and style shifts, we perform another ablation "target-turn", and summarize the results in the following table 4. This setting provides the patient’s prior questions but removes all prior responses (model or physician). Results show that removing prior model responses significantly improves performance (e.g., GPT-5 improves from 60.6% to 76.3%), confirming that models actively propagate their own errors, though a gap below the Oracle remains due to intrinsic difficulty.

6 Discussion↩︎

Patients often ask questions built on incorrect assumptions, requiring AI systems not just to answer but to identify and correct the underlying misconception [3]. These misconceptions can evolve across multi-turn conversations, as patients reinterpret prior guidance from the AI system. Yet existing evaluation frameworks mostly test isolated questions, missing how misconceptions emerge, persist, and shift over the course of a dialogue [13], [23].

Our results show that strong single-turn performance does not guarantee reliability over the course of a conversation. While several models demonstrate strong correction ability at the initial turn, this ability degrades rapidly as conversations progress, with substantial declines within the first few follow-ups. As performance drops, misconceptions increasingly go unresolved, and errors tend to persist rather than being corrected in later responses. This pattern suggests that turns are not independent: a missed correction shapes the context for everything that follows, producing path-dependent failures rather than isolated ones. This is consistent with prior work on sequential and dialogue systems [37], where earlier outputs influence later predictions and compound into larger failures over an interaction [25], [26].

The oracle-physician ablation supports this interpretation: replacing prior model-generated responses with physician answers consistently improves performance across all models, indicating that some of the decline is driven by earlier model outputs rather than the difficulty of any single turn. However, performance still falls short of perfect for some models even with this oracle context, suggesting that clean prior context alone is not sufficient. Together, these results point to two distinct contributors to failure: propagation of earlier errors, and the underlying difficulty of catching misconceptions at all.

These findings carry implications for how LLMs are evaluated and deployed in patient-facing settings. Benchmarks based on isolated queries miss the temporal dynamics observed here and can overestimate reliability in real interactions. Multi-turn evaluation reveals failure modes hidden in single-turn settings, including performance decline, persistence of errors, and limited recovery after mistakes [7], [16], [38]. Evaluation should account for interaction dynamics, not just single-turn accuracy.

This also points to a design implication. Since much of the decline traces back to uncorrected errors compounding over time, brief points of human input could break that chain [39]. One approach is lightweight micro-interactions, where a physician or other healthcare worker flags a response mid-conversation or can manually provide a correction [40], giving the model a chance to correct course before the error shapes later turns. The oracle-physician ablation supports the potential value of this: even partial, intermittent physician input could meaningfully improve responses over time.

More broadly, this also suggests a role for multi-stakeholder system design [41], where different healthcare workers contribute different kinds of oversight: physicians catching clinical errors, while nurses or care navigators watch for communication breakdowns or signs that a patient has misunderstood guidance. Distributing oversight this way could be more scalable than relying on physicians alone, since not every error requires the same level of expertise to catch, and it better reflects how care is delivered across multiple roles.

Beyond misconception handling, ThReadMed-QAoffers a realistic testbed for studying broader phenomena in longitudinal interactions, including error accumulation, safety drift, and sensitivity to evolving user input. Future work could use the dataset to develop and evaluate models built for multi-turn reliability. For instance, approaches that explicitly incorporate feedback, correction, and consistency across turns, or to fine-tune and align models to better detect and resolve misconceptions in realistic patient interactions. This matters most in clinical contexts, where uncorrected early errors can compound into larger misinformation over the course of an interaction.

6.0.0.1 Limitations

Several limitations of this work warrant consideration. First, the ground truth responses are drawn from r/AskDocs, an online forum, and are not equivalent to formal clinical documentation. Although responses are provided by verified physicians and subject to moderation, they may vary in completeness and reflect differences in clinical judgment, introducing potential noise in the reference answers. We only use the responses for oracle-physician ablation in this work. Second, the dataset is derived from Reddit and may reflect platform-specific biases. The user population is not representative of the general patient population, and interactions often follow informal, iterative patterns shaped by the platform. In addition, our filtering procedure retains only multi-turn threads with follow-up questions and a fully answered subset, which may over-represent more complex or engaged interactions and under-represent simpler, single-turn queries. Third, misconception identification and response evaluation rely on an LLM-as-a-Judge framework. Despite strong agreement with physician annotations, this approach may introduce labeling noise that can affect downstream metrics. Fourth, the number of failure-start pairs used to estimate recovery rates is limited, resulting in wide confidence intervals and restricting the strength of conclusions regarding recovery behavior. Fifth, comparisons between GPT-5 and other models should be interpreted cautiously, as GPT-5 outputs are stochastic while other models are evaluated deterministically. Finally, the dataset is restricted to English-language text and does not include multi-modal inputs, which are common in real clinical settings.

6.0.0.2 Ethical Considerations

While Reddit posts are publicly available, we recognize that users may not have originally intended their queries for large-scale research. To prioritize privacy, we strictly adhered to Reddit’s API terms of use and performed rigorous de-identification. We removed all usernames, post IDs, and specific timestamps, replacing them with unique anonymized identifiers (e.g., THREAD_001). No metadata that could lead to the re-identification of specific Reddit threads or users was included in our analysis.

7 Additional Dataset Details↩︎

7.1 Source and Format Conversion↩︎

We construct ThReadMed-QAfrom publicly available Reddit data from r/AskDocs, where users seek medical advice from healthcare professionals. The raw data consist of JSONL files containing one record per post and comment, with metadata fields including id, link_id, parent_id, body, author, created_utc, author_flair_text, and is_submitter.

To reconstruct conversational structure, we group comments by link_id and build a tree for each post, where top-level comments correspond to replies to the original post. This tree representation is used to extract linear multi-turn conversation threads.

7.2 Filtering and Preprocessing↩︎

We restrict posts to a fixed time window (January 2015 to June 2023) and retain only those with at least one comment to ensure engagement. We apply standard preprocessing, following [8], to remove low-quality or incomplete content. This includes:

• deleted or moderator-removed posts and comments,

• content from banned users or known bots,

• posts with minimal textual content (fewer than five words),

• image-only or non-textual submissions.

URLs are removed from both posts and comments. The same cleaning procedure is applied to comment trees, and only threads associated with retained posts are preserved.

7.3 Multi-turn Thread Construction↩︎

We extract linear patient–physician conversations from the comment trees using the following procedure.

7.3.0.1 Participant roles.

The original post is treated as the first patient question. Subsequent messages authored by the original poster (OP) are treated as follow-up questions. Responses from users with verified medical flair (e.g., Physician, Doctor) are treated as answers. Comments from non-clinicians are ignored when constructing the question–answer sequence.

7.3.0.2 Branch selection.

Each post may contain multiple comment branches. For each top-level reply, we construct a candidate linear conversation by traversing its descendants in chronological order. Among candidate branches, we select the one most relevant to the original question using a word-overlap heuristic between the post text and the first physician response. This favors coherent, on-topic exchanges.

7.3.0.3 Multi-turn requirement.

We retain only threads with at least two follow-up questions beyond the initial post, ensuring a minimum of three question–answer pairs per thread.

7.3.0.4 Quality filtering.

We remove threads with incomplete or low-quality responses, including those with excessive [deleted] or [removed] content or broken formatting. If the OP deletes their content mid-thread, the entire conversation is discarded.

7.4 Orphan Handling and Dataset Variants↩︎

Some conversations contain unanswered follow-up questions, which we refer to as orphans. These arise when a patient’s follow-up receives no physician response or when responses are removed during preprocessing.

We construct two dataset variants:

• Complete Dataset: all extracted multi-turn threads, including those with unanswered follow-ups.

• Fully Answered Subset: a filtered subset in which every question has a corresponding physician response.

To construct the fully answered subset, we only retain the threads where all questions are paired with physician responses and discard threads with fewer than three question-answer pairs. We then retain only threads with complete coverage across all turns.

7.5 Dataset Statistics↩︎

The Complete Dataset contains 9,741 threads and 37,191 question–answer pairs. The Fully Answered Subset contains 2,437 threads and 8,204 pairs, with physician responses for all questions.

Conversations exhibit varying depth, with most threads containing multiple follow-up questions and a long tail of deeper interactions. This structure enables analysis of model behavior across extended, realistic patient interactions.

7.6 Reproducibility↩︎

To facilitate reproducibility, we provide scripts for preprocessing, thread construction, and dataset filtering. All preprocessing steps and filtering criteria described above can be reproduced using the provided code ⁴.

8 LLM-as-a-Judge Validation↩︎

We use LLM-as-a-Judge in two distinct roles: (i) misconception identification: binary detection of false presuppositions in a patient question and (ii) scoring: ordinal assessment of whether a model response identifies and corrects those presuppositions. In both cases, we report agreement with expert labels and cohen’s \(\kappa\).

8.1 Misconception Identification↩︎

8.1.0.1 Gold standard and Sample:

We treat expert annotations (collected from ECFMG-certified M.D.) as the reference standard on a stratified multi-turn sample of patient questions \(N=310\) rows with expert labels in our annotations. Experts labeled each target patient turn for the presence of clinically relevant false presuppositions (YES/NO).

8.1.0.2 Judge setup and agreement:

The identification judge receives the patient question (and, when applicable, prior questions per our task definition) and must return a single binary LABEL \(\in \{\text{YES}, \text{NO}\}\). Table 5 summarizes exact agreement and Cohen’s \(\kappa\) for GPT-4o run with the rubric and prompt version used in the paper. Following common benchmarks for Cohen’s \(\kappa\), the coefficient falls in the substantial range (\(0.61\)–\(0.80\)).

8.2 Misconception Scoring and Debiasing↩︎

8.2.0.1 Gold Standard and Sample:

We curated a small held-out validation set of \(N=72\) instances with expert judgment scores. We use JSON as the output format, following previous work which showed better performance with structured outputs [42], [43].

8.2.0.2 Judge Setup and Agreement:

Given a patient turn that contains false presuppositions and a candidate model answer, a sharpness judge assigns an integer score in \(\{-1,0,1\}\) reflecting whether the answer fails to recognize the misconception (\(-1\)), shows partial awareness or incomplete correction (\(0\)), or clearly identifies and corrects it (\(1\)), per the rubric in the appendix 11. We report the agreement in Table 6.

8.2.0.3 De-biasing LLM Judgments:

We use multiple LLMs as judges (GPT-4o, Claude-Sonnet, and Llama3.3-70B) and aggregate their scores to reduce bias. GPT-4o serves as the primary judge (for tie-breaking) due to its strong agreement with expert judgment, supported by prior work [15], [28]. This reduces variance from any single model’s positional or stylistic biases and avoids optimistic pooling rules (e.g., always taking the most frequent score).

To provide greater transparency regarding judge behavior, we add per-class precision, recall, F1 scores in Table 7.

9 Additional Results↩︎

9.0.0.1 Turn-Wise Performance with Confidence Intervals:

In this section, we provide the detailed turn-wise performance metrics corresponding to Figure 5 in the main text. As multi-turn interactions progress, the number of eligible misconception-bearing questions naturally decreases. To make the evaluation sample size and statistical uncertainty explicit, the table below reports the exact number of question-answer pairs evaluated at each turn (n), alongside the Presupposition Correction Rate (PCR) and its 95% percentile bootstrap confidence intervals across all models. This ensures that performance estimates at later turns are properly contextualized by their underlying support.

4pt

10 Cost↩︎

10.1 Misconception Identification↩︎

We report the cost of running GPT-4o for misconception identification. This was run on 8,204 questions. We report this in Table 9.

10.2 Inference↩︎

After identifying the questions with misconceptions (604 question-answer pairs across 505 conversation threads), we run inference on them as mentioned in Section 4. We also run oracle inference for all questions except the ones in the first turn. We used 5 LLMs to generate responses. We report the cost in Table 10.

10.3 LLM-as-Judge Evaluation↩︎

We report API costs for running LLM-as-a-Judge evaluation for our 3 judge models for each of the 5 generations. We summarize these numbers in Table 11.

11 Prompts↩︎

{"Score": <0, 1>, "Explanation": [str]}

{"Score": <-1, 0, 1>, "Explanation": [str]}

References↩︎