1 Introduction↩︎

Online medical consultation has become an important component of China’s digital health system. Patients increasingly interact with physicians through asynchronous, platform-mediated encounters that combine free-text messages, structured medical records, and patient-uploaded images. This trend is driven by the uneven distribution of offline medical resources, the growing population of people with chronic diseases, and national digital health initiatives [1], [2]. For example, JD Health (https://ir.jdhealth.com), one of the largest online healthcare platforms in China, reported 217.7 million annual active users at the end of 2025. Its Internet Hospital handled more than 180 million consultations in 2024, averaging more than 490,000 consultations per day [3], [4]. The scale of these services has created a growing demand for AI systems that can assist physicians during online consultation.

Recent advances in large language models (LLMs) have made them promising assistants for online medical consultation and have been increasingly explored for applications such as medical question answering, consultation assistance, and clinical decision support [5]–[7]. However, whether these models can reliably support real-world online consultation remains largely unclear, since existing evaluations primarily test medical knowledge or simplified interactions rather than deployment-like consultation behavior. As illustrated in Figure 1, this mismatch has three dimensions. First, many benchmarks use clean exam-style cases, short-form QA, synthetic dialogues, or patient simulators, while real patients often communicate in incomplete, colloquial, fragmented, and sometimes inconsistent ways across multiple turns. Second, medical dialogue benchmarks are typically text-only, while multimodal medical benchmarks usually evaluate curated image-question pairs; in contrast, online physicians must interpret patient-uploaded photographs, laboratory reports, prescriptions, or imaging results in the context of the evolving dialogue. Third, physician responses are open-ended, case-specific, and safety-sensitive, so reference matching, lexical overlap, and multiple-choice metrics fail to capture clinical appropriateness, missing information gathering, unsupported claims, or unsafe recommendations.

To address these gaps, we introduce MedRealMM, a real-world multimodal benchmark for Chinese online medical consultation constructed from de-identified one-on-one consultations collected from JD Health. Figure 2 provides an overview of the benchmark construction pipeline. Specifically, rather than replaying entire consultation trajectories or relying on patient simulation, we identify Multimodal Clinical Challenge Points (MCCPs), namely clinically demanding points at which physicians must engage substantively, collecting information, interpreting images, performing differential diagnosis, recommending treatment, or communicating risk. Each MCCP is converted into a self-contained single-turn evaluation instance while preserving the original multimodal consultation context. The evaluation task is formulated as generating the physician’s next response given the same multimodal context. This formulation evaluates multimodal understanding in its original consultation setting, where patient-uploaded images are embedded in multi-turn dialogue and must be interpreted jointly with the textual context. Each case is associated with a case-specific evaluation rubric, initially generated by an LLM and subsequently refined through iterative physician feedback. This process reduces the cost of annotation relative to fully manual rubric construction while maintaining physician oversight throughout rubric refinement.

Our contributions are summarized as follows:

• We introduce MedRealMM, a real-world multimodal benchmark for Chinese online medical consultation. Each benchmark case preserves the patient-uploaded medical images available during the original consultation.

• We propose Multimodal Clinical Challenge Point (MCCP) extraction, a method that converts real multi-turn multimodal consultations into self-contained single-turn evaluation instances without resorting to patient simulation.

• We develop a physician-in-the-loop iterative rubric refinement protocol that combines LLM-initialized criteria with physician revisions, providing a case-specific evaluation grounded in real clinical judgment.

• We perform comprehensive evaluations of general-purpose, multimodal, and medical LLMs on MedRealMM, and find that top models match or exceed the treating physician on positive criteria yet trail on negatives, suggesting that patient safety remains an obstacle to deploying LLMs in real online consultation.

2 Related Work↩︎

Medical LLM benchmarks differ in the realism of consultation scenarios, the use of multimodal medical information, and the evaluation of open-ended responses. Table 1 summarizes the relationship between the existing benchmarks and MedRealMM.

2.1 Realism of Consultation↩︎

Early medical LLM benchmarks primarily evaluate medical knowledge through question answering, including PubMedQA [8], MedQA [9], MedMCQA [10], and MultiMedQA [11], [12]. Chinese benchmarks similarly focus on medical examinations and clinical knowledge assessment [13]–[18]. Although these resources measure medical knowledge, they do not evaluate the patient-physician interaction. Subsequent work moves toward consultation settings. Dialogue datasets such as MedDialog, MedDG, and MediTOD [19]–[21] provide real consultation transcripts but treat them as next-utterance prediction rather than performance evaluation. Recent benchmarks instead employ patient simulators or agent-based environments  [22]–[27] and synthetic conversations [28], [29] to evaluate interactive consultation. However, these benchmarks may not be able to reflect the distributional irregularities, such as fragmented and inconsistent behaviors, observed in real consultations.

The closest realistic settings include HealthBench Professional [30] and LiveMedBench [31]. HealthBench Professional focuses on evaluating clinician-facing tasks rather than online consultation. LiveMedBench is constructed from public medical forums but explicitly discards image-containing threads. In contrast, MedRealMM is built from private one-on-one patient-physician consultations in a deployed Chinese internet hospital and retains the patient-uploaded images that accompany them, reducing distribution mismatch and potential data contamination.

2.2 Multimodal Medical Consultation↩︎

Multimodal medical evaluation has largely focused on visual question answering over curated medical images, including PathVQA [32], SLAKE [33], PMC-VQA [34], OmniMedVQA [35], and GMAI-MMBench [36]. These benchmarks evaluate image understanding using isolated image-question pairs.

Conversely, existing medical dialogue benchmarks are predominantly text-only and either exclude image-containing consultations or do not collect visual information [19], [20], [26], [28]–[31]. A few recent benchmarks combine consultation scenarios with visual information, including 3MDBench [37] and the multimodal track of MedBench v4 [18], but they rely on curated or simulated images. In contrast, MedRealMM evaluates physician response generation using patient-uploaded images embedded within real consultation histories.

2.3 Evaluation of Open-Ended Responses↩︎

Because physician responses admit multiple clinically appropriate forms, exact-match and lexical-overlap metrics such as BLEU [38] and ROUGE [39] are often insufficient for evaluation. Recent work, therefore, adopts rubric-based evaluation using LLM judges. HealthBench [28] and HealthBench Professional [30] employ physician-written case-specific rubrics, providing clinically meaningful evaluation at a relatively high annotation cost. Subsequent approaches, including LiveMedBench and automated rubric pipelines [29], [31], [40], use LLM-generated rubrics with limited physician verification to improve scalability.

MedRealMM adopts a physician-guided rubric construction process in which case-specific rubrics are initialized by an LLM and iteratively refined by physicians. This design combines scalable rubric generation with continuous clinical supervision and supports the evaluation of real multimodal consultation cases.

4pt

3 Methods↩︎

3.1 Phase 1: Construction of a Real-World Multimodal Consultation Corpus↩︎

3.1.0.1 Data source.

MedRealMM is constructed from real patient-doctor interactions collected from the large-scale Chinese internet hospital platform JD Health between 2025 and 2026 [3], [4]. The platform serves patients nationwide across a broad spectrum of clinical departments and supports multimodal consultations, where both patients and physicians can upload medical images alongside textual descriptions of their symptoms.

Formally, each consultation is represented as an ordered multimodal interaction trajectory \[\begin{align} C=(e_1,e_2,\ldots,e_T),\;e_t=(c_t,m_t,s_t,\tau_t), \end{align}\] where \(c_t\) denotes the content of the event \(e_t\), \(m_t\subseteq\{\text{text},\text{image}\}\) its modality set (a single event can carry both text and image attachments), \(s_t\in\{\text{patient},\text{physician}\}\) the sender identity, and \(\tau_t\) the timestamp. This formulation preserves both the temporal ordering of events and the interleaving of textual and visual information.

3.1.0.2 Multistage corpus construction.

Starting from the raw record set \(\mathcal{C}_0\), we obtain the final corpus \(\mathcal{C}\) through a chain of filtering, balancing, and review operators \[\mathcal{C}_0 \xrightarrow{\,\Phi_{\text{img}}\,} \mathcal{C}_1 \xrightarrow{\,\Phi_{\text{cmpl}}\,} \mathcal{C}_2 \xrightarrow{\,\Phi_{\text{bal}}\,} \mathcal{C}_3 \xrightarrow{\,\Phi_{\text{deid}}\,} \mathcal{C}_4 \xrightarrow{\,\Phi_{\text{rev}}\,} \mathcal{C},\] where each operator performs one stage of corpus construction. (i) Image presence: \(\Phi_{\text{img}}\) retains only consultations containing at least one patient-uploaded medical image, \(\mathcal{C}_1=\{C\in\mathcal{C}_0\mid \exists\,e_t\in C,\,\text{image}\in m_t\}\), ensuring that every retained consultation contains patient-provided visual evidence. (ii) Completeness: \(\Phi_{\text{cmpl}}\) removes consultations that are excessively short, truncated, or otherwise unsuitable for meaningful evaluation. (iii) Department balance: \(\Phi_{\text{bal}}\) performs department-aware sampling to alleviate the severe specialty imbalance in the raw data. (iv) De-identification: \(\Phi_{\text{deid}}\) applies an LLM-assisted de-identification [41] that removes personally identifiable information from both text and images while preserving clinically relevant content; cases whose clinical interpretation depends primarily on identifiable visual information are excluded. (v) Manual review: \(\Phi_{\text{rev}}\) requires trained reviewers to verify both privacy protection and preservation of clinical utility. The final corpus \[\mathcal{C}=\{C_1,C_2,\ldots,C_N\}\] is used in the subsequent stages of benchmark construction. Detailed filtering rules, de-identification procedures, and privacy considerations are provided in Appendix 6.1.

3.2 Phase 2: Multimodal Clinical Challenge Point Extraction↩︎

A realistic evaluation of multi-turn online consultation would ideally replay the entire interaction. In practice, this requires patient simulation, which often fails to capture the complexity and unpredictability of real patient behavior [26], [29]. We instead identify clinically meaningful points within authentic consultation trajectories and formulate evaluation around these states. This leads to the notion of a Multimodal Clinical Challenge Point (MCCP).

3.2.0.1 Definition of MCCP.

Given a consultation trajectory \(C=(e_1,e_2,\ldots,e_T)\), an MCCP corresponds to a position \(t^* \in \{1,\ldots,T\}\) immediately before a physician response, at which the accumulated consultation state requires substantive clinical reasoning based on the clinical expertise of the responding physician. A valid MCCP must jointly satisfy the following:

(i) Clinical engagement \(\mathcal{A}(t^{*}; C)=1\): the expected physician response involves substantive clinical engagement (e.g., history taking, differential diagnosis, examination or imaging interpretation, risk stratification, treatment planning, or patient education) rather than routine acknowledgments or administrative replies.

(ii) Multimodal relevance \(\mathcal{M}(t^{*}; C)=1\): images uploaded in \(C_{\le t^{*}}\) can materially influence an appropriate physician response, including diagnostic reasoning, risk assessment, or treatment decisions.

A position \(t^{*}\) is admitted as an MCCP iff \(\mathcal{A}(t^{*};C)=1\wedge\mathcal{M}(t^{*};C)=1\).

3.2.0.2 MCCP extraction pipeline.

We implement the two extraction criteria through a two-agent pipeline. Let \(\mathcal{T}=\{t\;|\;e_{t+1} \text{is a physician response}\}\) denote the set of turns before a physician response in the consultation trajectory \(C\). Each agent \(\phi:(t,C)\rightarrow(l,s)\) takes a trajectory and a candidate turn as input, and returns a binary decision \(l\in\{0,1\}\) together with a selection score \(s\in\mathbb{R}\). Binary decisions correspond to the clinical engagement criterion \(\mathcal{A}\) and the multimodal relevance criterion \(\mathcal{M}\), respectively, while selection scores are used to rank multiple valid candidates.

Stage 1: Filtering for clinical engagement. Given the complete consultation trajectory \(C\), an LLM agent Cropper \(\phi_\mathcal{A}\) evaluates each physician response turn and retains turns that satisfy the clinical engagement criterion: \[(l_\mathcal{A}(t),\,s_\mathcal{A}(t))=\phi_{\mathcal{A}}(t,C),\quad \mathcal{T}_{\mathcal{A}}(C)=\left\{t\in\mathcal{T}\;\middle|\;l_\mathcal{A}(t)=1\right\}.\]

Stage 2: Verification of multimodal relevance. For each candidate turn, an LLM agent Verifier \(\phi_\mathcal{M}\) evaluates the consultation history available until the physician response and retains turns whose accompanying patient images are clinically necessary to determine an appropriate response: \[(l_\mathcal{M}(t),\,s_\mathcal{M}(t))=\phi_\mathcal{M}(t, C_{\le t}), \quad \mathcal{T}_{M}(C)=\left\{t\in\mathcal{T}_{\mathcal{A}}(C)\;\middle|\;l_\mathcal{M}(t)=1\right\}.\]

If multiple candidate turns remain, we select the MCCP as \(t^{*}=\arg\max_{t\in\mathcal{T}_{M}(C)}\left(s_\mathcal{A}(t)+s_\mathcal{M}(t)\right)\). Consultations with \(\mathcal{T}_\mathcal{M}(C)=\emptyset\) are discarded. The extracted MCCPs are independently validated by physician review (Appendix 6.3).

3.2.0.3 Benchmark instance construction.

For each retained consultation, the trajectory is truncated at its MCCP, and the benchmark instance is defined as \[x = C_{\le t^{*}}=\{e_1,e_2,\ldots,e_{t^{*}}\},\] where \(e_{t^{*}}\) denotes the patient turn, and \(x\) contains all textual and visual information available up to and including this patient turn. The prediction target is the turn of the physician immediately following \(y_{\text{target}}=c_{t^{*}+1}\). The evaluation task is to generate \(\hat{y}\sim p_\theta(\cdot\mid x)\) conditioned on this truncated multimodal state. This formulation transforms raw consultation logs into self-contained evaluation instances while yielding standardized evaluation instances derived directly from real consultation trajectories.

3.2.0.4 Patient-intent and consultation-stage annotation.

Online consultations vary substantially in both patient intents and consultation stages. Reporting only aggregate performance obscures these differences and makes model failures difficult to interpret. To enable a stratified and clinically interpretable evaluation, we further annotate each instance \(x\) with two structured labels: \[z_{\mathrm{intent}}=\phi_{\mathcal{I}}(x)\in\mathcal{I},\qquad z_{\mathrm{stage}}=\phi_{\mathcal{S}}(x)\in\mathcal{S},\] where \(\phi_{\mathcal{I}}\) is an LLM agent that predicts the dominant patient intent of a closed taxonomy \(\mathcal{I}\) (e.g., diagnosis seeking, examination interpretation, treatment consultation, medication guidance, follow-up), and \(\phi_{\mathcal{S}}\) predicts the consultation stage of a clinically motivated taxonomy \(\mathcal{S}\) aligned with standard medical-consultation practice (e.g., history taking, examination/image review, diagnostic reasoning, management planning). These annotations enable stratified evaluation, support balanced benchmark construction, and facilitate downstream error analysis.

3.3 Phase 3: Physician-Guided Case-specific Rubric Construction↩︎

Evaluating open-ended medical consultation responses cannot be reliably reduced to reference-answer matching [28], [31], as multiple clinically appropriate responses may exist for the same consultation state. We therefore construct a physician-guided, case-specific rubric for each benchmark instance.

3.3.0.1 Case-specific rubric generation.

Given a benchmark instance \(x\) and the corresponding original physician response \(y_{\text{orig}}\), an LLM agent Generator \(\phi_{\mathcal{R}}\) generates an initial rubric \[\mathcal{R}^{(0)}=\phi_{\mathcal{R}}(x, y_{\text{orig}})=\{(c_j,w_j)\}_{j=1}^{N_x},\] where each criterion \(c_j\) is a clinically meaningful evaluation criterion that should be satisfied or violated by an appropriate physician response, and \(w_j\in[-20,20]\setminus\{0\}\) denotes its signed importance. We divide the rubric into a positive set \(\mathcal{R}^{+}=\{j:w_j>0\}\) that rewards the inclusion of clinically desirable behaviors (e.g., correct differential, examination ordering, and safety advice), and a negative set \(\mathcal{R}^{-}=\{j:w_j<0\}\) that penalizes hallucinations, contradictions, or unsafe behaviors. The rubric size \(N_x\) is instance-dependent and adapts to the clinical complexity of each case.

3.3.0.2 Iterative rubric refinement.

In each round \(k\), physicians review \(\mathcal{R}^{(k)}\) and provide structured feedback \(\mathcal{F}^{(k)}\) identifying missing criteria, incorrect weights, or overlooked clinical considerations, such as unsupported diagnoses, misinterpretations of images, unsafe recommendations, or missing follow-up advice. A rubric-improvement Refiner agent \(\phi_{\mathcal{R}}^{\,\prime}\) then revises the rubric: \[\mathcal{R}^{(k+1)}=\phi_{\mathcal{R}}^{\,\prime}\!\left(x,\mathcal{R}^{(k)},\mathcal{F}^{(k)}\right).\] The procedure ends in round \(K=\min\{k:\mathcal{F}^{(k)}=\varnothing\}\), i.e., when the physicians approve the rubric without further revisions. Because JD Health imposes a strict per-turn length limit to encourage concise physician responses, we incorporate case-specific length constraints as rule-based evaluation criteria. We denote the resulting physician-approved rubric by \(\mathcal{R}^{*}=\mathcal{R}^{(K)}\) and use it for all downstream evaluations.

The original physician response is used only to initialize case-specific evaluation criteria rather than to define a reference answer. Through iterative physician refinement, criteria that are overly specific to the original response are revised or removed, resulting in a rubric that captures clinically acceptable behaviors instead of reference-specific wording.

3.4 Phase 4: Rubric-Grounded Automated Evaluation↩︎

MedRealMM evaluates open-ended medical consultation responses through rubric-grounded clinical assessment [28]: \[\begin{align} \mathrm{Grade}(x,y,\mathcal{R}^{*})\rightarrow S, \end{align}\] where \(x\) denotes the benchmark instance, \(y\) denotes the generated physician response, \(\mathcal{R}^{*}\) denotes the physician-approved case-specific rubric, and \(S\) denotes the resulting evaluation score.

3.4.0.1 Evaluation setup.

For each benchmark instance \(x=C_{\le t^{*}}\), an evaluated model \(M\) generates a physician response \(y\sim p_{M}(\cdot\mid x)\), which is then assessed against the physician-approved rubric \(\mathcal{R}^{*}=\{(c_j,w_j)\}_{j=1}^{N_x}\) obtained in Phase 3.3.

3.4.0.2 Criterion-level assessment.

For each criterion \(c_j\), an LLM agent Grader \(\phi_{\mathcal{J}}\) (LLM-as-a-judge [28], [42]) predicts a binary decision together with a rationale: \[\begin{align} (v_j, r_j)=\phi_{\mathcal{J}}\left(x,y,c_j\right),\qquad v_j\in\{0,1\}, \end{align}\] where \(v_j\) measures whether \(y\) satisfies \(c_j\) given the instance \(x\), and \(r_j\) is a textual rationale that explains why. \(\phi_{\mathcal{J}}\) does not condition on the original physician response \(y_{\text{orig}}\), so the generated answers are not penalized for stylistic deviation from the single observed reference.

3.4.0.3 Rubric-level aggregation.

Criterion-level scores are aggregated according to their signed importance weights. Since negative weights can make \(\sum_j w_j\) non-positive, we normalize only by the total positive weight and clip the resulting score to \([0,1]\): \[\begin{align} S(x,y,\mathcal{R}^{*})=\text{Clip}\left(\frac{\sum_{j=1}^{N_x}w_j v_j}{\sum_{j=1}^{N_x}\max(w_j, 0)}, 0, 1\right). \end{align}\]

3.4.0.4 Benchmark-level evaluation.

For an evaluated model \(M\), the final benchmark score is the case-level mean over the evaluation set \(\mathcal{D}\): \[\begin{align} \mathrm{Score}(M)=\frac{1}{|\mathcal{D}|}\sum_{i=1}^{|\mathcal{D}|}S\!\left(x_i,\,M(x_i),\,\mathcal{R}_i^{*}\right), \end{align}\] where \(x_i\in\mathcal{D}\). This rubric-grounded evaluation framework enables scalable assessment of multimodal medical consultation responses while preserving case-specific clinical expectations and multimodal reasoning requirements.

3.5 Dataset Statistics↩︎

MedRealMM contains 5,620 consultations spanning 64 clinical departments. Figure 3 further summarizes the distributions of patient intents, consultation stages, dialogue turns, image counts, and rubric complexity. Symptom inquiry is the most common patient intent, while treatment recommendation and patient Q&A account for the largest fraction of extracted MCCPs. The cases contain between 1 and 10 patient-uploaded images and an average of approximately 7 dialogue turns. The physician-in-the-loop rubric construction process yields 3 to 19 criteria per case (positive: 2 to 17, negative: 0 to 7), enabling fine-grained assessment of model responses.

4 Experiments↩︎

We first describe the experimental setup (Section 4.1) and summarize the main results (Section 4.2). We then investigate three questions corresponding to the key contributions of MedRealMM. Section 4.3 asks whether real multimodal consultation constitutes a distinct, unsolved capability, and tests the hypothesis that visual grounding rather than biomedical knowledge is the dominant bottleneck. Section 4.4 examines where the models break down along the clinical axes that matter for safe remote care. Section 4.5 assesses whether rubric-grounded LLM-as-a-judge scoring on real consultations is reliable enough to support these conclusions.

4.1 Experiment Settings↩︎

We evaluate 19 models as revealed in Table 4, spanning three dimensions: modality (multimodal vs.text-only), domain focus (general-purpose vs.medical-specialized), and access type (open-source vs.closed-source). Unless otherwise specified, Claude-Opus-4.7 [43] is used for rubric generation and refinement, while Gemini-3-Pro-Preview [44] serves as the Grader and de-identification agent. Rubric refinement is performed for three iterations to achieve convergence. MedRealMM evaluates clinical quality but not response latency; in deployment, physicians are expected to respond within roughly 20 seconds, and jointly measuring quality and latency is beyond the scope of this benchmark. See Appendix 7 for more details.

4.2 Main Results↩︎

Figure 4 (a) summarizes the overall performance of all models, together with the original real-world physician response (Online). The ordering is consistent: a closed-source frontier cluster leads general-purpose open-source models, which in turn lead medical-specialized models, and the physician response sits above all of them. Even the best model falls short of Online, and its absolute scores remain around \(50\), well below saturation, despite being at the frontier. Unlike prior medical benchmarks, every evaluation instance in MedRealMM is derived from an authentic consultation requiring joint reasoning over Chinese patient dialogue and uploaded medical images. Consequently, these scores better reflect deployment readiness than performance on exam-style medical benchmarks. See Section 7.4 for additional results and analysis.

4.2.0.1 The physician anchor leads by avoiding penalties, not by covering more positives.

Figure 4 (b) decomposes each model’s score into the average number of positive (reward) and negative (penalty) criteria its responses satisfy per case. The two components become decoupled among the highest-performing systems. Claude-Opus-4.6 and 4.7 actually meet more positive criteria than the Online physician response, but Online ranks above them because it triggers fewer negatives. Below the top, the joint pattern is simpler: lower-scoring models meet fewer positives and trigger more negatives. In a clinical setting, a single harmful recommendation can outweigh several correct ones [45]. These results suggest that avoiding clinically unsafe behavior, rather than maximizing positive coverage, is the key factor separating the treating physician from current frontier LLMs.

4.3 Is Visual Grounding the Main Bottleneck?↩︎

4.3.0.1 The closed-source advantage emerges primarily from multimodal capability.

Figure 5 compares the score of each multimodal model with and without patient images. Providing images raises frontier scores by 14 to 20 points, in some cases nearly doubling the text-only score. This behavior is consistent with the benchmark design: each MCCP is selected so that patient-uploaded images are clinically relevant to the subsequent physician response. On text-only input, the best open model (GLM-5.1 at 24.8) sits inside the frontier models’ own text-only band of 23.2 to 35.1; the closed-source advantage widens substantially once images are available. Because nearly all existing medical dialogue benchmarks are text-only, they would compress these systems into a much narrower performance range and obscure the multimodal capability required in real consultation.

4.3.0.2 Medical-specialized models trail general-purpose ones and gain little from images.

All medical-specialized models in our pool rank at the bottom. This trend is consistent with observations on the text-only LiveMedBench [31]; it persists and widens in the multimodal setting. This result likely reflects two factors. First, most medical-specialized models are text-only, so a multimodal benchmark places them at a structural disadvantage. Second, for the few that do accept images, the performance gain from image input is less than 6 points, substantially smaller than for general-purpose frontier models. These results suggest that current medical-specialized models are optimized primarily for text-based medical reasoning and derive limited benefit from visual evidence.

4.4 Where do Models Fail?↩︎

4.4.0.1 Specialty robustness masks a Chinese-specific coverage gap.

Figure 6 (a) reports performance after aggregating the 64 clinical departments into major specialty groups. Frontier models show relatively consistent performance across most clinical specialties. For example, Claude-Opus-4.7 achieves scores between 51 and 58 in nearly all departments, suggesting that variation across models is generally larger than variation across specialties. Psychiatry is a notable exception: all evaluated systems, including Online, score less than or equal to 40, indicating that psychiatric consultation remains a challenging setting under our evaluation protocol for both current LLMs and treating physicians. traditional Chinese medicine (TCM) exhibits a different pattern. While frontier models achieve performance comparable to their results in other specialties, the treating physician outperforms the strongest LLM by 14 points, representing the largest Online-LLM gap in the benchmark. This suggests that the gap between current multimodal LLMs and practicing physicians is particularly pronounced in TCM, underscoring the importance of evaluating models on authentic Chinese consultation data rather than benchmarks centered on Western medical practice.

4.4.0.2 Failures are uniform across patient intents but concentrate at specific consultation stages.

We further analyze model performance across patient intents and consultation stages, as shown in Figure 6 (b). Across the seven intent classes, the frontier models hold a narrow 40 to 60 band, so the type of patient goal alone does not determine difficulty. However, two consultation stages consistently emerge as the most difficult. Most models score below 40 on the combined diagnosis-with-explanation and treatment-recommendation stage, where multiple clinical decisions should be reasoned about together. The clearest gap to the physician response appears in history taking: Online reaches roughly 70, while every model falls below 50. Success at this stage typically requires asking for the missing information rather than answering immediately, the “know-when-to-ask” competence emphasized in interactive-consultation work [23], [25].

4.4.0.3 MedRealMM cleanly separates models and has no ceiling effect.

Figure 7 reports the per-case score distribution. Weak models are unimodal with a large spike at 0: a response that meets too few positive criteria or triggers too many negative ones is assigned the minimum score, regardless of whether it is broadly wrong or only narrowly inadequate. Frontier models, in contrast, spread broadly across the range with negligible mass at either tail. This contrasts with HealthBench [28], whose per-case distribution exhibits two peaks: a larger one near 0 and a smaller one near the ceiling. The high-end peak is absent from MedRealMM(even the strongest systems rarely score near 100 on a case), leaving substantial room for future improvements.

4.5 Is Rubric-grounded Evaluation Trustworthy?↩︎

We evaluate the reliability of the proposed evaluation protocol from two perspectives: (i) robustness to the choice of judge models and (ii) consistency with physician assessment.

4.5.0.1 Evaluation is consistent across judges and agrees with physicians.

To evaluate judge robustness, we score a sampled subset of 200 benchmark cases (1,797 rubric criteria) of Claude-Opus-4.7’s image-conditioned responses using three independent LLM judges (Gemini-3-Pro-Preview, GPT-5, and Kimi-K2.5). A physician panel independently evaluates the same model responses against the identical rubrics, providing a human reference for comparison. Table 2 reports the mean scores per-judge together with Gwet’s AC1 [46] for each pair of judges. Across the three LLM judges, mean scores differ by less than 3%, and pairwise Gwet’s AC1 remains consistently higher than 0.8, indicating that the benchmark is largely insensitive to the choice of judge model. The agreement between each LLM judge and the physician panel is comparable to the agreement among the LLM judges themselves, supporting the reliability of the automated evaluation protocol. However, the physician panel assigns scores approximately 7 points higher on average. This difference is concentrated in positive criteria: for GPT-5, physician-judge agreement is lower on positive criteria (AC1 = 0.768) than on negative ones (AC1 = 0.920), with the same trend observed for the other judges. This suggests that physicians are more likely to award positive criteria, leading to slightly higher overall scores.

4.5.0.2 Deliberative physician responses reveal additional human headroom.

The Online reference corresponds to the physician response produced during routine platform use under real-time clinical constraints. Therefore, it should be interpreted as an operational baseline rather than the best performance physicians could achieve under more favorable conditions. To assess whether higher-quality physician responses are achievable, we collect a second set of physician responses on a 200-case sample, where physicians are given ample time and free access to reference materials, and score these responses using the same case-specific rubrics. The physician subset is disjoint from the agreement subset to avoid potential bias introduced by prior exposure to the evaluation rubrics. The results are reported in Table 3. The deliberative physician response score is higher than both the Online anchor and the strongest LLM. This result supports the view that online physician responses provide a realistic operational baseline rather than a ceiling on human clinical performance and that substantial headroom remains for future medical LLMs.

5 Conclusion↩︎

We introduced MedRealMM, a real-world benchmark for multimodal Chinese online medical consultation built from authentic patient-physician interactions. By identifying Multimodal Clinical Challenge Points (MCCPs) from real consultation trajectories and pairing each case with a physician-refined, case-specific rubric, MedRealMM provides a scalable and clinically grounded alternative to synthetic patient simulation and reference-based evaluation. Experiments on 19 state-of-the-art LLMs show that current systems remain behind physicians in real online consultation. The largest gaps arise from integrating visual evidence, avoiding clinically unsafe behaviors, and adapting to different stages of the consultation process. These findings suggest that progress on conventional medical QA benchmarks does not necessarily translate into reliable performance in real-world online consultation. We hope MedRealMM will serve as a realistic benchmark for developing more capable multimodal medical LLMs. More broadly, we believe that extracting clinically meaningful challenge points from authentic interaction trajectories offers a practical paradigm for benchmarking interactive medical LLMs beyond patient simulation. Future work includes expanding specialty and language coverage, supporting multiple challenge points within a single consultation, and continuously updating the benchmark as clinical practice evolves.

6 Dataset Construction Details↩︎

6.1 Pipeline↩︎

Stage 1: Patient image availability. We retain a consultation only if the patient (not the doctor) uploads at least one medical image during the dialogue, e.g., skin/dermatoscopic photographs, lab reports, prescriptions, ultrasound printouts, or screen photographs of CT/MR/X-ray films. Stickers, emojis, and platform-injected images are excluded; image origin is decided from platform metadata.

Stage 2: Dialogue quality and complexity filtering. We discard a dialogue if its turn count falls outside \([5, 30]\), its total length exceeds 4,000 characters, more than half of the messages come from a single side, or it is closed by the patient with cancellation/refund-style messages and no diagnostic exchange.

Stage 3: Department distribution sampling. Raw traffic is heavily skewed toward dermatology, internal medicine, and pediatrics. We partition the pool by department and apply stratified sampling, capping high-volume departments and oversampling the long tail.

Stage 4a: Privacy-incompatible case removal. A case is removed in its entirety, before any masking, if redacting the patient’s identifying visual content would destroy the clinical signal, most commonly consultations whose diagnosis hinges on a full-frontal face photograph.

Stage 4b: Multimodal de-identification. Surviving cases undergo modality-specific masking on both text and images; the protocol is described in Appendix 6.2.

Stage 5: Human verification. Every remaining case is reviewed under a two-pass protocol: (i) a residual PII check on masked text and images, and (ii) a clinical-utility check confirming that masking did not destroy the clinical signal. High-sensitivity departments (pediatrics, mental health, reproductive health, oncology, and infectious disease) are reviewed in full, and a second annotator audits a 10% random sample of approved cases plus 100% of the high-sensitivity slice.

6.2 De-identification and Privacy↩︎

The protocol removes direct identifiers from both text and image modalities while retaining clinical attributes essential for downstream reasoning, e.g., age, sex, chief complaint, symptom duration, and image findings.

Text de-identification. We mask direct identifiers in both patient and doctor turns: personal/family names, government IDs, contact information, institutional identifiers (hospitals, medical-record and insurance numbers), sub-city geographic units, full dates, and online artefacts (URLs, order IDs), replacing each detected span with a category-typed placeholder (e.g., [NAME], [PHONE], [HOSPITAL], [DATE]). Detection combines rule-based regex for structurally regular identifiers with an LLM agent that catches identifiers in unconventional contexts.

Image de-identification. Image masking targets faces, embedded identifiers on reports and prescriptions (name, ID, medical-record number, hospital logo, barcode, QR code, and signature), burnt-in DICOM overlays on radiology screen photographs, identifying objects in the background, and EXIF metadata. Detected regions are removed with solid black rectangles rather than blurring. Cases whose clinical reasoning relies on a non-redactable identifier are removed earlier at Stage 4a rather than masked.

6.3 MCCP Validation↩︎

To verify that extracted MCCPs are clinically meaningful, we sampled 200 cases and asked physicians to make a binary judgement on whether each MCCP satisfies both clinical engagement and multimodal relevance. Only 1 of the 200 cases is rejected, giving an acceptance rate above 99%. We treat this as evidence that the two-agent extraction pipeline produces points physicians regard as appropriate for evaluation, rather than artefacts of the model’s own preferences.

6.4 Case Example↩︎

Figure 8 shows an example of real multimodal dialogue.

6.5 Physician-Guided Iteration Example↩︎

Figure 9 shows one iteration of the physician-in-the-loop rubric refinement loop (Section 3.3) on a case of tinea pedis. The original criterion bundled two unrelated requirements, adequate treatment duration and daily-care advice, into a single 8-point item. With the physician’s feedback, the refined output splits the criterion into two separately weighted items (5 + 3 points), so a response that gets the duration right but omits daily care no longer earns the full 8 points. Chinese rubric fields are translated to English here for readability.

7 Evaluation Details↩︎

7.1 Model Inference Settings↩︎

To ensure a fair and reproducible comparison, we evaluate a broad spectrum of contemporary large language models (LLMs) on MedRealMM.

7.1.0.1 Inference configuration.

For all closed-source models, we query the official provider APIs and generally follow each vendor’s recommended default generation configuration, including the default reasoning/thinking effort where applicable. For open-source models, we use each model’s officially recommended generation configuration as released on Hugging Face. The single exception is MedGemma-27B, whose default decoding parameters produce noticeably degraded Chinese outputs; we therefore adopt the model card’s recommended Chinese-friendly setting of temperature \(=0.7\), top-\(p\) \(=0.95\), and top-\(k\) \(=32\) for this model only. No model-specific prompt engineering, chain-of-thought triggers, or persona instructions are applied, so observed differences in performance reflect intrinsic model capability rather than prompt tuning.

7.1.0.2 Evaluation protocol.

Each model is invoked once per benchmark instance with greedy or sampling defaults as specified above, and the resulting response \(y\) is scored against the physician-approved rubric \(\mathcal{R}^{*}\) using the rubric-grounded grade described in Section 3.4. Table 4 lists every model evaluated together with its access endpoint, version/checkpoint identifier, default thinking effort, and source link.

7.2 Grading Prompt↩︎

Section 3.4 formulates the evaluation as a rubric-grounded clinical response assessment. We implement the Grader \(\phi_{\mathcal{J}}\) as a single LLM call invoked once per rubric criterion: given a benchmark instance \(x\), a generated physician response \(y\), and a single criterion \(c_j\) from the physician-approved rubric \(\mathcal{R}^{*}\), the agent returns a binary verdict \(v_j\in\{0,1\}\) together with a short natural-language rationale \(r_j\). For an instance with \(N_x\) criteria, the agent is therefore queried \(N_x\) times, yielding \(\{(v_j,r_j)\}_{j=1}^{N_x}\), which are then aggregated into the case-level score \(S(x,y,\mathcal{R}^{*})\) via the weighted formula in Section 3.4. The complete prompt template used for each call is shown in Figure 10. We use the same model as the Grader and the same decoding configuration across all evaluated systems to ensure comparability.

None

7.3 Evaluation Reliability↩︎

To check that MedRealMM’s rubric-grounded scores do not depend on the choice of judge models, we re-score the sampled 200-case subset with two additional LLMs (GPT-5 and Kimi-K2.5, together with Gemini-3-Pro-Preview) and an independent physician panel that applies the same rubrics and measure agreement between each judge model and the physician panel.

Each criterion verdict \(v_j\in\{0,1\}\) is binary and often skewed; most criteria are clearly satisfied or clearly violated by a given response, so Cohen’s \(\kappa\) is unstable on such data. We therefore use Gwet’s AC1 [46], which is more stable under skewed marginals. Let \(p_o\) be the observed agreement rate and \(\pi\) the average positive rate across the two raters; then \[\mathrm{AC1} \;=\; \frac{p_o - p_e}{1 - p_e}, \qquad p_e \;=\; 2\,\pi\,(1-\pi),\] with \(\mathrm{AC1}\in[-1,1]\) and \(1\) indicating perfect agreement. We compute AC1 per pair over all criterion-level verdicts on the subset.

7.4 Additional Results↩︎

7.4.0.1 Model performance is largely insensitive to image count and dialogue length.

Figure 11 reports model performance as a function of the number of patient-uploaded images and dialogue turns. Across both axes, most models exhibit little systematic variation: scores remain broadly stable as image count or dialogue length increases, with several models even performing slightly better on cases containing more images or longer histories. These results suggest that benchmark difficulty is driven primarily by the clinical demands of the extracted challenge point rather than by the amount of multimodal context. This observation is consistent with the MCCP construction procedure, which selects clinically challenging consultation states instead of simply favoring image-rich or long conversations.

7.4.0.2 Fine-grained specialties reveal localized physician advantages.

Figure 12 extends the department-level analysis in Figure 6 (a) to all clinical sub-specialties. Consistent with the main results, the Online physician response maintains a clear advantage across multiple traditional Chinese medicine (TCM) sub-specialties. The finer-grained analysis also reveals additional areas where physicians outperform current LLMs; for example, in pediatric dermatology the physician response exceeds the second-best model by roughly 10 points. These localized gaps are less apparent in the aggregated department-level results. We note that several sub-specialties at the bottom contain only a small number of cases, so their scores should be interpreted with appropriate caution.

8 Limitations↩︎

8.0.0.1 Single-language, single-platform scope.

MedRealMM is built entirely from Chinese-language consultations at one nationwide Internet hospital. Our findings do not directly transfer to other languages or platforms with different interaction norms, e.g., different image-upload prompts, different physician-response templates, and different triage routing. Cross-lingual and cross-platform generalization is left to future work.

8.0.0.2 Single-sample evaluation.

We score each model with one sampled response per case. HealthBench [28] reports that the worst-of-\(k\) scoring exposes a meaningful tail of unsafe or low-quality samples that mean-score evaluation hides. Running a \(k\)-sample evaluation on MedRealMM’s full set across all evaluated models is expensive and left to future work.

9 Ethical Considerations↩︎

The dialogue source is governed by the platform’s user agreement, which permits de-identified secondary use for research. We do not redistribute raw, pre-masking data, and we do not link cases across platforms.

MedRealMM is released under a Data Use Agreement (DUA) with the following core clauses:

• Permitted use. Research evaluation of medical LLMs and multimodal models, including academic publications and internal benchmarking by industry research groups.

• Prohibited use. Any attempt to re-identify patients or doctors or to link cases to external datasets for that purpose; redistribution of the raw data to parties who have not signed the DUA; and training of generative models intended to impersonate the source platform’s clinical service.

• Citation and contamination. Users must cite MedRealMM in any publication that reports results on it and must disclose if any MedRealMM content has been included in model training data.

• Patient withdrawal. Patients identifiable through the source platform may request, via the platform’s standard channels, that any case derived from their consultation be removed from future versions. Removed cases are documented in a public changelog without disclosing the requesting party.

• Distribution and access. The benchmark is hosted on a research-only access portal; downloaders must agree to the DUA before access is granted.

References↩︎