MedStreamBench: A Time-Aware Benchmark for Streaming and Proactive Medical Video Understanding
July 02, 2026
| Benchmark | #DS | Protocol | Temporal Tasks | |||||
|---|---|---|---|---|---|---|---|---|
| 3-5 (lr)6-9 | Eval Unit | Prefix-Only | Streaming | Retrospective | Present | Future | Proactive | |
| Cholec80 | 1 | frame / video | ||||||
| M2CAI16-Workflow | 1 | frame / video | ||||||
| JIGSAWS | 1 | gesture / video | ||||||
| LLaVA-Surg | 1 | clip / video | ||||||
| SurgPub-Video | 1 | clip / video | ||||||
| SurgOnAir | 1 | streaming video | ||||||
| MedHorizon | 1 | long video | ||||||
| MedStreamBench (Ours) | 22 | long video / stream | ||||||
5pt
1 Background↩︎
video understanding is becoming increasingly important for surgical intelligence, endoscopic screening, procedural education, and real-time clinical decision support. Unlike static medical images, medical videos encode evolving procedural context: instruments enter and leave the field, anatomical exposure changes over time, lesions may appear only transiently, and safety-relevant events often depend on the temporal order of visual evidence. Consequently, a clinically useful medical vision–language model should not only recognize what is visible, but also determine when sufficient evidence becomes available, when to abstain, and when to alert. This temporal discipline is essential in scenarios such as polyp screening, surgical workflow monitoring, view-quality assessment, and skill coaching, where premature conclusions can be misleading and delayed responses can reduce clinical utility.
Recent video-language benchmarks have substantially advanced the evaluation of multimodal large language models on video understanding, including long-video reasoning and online video analysis [1]–[4]. However, most existing benchmarks assume either access to a complete video or a fixed clip, and primarily measure whether the final answer is semantically correct. Such offline evaluation is insufficient for medical streaming video analysis, where models must respect the information actually available at a specified time point. In parallel, medical and surgical datasets have enabled progress in phase recognition, instrument detection, action triplet recognition, polyp detection, capsule-endoscopy abnormality recognition, and surgical visual question answering [5]–[13]. Yet these resources typically evaluate recognition or question answering in an offline setting, and do not explicitly test whether a model can answer, defer, or alert at the correct moment under a strict evidence window.
We introduce MedStreamBench, a diverse temporal benchmark for medical video question answering. To the best of our knowledge, MedStreamBench is the first benchmark designed to evaluate streaming medical video understanding, and
the first to formulate a proactive medical video question-answering paradigm. In the streaming paradigm, the same question is asked over progressively growing visual prefixes, and a model must output unanswerable until the evidence
becomes sufficient. In the proactive paradigm, the model continuously monitors a medical video stream and must respond with one of no_alert, uncertain, or alert: <reason>, triggering an alert only when
sufficient visual evidence appears. These two paradigms directly reflect clinically meaningful requirements: a model should avoid leaking future information, avoid unsupported early alarms, detect time-sensitive findings or workflow events promptly, and
maintain stable behavior once evidence is available.
As summarized in Table 1, recent medical video resources such as LLaVA-Surg [14], SurgPub-Video [15], SurgOnAir [16], and MedHorizon [17] extend evaluation beyond classical phase- or clip-level recognition; however, they still provide limited coverage of prefix-only answering, streaming evaluation, or future and proactive temporal reasoning. MedStreamBench is designed to be broad and comprehensive. It integrates 22 medical video and image-sequence datasets spanning laparoscopic surgery, open surgery, robotic skill assessment, hysterectomy, pituitary surgery, bariatric bypass surgery, colonoscopy, gastrointestinal endoscopy, and capsule endoscopy. The benchmark contains 5,419 question-answering records, including 4,663 single-turn items and 756 streaming items, which expand to 3,369 per-round streaming inference jobs. Its temporal taxonomy covers four complementary modes: retrospective questions about past evidence, present questions about the current visual state, future questions that require prediction or deferred answerability from partial evidence, and proactive monitoring questions that require timely alerts. The benchmark also contains both closed and open-ended questions, and combines original annotations, category labels, hybrid label-grounded generation, and conservative model-assisted mining to cover both high-confidence label-based tasks and visually grounded open-ended reasoning.
A central design principle of MedStreamBench is that every question is tied to an explicit visual evidence window. For single-turn tasks, the input is restricted to \([t_q - w,\, t_q]\), where \(t_q\) is the query time and \(w\) is the context window. For streaming future and proactive tasks, each round is evaluated only on \([t_q,\, t_c]\), where \(t_c\) is the current time for that round. This design prevents temporal leakage and converts medical video question answering from an offline recognition problem into a time-aware decision problem. For open-ended questions, MedStreamBench further uses fine-grained judge rubrics that separately assess semantic correctness, evidence grounding, and safety, improving fairness and reducing ambiguity compared with a single coarse match criterion. In particular, the benchmark asks: given only the frames available so far, should the model answer, abstain, remain uncertain, or raise an alert?
To support scalable construction and reproducible evaluation, we develop two accompanying systems. First, the MedStreamBench data engine harmonizes heterogeneous medical datasets into a unified temporal QA schema. It converts heterogeneous medical data, including structured annotations, directory labels, and weakly labeled videos, into unified temporal QA items, and then validates and materializes the final dataset-scoped benchmark files. Second, the MedStreamBench inference engine expands benchmark records into per-window or per-round jobs, samples visual inputs under a fixed temporal protocol, supports local and API-based multimodal inference, and evaluates predictions using content correctness, streaming responsiveness, and post-evidence stability. This separation between data construction and inference makes the benchmark extensible to new datasets, models, and temporal protocols.
Our contributions are threefold.
Time-aware formulation. We propose a time-aware medical video QA formulation that unifies retrospective, present, future, streaming, and proactive evaluation under explicit evidence-window constraints.
Diverse benchmark. We release a diverse benchmark built from 22 medical datasets, covering a wide range of procedures, anatomies, visual findings, task types, label sources, and temporal reasoning modes.
Reproducible engines. We provide data and inference engines that enable reproducible benchmark generation, per-round streaming evaluation, and metrics for both answer correctness and temporal behavior.
2 Summary↩︎
MedStreamBench is a time-aware benchmark for medical video question answering with explicit evidence windows, four temporal modes, and both single-turn and streaming evaluation. As illustrated in Figure 1 and Figure 2, it integrates 22 source datasets into a unified JSONL schema that records the question, answer, temporal specification, task mode, response protocol, and label provenance. The current release contains 5,419 QA records, including 4,663 single-turn items and 756 streaming items, which expand to 3,369 round-level inference jobs during evaluation. The benchmark combines annotation-derived, directory-derived, hybrid, and conservatively model-assisted items to broaden coverage while retaining provenance information for downstream analysis.
2.1 AI-Readiness↩︎
The benchmark is organized in a unified machine-readable schema with explicit temporal metadata, including query time, input range, and round-level streaming structure where applicable. For open-ended questions, MedStreamBench additionally provides fine-grained judge rubrics that separately assess semantic correctness, evidence grounding, and safety, enabling more consistent LLM-as-judge evaluation than a single coarse matching criterion. In addition to benchmark files, MedStreamBench is paired with a standardized inference and evaluation workflow that fixes frame sampling, prompt construction, and temporal scoring, making it suitable for controlled comparison of model behavior under evidence-constrained medical video understanding.
3 Discussion↩︎
3.1 Strengths↩︎
MedStreamBench introduces a temporal evaluation setting that is largely absent from existing medical video resources. Its main strengths are explicit evidence-window control, broad multi-source coverage, and evaluation of both semantic correctness and temporally appropriate response behavior.
3.2 Limitations↩︎
The underlying source datasets differ substantially in annotation granularity, label quality, visual scope, and temporal density, and some benchmark items are derived from directory labels, hybrid generation, or weak supervision rather than fully expert-authored QA annotation. In addition, not all weakly supervised or model-assisted samples received exhaustive manual visual review, and the fixed frame-sampling protocol may underrepresent brief events.
3.3 Bias and Representativeness↩︎
As a multi-source benchmark, MedStreamBench inherits distributional biases from its constituent datasets, including differences in procedure types, devices, recording styles, annotation practices, and case distributions. Aggregate scores should therefore be interpreted together with dataset-level or domain-level analyses, especially when comparing behavior across surgical, endoscopic, and skill-assessment settings.
| Dataset | Access | Dataset | Access | Dataset | Access |
|---|---|---|---|---|---|
| cholec80 | upstream terms | M2CAI16-Workflow | challenge terms | EndoVis2019 | challenge terms |
| AutoLapora | public release | LDPolypVideo | upstream terms | PitVis | public release |
| simsurgskill_2021 | public release | kvasir-capsule | public release | hyper-kvasir | public release |
| Colonoscopic-web | upstream terms | CVC-ClinicDB | public release | AlxSuture | upstream terms |
| AVOS | public release | Surgical-phase | will release | JIGSAWS | public release |
| CholecT45 | public release | CholecT50 | public release | HeiChole | challenge terms |
| BernBypass70 | upstream terms | MultiBypass140 | upstream terms | private_data | will release |
| Colonoscopic-addi | upstream terms |
4pt
3.4 Responsible Use↩︎
MedStreamBench is intended as a research benchmark for evaluating time-aware medical video understanding systems. It should not be interpreted as a clinical decision-support system or as a substitute for expert judgment. Weakly supervised and model-assisted items are included to broaden evaluation coverage, but they should not be treated as expert-certified clinical labels. Any deployment-oriented use would require additional task-specific validation, prospective testing, and appropriate human oversight.
4 Resource Availability↩︎
4.1 Data Release↩︎
The benchmark is released as dataset-scoped JSONL files following a unified temporal QA schema and is hosted at Venn2024/MedStreamBench. The release provides derived QA annotations and metadata under the Apache License 2.0, while the underlying image and video assets remain governed by the corresponding upstream sources.
4.2 Code Release and Access Policy↩︎
MedStreamBench is accompanied by a data construction pipeline and an inference/evaluation pipeline supporting benchmark generation, schema validation, frame sampling, prompt construction, streaming round expansion, and metric computation. Access to original videos or images remains subject to the terms, licenses, and availability conditions of the corresponding upstream datasets. The benchmark release therefore distributes derived QA JSONL files and associated metadata, but does not redistribute the original media unless permitted by the upstream source. When explicitly allowed by the corresponding upstream licenses or access terms, selected source media may also be redistributed or linked as part of the benchmark release, subject to those upstream conditions. Table 2 summarizes the current access status and license terms for the 22 source datasets covered by MedStreamBench.
4.3 Intended Use↩︎
The intended use of MedStreamBench is the evaluation of medical vision–language models under temporally restricted visual evidence. Representative use cases include benchmark comparison, study of answerability under partial observation, evaluation of streaming response behavior, and research on proactive monitoring in medical video analysis. The resource is not intended to serve as a stand-alone clinical system.
![Figure 3: Illustration of the four temporal reasoning settings in MedStreamBench. Given a surgical video timeline, a decision point T_q defines the latest observable evidence available to the model. Retrospective questions require understanding past events within the observation window [T_{q-w},T_q], while present questions focus on the current surgical state near T_q. Future questions evaluate whether models can predict upcoming events before the first answerable moment T_{\mathrm{event}} using only historical observations. Proactive streaming tasks further require the model to continuously monitor an evolving video prefix [T_q,T_c^k] and trigger alerts only when sufficient evidence becomes visible.](/article/2607.01751/four_mode.png "fig:")
5 Methods↩︎
5.1 Benchmark Overview↩︎
MedStreamBench is a time-aware medical video question-answering benchmark in which each item is associated with a visual source, an explicit temporal evidence window, a question, a reference answer, and metadata describing task type and label provenance. The release is stored as dataset-scoped JSONL files and covers both closed-choice and open-ended questions across retrospective, present, future, and proactive settings.
Each item follows a compact unified schema: \[x = (v, q, a, \tau, m, r, s),\] where \(v\) denotes the visual source, \(q\) is the question, \(a\) is the answer or reference answer, \(\tau\) denotes the temporal specification, \(m\) denotes the task mode, \(r\) contains evaluation rubrics or response protocols, and \(s\) stores source and quality metadata. The temporal specification includes the query time \(t_q\), the context window length \(w\), and the input range \([t_s, t_e]\). For streaming tasks, each item additionally contains a sequence of rounds with current time \(t_c\), round-specific input range, expected output, and answerability flag.
5.2 Data Construction Pipeline↩︎
The MedStreamBench data engine converts heterogeneous medical datasets into a unified temporal QA format. As illustrated in Figure 2, the pipeline is organized into three aligned stages: temporal surgical data preparation, label-grounded QA construction, and benchmark validation.
5.2.0.1 Temporal surgical data preparation.
We first standardize the source videos or image sequences and align them with available supervision signals. For structured datasets, these signals come from original annotations such as phases, tool labels, actions, masks, bounding boxes, or skill metrics; for other sources, they may also come from directory-defined categories or weak-label mining. Each target contains a visual source, a task specification, an answer signal, and an initial temporal window. For video datasets, timestamps are converted into seconds using dataset-specific metadata, whereas image-only datasets use the temporal range \([0, 0]\).
5.2.0.2 Label-grounded QA construction.
The aligned supervision signals are then converted into temporally grounded QA items. For label-grounded tasks, deterministic targets are rewritten as natural-language closed or open-ended questions while preserving the answer. For weakly labeled or unlabeled videos, the engine conservatively mines candidate QA items from visible findings, procedural states, tool interactions, or screening alerts. To broaden temporal coverage, this stage also produces present, future, streaming, and proactive items rather than only retrospective recognition examples.
5.2.0.3 Benchmark validation and materialization.
The resulting candidate items are aggregated, refined, and validated before release. The validator checks schema completeness, question and answer fields, formatting, media paths, temporal fields, and streaming rounds, and it enforces the temporal
contract that single-turn items end at \(t_q\) while streaming items use round-specific windows of the form \([t_q, t_c]\). Proactive items are additionally constrained to the response space
no_alert, uncertain, or alert: <reason>. Accepted items are then materialized into final dataset-scoped benchmark files for downstream inference and evaluation.
| Metric | Interpretation |
|---|---|
| Content correctness (\(C\)) | Semantic correctness under the evidence-constrained input. For closed questions, it is based on normalized exact match with optional semantic fallback; for open questions, it combines correctness, evidence grounding, and safety. |
| Responsiveness (\(R\)) | Temporal accuracy of the first positive answer or alert relative to the expected answerable or critical time. Early, delayed, and missed responses are penalized. |
| Stability (\(S\)) | Post-evidence consistency after the item becomes answerable, measuring whether the model remains positive in later rounds rather than reverting to abstention or omission. |
| Overall score (\(O\)) | Task-level summary score. For single-turn items, \(O=C\); for streaming future and proactive items, \(O=0.7\,C+0.3\,R\), prioritizing correctness while rewarding timely response. |
4pt
5.3 Temporal Task Formulation↩︎
MedStreamBench defines four temporal modes and two task modes. Single-turn items produce one model call per item, whereas streaming items produce one model call per round.
5.3.0.1 Retrospective and present single-turn QA.
For retrospective and present questions, the model receives a context window ending at the query time: \[I = [\max(0, t_q - w),\;t_q].\] Retrospective questions ask about events or states within the window, whereas
present questions focus on the visual state near \(t_q\). If the available evidence is insufficient, the expected behavior is unanswerable.
5.3.0.2 Future single-turn QA.
Future single-turn questions use the same input window: \[I = [\max(0, t_q - w),\;t_q],\] but ask about an event or state after \(t_q\). The model must reason from the available context
only and return unanswerable when the future event is not yet predictable.
5.3.0.3 Future streaming QA.
For future streaming questions, the item is evaluated over a sequence of rounds. At round \(k\), the model receives only the prefix: \[I_k = [t_q,\;t_c^{(k)}],\] where \(t_c^{(k)}\) is the current time for that round. Before the evidence becomes sufficient, the expected output is unanswerable; once the target event becomes answerable, the model should answer and remain stable in
subsequent rounds.
5.3.0.4 Proactive streaming QA.
Proactive questions also use round-specific prefixes \([t_q, t_c^{(k)}]\), but the response space is constrained to: \[y_k \in \{\texttt{no\_alert},\;\texttt{uncertain},\;\texttt{alert:
<reason>}\}.\] The model should alert only when sufficient evidence appears, while using no_alert or uncertain beforehand.
| Retrospective | Present | Future | Proactive | Overall | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (lr8pt)2-2 (lr8pt)3-3 (lr8pt)4-6 (lr8pt)7-9 (lr8pt)10-10 | Content | Content | Content | Resp. | Stab. | Content | Resp. | Stab. | Score | |||||||||||
| Gemini-2.5-Pro | 0.3428 | 0.4140 | 0.4384 | 0.3636 | 0.6927 | 0.6158 | 0.4494 | 0.4742 | 0.3728 | |||||||||||
| Qwen3-VL-4B | 0.2255 | 0.3491 | 0.4046 | 0.1870 | 0.3415 | 0.6428 | 0.2301 | 0.2114 | 0.2701 | |||||||||||
| Qwen3-VL-8B | 0.2008 | 0.2898 | 0.3260 | 0.0912 | 0.3293 | 0.6229 | 0.1499 | 0.1337 | 0.2324 | |||||||||||
| Qwen2.5-VL-7B | 0.2413 | 0.3446 | 0.3656 | 0.0533 | 0.4635 | 0.6255 | 0.0547 | 0.0525 | 0.2662 | |||||||||||
| InternVL3.5-8B | 0.2536 | 0.3461 | 0.3847 | 0.0710 | 0.3934 | 0.4441 | 0.0899 | 0.0813 | 0.2713 | |||||||||||
| LLaVA-OneVision-7B | 0.2498 | 0.2888 | 0.2890 | 0.0746 | 0.8142 | 0.5470 | 0.0579 | 0.0595 | 0.2572 | |||||||||||
| HuluMed-4B | 0.2231 | 0.2868 | 0.4327 | 0.1456 | 0.2399 | 0.4742 | 0.2088 | 0.2248 | 0.2578 | |||||||||||
| HuluMed-7B | 0.2298 | 0.2798 | 0.3851 | 0.1425 | 0.1961 | 0.6258 | 0.0677 | 0.0723 | 0.2601 | |||||||||||
| Lingshu-7B | 0.2329 | 0.3302 | 0.3417 | 0.0953 | 0.7180 | 0.6302 | 0.0514 | 0.0514 | 0.2586 | |||||||||||
5.5pt
Content: content accuracy. Resp.: responsiveness score. Stab.
: stability score.
5.4 Inference Protocol↩︎
The MedStreamBench inference engine expands each benchmark item into one or more inference jobs. Single-turn items yield one job, whereas future and proactive streaming items yield one job per round.
For video inputs, the engine samples frames uniformly at one frame every 2 seconds within the job-specific input range, including the endpoint when needed so that evidence at \(t_q\) or \(t_c\) remains visible. For image-sequence datasets, the input range is interpreted analogously as a frame-index interval. The textual prompt specifies the task mode, temporal mode, query time, input range, and the constraint
that no evidence beyond the designated endpoint may be used. Future streaming prompts require unanswerable before sufficient evidence appears, while proactive prompts restrict the response space to no_alert,
uncertain, or alert: <reason>.
5.5 Evaluation Metrics↩︎
MedStreamBench evaluates both answer correctness and temporal behavior. Table 3 summarizes the interpretation of the reported metrics; below we specify the corresponding scoring rules.
5.5.0.1 Content correctness.
For closed questions, predictions are evaluated by normalized exact match, with optional judge fallback for semantically equivalent mismatches. For open-ended questions, a rubric-based judge scores correctness, evidence grounding, and safety. The resulting score is normalized to \([0, 1]\): \[C = \frac{s_{\mathrm{correct}} + s_{\mathrm{ground}} + s_{\mathrm{safety}}}{5},\] where \(s_{\mathrm{correct}} \in \{0, 1, 2\}\), \(s_{\mathrm{ground}} \in \{0, 1, 2\}\), and \(s_{\mathrm{safety}} \in \{0, 1\}\). For streaming items, content correctness is averaged over rounds.
5.5.0.2 Responsiveness.
For future and proactive streaming items, responsiveness measures whether the first positive response occurs near the expected response time. For future streaming tasks, a positive response is any answer other than unanswerable; for
proactive tasks, it is an alert. Let \(t^*\) be the expected response time, \(\hat{t}\) the first predicted positive time, and \(\delta\) the
tolerance. A prediction receives the maximum responsiveness score if \[|\hat{t} - t^*| \leq \delta.\] Missed responses receive zero responsiveness, and premature or delayed responses are penalized according to their
temporal distance from \(t^*\).
5.5.0.3 Stability.
For streaming tasks, stability measures whether the model remains positive after the item becomes answerable: \[S = \frac{1}{|\mathcal{K}_{\geq t^*}|} \sum_{k \in \mathcal{K}_{\geq t^*}} \mathbb{1}[y_k \text{ is positive}],\] where \(\mathcal{K}_{\geq t^*}\) denotes rounds at or after the expected response time.
5.5.0.4 Overall score.
For single-turn items, the overall score is the content score: \[O = C.\] For streaming future and proactive items, the overall score combines correctness and responsiveness: \[O = 0.7\,C + 0.3\,R,\] where \(R\) is the responsiveness score.
6 Validation↩︎
6.1 Baseline description↩︎
Table 4 reports representative baselines from three groups: a closed-source model [18], five general-domain open-source models [19], [20], and three medical-domain models [10], [21]. All experiments were conducted on NVIDIA H100 GPUs under the same benchmark protocol.
6.2 Main Results↩︎
Table 4 summarizes initial baseline results on MedStreamBench. Together with the metric definitions in Table 3, it separates semantic accuracy from temporal responsiveness and stability across retrospective, present, future, and proactive settings. This separation makes it possible to distinguish models that perform well in offline recognition from models that are also temporally disciplined under prefix-only evidence. The overall score provides a compact benchmark-level summary by aggregating item scores under the task-specific evaluation rule defined in Section 4.2. Across the evaluated models, Gemini-2.5-Pro achieves the strongest overall performance, while InternVL3.5-8B is the best-performing open-source model and Lingshu-7B yields the strongest overall score among the medical-domain models. These aggregate results are useful for high-level comparison, but they should be interpreted together with the mode-specific columns because temporal strengths are not uniformly distributed across 4 task modes.
6.3 Organ-level analysis↩︎
Figure 4 aggregates dataset-level content performance by organ or procedural domain using sample-count-weighted averaging. Gemini-2.5-Pro achieves the strongest performance across most organ groups, whereas the best open or medical-domain models emerge in more specialized regions, including InternVL3.5-8B on the GI tract and capsule group, Qwen2.5-VL-7B on multi-organ open surgery, and HuluMed-4B on MIS robotic surgery. This heterogeneous pattern shows that benchmark performance is not uniform across clinical domains and motivates organ-level analysis beyond a single global average. Figure 1 and Figure 3 provide the corresponding resource-level and temporal-task context for interpreting these benchmark-level quantitative results.
6.4 Human Check↩︎
As a preliminary quality check for model-assisted items, we compared AI-generated answers with judgments from two independent human annotators on a random sample. Figure 5 shows agreement with both annotators in 60% of cases, with one annotator in 25%, and with neither annotator in 15%.