Scientific visualization (SciVis) has long served as an important medium for exploring, analyzing, and communicating complex scientific data, including simulation output, medical imaging, climate data, and other spatially grounded scientific phenomena [1], [2]. In contrast to information visualization (InfoVis), SciVis typically represents data with intrinsic spatial or physical structure. This distinction is closely tied to the notions of physically based data and given spatialization, which differentiate much of SciVis from InfoVis [3]. In addition, when interpreting SciVis, viewers need to reason about spatial structure, multivariate relationships, and time-varying or physically meaningful patterns encoded in the visual representation [2], [3].

Meanwhile, multimodal large language models (MLLMs) have rapidly advanced in their ability to process images alongside text and to support visualization interpretation, scientific communication, and visual analytics workflows [4]–[6]. However, existing evidence suggests that their performance on visualization tasks remains uneven, and current evaluations have focused primarily on InfoVis rather than SciVis [7], [8]. As a result, whether current MLLMs possess SciVis literacy remains an open question [9].

To address this gap, we benchmark MLLMs using the scientific visualization literacy assessment test (SVLAT) [10], a recently introduced instrument designed specifically to measure how well non-experts read, understand, and interpret scientific visualizations and illustrations. Because SVLAT spans multiple SciVis techniques and tasks under a closed-world design, it provides a suitable benchmark for evaluating SciVis understanding in MLLMs.

Using SVLAT, our goal is to examine whether current MLLMs demonstrate measurable SciVis literacy and, if so, where their strengths and weaknesses lie across techniques and task categories. More broadly, we position SciVis literacy as a missing benchmark dimension for multimodal AI systems. By doing so, we aim to contribute a clearer evaluation target for future work on trustworthy model assessment, model-assisted scientific communication, and human–AI interaction around scientific visual content.

1 Related Work↩︎

Visualization literacy assessment. Visualization literacy refers to a viewer’s ability to read, understand, and interpret data visualizations [11]–[13]. Boy et al. [14] introduced a principled approach based on item response theory (IRT) to assess visualization literacy, showing that literacy can be modeled as a latent ability rather than treated solely as raw accuracy. Building on this psychometric perspective, Lee et al. [15] developed VLAT, a standardized assessment for non-expert users that measures literacy across 12 visualization types and 8 visualization tasks. CALVI [16] extended the notion of visualization literacy beyond routine reading by focusing on critical thinking about misleading or erroneous visualizations. To improve practicality and reduce administrative time, later work introduced shorter, more specialized instruments. For instance, Mini-VLAT [17] condensed VLAT into a brief but effective short-form test while preserving much of its measurement utility. Adaptive versions of VLAT and CALVI [18] reduced testing burden while retaining reliability and validity. These studies show that visualization literacy is a mature topic for assessment within InfoVis.

SciVis literacy. Compared with InfoVis, SciVis introduces additional interpretive demands. Scientific visualizations often encode data with inherent spatial and physical structure, requiring users to reason about geometry, depth, topology, motion, and relationships among scalar, vector, or tensor fields. Prior work on SciVis tasks has emphasized the importance of task taxonomies tailored to scientific data analysis, especially for volume data and spatial reasoning [19]. These characteristics make SciVis literacy qualitatively different from InfoVis literacy in the context of conventional charts and graphs. Despite the importance of SciVis in scientific analysis and communication, standardized assessment in this area has remained largely absent until recently. SVLAT [10] is a recently proposed psychometrically grounded instrument for measuring SciVis literacy. SVLAT is designed around 8 canonical SciVis techniques and 11 interpretation tasks, enabling literacy to be measured in a way that better reflects the demands of scientific visual representations. Because SVLAT is standardized, validated, and explicitly constructed for scientific visual material, it provides an appropriate foundation for benchmarking MLLMs on SciVis literacy rather than relying on chart-oriented proxies.

MLLMs and visualization understanding. Work on machine understanding of visualized data initially developed through benchmark datasets for chart and figure question answering. Early datasets such as FigureQA [20] and DVQA [21] focused on synthetic scientific-style figures and bar charts, respectively, establishing controlled testbeds for visual reasoning over plotted data. PlotQA [22] moved closer to realistic scientific plots by introducing open-vocabulary and numerical answers, while ChartQA [23] expanded the task to charts requiring both visual and logical reasoning. More recent benchmarks have further stressed the limitations of current MLLMs. CharXiv [24] evaluated realistic charts collected from scientific papers and showed that prior benchmarks can overestimate model capability, and MultiChartQA [25] studied multi-chart reasoning, where models must integrate information across several related visualizations. Beyond chart-only settings, SPIQA [26] evaluated multimodal question answering over figures and tables embedded in scientific papers, extending the problem toward document-grounded scientific figure understanding.

Within the visualization community, recent work has begun to study MLLMs through the lens of visualization literacy itself. Bendeck and Stasko [7] evaluated GPT-4 on a suite of visualization-literacy-related tasks and found a mixed pattern of strengths and weaknesses: the model could identify higher-level trends and extreme values, but it struggled with precise value retrieval, color discrimination, and consistency. Hong et al. [8] then examined whether LLMs truly possess visualization literacy by testing them on modified VLAT-style visualizations designed to probe generalization and reliance on visual evidence rather than prior knowledge. Their findings suggest that model performance can degrade substantially under benign visual modifications, raising concerns about robustness and genuine visual grounding. Das et al. [27] further showed that structured prompting can substantially improve performance on VLAT-style tasks, highlighting that measured capability depends not only on the model itself but also on the evaluation protocol. Complementing these studies, Varona et al. [28] analyzed visualization-literacy barriers in MLLMs by qualitatively coding erroneous responses, providing a taxonomy of failure modes that helps explain why models struggle even when benchmark accuracy appears competitive. Dong and Crisan [29] further probed the internal reasoning behavior of vision-language models for chart understanding, arguing that answer correctness alone is insufficient for characterizing visualization literacy in models.

Most existing datasets, evaluations, and prompting studies focus on InfoVis-style charts or chart-like figures rather than SciVis-native representations. Together, these studies show that evaluation of MLLMs on visualization tasks remains largely chart-centric, leaving SciVis-native assessment comparatively underexplored.

2 Methodology↩︎

Assessment instrument. We evaluate MLLMs using SVLAT [10], a standardized assessment of SciVis literacy. SVLAT consists of 49 items based on 18 scientific visualizations and illustrations, including static images and animations, spanning 8 SciVis techniques and 11 task types. It follows a closed-world design in which each item is intended to be answerable using only the provided visualization and caption, making it suitable for assessing SciVis literacy in MLLMs while limiting reliance on external knowledge.

Model selection and configuration. We evaluate both closed-source and open-source MLLMs. The closed-source models are GPT-5.4 (GPT)[30], Claude-Opus-4.6 (Claude)[31], and Gemini-3.1-Pro-Preview (Gemini)[32], all accessed via the OpenRouter API for consistent deployment. For animation items, GPT and Claude were evaluated using frame extraction (1 frame per second), as their APIs did not support direct video input at the time of evaluation. The open-source models are Qwen3.5-9B (Qwen)[33], InternVL3.5-8B (InternVL)[34], and LLaVA-OneVision-1.5-8B-Instruct (LLaVA-OneVision)[35]. All models were evaluated with temperature set to 0 and max_tokens set to 300 to improve reproducibility and consistency across models and runs. Setting the temperature to 0 makes decoding as deterministic as possible, thereby reducing sampling variability in both answer selection and rationale generation. We set max_tokens to 300 to ensure sufficient space for the required JSON output and brief explanation while avoiding unnecessarily long responses. Because some backend nondeterminism may remain, we repeated each evaluation 10 times and report the averaged results.

Prompt design. We use a single instruction prompt for all models to standardize the evaluation setting. The prompt frames the model as a SciVis expert and explicitly constrains it to use only the provided visualization and caption. It also requires the model to return a structured JSON object containing the selected option, the full answer text, and a brief rationale. To discourage unsupported guessing, the prompt includes a "Not sure" option when the answer cannot be determined confidently from the given evidence. The prompt used in our experiments is shown in Figure 6. This prompt is designed to align with the closed-world principle of SVLAT and to make model outputs easier to parse automatically.

Evaluation protocol. We evaluate each model on SVLAT one question at a time. For each item, we run the model 10 times using different random seeds and compute the average score across runs. This repeated-run protocol helps reduce the impact of residual sampling variability or backend nondeterminism. In addition to the selected answer, we record the model’s rationale for each response. These explanations are used to examine whether the model’s reasoning is grounded in the provided visualization and caption rather than unsupported prior knowledge or speculation. Although the rationale is not used directly for scoring accuracy, it provides qualitative evidence about how the model arrives at its answers and whether incorrect responses stem from misinterpretation of visual evidence, overconfident guessing, or reliance on external knowledge. We also analyze performance by SciVis techniques and task categories to identify systematic strengths and weaknesses. To contextualize MLLM capabilities, we compare accuracy against the human performance baseline from the original SVLAT study [10], established via an online tryout with 485 non-expert participants (246 females; ages 18–65; education ranging from high school to postgraduate).

3 Results↩︎

3.1 Overall Performance↩︎

Table 1 reports the overall performance of the six evaluated MLLMs across 10 runs and of 485 human participants on SVLAT. Gemini is the strongest model by a wide margin, achieving 90.9% on image items, 82.9% on animation items, and 88.6% overall, exceeding human performance of 76.4%, 74.5%, and 75.9%, respectively. The three open-weight models trail humans in combined accuracy, with Qwen performing best (68.8% overall), followed by InternVL (64.3%) and LLaVA-OneVision (64.1%). On animation items, Gemini remains the top-performing model. Via frame extraction, GPT matches human-level accuracy (73.6% vs.%), while Claude scores the lowest of all models and humans at 59.3%. Among the open-source models, LLaVA-OneVision (73.6%) and InternVL (72.9%) come close to the human level, and both perform better on animation items than on static image items, whereas Qwen remains lower at 67.1%. MLLMs exhibit greater performance heterogeneity across item types (SD: 3.9–11.7) than humans (SD: 2.2–4.7).

3.2 Performance by Techniques↩︎

Human and model performance differ substantially across techniques, as shown in Figure 1. Humans exhibit the most balanced pattern, with mean accuracy ranging from 68.9% to 88.8% across all techniques, whereas the MLLMs show much larger fluctuations. Among the evaluated models, Gemini is the strongest and broadest-performing system. It achieves 100.0% on Surface Rendering and Volume Rendering, 95.0% on Mixed Rendering, 95.7% on Scientific Illustration, and 91.8% on Glyph, Mesh, Plot, exceeding human performance on the same techniques. Closed-source models consistently outperform open-source models across Surface Rendering, Mixed Rendering, and Glyph, Mesh, Plot, with the gap being most pronounced on the latter.

Across all techniques, Scientific Illustration is the most accessible to humans and all models. Human accuracy reaches 79.5%, while all models perform strongly, ranging from 81.4% to 95.7%. In contrast, Texture-Based Visualization and Integration-Based Visualization reveal a clearer gap between human and model performance. Humans achieve high accuracies on both techniques, at 82.0% and 88.8%, respectively. LLaVA-OneVision is the main exception on Texture-Based Visualization (90.0%), where all other models fall substantially below it. On Integration-Based Visualization, LLaVA-OneVision (82.5%), GPT (80.0%), and Gemini (75%) remain competitive with humans, whereas other models are notably lower. Per-item accuracies for MLLMs and humans, categorized by techniques, are furnished in Appendix 5.

3.3 Performance by Tasks↩︎

Figure 2 shows that performance varies substantially by task. Across all categories, Search and Spatial Understanding tasks are the most accessible, whereas Quantitative Estimation–Relative Estimation (Quantitative) is the most difficult; indeed, it is the hardest task for every category, with accuracy remaining below 50.0% throughout. Humans perform particularly well on Spatial Understanding–Absolute (88.1%), Spatial Understanding–Intersection (87.7%), and Search–Counting (85.6%), and the models follow the same broad pattern.

Among all models, Gemini performs best, achieving the highest accuracy on nearly every task. The main exception is Pattern Recognition–Repetition and Quantitative Estimation–Relative Estimation (Quantitative), where Gemini drops to 50.0% and 45.0%, respectively. For Spatial Understanding–Absolute, all three closed-source models max out, whereas open-source models are notably lower. The open-source models are much less competitive with humans. Qwen and InternVL exceed human performance only on Search–Presence/Absence and Spatial Understanding–Intersection, while LLaVA-OneVision exceeds humans only on Spatial Understanding–Intersection. On the remaining tasks, the open-source models perform at or below the human level. It is worth noting that Quantitative Estimation–Relative (Quantitative), Spatial Understanding–Intersection, and Pattern Recognition–Repetition are each represented by only one or two items in the current benchmark, so the above findings for these categories should be treated as preliminary.

3.4 Error Analysis↩︎

The preceding results show that MLLM performance on SVLAT varies substantially across techniques, tasks, and model families. To better understand the sources of these differences, we qualitatively examine model rationales to identify recurring patterns in how models interpret, misinterpret, or fail to ground visual evidence.

Fine-grained quantitative misestimation. Models struggle with precise quantitative judgment when the answer depends on careful visual estimation rather than coarse pattern recognition. In Item 3, Gemini correctly recognized that “the map’s contour interval is 40 feet” and located waypoint D, stating that it “lies between the 6880 ft and 6840 ft contour lines” (see Figure 3). However, it misread waypoint A by claiming that it is “located exactly on the thick index contour line labeled 7200,” even though A lies very close to the 7200 contour line while remaining between the 7200 ft and 7240 ft contour lines. A similar problem appears in Item 41 “What is the approximate radius of the lipid envelope?” (refer to Appendix 5). Gemini used the scale bar correctly, but estimated the diameter to be “approximately five times the length of the scale bar,” yielding a radius of 50 nm; Claude produced the same overestimate across all 10 runs, and GPT did so in 3 of 10 runs. By contrast, the open-source models showed no consistent bias, producing radius estimates ranging from 30 to 50 nm across runs. The correct diameter is approximately 4 times the 20 nm scale bar, giving a radius of about 40 nm. These examples suggest that the models can often identify the relevant quantitative encoding and coarse measurement strategy, a weakness reflected in the particularly low MLLM scores on Quantitative Estimation–Relative Estimation (Quantitative).

Flow-direction interpretation error. Texture-based visualizations exposed difficulty in decoding local flow direction from animated vector fields. In Item 20, Gemini and Qwen incorrectly concluded that the winds near Chicago and Houston flow in different directions, reasoning that Chicago moves “from west to east” while Houston moves “from south to north.” In fact, the winds in both regions flow in roughly the same direction, approximately from northwest to southeast (see Figure 4). This pattern of difficulty extends even to models that answered correctly: InternVL identified the same direction at both cities but misread it as “west to east,” while LLaVA avoided any specific direction, stating only that arrows “point in a similar direction.” This suggests that the model misread local motion in the animated flow field, possibly by confusing streak orientation with the actual flow direction. Because most models correctly identified the hurricane’s direction in Item 9 (refer to Appendix 5), the difficulty appears specific to dense texture-based motion decoding rather than to directional reasoning more generally.

Encoding-mapping error during higher-level SciVis interpretation. Some failures occur when the model makes a higher-level interpretation, but grounds it in an unsupported visual mapping. In Item 35, Gemini inferred that fiber density was indicated by the opacity and solidness of the colored regions, and on that basis concluded that region C had the highest density because it appeared as a “highly opaque, solid red mass” (see Figure 5). Claude reached the same wrong conclusion across all 10 runs, but via a different unsupported mapping: it invoked anatomical prior knowledge, stating that region C “corresponds to the splenium of the corpus callosum ... densely packed with fibers of multiple orientations (red, green, and blue),” thereby grounding its density judgment in imported neuroanatomical knowledge and color variety rather than actual fiber concentration in the image. The item requires a SciVis judgment on fiber density, which can be estimated from the overall concentration and fiber overlap. Instead, both models justified their answer with unsupported cues even though the caption specifies that color encodes fiber orientation. The error, therefore, is not simply that the model made a higher-level inference, but that it grounded this inference in an unsupported encoding rather than in the actual evidence provided by the visualization and caption.

4 Conclusions and Future Work↩︎

Our results show that current MLLMs do not yet exhibit uniform SciVis literacy. Performance varies substantially across techniques and tasks. Across techniques, the models perform best on Scientific Illustration, while Gemini also performs strongly on Surface Rendering, Volume Rendering, and Mixed Rendering. Across tasks, their strongest results are in Search and Spatial Understanding, particularly in Presence/Absence, Counting, and Intersection. However, the open-source models remain less robust overall and show narrower, more uneven strengths. The qualitative analysis shows that they reflect recurring weaknesses in fine-grained quantitative estimation, flow-direction interpretation in texture-based visualizations, and unsupported encoding inference during higher-level SciVis interpretation. These patterns suggest that current MLLMs can often identify the general structure of a task, but still struggle when success depends on precise measurement, decoding local motion in dense dynamic fields, or grounding higher-level judgments in the actual encoding specified by the visualization and caption.

This study has several limitations. GPT and Claude were evaluated on animation items via frame extraction, as their APIs did not support direct video input during the evaluation. In addition, the evaluation is based on a single benchmark, SVLAT, and a single standardized prompt format, and the open-source comparison is limited to lightweight models. Additionally, some tasks are represented by only one or two items, which limits the reliability of per-task estimates and may not fully capture the range of difficulty within those tasks. Future work should extend this benchmark to larger and more diverse models, richer prompting and tool-use settings, and additional SciVis-native evaluation instruments. Overall, our findings show that current MLLMs are selectively capable rather than broadly SciVis literate. SciVis literacy is, therefore, a necessary benchmark dimension because scientific visualizations require forms of spatial, temporal, and encoding-grounded interpretation that are not captured by general multimodal evaluations.

Figure Credits↩︎

Figure 3: Terrain Map of the Roberts Mountains, Nevada—Image adapted from USGS National Geologic Map Database. Figure 4: Wind Map of the United States—Animation from hint.fm. Figure 5: Human Brain Fiber Bundles—Image adapted from Zeynep Saygin / MIT Koch Institute.

5 Additional Details and Results↩︎

References↩︎