Navigating the Mirage: A Dual-Path Agentic Framework for Robust Misleading Chart Question Answering
March 30, 2026
Abstract
Despite the success of Vision-Language Models (VLMs), misleading charts remain a significant challenge due to their deceptive visual structures and distorted data representations. We present ChartCynics, an agentic dual-path framework designed to unmask visual deception via a “skeptical” reasoning paradigm. Unlike holistic models, ChartCynics decouples perception from verification: a Diagnostic Vision Path captures structural anomalies (e.g., inverted axes) through strategic ROI cropping, while an OCR-Driven Data Path ensures numerical grounding. To resolve cross-modal conflicts, we introduce an Agentic Summarizer optimized via a two-stage protocol: Oracle-Informed SFT for reasoning distillation and Deception-Aware GRPO for adversarial alignment. This pipeline effectively penalizes visual traps and enforces logical consistency. Evaluations on two benchmarks show that ChartCynics achieves 74.43% and 64.55% accuracy, providing an absolute performance boost of \(\sim\)29% over the Qwen3-VL-8B backbone, outperforming state-of-the-art proprietary models. Our results demonstrate that specialized agentic workflows can grant smaller open-source models superior robustness, establishing a new foundation for trustworthy chart interpretation.
1 Introduction↩︎
![Figure 1: Overview of ChartCynics resolving an “Inverted Axis” deception. (Left) A counter-intuitive real-world case [1]: The inverted Y-axis creates a visual illusion of declining deaths, hijacking standard VLM attention despite the underlying numerical increase. (Middle) Our dual-path architecture decouples perception from extraction. Crucially, pure OCR lacks spatial context, causing entity misalignment (e.g., confusing axis ticks with data points). The Diagnostic Vision Path detects the scale anomaly via ROI cropping and generates an Action Directive to guide the OCR Path in accurate entity mapping. (Right) The Agentic Summarizer resolves cross-modal conflict by combining the complementary information: vision-derived structural baseline and precise OCR numerals to correctly infer a numerical increase.](/article/2603.28583/figure/figure1.png "fig:")
Visualizations are ubiquitous in the data-driven world, serving as powerful tools for communicating complex numerical information. However, this persuasive power is frequently compromised by misleading charts: visualizations designed to distort data perception through tactics such as axis manipulation, cherry-picking, or disproportionate encoding [2]–[6]. The visualization and human-computer interaction communities have extensively studied how these deceptive designs manipulate human audiences, yielding well-established taxonomies and empirical evaluations [7]–[11], as well as literacy assessments [12], [13].
Recently, as Visual Language Models (VLMs) [14], [15] become increasingly central to automated chart understanding, a critical new challenge has emerged: ensuring their robustness against these misleading visualizations. Traditional chart comprehension tasks primarily evaluate the extraction of explicit visual facts under benign assumptions [16]–[18]. In contrast, the emerging task of Misleading Chart Question Answering (MQA) shifts the paradigm toward visual critical reasoning, requiring models to actively detect manipulations and resolve cross-modal contradictions. However, recent studies reveal a significant gap: on targeted benchmarks like the Misleading ChartQA dataset, state-of-the-art VLMs consistently score below 50% accuracy[19]. As corroborated by diverse evaluations[20], [21], these models perform no better than untrained human observers because they fail to maintain logical consistency against deceptive cues.
The current methods for automatically interpreting charts mainly follow two patterns. The first paradigm, the end-to-end VLM path, leverages multimodal models for direct visual reasoning [14], [22]–[24]. Although these models perform well on standard benchmarks such as ChartQA [16], they process charts holistically, favoring macro-level visual heuristics (e.g., the overall slope of a line or relative bar heights) over fine-grained structural cues like exact tick values. As highlighted in Figure 1, this architectural limitation makes them highly vulnerable to malicious encodings. For instance, when a Y-axis is inverted (zero at the top), the visual encoder’s attention is hijacked by the steep downward physical slope. This forces the model into “cognitive sycophancy,” erroneously predicting a numerical decline and completely ignoring the actual increasing data values.
The second paradigm, the OCR-enhanced pipeline, attempts to circumvent perceptual pitfalls by linearizing charts into structured text [25]–[28]. While this approach successfully retrieves numerical literals, it introduces a severe structural dilemma: standalone OCR models eliminate vital spatial and layout semantics. As shown in the OCR path of Figure 1, without visual context, the model extracts an unstructured sequence of characters (e.g., 873, 2005, 721). Consequently, it suffers from catastrophic entity misalignment, lacking the capability to distinguish whether a number is a background axis tick or a foreground data label. Without a visual anchor to establish the mapping baseline, these extracted “cold facts” are practically unusable for reasoning.
An ideal solution lies in the synergistic fusion of these two paths, moving beyond passive description toward active verification: much like a human auditor who cross-verifies a visually deceptive trend against literal numerical data. However, integrating diagnostic vision with data-driven reasoning presents three non-trivial challenges that correspond to the perception, reasoning, and optimization levels:
Diagnostic Locality (Perception Level): Standard VLMs process charts holistically, making it difficult to force fine-grained inspections on easily overlooked areas, such as non-zero axis baselines or manipulated legends, where visual deceptions are often anchored.
Conflict Arbitration (Reasoning Level): Even if both visual trends and precise OCR numerals are extracted, a robust mechanism is required to rationally resolve the inherent cross-modal contradictions when these two signals provide opposing evidence.
Reasoning Calibration (Optimization Level): How to shift the model from a passive observer to a “skeptical auditor.” It requires specialized training interventions to suppress the model’s pre-trained visual biases and prioritize evidence over misleading heuristics.
To address these challenges, we propose ChartCynics, an agentic dual-path framework designed to unmask visual deception. ChartCynics employs a “skeptical” philosophy that decouples structural perception from literal extraction. As illustrated in Figure 1, our Diagnostic Vision Path acts as a structure sensor: it utilizes strategic element cropping to detect anomalies (e.g., an inverted scale) and provides the essential spatial semantics. Guided by this structural baseline, the OCR-Driven Data Path can perform precise entity mapping without misalignment. Finally, to resolve cross-modal conflicts, we introduce an Agentic Summarizer that executes a detective Chain-of-Thought (D-CoT). Rather than simply discarding visual cues, it synthesizes these complementary modalities, using the visual structural context to ground the numerical facts, thereby synthesizing a robust and rational answer.
A core contribution is our two-stage optimization strategy for the Summarizer. First, we perform Supervised Fine-Tuning (SFT) to inject a detective Chain-of-Thought (CoT). This step establishes the structural framework for skepticism, enabling the model to follow a rigorous 5-step verification process. Second, we apply Group Relative Policy Optimization (GRPO) with a deception-aware reward function. By specifically penalizing the selection of misleading “trap” answers, GRPO aligns the model to maintain logical consistency even when faced with high-confidence visual illusions.
Experimental results demonstrate that while a standard Qwen3-VL-8B baseline achieves only \(45.57\%\) accuracy on deceptive benchmarks, ChartCynics reaches a state-of-the-art \(74.43\%\). Notably, the integration of SFT provides a foundational \(22.95\) percentage point boost over the baseline, while GRPO further optimizes the reasoning trajectory to resolve complex cross-modal contradictions. Furthermore, evaluations on a mixed standard-and-misleading benchmark reveal a critical advantage: unlike typical defensive mechanisms that suffer from “over-skepticism” (falsely penalizing benign data), ChartCynics actually enhances fundamental chart comprehension. Our systematic structural investigation yields superior accuracy even on standard, non-misleading visualizations compared to specialized chart-parsing SOTA models.
Our contributions are summarized as follows:
We present ChartCynics, an agentic dual-path framework that integrates diagnostic cropping and OCR reasoning for robust defense against misleading visualizations.
We introduce a two-stage training protocol combining SFT and GRPO, which equips MLLMs with both a structured investigative CoT and an alignment-based defense against visual traps.
We demonstrate that the fusion of agentic workflows and reinforcement learning significantly enhances critical thinking, achieving \(74.43\%\) accuracy across challenging benchmarks.
We validate the generalized robustness of our framework, proving that equipping models with a “skeptical lens” does not induce over-skepticism. ChartCynics not only unmasks deception but also improves basic data extraction capabilities on standard, non-misleading charts.
2 Related Work↩︎
2.1 Visual Deception and Chart Literacy↩︎
Traditional chart comprehension research primarily focused on factual extraction from natural visualizations, as seen in early datasets like FigureQA [29], DVQA [30], PlotQA [31] and Chart-HQA [32].
Misleading charts, which utilize deceptive encodings (such as axis truncation and inverted axes) to exploit human cognitive biases, have long been a subject of concern in visual communication [2], [7]. Recent benchmarks like Misleading ChartQA [19], LEAF-QA [33] and CHARTOM [21] have further shown that even advanced VLMs are highly susceptible to these visual “traps,” often performing no better than general audiences in identifying “Theory-of-Mind” deceptive patterns [34].
2.2 Multimodal Reasoning for Structured Data↩︎
The integration of VLMs has significantly advanced ChartQA capabilities, particularly through large-scale pre-training on chart-specific tasks [16], [33]. Models like MATCHA [26] and DEPLOT [25] leverage math reasoning and plot-to-table translation to link pixels and structured data. Despite this, holistic image processing based on backbones like ViT [35] or Pix2Struct [36] often overlooks fine-grained anomalies, such as non-uniform axis scales. Recent works like ChartX [17] and AskChart [27] have explored augmenting VLMs with OCR and textual enhancement to provide numerical constraints. Similar fine-grained feature isolation and modal alignment strategies have also been verified effective in multimodal misinformation detection [37].
However, VLMs still exhibit “weak grounding” in chart elements [38], and effectively resolving contradictions between visual representation and raw data remains an open challenge in complex real-world scenarios [18].
2.3 Supervised Fine-Tuning and RL-based Alignment↩︎
Recent advancements in VLM optimization have shifted from general instruction following to specialized reasoning alignment [39], [40]. Supervised Fine-Tuning (SFT) remains a cornerstone for injecting domain-specific knowledge, such as the “skeptical” investigative Chain-of-Thought (CoT) [41] and reflective interaction mechanisms like PointCoT [38].
To further refine this reasoning, researchers have turned to Reinforcement Learning (RL), specifically Group Relative Policy Optimization (GRPO) [42], to align model outputs with complex objectives without excessive computational overhead. By employing MCTS-guided sample selection to identify appropriately challenging instances [43] and designing a reward function that explicitly penalizes misleading distractors, Wang et al. achieved high accuracy in challenging samples dataset. This demonstrates the power of targeted RL alignment for robust chart interpretation and its ability to bridge the gap in visual Theory-of-Mind, specifically addressing human-centric “Mind” questions [21].
3 Methodology↩︎
3.1 Concept and Problem Definition↩︎
We first define the general task of Chart Question Answering (ChartQA) and then formalize the visual deception mechanism to establish the definition of Misleading ChartQA.
General ChartQA Task Formalization. Given a chart image \(\mathcal{I}\) and a natural language question \(\mathcal{Q}\) with a set of candidate options \(\mathcal{O} = \{a_1, a_2, ..., a_n\}\), the goal of a ChartQA system is to predict the correct answer \(a^* \in \mathcal{O}\). Traditional VLM approaches typically model this as a direct mapping \(P(a \mid \mathcal{I}, \mathcal{Q})\), defining the objective as: \[a^* = \arg\max_{a \in \mathcal{O}} P(a \mid \mathcal{I}, \mathcal{Q}) \label{eq:mqa95objective}\tag{1}\]
Visual Deception and Trap Answers. A misleading chart \(\mathcal{I}\) incorporates a manipulation function \(f_{deceptive}(\mathcal{D}) \to \mathcal{I}\), where \(\mathcal{D}\) is the raw data. This function ensures that the perceived visual trend \(\mathcal{V}_{trend}\) contradicts the actual numerical relationship \(\mathcal{D}_{rel}\). A Trap Answer \(a_{trap} \in \mathcal{O}\) is thus defined as: \[a_{trap} = \text{Inference}(\mathcal{V}_{trend}) \neq a^* = \text{Inference}(\mathcal{D}_{rel})\]
Formalizing Misleading ChartQA. Building upon the above, we define Misleading ChartQA as a specialized task where \(\mathcal{O}\) explicitly contains at least one \(a_{trap}\). Unlike standard ChartQA, where visual cues and data usually align, Misleading ChartQA presents a scenario where \(\mathcal{O}\) contains a Trap Answer \(a_{trap}\) derived from \(f_{deceptive}\). The challenge lies in complementary reasoning: the system must leverage \(\mathcal{V}_{trend}\) to understand the user’s perceptual context while simultaneously using \(\mathcal{D}_{rel}\) to ground the final answer in numerical truth, effectively resolving the inherent conflict between the two.
The Proposed Agentic Objective. To achieve this, our ChartCynics framework introduces a Misleading Taxonomy \(\mathcal{T}\) as expert prior knowledge to guide the dual-path arbitration. Instead of simply prioritizing one path over the other, the agent performs Inconsistency-Aware Fusion, modeling the objective as: \[a^* = \arg\max_{a \in \mathcal{O}} P(a \mid \mathcal{P}_v(\mathcal{I}), \mathcal{P}_d(\mathcal{I}), \mathcal{Q}, \mathcal{T}) \label{eq3}\tag{2}\] where \(\mathcal{P}_v\) and \(\mathcal{P}_d\) represent information extracted from the Visual Path (capturing semantic context) and the Data Path (ensuring numerical precision), respectively.
The “skeptical” dual-path philosophy of ChartCynics does not aim to ignore visual intuition, but rather to decouple it from the final decision-making process to prevent “blind” trust. This allows the agent to: (1) identify cross-modal deviations, (2) synthesize responses that calibrate visual perceptions via data verification, and (3) utilize RL-based optimization to learn optimal balancing weights between \(\mathcal{P}_v\) and \(\mathcal{P}_d\) across diverse deception types.
To operationalize this objective and philosophy, we design a comprehensive architecture as illustrated in Figure 2: the framework implements a dual-path pipeline that decouples visual heuristics from OCR-based numerical facts, feeding them into a Summarizer Joint Inference module. To internalize this investigative logic, Figure 2b details our two-stage optimization strategy: Oracle-Informed SFT for reasoning distillation and Deception-Aware GRPO for adversarial alignment. This unified architecture ensures that \(a^*\) is the result of rigorous conflict resolution rather than visual sycophancy.
3.2 Vision Path: Diagnostic-Augmented Investigation↩︎
As illustrated in Figure 3, the Vision Path operationalizes the primary visual component \(P_v(I)\) of the agentic objective (Eq 2 ). Rather than a holistic glance, we decouple perception from reasoning through a two-agent architecture: the Diagnostic Agent (\(f_{diag}\)) and the Reasoning Agent (\(f_{reason}\)). The goal of this path is to generate a visual-perspective report that identifies potential traps before they reach the final fusion stage.
3.2.1 Structural ROI Extraction and Semantic Padding↩︎
To capture fine-grained deceptive cues (e.g., tick labels or small legend markers), we implement an automated Region of Interest (ROI) extraction module. Unlike heuristic-based cropping, we utilize a graphic element detection module to precisely localize the bounding boxes of critical chart components, defining the set \(\mathcal{C}_{roi}\) = {title, legend, xaxis, yaxis}.
To ensure semantic completeness, we apply a padding mechanism. For each \(c \in \mathcal{C}_{roi}\), the module calculates the spatial spread \((x\_spread, y\_spread)\) of detected elements. For instance, legend ROIs are expanded to ensure color-coded markers are included, while axis ROIs are padded vertically to encompass the full extent of tick labels. This ensures that the Diagnostic Agent performs literal “reading” rather than visual “estimation” of the scale.
3.2.2 Decoupled Agentic Workflow.↩︎
A core innovation of ChartCynics is the functional isolation designed to eliminate confirmation bias [44].
As shown in Figure 3a, the Diagnostic Agent executes a Blind Test. It is provided only with \(\mathcal{I}\) and the high-resolution ROI crops, while \(\mathcal{Q}\) and \(\mathcal{O}\) are strictly withheld. This forces the agent to act as a neutral auditor: \[\mathcal{R}_{diag} = f_{diag}(\text{Crops}(\mathcal{I}, \mathcal{C}_{roi}), \mathcal{T})\] where \(\mathcal{R}_{diag}\) is the resulting diagnostic Report. This prevents “Self-Prompt Pollution” where the model might otherwise hallucinate visual evidence to support a pre-conceived answer derived from the question [45], [46].
The agent sequentially examines each crop for visual anomalies defined in \(\mathcal{T}\). By amplifying these localized regions, the process uncovers deceptive structural cues that are otherwise masked by the intuitive global visual trend \(\mathcal{V}_{trend}\).
3.2.3 Cognitive Anchoring and Inference↩︎
The Vision Path produces in a structured report \(\mathcal{R}_{diag} = \{\text{Diagnosis}, \text{Action Directive}\}\). The Diagnosis provides factual identification (e.g., “The y-axis starts at 15,000”), while the Action Directive provides prescriptive logic (e.g., “Ignore visual heights; read literal tick values”).
To ensure objective inference, the Reasoning Agent (Figure 3b) follows an unconditional trust policy. It is required to explicitly anchor its Chain-of-Thought (CoT) to \(\mathcal{R}_{diag}\) in its first reasoning step: \[a^* = f_{reason}(\mathcal{Q}, \mathcal{O}, \mathcal{R}_{diag})\]
By forcing the model to reiterate the Action Directive, we shift the reasoning trajectory from an uncalibrated visual prior \(P(a_{trap} \mid \mathcal{V}_{trend})\) to a comprehensive posterior \(P(a^* \mid \mathcal{V}_{trend}, \mathcal{D}_{rel}, \mathcal{T})\) that fuses both modalities. This mechanism effectively transforms the “Trap Answer” into a detectable anomaly, leading to the final correct inference \(a^*\).
3.3 Data Path: OCR-Driven Serialization↩︎
To provide an orthogonal verification layer against visual illusions \(\mathcal{V}_{trend}\), the Data Path (\(\mathcal{P}_d\)) is engineered to reconstruct the underlying numerical relationship. This path bypasses deceptive visual encodings (e.g., manipulated areas, lengths, or angles) by establishing a “literal backbone” through structured text and numeric data extraction.
3.3.1 Structural Data Serialization↩︎
Rather than relying on visual rendering, the first stage directly reveals the graphical attributes of the chart image \(\mathcal{I}\). We utilize an advanced multimodal OCR parsing module to extract all explicit textual and numerical entities, such as axis scales, data labels, and legends, and serialize them into a unified, structured Markdown format \(\mathcal{M}_{ocr}\): \[\mathcal{M}_{ocr} = f_{extract}(\mathcal{I})\]
As this Markdown representation relies strictly on the parsed literal characters rather than their spatial positioning, it effectively recovers the numerical data \(\mathcal{D}\) without being tainted by the deceptive visual manipulation \(f_{deceptive}\) applied to the chart.
3.3.2 Calibration Directives for Downstream Reasoning↩︎
While \(\mathcal{M}_{ocr}\) provides objective numerical evidence, optical extraction can sometimes capture numbers from arbitrary text or hallucinate when labels are absent. Therefore, rather than hard-coding a pre-processing filter, the Data Path establishes a set of Calibration Directives that are passed to the downstream Reasoning Agent to evaluate \(\mathcal{M}_{ocr}\) dynamically during its Chain-of-Thought (CoT).
Dynamic Trust Evaluation: Not all extracted values in \(\mathcal{M}_{ocr}\) hold equal epistemic weight. The framework defines a dual-tier trust principle: (1) High Trust applies to values originating from explicit data labels plotted directly on elements, treating them as immutable ground truth. (2) Low Trust (Skepticism) applies when the chart relies solely on axis ticks without direct labels, or when visual elements exceed the canvas boundary. In such cases, the agent is instructed to treat the OCR data as potentially imprecise and rely more heavily on visual diagnostic deductions.
Temporal and Structural Integrity Check: To counter temporal deceptions like Inappropriate Aggregation or Inappropriate Ordering, the Data Path mandates an integrity audit rule. The agent is explicitly instructed to scrutinize the sequence of categories in \(\mathcal{M}_{ocr}\). For instance, it must check whether the time-series is deliberately reversed, or if the final data point represents a deceptive “incomplete period” masquerading as a full cycle.
3.4 Agentic Fusion with Detective Chain-of-Thought↩︎
The Agentic Fusion module acts as the Summarizer (\(\mathcal{S}\)), tasked with integrating the Diagnostic Report \(\mathcal{R}_{diag}\) from the Vision Path (\(\mathcal{P}_v\)) and the Calibrated Data \(\mathcal{M}_{ocr}\) from the Data Path (\(\mathcal{P}_d\)). As defined in the task formalization, its objective is to maximize the posterior \(P(a \mid \mathcal{P}_v, \mathcal{P}_d, \mathcal{Q}, \mathcal{T})\) by explicitly identifying and neutralizing the manipulation function \(f_{deceptive}\). Before detailing the training process, we first mathematically formalize the ideal reasoning policy that the Summarizer must follow via structured prompting to achieve this objective.
3.4.1 Conflict Arbitration via Evidence Weighting↩︎
To resolve discrepancies between visual heuristics and numerical literals, the Summarizer \(\mathcal{S}\) implements a hierarchical weight system governed by two Golden Rules of Evidence:
Rule I: Heuristic Calibration. Rather than a naive zero-sum arbitration that bluntly discards information from one conflicting path, the Summarizer recontextualizes it using the structural anomaly \(\tau \in \mathcal{R}_{diag}\). Formally, the reasoning prompt acts as a structural constraint, forcing the agent’s generation trajectory to align the visually-derived inference with the data-derived truth. The agent recognizes that the calibrated visual posterior leads to the same optimal decision as the numerical data: \[\arg\max_{a \in \mathcal{O}} P(a \mid \mathcal{V}_{trend}, \tau) = \arg\max_{a \in \mathcal{O}} P(a \mid \mathcal{D}_{rel}) = a^*\] This ensures the model logically translates the visual shape (e.g., recognizing that a physical downward slope \(\mathcal{V}_{trend}\) on an inverted axis \(\tau\) perfectly corroborates a numerical increase derived from \(\mathcal{D}_{rel}\) in Figure 1).
Rule II: Dynamic Trust Calibration. The epistemic weight of \(\mathcal{M}_{ocr}\) is conditioned on the Trust Level flags. If \(\mathcal{M}_{ocr}\) contains High Trust labels, it serves as the immutable ground truth for \(D_{rel}\). Conversely, if labels are absent and \(\mathcal{R}_{diag}\) indicates a scale violation (e.g., “Exceeding the Canvas”), the agent treats \(\mathcal{M}_{ocr}\) as a qualitative indicator and prioritizes the Action Directives for relative reasoning.
3.4.2 The Five-Step Detective Framework (D-CoT)↩︎
The reasoning process is operationalized via a structured Detective Chain-of-Thought (D-CoT), designed to transform the model from a passive describer into a skeptical auditor. This sequence ensures that the final selection \(a^*\) is a logical consequence of debunking the \(f_{deceptive}\) mechanism:
Perception Audit (Prior Calibration): The agent first reiterates the Action Directives from \(\mathcal{R}_{diag}\). This step acts as a cognitive anchor to re-evaluate (rather than strictly suppress) the high-confidence prior \(P(a_{trap} \mid \mathcal{V}_{trend})\) before performing any calculations.
Numerical Anchoring (\(D_{rel}\) Reconstruction): The agent maps categories and values from \(\mathcal{M}_{ocr}\) to the entities in \(\mathcal{Q}\). It identifies whether the data represents the “actual relationship” \(\mathcal{D}_{rel}\) or just a partial subset of the visual representation.
Deception Mapping: Using the Taxonomy \(\mathcal{T}\) as a logic gate, the agent classifies the chart’s specific manipulation subtype. For example, if mapped to Inappropriate Ordering, the agent is triggered to re-sort the \(\mathcal{M}_{ocr}\) sequence before comparison.
Sufficiency & Integrity Check: The agent evaluates if the combined evidence (\(\mathcal{R}_{diag} + \mathcal{M}_{ocr}\)) is sufficient to resolve the contradiction. If the trust levels are conflicting or inadequate, it is permitted to output a “Cannot be Inferred” conclusion to avoid hallucination.
Adversarial Trap Rejection: In the final step, the agent must explicitly prove why the “Trap Answer” \(a_{trap}\) (derived from \(\mathcal{V}_{trend}\)) is a mathematical fallacy. By articulating the discrepancy between the visual illusion and the grounded data, the model ensures \(a^*\) is selected through critical elimination rather than pattern matching.
3.5 Optimization: SFT and GRPO↩︎
To internalize the reasoning heuristics and resolve multimodal conflicts, we implement a two-stage optimization pipeline: Oracle-Informed Distillation followed by Deception-Aware Group Relative Policy Optimization (GRPO).
3.5.1 SFT via Oracle-Informed Reasoning Distillation↩︎
We perform Supervised Fine-Tuning (SFT) using a dataset generated by a larger model (Qwen3-VL-32B). To address the information disparity between the source model (which has access to raw CSV data) and the target model (which only processes visual pixels), we employ a specific prompting strategy. During data generation, the larger model utilizes ground-truth CSV data and trap labels to ensure factual accuracy. However, it is strictly constrained to generate reasoning chains based only on visible chart elements. This process converts inaccessible structured data into visually-grounded logical traces, including \(\langle Visual\_Heuristic \rangle\), \(\langle OCR\_Validation \rangle\), and \(\langle Ambiguity\_Resolution \rangle\). Consequently, the target model learns to derive conclusions from evidence present within the image rather than relying on external metadata.
3.5.2 Deception-Aware GRPO↩︎
To evolve the model beyond simple imitation, we apply Group Relative Policy Optimization (GRPO) [42] to align its reasoning process with adversarial skepticism. Unlike DPO [47], which relies on binary preferences, GRPO allows us to utilize a multi-dimensional continuous reward function \(R_{total}\) to evaluate the quality of the generated reasoning trace within a sampled group \(G\):\[R_{total} = w_1 R_{fact} + w_2 R_{contra} + w_3 R_{logic} + w_4 R_{fmt} + R_{shaping}\]where the components are defined as follows:
Numerical Grounding (\(R_{fact}\)): To ensure robustness against OCR noise, \(R_{fact}\) calculates the Spearman rank correlation between numeric tokens extracted from the \(\langle OCR\_Validation \rangle\) tag and the Oracle CSV. By rewarding the correlation of trends (rank) rather than absolute pixel-to-value accuracy, we encourage the model to prioritize relative data relationships.
Semantic Contradiction (\(R_{contra}\)): To prevent “reward hacking” via keyword stuffing, \(R_{contra}\) combines keyword hit rates (e.g., “non-zero baseline”) with a semantic overlap constraint. The model is rewarded only if its identified contradiction aligns with the expert-annotated explanation, ensuring logical consistency across the 5-step D-CoT trajectory.
Structural Enforcement (\(R_{fmt}\)): A binary gatekeeper reward is applied to penalize “shortcut” generations. If the output fails to strictly follow the four-step XML-tagged structure, it incurs a significant baseline deduction, leveraging the LLM’s auto-regressive nature to anchor the final answer in the preceding reasoning steps.
Asymmetric Shaping (\(R_{shaping}\)): We implement an adversarial penalty to combat visual sycophancy. While a correct answer yields \(+1.0\), selecting the specific misleading distractor (the trap) triggers a heavy penalty of \(-2.0\). This asymmetry forces the policy to prioritize “trap avoidance” as a primary objective, effectively training the model to be inherently skeptical of deceptive visual cues.
By optimizing for the group-relative advantage of these fine-grained rewards, the model learns to navigate the tension between visual intuition and factual evidence without the computational overhead of a dedicated Critic network.
4 Experiments↩︎
This section evaluates the efficacy of ChartCynics in mitigating visual misleading. We detail the experimental configuration, followed by a mechanistic analysis of performance gains, ablation experiments, and the impact of our specialized RL alignment.
| MC (\(N=305\)) | CDCC (\(N=110\)) | |||||
|---|---|---|---|---|---|---|
| 2-4 (lr)5-7 Model & Configuration | Acc \(\uparrow\) | WM \(\downarrow\) | WO \(\downarrow\) | Acc \(\uparrow\) | WM \(\downarrow\) | WO \(\downarrow\) |
| Standard Multimodal Baselines | ||||||
| ChartMoE [48] | 33.11 | 43.28 | 23.61 | 28.18 | 46.36 | 25.45 |
| Gemini-3.1-Pro [49] | 70.49 | 19.34 | 10.16 | 68.18 | 14.54 | 17.27 |
| Structural Mitigation Heuristics | ||||||
| Qwen3-VL-8B + Table-based QA [20] | 49.18 | 29.51 | 21.31 | 36.39 | 20.91 | 42.73 |
| Qwen3-VL-8B + Redrawn [20] | 44.26 | 35.41 | 20.33 | 38.18 | 20.91 | 40.91 |
| Ours: ChartCynics Framework over Base Models | ||||||
| Qwen3-VL-8B (Base Model) [50] | 45.57 | 40.00 | 14.43 | 35.45 | 31.82 | 32.73 |
| + ChartCynics (Train-free) | 60.66 | 24.26 | 15.08 | 47.47 | 16.16 | 36.36 |
| + SFT + GRPO (Full) | 74.43 | 11.15 | 14.42 | 64.55 | 8.18 | 27.27 |
| GPT-o4-mini (Base Model) [51] | 55.08 | 34.43 | 10.49 | 69.09 | 17.27 | 13.64 |
| + ChartCynics (Train-free) | 71.80 | 18.69 | 9.51 | 79.09 | 9.09 | 11.82 |
| Gemini-2.5-Flash (Base Model) [49] | 67.21 | 20.98 | 11.80 | 62.73 | 17.27 | 20.00 |
| + ChartCynics (Train-free) | 71.67 | 13.67 | 14.67 | 75.45 | 7.27 | 17.27 |
4.1 Datasets and Evaluation Benchmarks↩︎
To assess our framework’s robustness and generalization, we evaluate ChartCynics across three distinct benchmarks. Detailed construction protocols, sampling strategies, and statistics for all datasets are presented in the supplementary material.
Misleading ChartQA (MC) [19]: Our primary benchmark for structural visual deception. We utilize the training set (2,619 samples) for our SFT and GRPO optimization, and evaluate on the official test set (305 samples).
Curated Deceptive Chart Collection (CDCC) [20]: To test resilience against real-world epistemic conflicts, we compile a secondary benchmark of 110 expert-validated deceptive visualizations aggregated from established studies in the HCI community [5], [13], [52].
Mixed Standard and Misleading Benchmark (MSMB) [19], [53]: To ensure our “skeptical auditor” does not suffer from over-skepticism on benign data, we construct a balanced evaluation set of 244 charts (122 standard [53], 122 misleading [19]) to verify fundamental chart comprehension remains intact.
4.2 Metrics↩︎
To evaluate both authentic reasoning capabilities and specific vulnerabilities to visual traps, we adopt three core metrics introduced in [19]. Formal definitions and detailed error attribution criteria are available in the supplementary material.
Accuracy (Acc): The standard grounded accuracy indicating the percentage of correctly answered queries.
WM (Wrong due to Misleader): A targeted metric tracking errors strictly caused by cognitive sycophancy, where the model’s prediction falls for the visual trap.
WO (Wrong due to Other factors): A general error metric tracking failures unrelated to visual deception (e.g., basic OCR failures or instruction-following errors).
4.3 Baselines↩︎
To evaluate ChartCynics, we benchmark against three categories of state-of-the-art methods representing the paradigms discussed in our Introduction. First, to represent the visual-first paradigm, we evaluate proprietary VLMs (GPT-o4-mini [51], Gemini-2.5-Flash, and Gemini-3.1-Pro [49]) and our foundational backbone, Qwen3-VL-8B [50], to directly quantify the net gains of our dual-path architecture. Second, we include ChartMoE [48] as a domain-specific baseline to test if standard chart proficiency yields adversarial robustness. Finally, representing the data-first (OCR-enhanced) paradigm, we compare against two inference-time structural mitigations from [20]: Table-based QA and Visualization Redrawing. This contrast effectively highlights the superiority of our synergistic arbitration over rigid, single-path de-rendering techniques.
4.4 Implementation Details↩︎
The training and evaluation of ChartCynics are conducted on a cluster of 4\(\times\) NVIDIA A800 (80GB) GPUs. For dual-path input processing, we utilize nemotron-graphic-elements-v1 [54] for precise Region-of-Interest (ROI) bounding box extraction in the Vision Path. For the Data Path, we employ
LlamaParse [55] (configured with GPT-4o) as the underlying multimodal parsing engine—to robustly convert complex chart images
into structured Markdown. We initialize our framework using the open-weight Qwen3-VL-8B as the foundational reasoning backbone and proceed with a two-stage optimization pipeline.
Stage 1: Oracle-Informed SFT. To bridge the epistemic gap between pixels and underlying data, we perform Supervised Fine-Tuning (SFT) on 5,238 high-quality reasoning chains. These chains are distilled from the 2,619 samples of the Misleading ChartQA training set [19] using a Qwen3-VL-32B teacher.
Stage 2: Deception-Aware GRPO. Following SFT, we utilize GRPO for model alignment, employing a group size of \(G=8\). The multi-objective reward function \(R_{total}\) is defined with the following coefficients: \(w_{fact}=0.20\), \(w_{contra}=0.25\), \(w_{logic}=0.20\), and \(w_{fmt}=0.10\). Furthermore, we implement an asymmetric reward shaping strategy, applying a \(-2.0\) penalty for selecting “misleading trap” distractors and a \(+1.0\) reward for ground-truth alignment.
4.5 Main Results↩︎
To comprehensively evaluate our approach, Table [tab:main95results] presents a detailed performance comparison across the MC and the CDCC. By analyzing the overall Accuracy (Acc) and the error breakdowns, we distill three primary observations:
Superiority of Dual-Path Skepticism (Zero-Shot & Aligned). To demonstrate broad applicability, we evaluate ChartCynics across backbones ranging from open-source to proprietary systems. The training-free framework shows immediate universality, boosting Qwen3-VL-8B by +15.09% (to 60.66%) and GPT-o4-mini by +16.72% (to 71.80%) on MC. Crucially, our two-stage optimization (SFT+GRPO) propels the 8B model to a state-of-the-art 74.43% accuracy. This +28.86% absolute gain enables the smaller model to outperform much larger proprietary systems, notably surpassing Gemini-3.1-Pro (70.49%). This robust alignment further translates to the CDCC, where the optimized 8B model nearly doubles its baseline accuracy (35.45% to 64.55%).
Mechanistic Suppression of Deception. The error breakdowns validate our Complementary Fusion hypothesis. Across all backbones, ChartCynics drastically reduces the WM rate without triggering a compensatory spike in the WO rate. For instance, the training-free GPT-o4-mini drops its WM rate nearly two-fold (from 34.43% to 18.69%) on MC. Most notably, the fully optimized Qwen3-VL-8B crushes its WM rate from 40.00% down to 11.15%. This confirms that the framework successfully anchors its reasoning in structurally-grounded OCR evidence rather than simply guessing blindly to avoid traps.
Outperforming Structural Mitigation Heuristics. We investigate whether visual deception can be mitigated simply by removing the visual modality. As shown in the middle section of Table [tab:main95results], converting deceptive charts into raw tables (Table-based QA) or standardizing them (Redrawn) yields marginal gains or even degradation for Qwen3-VL-8B (+3.61% and -1.31%, respectively). This confirms that simply de-rendering an image into text is insufficient; models require the active, complementary reasoning provided by ChartCynics (74.43%) to navigate complex structural deceptions.
Beyond deceptive scenarios, a critical concern is whether such defensive mechanisms degrade performance on benign data. We evaluate our training-free framework against ChartMoE [48] on the Mixed Benchmark (Table [tab:mixed95benchmark]). While ChartMoE degrades severely on visual deceptions (31.97%), ChartCynics not only doubles this accuracy (68.03%) but also surprisingly outperforms ChartMoE on standard charts (94.26% vs. 88.52%). The dual-agent structural investigation enhances fundamental data extraction across all chart types, proving our framework is a generalized solution rather than a narrow patch.
| Data Subset | ChartMoE | ChartCynics |
| (SOTA) | (Train-free) | |
| Standard (\(N=122\)) | 88.52% (108) | 94.26% (115) |
| Misleading (\(N=122\)) | 31.97% (39) | 68.03% (83) |
| Overall (\(N=244\)) | 60.25% (147) | 81.15% (198) |
4.6 Ablation Study↩︎
To isolate the contributions of our proposed framework, we conduct a comprehensive ablation study using MC (Table 3).
Architectural Necessity of Complementary Fusion. We first evaluate the necessity of merging visual structure with OCR literals. As shown in the upper section of Table 3, removing the visual diagnostic path (OCR Only) yields the highest misled rate (WM: 30.16%), confirming that unstructured OCR data lacks the spatial semantics required for accurate mapping. Conversely, enhancing the model with local crops (VLM + Crop) effectively reduces observational errors (WO drops to 15.41%) but fails to resolve the underlying deception. Only the full ChartCynics architecture successfully leverages multimodal complementarity to minimize both WM and WO simultaneously.
Impact of Optimization Stages. We further validate our two-stage optimization strategy (lower section of Table 3). Applying SFT alone provides a substantial performance leap (Acc: 68.52%), distilling the investigative D-CoT logic into the model. Crucially, the subsequent GRPO alignment propels accuracy further to 74.43% and compresses the WM rate to a mere 11.15%. Notably, while GRPO drastically diminishes susceptibility to visual traps, it maintains a low rate of errors (WO: 14.42%), proving that our reward shaping successfully aligns the model’s cognitive circuits toward skeptical evidence-based reasoning rather than blind guessing.
| Setting | Acc \(\uparrow\) | WM \(\downarrow\) | WO \(\downarrow\) |
|---|---|---|---|
| Architectural Components (Train-free) | |||
| VLM (Standard) | 52.13 | 27.21 | 20.66 |
| VLM + Crop | 57.38 | 27.21 | 15.41 |
| OCR Only (Vision-free) | 46.89 | 30.16 | 22.95 |
| Full ChartCynics (Train-free) | 60.66 | 24.26 | 15.08 |
| Optimization Stages (Full Architecture) | |||
| Full ChartCynics (Train-free) | 60.66 | 24.26 | 15.08 |
| + SFT Only | 68.52 | 15.08 | 16.39 |
| + SFT + GRPO (Full) | 74.43 | 11.15 | 14.42 |
5 Conclusion↩︎
This paper presents ChartCynics, an agentic dual-path framework that fortifies VLMs against misleading visualizations by decoupling intuitive perception from rigorous verification. By synergizing a Diagnostic Vision Path with an OCR-Driven Data Path, our approach systematically unmasks deceptive structures through a skeptical reasoning paradigm. Experimental results show that ChartCynics achieves a state-of-the-art 74.43% accuracy on the Qwen3-VL-8B backbone, outperforming advanced proprietary models. We demonstrate that while SFT via Oracle-Informed Reasoning Distillation is crucial for distilling structured investigative logic, the subsequent Deception-Aware GRPO is essential for penalizing visual sycophancy and enforcing logical consistency. Ultimately, ChartCynics provides a robust foundation for trustworthy multimedia AI, ensuring that automated data interpretation remains grounded in factual reality rather than deceptive visual heuristics.