HiEvi-RAG: Hierarchical Evidence-Driven Reasoning for Long Document Understanding
January 01, 1970
Abstract
Retrieval-Augmented Generation (RAG) streamlines long-document understanding by leveraging retrieval mechanisms to restrict input images to a highly curated subset. However, existing multimodal RAG pipelines primarily face two critical challenges: first, standard semantic similarity retrievers frequently fetch topically overlapping yet answer-void distractor pages that mislead downstream generation; second, rigid single-pass pipelines heavily depend on initial retrieval success, where any omission of core evidence inevitably causes cascading errors. To address these challenges, we introduce HiEvi-RAG, a hierarchical, evidence-driven multimodal RAG framework for closed-domain document understanding. HiEvi-RAGsystematically factorizes complex queries into a cooperative four-stage pipeline: (1) hierarchical question decomposition to break multi-hop root queries into atomic child questions; (2) coarse visual page retrieval leveraging a multimodal retriever to fetch candidate pages based on semantic similarity; (3) fine-grained page verification via EviAgent, a specialized multi-page verifier trained with GRPO to execute cross-page reasoning over multi-image blocks; and (4) memory-guided iterative generation that leverages accumulated sub-question context to execute multi-round, dynamic reasoning over the prioritized sequence. Extensive evaluations across four benchmarks demonstrate the robust efficacy and synergy of our framework, which significantly outperforms existing open-source baselines and exceeds the strongest reported baseline by an average of 8.05% in accuracy.
1 Introduction↩︎
Closed-domain document understanding has become an indispensable core capability for practical AI systems deployed across various sectors, including science, finance, enterprise knowledge management, and technical support [1]–[5]. In these scenarios, users typically pose queries over a proprietary document collection, where the answers are sparsely distributed across hundreds of pages containing diverse and rich information such as plain text, tables, charts, and illustrations [6], [7]. However, due to the prohibitive number of images in these closed-domain documents, feeding the entire collection into a model in an end-to-end manner easily exceeds the context window limits of existing LLMs.
To address this bottleneck, a growing body of work on Retrieval-Augmented Generation (RAG) has emerged, which leverages retrieval mechanisms to restrict the number of input images to a highly curated subset prior to answer generation [2], [8]–[10]. Initially, to bypass traditional OCR limitations and capture multimodal information, ColPali [11] extends late interaction to rendered page images, while M3DocRAG [9] establishes this method as the core interface for multi-page QA. Subsequently, research shifted toward filtering distractors and optimizing ranking precision: MDocAgent [12] and SimpleDoc [13] refine results via module collaboration or iterative cascading; RagVL [14] and MM-R5 [15] employ instruction-tuning or reinforcement learning to train strong VLM rerankers; and MoLoRAG [16] leverages page topology graphs for logic-aware traversal. Concurrently, DocAgent [17] and ALDEN [18] utilize memory feedback or active exploration protocols to gather evidence dynamically, while DMAP [19] constructs human-aligned structural maps for global document comprehension. Despite these advancements, two challenges remain: first, relying on semantic similarity misleads models with topically related yet answer-void pages; second, single-pass pipelines rigidly depend on initial retrieval, where any omission of core evidence makes correct generation impossible.
To address these challenges, we propose HiEvi-RAG, a four-step framework for long-document understanding. First, hierarchical question decomposition breaks down the complex, multi-hop root question into multiple atomic sub-questions, reducing retrieval difficulty while assisting the root question’s resolution from different logical levels. Second, coarse-grained page retrieval utilizes a multimodal retriever (e.g., ops-ColQwen3) to fetch the top-\(k\) pages based on semantic similarity for each question, including both root and sub-questions. Third, for fine-grained evidence verification, inspired by DocR1 [20], we employ GRPO [21] to train EviAgent, an evidence-aware multi-page reasoning model capable of cross-page reasoning and grounding over multi-image inputs to pinpoint question-relevant evidence. In this stage, for sub-questions, we strictly retain only the evidence pages identified by EviAgent; for the root question, EviAgent’s verified evidence pages are prioritized at the front, while non-evidence pages follow in order of their initial semantic similarity, thereby achieving precise reranking. Fourth, during memory-guided iterative generation, an empty memory is initialized. The sub-questions and their corresponding evidence pages are fed into the model, with the sub-questions, generated reasoning traces, and intermediate answers subsequently accumulated in the memory. For the root question, the model takes the root query, the accumulated memory, and a batch of images as input. If the current memory and image batch are insufficient to resolve the question, the system appends the latest reasoning trace to the memory and fetches the next batch of images for iterative reasoning.
To verify the effectiveness of our approach, we conduct extensive experiments and analyses across four distinct datasets. The results demonstrate that our method significantly outperforms existing open-source baselines, exceeding the previous state-of-the-art by an average of 8.05%. Furthermore, we perform comprehensive ablation studies to validate the individual efficacy of each proposed module.
Our contributions can be summarized as follows:
We propose HiEvi-RAG, a four-step multimodal RAG framework for long-document QA that unifies hierarchical question decomposition, coarse-to-fine retrieval, and memory-guided iterative generation.
We introduce a fine-grained verification mechanism via EviAgent, which performs cross-page reasoning over multi-image inputs to precisely ground answer-supporting evidence.
We formulate a memory-guided iterative generation process that utilizes sub-question reasoning traces to assist the root query while dynamically updating an explicit memory across multiple rounds to eliminate cascading retrieval errors.
2 Related Work↩︎
2.1 Document Visual Question Answering↩︎
Advancements in DocVQA have bifurcated into OCR-based and OCR-free paradigms. OCR-based methods convert pages into structured textual and spatial tokens to feed downstream reasoning frameworks [22]–[24], yet their performance remains inherently bounded by compounding parsing errors and the serialization loss of visual context. Conversely, OCR-free approaches directly train VLMs on raw document images, evolving from architectural scaling for resolution management [25]–[27] to evidence-aware reasoning via reinforcement learning [20], [28]. While these methods substantially enhance single- or multi-page reader execution, effectively navigating and distilling massive long-document repositories remains a largely orthogonal challenge.
2.2 Retrieval-Augmented Generation↩︎
Multimodal RAG methodologies have transitioned from coarse-grained text-retrieval extensions to visual-centric, layout-aware document discovery. Early pipelines leverage visual retrievers and VLM-based rerankers to screen candidate pages independently based on superficial similarity scoring [9], [11], [14], [15]. More recent efforts incorporate iterative multi-agent navigation, memory feedback, or topological page graphs to capture structural inter-page logic [16]–[19]. Despite these scaling improvements, existing pipelines still evaluate retrieved pages as isolated query-image pairs, leaving them highly susceptible to topically overlapping but answer-void distractors. To bridge these gaps, HiEvi-RAGcasts validation as a joint, cross-page verification problem and bypasses cascading failure modes via memory-guided iterative reasoning. Detailed related works are deferred to Appendix 6.
3 Method↩︎
3.1 Overview and Task Formulation↩︎
Let a document be represented as an ordered sequence of page images \(D=\{p_1, p_2, \ldots, p_N\}\) and \(q\) denote a user question in a closed-domain setting. The final answer is assumed to be grounded in \(D\), although the supporting evidence may be sparsely distributed across multiple non-contiguous pages. When the document length \(N\) scales to hundreds of pages, direct end-to-end VLM ingestion becomes computationally prohibitive due to context window constraints. To bypass this bottleneck, HiEvi-RAGdecomposes long-document QA into four cooperative stages: (1) hierarchical question decomposition, (2) coarse visual page retrieval, (3) evidence-aware page verification, and (4) memory-guided iterative generation. A core design rationale of our framework is the multi-granularity operation across stages: coarse retrieval is strictly recall-oriented, EviAgentconducts fine-grained verification over grouped page candidates via cross-page reasoning, and the final generator iteratively consumes the verified evidence sequence backed by an explicit memory. The overall pipeline is illustrated in Figure 1.
3.2 Hierarchical Question Decomposition↩︎
Given a complex root question \(q\), this stage constructs a shallow reasoning hierarchy, as illustrated in Figure 1 (1). Specifically, a lightweight LLM extracts key entities, structural attributes, constraints, and logical relations from \(q\) to instantiate a set of atomic child questions: \[\mathcal{Q}(q)=\{q_1, q_2, \ldots, q_m\}, \qquad m \leq 5.\] Each child question \(q_i\) is formulated as a distinct natural query covering a non-redundant subset of the global information need, while strictly avoiding near-paraphrases of the full prompt.
Crucially, partitioning the global query into localized targets reduces the inherent difficulty of multi-hop reasoning. This upfront decomposition prevents retrieval omissions typical of standalone root queries, ensuring core evidence is captured before downstream generation. The corresponding prompt template and example are detailed in Appendix 7.1 and 10.
3.3 Coarse Visual Page Retrieval↩︎
Following decomposition, we employ Ops-ColQwen3-4B [29], a ColPali-style late-interaction multimodal retriever [11], [30], to construct a candidate page pool from rendered
document images. As illustrated in Figure 1 (2), this retrieval stage is split into offline index construction and online query matching.
3.3.0.1 Offline Index Construction.
Each document page image \(p_i \in D\) is encoded offline into a multi-vector representation to preserve fine-grained visual features: \[\mathbf{v}_{p_i} = \mathcal{E}_{\text{page}}(p_i), \qquad i=1,2,\ldots,N.\] The derived page embeddings are subsequently stored to build the global document index.
3.3.0.2 Online Query Matching.
Upon receiving a root question \(q\), the stage online encodes both \(q\) and its derived atomic child questions \(q_i \in \mathcal{Q}(q)\) to obtain their respective text embeddings: \[\mathbf{v}_q = \mathcal{E}_{\text{query}}(q), \quad \mathbf{v}_{q_i} = \mathcal{E}_{\text{query}}(q_i).\] These query embeddings are then matched against the offline page index using the late-interaction operator to compute similarity scores across all document pages. Based on these scores, we rank and retain the Top-\(K\) (\(K=20\)) candidate pages for the root question and each child question, denoted as \(T_q\) and \(T_{q_i}\) respectively.
3.4 Evidence-Aware Page Verification↩︎
To filter out topically related but answer-void distractors from the coarse retrieval pool, we introduce EviAgent, a specialized model trained to perform binary evidence page verification via cross-page reasoning. As illustrated in Figure 1 (3), this stage takes the coarse candidate lists \(T_q\) and \(T_{q_i}\) as input, partitions them into grouped-\(k\) sliding windows, and leverages the trained agent to output structured decisions.
3.4.0.1 Training EviAgentwith Evidence-Aware GRPO.
Inspired by DocR1 [20], we train EviAgentusing an Evidence-Aware Group Relative Policy Optimization (EviGRPO) pipeline to specialize a VLM for multi-page document verification. The policy is optimized across six open-source multi-page datasets [31]–[36] with statistics detailed in Appendix 9.
During the training phase, the VLM is constrained to generate a structured content containing three mandatory fields: \(o = (\tau, \mathbf{y}, a)\). The corresponding prompt template is detailed in Appendix 7.2.
Reasoning Trace (\(\tau\)): Contained within the
<think>tag, \(\tau\) encapsulates intra-page visual inspection and cross-page reasoning.Evidence Decisions (\(\mathbf{y} \in \{T, F\}^k\)): Enclosed within the
<evidence_page>tag, \(\mathbf{y}\) enforces a strict one-to-one binary sequence matching the exact order of input images to prevent evaluation omissions.Final Answer (\(a\)): Wrapped within the
<answer>tag, \(a\) delivers the terminal text response derived from the verified evidence.
The policy is optimized via a weighted joint reward function: \[R = \lambda_{\text{fmt}} r_{\text{fmt}} + \lambda_{\text{evi}} r_{\text{evi}} + \lambda_{\text{acc}} r_{\text{acc}},\] where \(r_{\text{fmt}}\) enforces schema adherence, and \(r_{\text{acc}}\) evaluates final answer correctness. The evidence reward \(r_{\text{evi}}\) computes the \(F_1\) score against ground-truth page labels to accurately measure verification performance and prevent reward hacking.
3.4.0.2 Fine-Grained Page Validation.
During online inference, each retrieved candidate list is split into \(L = \lceil K/k \rceil\) blocks of size \(k\): \(b_t^k = \{p_{(t-1)k+1}, \ldots, p_{\min(tk,K)}\}\), where \(t \in \{1, \ldots, L\}\) is the block index. Each block is verified jointly by the trained EviAgent. To prevent information fragmentation across boundaries, all pages verified as true evidence are aggregated into a final consolidated block for a global cross-block synthesis. The refined signals are then routed to two distinct downstream policies:
Discarding for Child Questions: For each child question \(q_i\), we extract only the verified evidence pages: \[E(q_i) = \{p \in T_{q_i} : \mathbf{y}_p = T\}.\] Non-evidence pages are discarded to restrict downstream generation strictly to localized, verified facts.
Reranking for the Root Question: For the root question \(q\), let \(E(q)\) denote the verified evidence page set. We construct a prioritized sequence \(\Pi(q)\) while preserving the original retriever ranking order within each partition: \[\Pi(q) = E(q) \oplus (T_q \setminus E(q)),\] where \(\oplus\) denotes sequence concatenation. Both \(E(q)\) and the remaining subset \(T_q \setminus E(q)\) strictly inherit the original ranking order determined by the coarse visual retriever. Non-evidence pages are retained as a backup suffix to secure recall for downstream iterative reasoning.
3.5 Memory-Guided Iterative Generation↩︎
This final stage leverages the resolved child questions and the prioritized sequence \(\Pi(q)\) to generate the terminal answer via a two-step, memory-enhanced pipeline. As illustrated in Figure 1 (4), the process consists of initializing an explicit memory layer with sub-question contexts and iteratively reasoning over the root question.
3.5.0.1 Initializing Memory with Sub-Questions.
Before addressing the root query, the framework sequentially resolves each child question \(q_i \in \mathcal{Q}(q)\) over its verified evidence block \(E(q_i)\) using EviAgent: \[(\tau_i, \mathbf{y}_i, a_i) = \mathcal{A}\big(q_i, E(q_i)\big),\] where \(\mathcal{A}\) denotes the generative policy of EviAgent. Initializing the memory as \(M_0 = \emptyset\), the state updates cumulatively by appending the complete triplets: \[M_i = M_{i-1} \cup \{(q_i, \tau_i, a_i)\}.\] This mechanism distills sub-question verification into informative textual facts, seeding the working memory with both mid-level execution traces and terminal local answers.
3.5.0.2 Iterative Reasoning for the Root Question.
Upon consolidating the sub-question memory \(M_m\) (where \(m = |\mathcal{Q}(q)|\)), HiEvi-RAGevaluates the root question \(q\) by scanning the prioritized sequence \(\Pi(q)\) via a sliding window of size \(k\). At execution round \(t\), EviAgentingests the root query, the persistent memory state, and the \(t\)-th evidence block \(b_t^k \subset \Pi(q)\): \[(\tau_t,
\mathbf{y}_t, a_t) = \mathcal{A}\big(q, b_t^k, M_{m+t-1}\big).\] The execution terminates immediately, returning \(a_t\) as the global response if \(a_t \neq \alpha_{\emptyset}\),
where \(\alpha_{\emptyset}\) denotes the designated abstention token (i.e., NOT_ANSWERABLE). Otherwise, the current reasoning trace \(\tau_t\) is appended to the memory to guide
the next window: \[M_{m+t} = M_{m+t-1} \cup \{(q, \tau_t)\}.\] This iterative accumulation continues until a valid answer is produced or \(\Pi(q)\) is exhausted. By maintaining text-based
memory across rounds, this recurrent design forces the model to synthesize historical context globally, preventing critical information omission inherent in local window constraints.
4 Experiments↩︎
| Benchmark | # QA Samples | Min. Pages | Max. Pages | Avg. Pages |
|---|---|---|---|---|
| PaperTab | 393 | 2 | 152 | 10.7 |
| FetaTab | 1016 | 2 | 216 | 16.3 |
| MMLongBench | 1,082 | 9 | 468 | 47.8 |
| LongDocURL | 2,325 | 51 | 149 | 89.0 |
| Method | Param. | PaperTab | FetaTab | MMLongBench | LongDocURL |
| (Acc.) | (Acc.) | (Acc.) | (Acc.) | ||
| M3DocRAG [9] | 7B | 28.5 | 63.8 | 36.2 | 49.0 |
| MDocAgent [12] | 7B | 30.0 | 66.3 | 38.5 | 46.9 |
| MoLoRAG+ [16] | 7B | 31.0 | 69.2 | 41.0 | 51.9 |
| ALDEN [18] | 7B | 24.5 | 62.3 | 39.2 | 55.1 |
| URaG [37] | 7B | – | – | 33.8 | 52.2 |
| Doc-\(V^\star\) [28] | 7B | – | – | 42.1 | 56.3 |
| 8B | 44.0 | 72.9 | 48.2 | 65.7 |
| Variant | PaperTab | FetaTab | MMLongBench | LongDocURL | Avg. |
|---|---|---|---|---|---|
| 44.0 | 72.9 | 48.2 | 65.7 | 57.7 | |
| w/o Evidence-Aware Verification | 39.7 (-4.3) | 69.6 (-3.3) | 44.5 (-3.7) | 59.1 (-6.6) | 53.2 (-4.5) |
| w/o Hierarchical Decomposition | 41.3 (-2.7) | 70.5 (-2.4) | 45.8 (-2.4) | 62.4 (-3.3) | 55.0 (-2.7) |
| w/o Iterative Reasoning | 41.9 (-2.1) | 72.0 (-0.9) | 46.1 (-2.1) | 63.2 (-2.5) | 55.8 (-1.9) |
4.1 Experimental Setup↩︎
4.1.0.1 Benchmarks.
We evaluate HiEvi-RAGon four benchmarks: PaperTab and FetaTab [1], which target question answering over scientific and Wikipedia-style tables; and MMLongBench [6] and LongDocURL [3], which feature extended multimodal documents stressing cross-page retrieval, visual grounding, and multi-hop reasoning. Structural statistics are provided in Table 1.
4.1.0.2 Implementation Details.
For hierarchical question decomposition, a lightweight Qwen3.5-4B model is deployed by default to generate at most five child questions. The baseline coarse retriever is configured as ops-ColQwen3-4B [29]. Both root and child queries retrieve the Top-\(K\) (\(K=20\)) candidate pages, which are
subsequently verified using a grouped window size of \(k=7\). We instantiate EviAgentusing Qwen3-VL-8B-Instruct [38] as the foundational backbone. For the joint reward function, the balancing scaling weights are explicitly set as \((\lambda_{\text{fmt}}, \lambda_{\text{evi}},
\lambda_{\text{acc}}) = (0.1, 0.5, 0.4)\), prioritizing the evidence verification performance. Training is executed on an \(8\times\) NVIDIA A100 GPU cluster.
4.1.0.3 Metrics.
For MMLongBench and LongDocURL, we follow their original evaluation protocols, employing standard accuracy and rule-based string matching tailored to various answer types. For PaperTab and FetaTab, we adopt an LLM-as-a-judge paradigm leveraging Qwen3.5-27B as the evaluator to compute binary accuracy (0 or 1) by assessing whether the generated response semantically matches the ground truth. Additionally, retrieval-related performance is quantified via Recall@K, NDCG@K, and MRR@K, with detailed mathematical formulations deferred to Appendix 8.
| Method | Generator | Retriever | PaperTab | FetaTab | MMLongBench | LongDocURL |
| (Acc.) | (Acc.) | (Acc.) | (Acc.) | |||
| MoLoRAG+ [16] | Qwen2.5-VL-7B-Instruct | ColQwen2.5 | 31.0 | 69.2 | 41.0 | 51.9 |
| Qwen2.5-VL-7B-Instruct | ColQwen2.5 | 34.2 | 69.1 | 42.6 | 55.4 | |
| Qwen2.5-VL-7B-Instruct | Ops-ColQwen3-4B | 38.3 | 71.4 | 45.0 | 60.2 | |
| Qwen3-VL-8B-Instruct | ColQwen2.5 | 36.0 | 70.8 | 43.8 | 57.1 | |
| Qwen3-VL-8B-Instruct | Ops-ColQwen3-4B | 44.0 | 72.9 | 48.2 | 65.7 |
4.2 Main Results↩︎
Table 2 presents a performance comparison across the four benchmarks. HiEvi-RAGconsistently outperforms all competitive baselines, validating the synergistic effects of its core architectural components.
On PaperTab and FetaTab, HiEvi-RAGachieves 44.0% and 72.9% accuracy, yielding absolute improvements of +13.0% and +3.7% over the strongest baseline (MoLoRAG+), respectively. Tabular documents typically require exact cell alignment; standard retrievers frequently introduce false positives due to repetitive schema structures. HiEvi-RAGmitigates this bottleneck by executing fine-grained verification via EviAgent, effectively isolating answer-void distractors.
Similarly, on MMLongBench and LongDocURL, HiEvi-RAGsecures 48.2% and 65.7% accuracy, outperforming Doc-\(V^\star\) by +6.1% and +9.4%, respectively. These tasks necessitate cross-page retrieval and multi-hop reasoning, where single-pass pipelines easily omit critical information. The consistent performance gains demonstrate the efficacy of our memory-guided iterative generation.
4.3 Ablation Studies↩︎
4.3.0.1 Contribution of Individual Modules.
To isolate the empirical impact of each architectural phase in HiEvi-RAG, we evaluate three ablation variants across all benchmarks (Table 3). Disabling any single stage yields a consistent performance degradation, validating the synergy of our core components.
Specifically, w/o Evidence-Aware Verification drops the average score by -4.5% absolutely. Without Step 3 verification, answer-void distractors occupy the front of the sequence, misleading the generator into premature early stopping and missing genuine evidence. Furthermore, initializing memory over unverified Top-\(K\) inputs introduces catastrophic cascading noise. Meanwhile, w/o Hierarchical Decomposition compromises the average score by -2.7%. This demonstrates that executing complex queries as standalone prompts without HiEvi-RAG’s Step 1 induces severe retrieval omissions, whereas atomic decomposition guarantees the upfront recall of dispersed evidence. Lastly, w/o Iterative Reasoning consistently degrades results by -1.9% on average. This variant deactivates the sliding window in HiEvi-RAG’s Step 4 and force-feeds the Top-5 pages in a single pass. The result proves that while verification serves as an effective filter, iterative reasoning acts as an indispensable safety net against information omission under local window constraints.
4.3.0.2 Backbone and Retriever Sensitivity.
To evaluate the robustness of HiEvi-RAGunder varying underlying architectures, we perform a grid sensitivity analysis by cross-combining different generative backbones and visual retrievers (Table 4).
Specifically, when deploying the identical backbone and retriever configuration as MoLoRAG+ (Qwen2.5-VL-7B-Instruct + ColQwen2.5), HiEvi-RAGoutperforms this baseline in almost all benchmarks. It
delivers clear absolute improvements of +3.2% on PaperTab and +3.5% on LongDocURL, demonstrating that our primary advantages stem strictly from architectural innovations rather than the scaling of foundational models. Furthermore, scaling up either the
retriever to Ops-ColQwen3-4B or the generator to Qwen3-VL-8B-Instruct provides independent performance increments across benchmarks. The combination of both upgraded components yields the best performance, proving that our
framework effectively leverages stronger retrieval and generation capabilities to maximize final accuracy.
| Top-\(K\) | Method | MMLongBench | LongDocURL | ||||
|---|---|---|---|---|---|---|---|
| 3-5 (lr)6-8 | Recall | NDCG | MRR | Recall | NDCG | MRR | |
| 1 | Ops-ColQwen3-4B (baseline) | 54.2 | 70.4 | 70.4 | 53.5 | 73.4 | 73.4 |
| + Qwen3-VL-8B-Instruct | 54.2 \((+0.0)\) | 70.6 \((+0.2)\) | 70.6 \((+0.2)\) | 52.9 \((-0.6)\) | 72.1 \((-1.3)\) | 72.1 \((-1.3)\) | |
| + | 59.4 \((+5.2)\) | 78.3 \((+7.9)\) | 78.3 \((+7.9)\) | 57.3 \((+3.8)\) | 78.8 \((+5.4)\) | 78.8 \((+5.4)\) | |
| 3 | Ops-ColQwen3-4B (baseline) | 75.2 | 73.8 | 77.2 | 74.8 | 73.7 | 81.1 |
| + Qwen3-VL-8B-Instruct | 75.6 \((+0.4)\) | 74.4 \((+0.6)\) | 77.6 \((+0.4)\) | 73.7 \((-1.1)\) | 72.5 \((-1.2)\) | 79.8 \((-1.3)\) | |
| + | 80.1 \((+4.9)\) | 80.6 \((+6.8)\) | 83.7 \((+6.5)\) | 77.5 \((+2.7)\) | 77.3 \((+3.6)\) | 84.9 \((+3.8)\) | |
| 5 | Ops-ColQwen3-4B (baseline) | 81.4 | 76.2 | 78.2 | 81.0 | 76.4 | 82.0 |
| + Qwen3-VL-8B-Instruct | 81.8 \((+0.4)\) | 76.6 \((+0.4)\) | 78.5 \((+0.3)\) | 80.2 \((-0.8)\) | 75.4 \((-1.0)\) | 80.7 \((-1.3)\) | |
| + | 85.2 \((+3.8)\) | 82.2 \((+6.0)\) | 84.4 \((+6.2)\) | 83.2 \((+2.2)\) | 79.7 \((+3.3)\) | 85.6 \((+3.6)\) | |
4.3.0.3 Effectiveness of Fine-Grained Page Validation.
To evaluate the impact of the verification stage on retrieval quality, we compare the retrieval metrics of the baseline against the reranked outputs generated by our verifier both before and after training under various Top-\(K\) settings on MMLongBench and LongDocURL (Table 5).
Specifically, deploying an untuned Qwen3-VL-8B-Instruct yields marginal or even detrimental performance shifts. While it provides negligible fluctuations on MMLongBench (e.g., a modest \(+0.2\) point increase in NDCG@1 at Top-1), it consistently degrades retrieval precision on LongDocURL, with NDCG slipping by \(-1.3\) points at Top-1 and \(-1.0\) points at Top-5. This demonstrates that off-the-shelf VLMs lack the necessary alignment to distinguish authentic evidence from complex multimodal distractors. In sharp contrast, our trained EviAgent consistently and substantially outperforms the baseline. It elevates the NDCG@1 from 70.4% to 78.3% on MMLongBench, and from 73.4% to 78.8% on LongDocURL. Furthermore, at Top-5, EviAgentpushes the final Recall to 85.2% and 83.2% on the respective benchmarks. These comprehensive gains across Recall, NDCG, and MRR solidly confirm EviAgent’s superior capacity in identifying authentic evidence and filtering out answer-void noise.
4.3.0.4 Sensitivity to Block Size.
To evaluate the impact of the block size \(k\) defined in HiEvi-RAG’s Step 3, we analyze the retrieval quality under varying scales across MMLongBench (Figure 2). The empirical results indicate that both excessively large and excessively small values of \(k\) degrade retrieval quality.
Specifically, a moderate block size (e.g., \(k=7\)) achieves the optimal balance and yields the highest retrieval metrics. Setting \(k\) too small isolates candidate pages. This constrains the model’s capacity for cross-page reasoning and comparison. Conversely, expanding \(k\) excessively degrades performance. This drop is driven by visual context overload, where the excessive input scale dilutes model attention. Therefore, maintaining a tightly bounded block size \(k\) is essential to maximize page validation accuracy.
5 Conclusion↩︎
In this work, we presented HiEvi-RAG, a hierarchical, evidence-driven multimodal RAG framework for closed-domain long-document understanding, whose core innovation lies in its cooperative four-stage pipeline. Specifically, the framework first employs hierarchical question decomposition to partition complex multi-hop queries into atomic child questions, drastically reducing initial retrieval difficulty. Second, coarse visual retrieval leverages rendered page indexing to guarantee upfront evidence recall. Third, fine-grained page verification utilizes the EviAgentto perform cross-page reasoning over multi-image blocks, precisely suppressing topically related yet answer-void distractors. Fourth, memory-guided iterative generation accumulates sub-question traces into an explicit text-based memory, dynamically executing sliding-window reasoning to eliminate cascading information omissions. Extensive experiments across four benchmarks demonstrate that HiEvi-RAGsignificantly outperforms existing open-source baselines. Furthermore, comprehensive ablation studies analyses solidly validate the individual efficacy and systemic synergy of each cooperative phase within our pipeline.
Limitations↩︎
Despite the state-of-the-art performance achieved by HiEvi-RAG, several inherent limitations warrant further investigation.
First, our framework introduces non-trivial computational overhead during the inference stage. Specifically, the hierarchical question decomposition, the multi-block page verification via the EviAgent, and the subsequent memory-guided generation inherently mandate multiple sequential LLM/VLM invocations. This compound multi-stage architecture inevitably increases cumulative response latency and inference costs compared to standard single-pass pipelines.
Second, the memory-guided generation mechanism relies heavily on the quality of textual intermediate reasoning traces; if the initial sub-questions yield severely hallucinated or biased summaries, such errors may cascade into the persistent memory layer and occasionally misguide the terminal root-question synthesis.
Third, our framework was optimized and evaluated predominantly on born-digital electronic documents with pristine rendering. Its zero-shot generalization capabilities across real-world handwritten inputs, heavily degraded physical scans with text artifacts, or scene-text images captured in natural environments require broader empirical validation in future work.
Ethical Considerations↩︎
HiEvi-RAGis designed for closed-domain long-document understanding and may be applied to documents containing sensitive or proprietary information. All experiments in this paper are conducted on publicly available benchmarks and open-source datasets. We do not collect new user data or perform human-subject annotation. Nevertheless, deployment in real-world enterprise or financial scenarios should ensure proper access control, data privacy protection, and auditing of generated answers. Because the system may still produce incorrect or unsupported answers when retrieval or evidence verification fails, we recommend using the framework as an assistive tool rather than an autonomous decision-making system in high-stakes domains.
6 Detailed Related Work↩︎
6.1 Document Visual Question Answering↩︎
Recent advancements in DocVQA primarily diverge into OCR-based and OCR-free paradigms [23], [39]–[43]. OCR-based methods serialize document pages into structured textual representations to serve as inputs for downstream reasoning. Representative architectures leverage multimodal pretraining or specialized attention mechanisms to fuse textual, visual, and layout features, as exemplified by UDOP [22], DocFormerv2 [44], and DocLLM [23]. To enhance efficiency and structured generation, LayoutLLM [45] frames layout-sensitive reasoning as an instruction-tuning problem, whereas LayTokenLLM [24] employs compact layout tokens to represent spatial bounding boxes without expanding lengthy coordinate sequences. For structured data, TaPERA [46] incorporates content planning and execution-based reasoning to improve faithfulness in long-form table QA. Although highly effective when structural parsing is reliable, the performance of these methods remains intrinsically upper-bounded by cascading parsing errors and the inevitable loss of visually-situated evidence during text serialization.
OCR-free methods bypass external parsing pipelines by training VLMs to comprehend document images end-to-end. Early foundational architectures prioritized enhancing reading capabilities under constrained token budgets; within this cohort, UReader [25] introduces auxiliary reading and key-point generation tasks, KOSMOS-2.5 [47] pretrains on text-intensive images for structured text generation, DocPedia [26] processes visual inputs in the frequency domain, TextMonkey [48] incorporates high-resolution modeling with token filtering, and the mPLUG-DocOwl series [27], [49], [50] employs feature compression to scale from single-page to multi-page document understanding. More recently, frameworks have evolved toward specialized reasoning and optimization protocols. Specifically, GOT [51] frames vision-based reading as a unified OCR-free generator, whereas DLaVA [52] and Marten [53] implement visual answer localization and mask prediction to enhance model interpretability. To boost efficiency, DocVLM [54] uses compact textual queries to lower visual token costs, while MACT [55] orchestrates collaborative multi-agent reasoning at test time. To shift from passive reading to active policy optimization, DocR1 [20] and Doc-\(V^\star\) [28] leverage evidence-guided GRPO to explicitly train visual reasoning trajectories for multi-page environments.
6.2 Retrieval-Augmented Generation↩︎
RAG methodologies have evolved from traditional text passage retrieval to multi-modal and structure-aware document discovery. Following the text-centric foundations laid by passage retrieval [8] and token-level late interaction [30], visual RAG systems have increasingly focused on page-level document interfaces. ColPali [11] extends late interaction to rendered page images, while M3DocRAG [9] establishes this approach as a core interface for multi-page QA. To decouple retrieval from reading, MDocAgent [12] and SimpleDoc [13] utilize multi-module collaboration and iterative cascading, while RagVL [14] and MM-R5 [15] employ instruction-tuning or reinforcement learning to train specialized visual rerankers. Concurrently, efforts have been dedicated to structured or active navigation to handle complex layouts: MoLoRAG [16] leverages page topology graphs for logic-aware traversal, DocAgent [17] and ALDEN [18] introduce memory feedback and active exploration protocols, and DMAP [19] constructs human-aligned structural maps to guide global understanding.
7 Prompt Templates↩︎
7.1 Prompt For Hierarchical Question Decomposition↩︎
The system prompt for hierarchical question decomposition is detailed in Figure 3. Given the root question as input, the model is strictly constrained to output a maximum of five atomic child questions, each formatted strictly as an interrogative sentence.
7.2 Prompt For EviAgent↩︎
The system prompt used for both training and inference of EviAgentis provided in Figure 4. Given a question and a set of input pages, EviAgentis required to produce structured reasoning, make a binary evidence decision for each page, and generate the final answer.
8 Retrieval Metrics↩︎
Let \(G_q\) denote the gold evidence-page set for question \(q\), and let \(R_q^{K}=(r_1,\ldots,r_K)\) denote the ranked top-\(K\) retrieved pages. We report three standard retrieval metrics.
8.0.0.1 Recall@K.
Recall@K measures the fraction of gold evidence pages recovered in the top-\(K\) results: \[\mathrm{Recall@K}(q) = \frac{|G_q \cap R_q^{K}|}{|G_q|}.\] Higher Recall@K indicates better evidence coverage.
8.0.0.2 NDCG@K.
Normalized Discounted Cumulative Gain emphasizes retrieving gold pages early in the ranked list. With binary relevance labels \(\mathrm{rel}_i=\mathbb{I}(r_i \in G_q)\), we compute \[\mathrm{DCG@K}(q)=\sum_{i=1}^{K}\frac{2^{\mathrm{rel}_i}-1}{\log_2(i+1)},\] and normalize it by the ideal ranking: \[\mathrm{NDCG@K}(q)=\frac{\mathrm{DCG@K}(q)}{\mathrm{IDCG@K}(q)}.\] This metric rewards both correctness and ranking quality.
8.0.0.3 MRR@K.
Mean Reciprocal Rank focuses on how early the first relevant page appears: \[\mathrm{RR@K}(q)= \begin{cases} \frac{1}{\min \{i \mid r_i \in G_q\}}, & \text{if } G_q \cap R_q^K \neq \emptyset, \\ 0, & \text{otherwise.} \end{cases}\] MRR@K is the average reciprocal rank over all questions.
9 Training Datasets↩︎
Table 6 details the statistics of the multi-page training data utilized for EviAgent. To ensure a balanced and comprehensive optimization, we uniformly sample 1,000 instances from each of the six constitutive datasets, with all selected samples inherently featuring multi-page visual contexts:
DUDE [31] and MP-DocVQA [32] supply rich structural information from real-world complex electronic layouts and scanned industry documents.
MultiHiertt [33] and pdfQA [34] inject demanding financial reports and scientific literature requiring cross-page reasoning over hierarchical tables and textual narratives.
SciEGQA [35] provides dense academic papers with tightly coupled illustrations, charts, and mathematical proofs.
SlideVQA [36] introduces sequential presentation slides characterized by sparse, highly stylized cross-page visual elements.
| Input Page | Evidence Page | ||||||
|---|---|---|---|---|---|---|---|
| 3-5 (lr)6-8 Dataset | # Samples | Min | Max | Avg | Min | Max | Avg |
| DUDE | 1000 | 2 | 20 | 6.93 | 1 | 8 | 1.16 |
| MP-DocVQA | 1000 | 2 | 20 | 6.94 | 1 | 1 | 1.00 |
| MultiHiertt | 1000 | 3 | 7 | 4.06 | 1 | 3 | 1.48 |
| pdfQA | 1000 | 4 | 10 | 9.95 | 1 | 10 | 1.42 |
| SciEGQA | 1000 | 2 | 20 | 11.42 | 1 | 2 | 1.50 |
| SlideVQA | 1000 | 20 | 20 | 20.00 | 1 | 3 | 1.40 |
10 Case Study↩︎
Figure 5 illustrates a concrete execution trace of our decomposition layer. The complex, multi-hop root query requires concurrent visual counting and cross-year mathematical synthesis. Directly retrieving pages using this intricate query typically misleads standard semantic retrievers. By partitioning the root query into three localized, atomic child questions, our framework effectively simplifies the retrieval objectives, ensuring all disjoint tabular and visual evidence pages are fully recalled without cascading omissions.
References↩︎
Corresponding author.↩︎