-2.0ex 1.5ex Introduction

Long-context understanding has emerged as a critical capability for large language models (LLMs), as real-world applications such as question answering, summarization, and complex reasoning require models to recognize and utilize relevant evidence across tens or hundreds of thousands of tokens [1]–[5]. Supervised fine-tuning (SFT) on synthesized long-context instruction data has emerged as a promising and scalable solution [6]–[12]. However, existing long-context SFT datasets suffer from three compounding limitations. 1) Limited task coverage. Without a systematic task taxonomy to guide synthesis, prior work concentrates on a narrow set of task types (e.g., multi-hop QA [6]–[9]), leaving diverse real-world capabilities such as temporal reasoning, aggregation, and state tracking insufficiently supervised. 2) Insufficient instruction difficulty. Prior work often generates questions directly from raw documents without modeling evidence structures, where evidence spans may depend on each other in the form of chains, trees, or graphs [6], [11], [12]. This lack of structural modeling and difficulty stratification naturally biases the generated data toward easy, locally answerable questions, allowing models to exploit shortcuts rather than learn genuine cross-paragraph reasoning. 3) Lack of faithfulness supervision. Without supervision that anchors each reasoning step to source evidence [13], models may rely on parametric knowledge rather than the source context, potentially yielding unfaithful reasoning inconsistent with the document.

To this end, we propose LongCrafter, a data synthesis framework that addresses these limitations by coupling a comprehensive task taxonomy with a structured, evidence-grounded pipeline. Unlike prior approaches that treat data synthesis as a plain text generation task [6], [11], [12], LongCrafter treats it as a structured crafting problem, ensuring that both instructions and responses are deliberately designed around a specified task type and grounded in explicitly located evidence spans rather than relying on surface-level narrative imitation. Underpinning the framework is a hierarchical task taxonomy that serves as a global generative prior throughout the pipeline. As illustrated in Figure 1, it organizes long-context understanding into local/shallow and global/deep levels and groups tasks by their required capability (e.g., retrieval, ordering, tracking, and multi-evidence reasoning), yielding 32 fine-grained task types. Building on this taxonomy, LongCrafter operates through three stages. (1) Long Context Construction: We group the corpus by source and pair each source with candidate task types. For single-document tasks, each document is matched to a suitable task. For multi-document tasks, we select a document as the primary reference point, then retrieve and filter complementary documents based on semantic relevance and task alignment via hybrid BM25–dense retrieval to form the final long-context input. (2) Evidence-Constraint Graph Construction: We parse each document into a structured evidence graph by extracting candidate evidence spans, selecting task-relevant spans as nodes, and connecting them with dependency edges that capture cross-paragraph relations. The resulting graph provides explicit evidence anchors for downstream instruction and response generation. (3) Instruction-Response Pair Synthesis: We generate instructions and responses strictly conditioned on the evidence graph, where instructions must jointly utilize all key nodes to prevent shortcut answering, and responses are derived step-by-step with each reasoning step citing the corresponding evidence verbatim from the source. The proposed pipeline ensures broad task coverage across diverse capability types, with particular emphasis on scarce yet critical tasks that require integrating information across multiple dispersed evidence spans, while guaranteeing that responses constitute faithful and logically coherent reasoning grounded in fine-grained contextual evidence.

Extensive experiments demonstrate that models trained on LongCrafter-generated data consistently outperform prior long-context SFT baselines on LongBench [1], LongBench v2 [2], and LooGLE [3], achieving the highest All-Overall scores on both Qwen2.5-7B and LLaMA-3.1-8B backbones (45.15% and 45.71%) and improving over the corresponding official post-trained models by +2.41 and +5.25 points, respectively. Data analysis further shows that LongCrafter-generated data achieves superior task diversity and difficulty distribution compared to existing datasets. Further analysis reveals that LongCrafter-trained models exhibit strong positional robustness in evidence identification and concentrate attention precisely on true evidence documents, effectively mitigating the “lost in the middle” phenomenon [4].

The main contributions of this paper are as follows:

• We construct a hierarchical task taxonomy of 32 fine-grained task types spanning local/shallow to global/deep long-context understanding, providing systematic and comprehensive task coverage for long-context SFT data construction.

• We propose LongCrafter, which decomposes documents into evidence graphs modeling cross-paragraph dependencies and generates instruction–response pairs strictly conditioned on located evidence spans, thereby producing data with sufficient instruction difficulty and faithful, evidence-grounded supervision.

• We validate LongCrafter on comprehensive long-context benchmarks, where LongCrafter-trained models consistently outperform all baselines; further analysis confirms that the synthesized data achieves superior diversity and quality, and that trained models precisely locate evidence spans with strong positional robustness.

-2.0ex 1.5ex Methodology This section details LongCrafter, which couples a hierarchical task taxonomy with a three-stage pipeline. The taxonomy acts as a global generative prior that constrains the entire construction process, while the pipeline of Long Context Construction, Evidence-Constraint Graph Construction, and Instruction-Response Pair Synthesis produces training samples in a structured and traceable manner. To ensure broad domain coverage, we collect and preprocess documents from diverse web sources, forming a rich corpus, with further details provided in Appendix [sec:subsec:data95construction]. -1.8ex 0.8ex Task Taxonomy At the foundation of LongCrafter is a task taxonomy grounded in real-world long-context capability requirements, which serves as a global generative prior for the entire pipeline. As detailed in Table 4, the taxonomy organizes long-context understanding into two levels: local/shallow tasks that depend on localized evidence, and global/deep tasks that depend on multiple interdependent pieces of evidence. Tasks are further grouped by their required capability (e.g., retrieval, ordering, tracking, and multi-evidence reasoning), yielding 32 fine-grained task types that span both single- and multi-document scopes. Each fine-grained task type steers all downstream stages of LongCrafter, from evidence graph decomposition to instruction generation and response attribution.

-1.8ex 0.8ex Data Synthesis Guided by the task taxonomy, the LongCrafter pipeline produces training samples through three sequential stages: Long Context Construction (Stage 1), Evidence-Constraint Graph Construction (Stage 2), and Instruction-Response Pair Synthesis (Stage 3).

0.0.1 Stage 1: Long Context Construction↩︎

We first perform a coarse-grained matching: the corpus is grouped by source, and each source is paired with its candidate fine-grained task types (e.g., code documents are matched to code-understanding tasks), narrowing the search space for the subsequent document-level matching. We then perform document-centric matching, associating each document with applicable task categories along two paths determined by document scope.

0.0.1.1 Single-document matching.

Each document is evaluated against all single-document task categories among its candidates. Tasks that can be meaningfully constructed from the document alone—such as global summarization or intra-document multi-hop reasoning—are recorded as its candidates, and the document is split into equal-sized chunks as the context for Stage 2.

0.0.1.2 Multi-document matching.

Each document is also evaluated as a root document for multi-document tasks. If suitable, an LLM generates retrieval queries conditioned on the root document and the target task type, and a hybrid BM25–dense strategy retrieves complementary documents from the remaining corpus. An LLM then performs joint relevance and diversity filtering to assemble the final context with the root document, keeping the set topically related yet non-redundant—a prerequisite for meaningful cross-document evidence graphs in Stage 2. The assembled context is then split into equal-sized chunks and passed to Stage 2.

0.0.2 Stage 2: Evidence-Constraint Graph Construction↩︎

Stage 2 is the core component that distinguishes LongCrafter from existing methods. Rather than generating questions directly from raw context, LongCrafter first decomposes the context into a structured evidence graph that makes cross-paragraph dependencies explicit. The graph is built via an Extract-then-Construct procedure: exhaustive candidate span extraction followed by task-driven graph construction.

0.0.2.1 Step 1: Exhaustive Span Extraction.

Given context \(\mathcal{C} = \{c_1, c_2, \ldots, c_n\}\) (where \(c_i\) denotes the \(i\)-th chunk) and target task type \(t\), an LLM first generates task-specific guidance based on \(t\), and then extracts all candidate evidence spans from each chunk \(c_i\) with reference to the full context. For each candidate, the model records (1) a verbatim text snippet with its chunk-level source location, and (2) the task-necessary key information carried by the span (e.g., the event and date for timeline reasoning). This yields a dense candidate node set \(V^{+} = \bigcup_{i=1}^{n} V^{+}_i\), where \(V^{+}_i\) denotes all candidates from chunk \(c_i\), avoiding premature discarding of potentially necessary evidence that proves crucial only in cross-chunk reasoning.

0.0.2.2 Step 2: Task-Driven Graph Construction.

Given the full candidate set \(V^{+}\), the model performs a global reasoning pass over the complete context to construct the minimal sufficient evidence graph \(G = (V, E)\) with \(V \subseteq V^{+}\), where:

• Node \(v_i \in V\): an evidence span, comprising (1) a verbatim citation snippet precisely located to a specific paragraph, and (2) the task-necessary key information it carries (Necessary Notes);

• Edge \(e_{ij} \in E\): a directed edge encoding the cross-paragraph dependency between nodes \(v_i\) and \(v_j\) (e.g., temporal, causal, or coreference relations).

Concretely, the model selects the candidate spans jointly necessary and sufficient for task type \(t\), and establishes directed dependency edges \(E\) between the selected nodes, annotating each with its cross-chunk relation type to capture their logical relationships. A valid evidence graph must satisfy two conditions: (1) the nodes and edges are indispensable for the target task, ensuring the graph faithfully reflects task requirements such as reasoning complexity; (2) the node set uniquely supports the correct answer. This two-level design ensures that \(V\) constrains which evidence spans are necessary, while \(E\) constrains how they are logically connected—guiding Stage 3 to generate questions that probe inter-span dependencies rather than isolated facts. As a safeguard for global/deep task types, if the selected nodes all originate from the same chunk, targeted re-extraction is triggered to ensure \(G\) captures genuine multi-span evidence.

0.0.3 Stage 3: Instruction-Response Pair Synthesis↩︎

Stage 3 generates the final instruction-response pairs conditioned on the evidence graph \(G\) produced by Stage 2.

0.0.3.1 Instruction generation.

Taking task type \(t\), evidence graph \(G\), and context \(\mathcal{C}\) as input, an LLM generates questions centered on the nodes and edges of \(G\), under task-specific constraints (e.g., for global/deep tasks, answering correctly must jointly utilize all key nodes in \(G\) and cannot be resolved from any single local passage), thereby ensuring controllable difficulty by construction.

0.0.3.2 Response generation.

Responses adopt a step-by-step citation format: each reasoning step quotes the corresponding evidence verbatim from the source, and the final answer is logically derived from the resulting citation chain, with no external parametric knowledge introduced. This format inherits the node location information and edge dependency ordering from \(G\), ensuring end-to-end correspondence between responses and the original context.

The generated pairs are then validated by an LLM that checks whether each instruction is unambiguous, answerable from the given context, and admits a unique correct answer; pairs failing any criterion are discarded. Using GLM-5 throughout the entire pipeline, we generate 2,000 high-quality long-context training samples.

-1.8ex 0.8ex Dataset Analysis To validate that LongCrafter constructs higher-quality training data, we analyze it across three dimensions: diversity, instruction difficulty distribution, and response quality.

0.0.3.3 Diversity.

As data diversity is a recognized factor in dataset quality [14], we evaluate our method across six diversity metrics: Unique 3-grams [15], Mean L2 Distance, Mean Cosine Distance, KNN Distance [16], Cluster Inertia [17], and Radius [18]. As shown in Figure 7, our method achieves the best performance on all six, reflecting the broad semantic coverage of our taxonomy-guided construction.

0.0.3.4 Instruction Difficulty Distribution.

To quantify instruction difficulty, we define three dimensions: evidence locality, computation or transformation required, and distractor strength, each scored by GPT-5 and averaged into a final composite difficulty score. As shown in Figure 3 (left), our method achieves the most balanced difficulty distribution across all four bins (0–1, \(>\)​1–2, \(>\)​2–3, \(>\)​3–4), with a notably higher proportion of high-difficulty samples (score \(>\)​3) than baselines. In contrast, directly synthesized baselines such as LongAlign, LongReward, and LongMagpie concentrate in lower difficulty ranges, while LongFaith, built upon human-annotated instructions, skews toward medium difficulty. Furthermore, as shown in Figure 3 (middle), information-intensive tasks in our taxonomy exhibit substantially higher difficulty scores, confirming that dataset difficulty can be systematically steered by task type and that our pipeline can flexibly generate data at targeted difficulty levels.

0.0.3.5 Response Quality.

Following the four quality dimensions defined by [10], we employ an LLM-as-judge protocol to evaluate response quality across Helpfulness, Logicality, Faithfulness, and Completeness. As shown in Figure 3, our method achieves the highest scores on all four dimensions (3.88, 3.91, 3.94, and 3.90), substantially outperforming all baselines. The advantage is most pronounced on Faithfulness, where the lowest baseline scores only 1.59 (LongMagpie), indicating that without explicit grounding, models tend to inject parametric knowledge into the reasoning chain; in contrast, our evidence-grounded pipeline enforces strict adherence to the context and eliminates this failure mode at the source.

0.0.3.6 Human Evaluation.

Following [19], we conduct a human–LLM agreement study to verify the reliability of our LLM-as-judge evaluation. We sample 500 instances (100 per dataset), each independently scored by five expert annotators using our automatic-assessment criteria, with scores averaged as the final human ratings. As shown in Table 2, Spearman’s correlation between LLM and human scores is statistically significant (\(p<0.001\)) for both dimensions, moderate for instruction difficulty (\(\rho=0.63\)) and strong for response quality (\(\rho=0.74\)), indicating strong alignment. Moreover, experts judge that 98% of LongCrafter instructions satisfy the reasoning-complexity requirement of their assigned task type, confirming that our taxonomy-guided synthesis reliably controls difficulty through task type selection.

-2.0ex 1.5ex Experiments In this section, we describe our experimental setup and present the main results of LongCrafter on long-context understanding benchmarks. -1.8ex 0.8ex Experimental Setup

0.0.3.7 Model Training.

We conduct experiments on two representative open-source base models: Qwen2.5-7B [20] and LLaMA-3.1-8B [21], both supporting a native context length of 128K tokens. All models are trained using LoRA with a learning rate of \(5\times10^{-5}\) for 2 epochs, with LoRA rank 32, alpha 64, and dropout 0.1.

0.0.3.8 Baselines.

We compare LongCrafter against four representative long-context SFT data construction methods. LongAlign [6] constructs long instruction-following data from diverse document sources using Self-Instruct. LongMagpie [12] is a self-synthesis framework that leverages aligned LLMs to auto-regressively generate contextually relevant instructions given a document. LongFaith [8] synthesizes attribution-based reasoning data grounded in the human-annotated MuSiQue dataset. LongReward [10] filters SFT data by scoring synthesized responses along four dimensions: helpfulness, logicality, faithfulness, and completeness. Following prior work, we sample 2,000 examples from each dataset for model training to ensure a fair comparison. In addition, we include the officially released post-trained models of LLaMA and Qwen as strong reference baselines.

0.0.3.9 Evaluation.

We evaluate on three benchmarks covering diverse long-context task types and difficulty levels. LongBench [1] includes four subtasks spanning single-document and multi-hop reasoning: Qasper, HotpotQA, MuSiQue, and 2WikiMultihopQA. LongBench v2 [2] consists of 503 challenging multiple-choice questions sourced from realistic long-context scenarios across six task categories. LooGLE [3] evaluates long-dependency QA through four subtasks: Comprehension & Reasoning, Computation, Timeline Reorder, and Multiple Information Retrieval. Further details are provided in Appendix [sec:subsec:evaluation].

-1.8ex 0.8ex Main Results

Overall Performance. As shown in Table 1, models trained on LongCrafter data attain the highest All-Overall scores on both backbones (45.15% on Qwen2.5-7B and 45.71% on LLaMA-3.1-8B), ranking first on the large majority of individual subtasks and surpassing the second-best SFT baseline by +5.80 and +5.30 points, respectively. This cross-architecture consistency indicates that the gains stem from the data construction design itself rather than any backbone-specific inductive bias, and that LongCrafter’s broad task diversity yields strong generalization rather than overfitting to a particular task distribution.

Gains on Difficult Tasks. On LongBench v2, which targets deeper reasoning over realistic long contexts, LongCrafter achieves the best overall results among all SFT variants, reaching 29.16% on Qwen2.5-7B and 27.24% on LLaMA-3.1-8B. On LooGLE, whose tasks require integrating evidence dispersed across multiple locations, it leads across all four subtasks, with an especially large margin on Timeline Reasoning on LLaMA, where it scores 53.0% against 40.0% for the second-best SFT baseline.

Efficiency. With only 2,000 training samples, LongCrafter outperforms the officially post-trained models on the All-Overall metric, with gains of +2.41 and +5.25 points on Qwen2.5-7B and LLaMA-3.1-8B, respectively. This suggests that carefully designed, evidence-grounded data can drive substantial long-context gains even at minimal data scale.

-1.8ex 0.8ex Position-Robust Evidence Localization

0.0.3.10 Evidence Position Robustness

To test whether models trained on our data learn content-based evidence localization rather than positional shortcuts, we follow [22] and place the evidence document at positions 1, 5, 10, 15, and 20 among 20 candidates, directly probing the “lost in the middle” phenomenon [4]. As shown in Figure 4, LongCrafter-trained models maintain near-perfect retrieval accuracy across all positions on both LLaMA-3.1-8B and Qwen2.5-7B, remaining stable as the evidence moves toward the tail. In contrast, all baselines exhibit pronounced positional decay: on LLaMA-3.1-8B, LongFaith drops most sharply to roughly 71% at position 20, while on Qwen2.5-7B, LongReward collapses to about 40%, with the remaining baselines showing consistent declines on both backbones.

0.0.3.11 Attention Mass Analysis

To probe the underlying mechanism, we analyze attention mass following [23], measuring how strongly a model concentrates attention on the evidence document relative to others. For a response of \(T\) generated tokens, we compute the Evidence Attention Mass (EAM) for each layer \(\ell\) and head \(h\) by averaging the attention allocated to gold evidence tokens over all decode steps: \[\overline{\text{EAM}}^{(\ell,h)} = \frac{1}{T} \sum_{t=1}^{T} \sum_{j \in \mathcal{E}_t} a^{(\ell,h)}_{t,j},\] where \(a^{(\ell,h)}_{t,j}\) is the attention probability of the last generated token to key position \(j\) at step \(t\), and \(\mathcal{E}_t = \{ j \in \mathcal{E} \mid j < K_t \}\) is the set of gold evidence token indices visible within the current key length \(K_t\). We report this response-level EAM by averaging over all layers, all heads, and samples with the same evidence position. As shown in Figure 5, LongCrafter-trained models achieve the highest attention mass across all positions on both backbones, and sustain it even at position 20, where all baselines reach their lowest values. Together, these results show that our evidence-graph-guided training consistently improves evidence position robustness, enabling models to locate and attend to relevant evidence regardless of its position.

-1.8ex 0.8ex Ablation

We conduct ablation studies from two perspectives: component ablations and data distribution ablations. The results are shown in Table 3 and Table ¿tbl:tab:ablation?.

0.0.3.12 Component Ablations.

We first ablate LongCrafter’s core components, including Evidence Graph Construction (EGC) and Evidence-Based Citation (EBC). Removing EGC substantially lowers the difficulty and quality of synthesized data: the average difficulty score of Global/Deep tasks drops from 2.59 to 1.74. To further assess data quality, we manually inspect 100 instructions generated without EGC and find that about 30% are clearly low-quality, including insufficiently challenging instructions, answers that are not faithful to the original context, or flawed reasoning logic due to the absence of explicit evidence-graph edges that support logical dependencies among evidence spans.

The downstream results show the same trend. Removing EGC causes the largest performance drop, reducing the average score by 17.71 points on Qwen2.5-7B and 15.92 points on LLaMA-3.1-8B. Removing EBC also consistently hurts performance, with average drops of 5.25 points and 4.08 points, respectively. These results indicate that EGC improves both the difficulty and quality of generated instructions, while providing reliable evidence dependencies for synthesizing high-quality answers. EBC further strengthens faithful supervision, encouraging models to reason based on the original evidence.

0.0.3.13 Data Distribution Ablations.

We further study the effect of data distribution by comparing Easy-only, Hard-only, and Low-diversity settings. The Easy-only setting samples only local-information-dependent tasks, while the Hard-only setting samples only global-information-dependent tasks. Both underperform the full LongCrafter setting, with average drops of 8.29/6.80 points and 7.24/5.15 points on Qwen2.5-7B/LLaMA-3.1-8B, respectively. The Low-diversity setting, which samples only from common base task types, also reduces the average score by 7.72/6.78 points. These results indicate that robust long-context generalization is best achieved by covering diverse task types across a balanced difficulty distribution.

Overall, the proposed task taxonomy ensures balanced and diverse task distributions for better long-context generalization, while EGC and EBC ensure instruction difficulty, data quality, and faithful supervision.

3.8pt

-1.8ex 0.8ex Case Study

Figure 6 presents a representative example of how evidence-grounded reasoning affects answer faithfulness. The question requires a three-hop chain: identifying the shared director of Vera Cruz and Kiss Me Deadly, locating his 1964 film, and naming the actress who made her final film appearance in it. The baseline (LongFaith) locates the correct film but attributes the final role to Bette Davis, a prominent co-star, rather than the evidence-supported answer; lacking per-step citation, it conflates the most salient entity in the passage with the one actually stated by the evidence, a subtle drift that propagates to the final answer. In contrast, the LongCrafter-trained model grounds each reasoning step in fine-grained evidence spans, sequentially identifying Robert Aldrich, Hush…Hush, Sweet Charlotte, and Mary Astor as the answer, with its evidence-based citation suppressing the tendency to default to salient but unsupported entities and thus avoiding this failure mode.

-2.0ex 1.5ex Related Work Long-Context Understanding Enhancement. Many methods aim to extend the long-context capabilities of LLMs. Position encoding extensions, including rotary position embedding variants [24], [25] and position interpolation [26], help models generalize to longer sequences, while continued pre-training on long documents strengthens long-range dependency modeling [27], [28]. Beyond pre-training, supervised fine-tuning (SFT) on long-context instruction data equips models with instruction-following and reasoning over extended contexts [6], [7], [9], and more recent work applies reinforcement learning [5], [10] and preference optimization [29], [30] to further improve response quality. Our work targets the SFT stage, focusing on how to synthesize high-quality long-context instruction data with broad task coverage and faithful evidence grounding.

Long-Context Training Data Synthesis. Prior synthesis methods fall into two lines. The first targets instruction diversity, constructing question–answer pairs through direct synthesis [6], [9], [11], [12], multi-agent synthesis [7], [31], or graph-based multi-hop synthesis [32], but is largely confined to narrow task formats with limited reasoning complexity, offers little systematic control over task type and difficulty, and provides no guarantee of response faithfulness. The second targets response quality, improving answer faithfulness and traceability via chain-of-citation constraints [8], fine-grained citation [33], or multi-dimensional preference signals [10], yet introduces evidence constraints only after question generation, leaving the evidence structure of the instructions themselves unconstrained. In contrast, LongCrafter moves evidence modeling upstream by building an explicit evidence graph before instruction generation, jointly ensuring paragraph-level grounding, controlling reasoning complexity through cross-paragraph dependencies, and enabling systematic coverage across diverse task types.

-2.0ex 1.5ex Conclusion In this paper, we propose LongCrafter, a structured data synthesis framework for long-context SFT that simultaneously addresses limited task coverage, insufficient instruction difficulty, and the lack of faithfulness supervision. By establishing a comprehensive task taxonomy, constructing explicit evidence graphs that capture cross-paragraph dependencies, and enforcing citation-grounded response generation, LongCrafter produces training data that is task-diverse, difficulty-calibrated, and end-to-end faithful to source evidence. Extensive experiments across three benchmarks and two model families demonstrate consistent state-of-the-art performance, with particularly prominent advantages on high-difficulty tasks and strong positional robustness in evidence localization. Our ablation studies further confirm the contribution of the evidence graph and citation-grounded supervision, underscoring the effectiveness of the LongCrafter pipeline for developing capable long-context LLMs.

-2.0ex 1.5ex Limitations The multi-pass evidence graph construction introduces higher synthesis cost compared to single-pass baselines; training a dedicated graph construction model to amortize this cost is a promising direction for future work. Additionally, although LongCrafter covers a broad task taxonomy, certain capability dimensions such as long-horizon agent tasks and life-long dialogue scenarios lie outside its current scope and warrant further investigation.

-2.0ex 1.5ex Appendix

-1.8ex 0.8ex Data Construction Details

0.0.3.14 Corpus Collection.

We construct a cross-domain bilingual long-context corpus spanning 11 domains: dialogue, academic papers, structured data, source code, judicial documents, legal statutes, news, fiction, scripts, biographies, and knowledge graphs. Data are sourced from publicly available collections, including WildChat, arXiv, ChinaXiv, GitHub, Project Gutenberg, Wikipedia, and Wikidata, among others. Each domain naturally exhibits distinct long-context structures, such as hierarchical provisions in legal texts, cross-file dependencies in code, and entity-relation networks in knowledge graphs, providing structurally diverse training contexts.

0.0.3.15 Filtering and Cleaning.

We apply source-level and sample-level filtering to ensure data quality. Source-level filtering removes documents with failed parsing, encoding errors, unclear provenance, or duplicate content. Sample-level filtering requires at least 50% target-language characters and 50% non-empty line deduplication rate, and discards texts dominated by footnotes, numbered lists, or web noise. All documents are converted to Markdown or plain-text format using source-specific parsers, followed by domain-specific cleaning rules.

0.0.3.16 Long-context Sample Construction.

We construct long-context samples by applying length thresholds: single-document samples require at least 5,000 Chinese or 15,000 English characters, while multi-document samples are assembled within controlled length ranges per domain. Overlong documents are truncated at paragraph boundaries, and no document is reused across the constructed data.

-1.8ex 0.8ex Prompts

This section enumerates all prompts utilized in our evaluation framework. Prompt 1 presents the response quality evaluation prompt for long-context prompt-response data. Following [10], it evaluates each response along four dimensions: Helpfulness, Logicality, Faithfulness, and Completeness. Prompt 2 describes the instruction difficulty estimation prompt. Inspired by [34], it measures difficulty from three dimensions: evidence_locality, computation_or_transformation, and distractor_strength. Prompt 3 presents the answer correctness judge used for evaluation.

-1.8ex 0.8ex Diversity

Figure 7 reports the diversity comparison among different training datasets using six complementary metrics. Unique 3-grams measures lexical-level diversity by counting distinct 3-gram patterns. Mean L2 Distance and Mean Cosine Distance quantify the average pairwise dispersion of response embeddings, reflecting the global semantic spread of the dataset. KNN Distance measures local neighborhood dispersion, where larger values indicate that samples are less concentrated around near-duplicate or highly similar responses. Cluster Inertia and Radius further characterize the embedding-space distribution from a clustering perspective, capturing how broadly samples are distributed across semantic regions. For a fair comparison, all metrics are computed under the same sampling size and embedding configuration.Higher values indicate broader lexical or semantic coverage.

-1.8ex 0.8ex Evaluation. LongBench [1] is a multi-task benchmark for long-context understanding, from which we select four subtasks spanning single-document comprehension and multi-hop reasoning across multiple documents: Qasper, HotpotQA, MuSiQue, and 2WikiMultihopQA. LongBench v2 [2] targets deeper understanding and reasoning over realistic long-context multitasks, consisting of 503 challenging multiple-choice questions with contexts ranging from 8K to 2M words across six major task categories; LooGLE [3] evaluates long-context understanding through four long-dependency QA subtasks, namely Comprehension & Reasoning, Multiple Information Retrieval, Timeline Reorder, and Computation, which go beyond simple fact lookup and require models to track, aggregate, and reason over interdependent evidence dispersed across long documents.

To prevent data contamination, we first applied MinHash-based deduplication to filter out training samples with high similarity to the evaluation sets, followed by manual review to further ensure no data leakage between the training and test sets. Following prior work, we use GPT-5 to judge answer correctness. A manual spot-check shows that GPT-5’s judgments reach 99.5% agreement with human verification. For LongBench v2, we run each experiment three times with a temperature of 0.1 and report the average score. For LongBench and LooGLE, we set the temperature to 0 for deterministic evaluation.

3pt

{
  "helpfulness": {
    "score": number,
    "reason": "brief reason"
  },
  "logicality": {
    "score": number,
    "reason": "brief reason"
  },
  "faithfulness": {
    "score": number,
    "reason": "brief reason"
  },
  "completeness": {
    "score": number,
    "reason": "brief reason"
  }
}

References↩︎