1 Introduction↩︎

Question answering (QA) systems have achieved strong performance on curated, homogeneous corpora, ranging from reading comprehension [1] to open-domain text retrieval [2] and multi-hop reasoning [3]. However, real-world enterprise and scientific knowledge is rarely stored as flat text. Instead, it often resides in data lakes: heterogeneous, weakly structured collections of tables, passages, and linked metadata.

Answering natural language questions over data lakes requires navigating siloed sources, combining cross-modal evidence, and maintaining explicit provenance. However, modern retrieve-and-synthesize architectures [4], [5] lack evaluation environments that simultaneously stress discovery, cross-source composition, and faithful attribution over heterogeneous corpora. Existing benchmarks bypass the noise, scale, and weak metadata characteristic of real data lakes, evaluating either pure data discovery without natural language synthesis, or reasoning over already-clean, schema-aligned text.

To bridge this gap, we introduce LakeQuest, a benchmark designed to evaluate the end-to-end QA pipeline over realistic data lakes. LakeQuest provides a shared construction and evaluation framework instantiated across three diverse domains: AI/ML metadata, a realistic retail bank, and multimodal biomedical drug information. Rather than flattening these domains into a uniform representation, LakeQuest preserves their native structural heterogeneity as data lakes. Each benchmark instance pairs a natural language question with a reference answer and exact, modality-aware evidence pointers (e.g., specific table rows or passage spans). Furthermore, because LakeQuest preserves the native structure of the data lakes, it inherently captures factual redundancy. A system might successfully synthesize a grounded answer by discovering valid alternative provenance (e.g., a redundant table or densely informative passage) even if the target gold artifact is missed. This explicit attribution enables precise isolation of system failures into discovery errors (failing to retrieve evidence) versus reasoning errors (failing to synthesize an answer from the retrieved evidence) [6], [7].

We construct LakeQuest using a reproducible five-stage pipeline encompassing domain instantiation, language-model-driven QA synthesis, structural filtering, semantic validation, and rigorous human review. The resulting benchmark contains 9,846 human-validated QA pairs that entail single-source lookup, cross-source composition, and multi-hop reasoning.

Evaluating a diverse suite of baselines—from zero-shot reasoning [8] and standard retrieval-augmented generation (RAG) [4] to structured pipelines and agentic tool-use methods such as XMode [9] and ReSP [10]—we find that the bottlenecks in data lake QA remain fundamentally unsolved and are strongly domain-dependent. Specifically, accurate retrieval does not guarantee faithful reasoning when cross-modal composition is required, and even agentic strategies struggle with multi-hop discovery over weakly structured metadata.

Our primary contributions are: (1) The LakeQuest Benchmark: A human-validated dataset of over 9.8k questions evaluating grounded reasoning across three heterogeneous data lakes (AI/ML, Retail Banking, and Biomedical Drug). (2) A Reproducible Construction Framework: A five-stage protocol for synthesizing, filtering, and validating QA pairs with explicit evidence traces over weakly structured corpora. (3) Empirical Analysis of Modern QA: Extensive baseline evaluations revealing that high retrieval quality does not consistently translate into accurate, faithful answers—especially when multi-step or cross-source composition is required.

2 Related Work↩︎

Table 1 provides a structural comparison between LakeQuest and existing QA and data discovery benchmarks across modalities, evidence tracing, and evaluation scope.

3.0pt

Data Discovery and Integration. Data management literature extensively studies how to discover, link, and integrate datasets [11], [12]. Accordingly, benchmarking efforts in this space, such as LakeBench [13] and CMDL [14], evaluate the retrieval of joinable tables or cross-modal artifacts. While essential for data discovery, these benchmarks remain purely retrieval-focused and do not evaluate natural language synthesis.

Text-centric and Multi-hop QA. In the NLP community, QA benchmarks have evolved from single-document reading comprehension [1] to open-domain retrieval [2] and multi-hop reasoning with HotpotQA [3]. While foundational, these datasets largely assume text-centric evidence within curated, relatively homogeneous corpora (e.g., Wikipedia). Real data lakes, however, are characterized by weak metadata, fragmented entity references, and a mixture of semi-structured objects, making reasoning pathways noisier than those found in standard text-only multi-hop settings.

Hybrid and Multimodal QA. A closer line of work evaluates QA over multiple evidence types. HybridQA [15] and OTT-QA [16] require reasoning across tables and text, while TAT-QA [17] emphasizes numerical reasoning over financial reports. MultiModalQA [18] expands the scope to joint reasoning over text, tables, and images. While these benchmarks require cross-modal synthesis, they generally operate over highly curated, schema-aligned, or task-specific corpora. They do not model the core challenges of enterprise data lakes: navigating siloed, weakly linked sources, resolving entities across heterogeneous files, and maintaining strict, modality-aware provenance for auditability.

Retrieval-Grounded and Agentic Systems. Methodologically, LakeQuest provides a necessary testbed for modern retrieve-and-synthesize pipelines. RAG [4] and fusion-in-decoder (FiD) [19] established paradigms for grounding generation in retrieved contexts, while recent methods propose structured decomposition [20], iterative tool use [5], [21], and multi-agent orchestration like Symphony [22]. By providing a heterogeneous environment with explicit evidence requirements and realistic discovery bottlenecks, LakeQuest enables the community to rigorously evaluate whether these control strategies can handle the noise and scale of real-world data lakes.

3 The LakeQuest Framework↩︎

To rigorously evaluate QA over heterogeneous data lakes, we require a framework that isolates source discovery from cross-modal synthesis, scales across domains, and maintains strict provenance. LakeQuest operationalizes this through a formalized task definition and a reproducible construction pipeline.

3.1 Problem Formulation and Task Taxonomy↩︎

We formulate grounded data lake QA: given a data lake \(\mathcal{D} = \{d_1, \dots, d_n\}\) (where \(d_i\) is a table, passage, or metadata record) and a natural language question \(q\), a system \(f\) must return a textual answer \(\hat{a}\) and a precise supporting evidence set \(\hat{E} \subseteq \mathcal{D}\): \(f(q, \mathcal{D}) \rightarrow (\hat{a}, \hat{E})\). Modern agents decompose this operation into retrieve-and-synthesize workflows: \(\hat{E} = R(q, \mathcal{D})\) and \(\hat{a} = G(q, \hat{E})\). By requiring systems to output \(\hat{E}\), LakeQuest allows us to independently evaluate the retrieval mechanism \(R\) (discovery) and the generator \(G\) (synthesis).

To systematically test this pipeline, LakeQuest is organized into four distinct reasoning families: single-source lookups requiring fact extraction from isolated objects; cross-source composition necessitating the combination of facts across multiple heterogeneous objects (e.g., joining a policy with a transaction); multi-hop reasoning requiring chained inference over intermediate entities or linked metadata; and anchor-based retrieval, which conditions questions on a selected entity, forcing the agent to reason over related entities amidst semantically close distractors.

Unlike standard text QA benchmarks where evidence is simply a document ID, LakeQuest enforces modality-aware provenance. Each benchmark instance is defined by the tuple \(x_i = (q_i, a^*_i, E^*_i, m_i)\), where \(E^*_i\) resolves to concrete sub-document units (e.g., specific table rows or exact passage spans) and \(m_i\) tracks the question family and domain metadata. Figure 2 illustrates a concrete cross-source instance from the Retail Bank data lake.

3.2 The Shared Five-Stage Pipeline↩︎

Manually authoring tens of thousands of complex, cross-modal QA pairs is unscalable, while purely synthetic datasets often suffer from hallucinations and trivial reasoning pathways. To balance scale with rigor, we construct LakeQuest using an LLM-in-the-loop pipeline consisting of five shared stages across all domains (a visual schematic of the Retail Bank data lake question-answer synthesis process is provided in App 16).

The five shared stages include: (1) Lake Instantiation, preserving the domain’s native schema rather than homogenizing documents; (2) Candidate Synthesis, where a strong language model (OpenAI GPT-5, Feb. 1, 2026) synthesizes a candidate question \(q\), answer \(a^*\), and exact evidence pointers \(E^*\) from a sampled evidence subgraph; (3) Structural Filtering, which programmatically removes malformed outputs or deictic questions; (4) Semantic Filtering, utilizing an LLM-as-a-judge to reject trivial candidates answerable via parametric memory alone; and (5) Human Validation, the definitive gate for benchmark inclusion, assessing answerability and attribution. Exact prompts and rubrics are detailed in App 11.

3.3 Human Validation and Quality Control↩︎

To guarantee evaluation fidelity, all items passing the automated filters are subjected to rigorous human validation. Annotators were tasked with independently verifying two criteria: Answerability (can the question be answered exclusively using the cited evidence?) and Attribution (is the generated answer completely faithful to the cited text/table?). The use of human annotators was approved by the ethics review board at the authors’ institution. Prolific annotators (paid $12 USD/hr) independently verified each item under this protocol. Inter-annotator agreement (Fleiss’ \(\kappa\)) demonstrates strong consistency across raters: \(0.82\) for Answerability and \(0.85\) for Attribution. Full annotator guidelines, including a screenshot of the custom validation interface (Figure 3), are provided in App 12. Only instances that achieved strict human consensus were retained in the final LakeQuest release.

4 The LakeQuest Dataset — Instantiating the Three Data Lakes↩︎

To evaluate the framework across diverse reasoning requirements, LakeQuest instantiates three data lakes: an AI/ML metadata repository, a synthetic retail banking system, and a biomedical corpus. Table 2 outlines the high-level abstractions used to preserve the native structures of these domains. Furthermore, Table 3 details the final benchmark coverage and human annotation outcomes across these three lakes.

4.0.0.1 AI/ML Data Lake

The AI/ML data lake focuses on entity disambiguation and relation chaining over weakly linked metadata. We construct the corpus using a snapshot of the Hugging Face Hub API [23]. The resulting lake consists of model cards and dataset cards, which are inherently semi-structured. We instantiate 5 distinct question families in this domain, yielding 3,578 human-validated questions (see Table 3 for a high-level summary and Table 7 in App 10 for the detailed breakdown).

These tasks map to our core reasoning taxonomy by requiring systems to deduce specific hyperparameters from isolated cards (single-source), synthesize constraints by jointly reasoning over explicitly linked model and dataset cards (cross-source), chain relations across the metadata graph to link datasets and subsequent models (multi-hop), and formulate queries around specific anchor entities despite lexical drift (anchor-based).

4.0.0.2 Retail Bank Data Lake

The Retail Bank data lake evaluates grounded decision-making under strict policy constraints. Because real operational banking data is highly restricted, we generate a high-fidelity synthetic lake by deriving generation seed examples and instantiating an open-banking-inspired relational schema (client, account, transaction, and policy entities). We then populate client profiles, transaction ledgers, and institutional policy documents. We further validate that transaction distributions are similar between the synthetic lake and original real banking data by matching graph density on client profiles. Tasks are formulated across 9 persona-driven workflows (e.g., a fraud analyst investigating a transaction, or an auditor reviewing compliance), yielding 2,964 validated questions (detailed breakdown provided in Table 8, App 10).

These workflows map to core reasoning families: single-source lookups for retrieving isolated facts (e.g., account tiers or policy limits); cross-source composition to combine unstructured policy rules with structured transaction ledgers (e.g., checking if a wire transfer violates a tiered policy); multi-hop reasoning to trace complex fund flows across multiple tables before applying operational rules; and anchor-based retrieval to disambiguate compliance policies based on client attributes amidst overlapping regional guidelines.

4.0.0.3 Drug Data Lake

The Drug data lake stresses multimodal evidence integration. We construct this corpus by fusing structured XML records from DrugBank 6.0 [24] with open-access scientific literature indexed in PubMed [25]. This heterogeneous mixture of dense text and structured properties instantiates 5 question families, producing 3,304 validated questions (see Table 9 in App 10 for statistical breakdown).

The reasoning families in this domain stress multimodal evidence integration: single-source lookups extract precise values from isolated DrugBank tables or PubMed abstracts; cross-source composition synthesizes answers by linking unstructured text passages (e.g., mechanism of action) to structured tables (e.g., biochemical properties); multi-hop reasoning chains relationships across modalities (e.g., identifying a target in a table, then finding passages detailing its metabolic pathway); and anchor-based retrieval requires identifying correct active compounds amidst structurally similar distractors and complex aliases.

4.0.0.4 Splits and Licensing

To prevent contamination, we define (open) validation (20%) and test (80%) splits at the entity rather than question level (i.e., \(\mathrm{Entity}(\mathrm{val}) \cap \mathrm{Entity}(\mathrm{test}) = \emptyset\)). The LakeQuest benchmark and underlying corpora are hosted on Hugging Face at anonymousurl. The AI/ML and Retail Bank subsets are released under an MIT license. For the Drug subset, we release benchmark annotations and evidence pointers, while use of underlying DrugBank tables remains subject to DrugBank licensing terms and PubMed-linked content remains subject to NLM/PubMed access policies and publisher rights.

5 Experimental Setup and Baselines↩︎

To benchmark grounded reasoning over data lakes, we define a unified evaluation contract and test representative QA baselines ranging from direct prompting to agentic tool use.

5.0.0.1 Evaluation Metrics

Because data lake QA serves both high-assurance settings (where hallucination is unacceptable) and exploratory analytics (where cost and speed are critical), we treat evaluation as a multi-dimensional quality-cost trade-off. We evaluate systems across four axes:

• Accuracy (Correctness): We report binary correctness using rubric-based LLM-as-a-judge for baseline runs (including direct-answer and reasoning-answer correctness where applicable), and human-judged correctness for annotated benchmark slices in App 10.

• Faithfulness (Attribution): Whether the predicted answer \(\hat{a}\) is supported by the predicted evidence set \(\hat{E}\). We operationalize this using Attributable to Identified Sources (AIS) style judgments [26], aligned with the human attribution protocol used throughout this benchmark.

• Retrieval Quality: Domain/modality-aware recall of gold evidence \(E^*\) in \(\hat{E}\) (e.g., card retrieval in AI/ML, policy and table recall in Retail Bank, and passage/table recall in Drug). For unstructured evidence, success is chunk-level span inclusion; for structured evidence, we require retrieval of the referenced source object and apply row/field-level checks when explicit row-level pointers are available. Because real-world data lakes inherently contain redundant information (e.g., same fact appearing in multiple tables/passages), this metric serves as a strict lower bound. A system may retrieve valid alternative evidence that correctly answers the question despite missing the specific gold artifact.

• Token Usage: End-to-end retrieval-plus-synthesis token footprint, reported as total model tokens (and normalized per-query token count) under the evaluation-time configuration.

5.0.0.2 Evaluated Baselines

We establish baseline performance using models with different levels of agentic complexity. We use GPT-5 as the generator (kept fixed within each experiment suite) [27] and FAISS-backed text-embedding-3-small for retrieval [28]. Comprehensive details, including system instructions, retrieval formatting templates, and agentic tool-use prompts for the evaluated systems, are provided in App 11.

• Zero-Shot Prompting: A naive baseline that directly prompts the generator LLM without explicit retrieval, testing the limits of parametric memory and instruction following.

• Retrieval-Augmented Generation (RAG) [4]: Used as our baseline for the document-only AI/ML data lake, this is a standard retrieve-then-generate pipeline. We chunk unstructured passages and structured metadata cards into overlapping text segments before indexing; we then retrieve the top-\(k\) most relevant chunks and synthesize a final answer. We experiment with different values of \(k\) and finalize on \(k=5\), which consistently attains strong performance.

• Agentic and Structured Pipelines (XMode, ReSP) [9], [10]: Standard RAG is structurally insufficient for Retail Bank and Drug because it cannot natively execute the relational row operations that navigate Retail Bank transaction records and Drug ledgers. Instead, we evaluate stronger multi-step baselines that go beyond single-shot text retrieval by interleaving reasoning and tool use. In our implementation, XMode acts as a tool-augmented RAG system: it uses an iterative loop with passage retrieval from a FAISS index alongside direct query/SQL access to structured tables. ReSP follows an iterative retrieve–summarize–plan style decomposition.

6 Results and Cross-Domain Analysis↩︎

We evaluate baselines across the three domains to understand where modern retrieve-and-synthesize systems fail. Across domains, retrieval quality alone is insufficient; grounded data-lake QA requires stronger cross-source composition and attribution mechanisms.

6.1 AI/ML Data Lake: Discovery is the primary bottleneck↩︎

The AI/ML lake is challenging due to scale—it is built from a Hugging Face model-card corpus with over 1M cards and normalized into a large model–dataset graph with on-demand card-text retrieval—and the inconsistencies between how different authors document artifacts. As shown in Table 4, the RAG baseline exhibits a notable gap between exact-match Recall (\(R\)) and Accuracy (\(A\)). For example, on isolated Model questions, RAG achieves 0.35 Accuracy despite a recall of 0.09. This gap validates that real data lakes contain factual redundancy, for example from models that have been duplicated with minor modifications; the retriever frequently finds valid alternative evidence to synthesize the correct answer even if it misses the specific gold artifact (\(E^*\)). This reinforces that \(R\) acts as a strict lower bound and suggests improved retrieval as a logical next step for metadata QA. In summary, the wide gap between modest recall and reasoning accuracy suggests that navigating weakly linked metadata via dense retrieval is insufficient for reliable entity disambiguation and relation chaining.

6.2 Retail Bank Data Lake: A failure of grounding, not retrieval↩︎

Table 5 reports performance on the Retail Bank lake. This domain is distinctive because benchmark items are framed as operational decisions requiring both unstructured policy retrieval and structured table lookups. The XMode baseline generally achieves high Policy Recall (0.85+ for seven of the nine personas, though struggling on the Data Analyst and Salesperson workflows), but Table Recall varies significantly across the board. More importantly, final answer accuracy is often much lower than policy recall. This exposes a synthesis gap: while agentic systems reliably retrieve abstract governing rules, they fundamentally struggle to ground those rules against specific operational ledgers to execute a faithful decision. Notably, the zero-shot baseline achieves high accuracy on certain workflows (e.g., 0.88 for the Financial planner). While our semantic filter successfully removes simple factoid lookups, frontier LLMs possess strong parametric priors for standard financial principles, allowing them to often deduce correct operational outcomes without context. However, because this zero-shot reasoning lacks explicit provenance (Faithfulness is strictly zero), it highlights why our benchmark requires grounded synthesis; in an enterprise setting, an accurate but ungrounded answer remains an unauditable liability.

6.3 Drug Data Lake: Joint tabular QA remains challenging↩︎

Table 6 reports performance on the Drug data lake, which stresses multimodal question answering across passages and tables. The strongest results appear on Passage questions, where XMode achieves near-perfect recall (0.98), accuracy (0.97), and faithfulness (0.97). Passage-hop questions remain relatively strong: despite Passage Recall dropping to 0.72, answer accuracy is 0.86, suggesting systems can often synthesize correct answers from partially recovered text. This gap indicates that retrieving the correct table is often not sufficient; the remaining challenge lies in extracting the right fields and composing them into a correct answer. Difficulty increases further for Table-hop questions, where Table Recall drops to 0.48 and accuracy to 0.44, showing that multi-step reasoning over structured biomedical evidence remains substantially harder than single-table lookup.

The mixed Passage-Table family further reinforces this point. Although exact retrieval is very strong in both modalities—0.73 Passage Recall and 0.98 Table Recall—end-to-end accuracy reaches only 0.66. In other words, the main bottleneck is no longer finding the relevant evidence, but correctly integrating heterogeneous evidence across text and tables. Taken together, these results suggest that the Drug data lake is best characterized not as a retrieval-limited setting, but as a cross-modal synthesis setting in which joint reasoning over structured and unstructured biomedical evidence remains the main challenge.

6.4 Cross-Domain Takeaways↩︎

Across all three lakes, a consistent narrative emerges: high retrieval quality alone does not guarantee correct reasoning. We highlight two major takeaways for future system design:

The Divergence of Discovery vs. Synthesis: By isolating source discovery from synthesis using a controlled context-bundle ablation (App 14), we find that failure modes are strictly domain-dependent. When provided with perfect gold evidence, AI/ML accuracy saturates at 98.7%, proving that the AI/ML lake is purely a discovery bottleneck. Conversely, Drug lake accuracy only reaches 62.5% even with perfect evidence, demonstrating that cross-modal reasoning over dense biomedical tables remains fundamentally broken in LLMs, independent of retrieval performance.

The Compounding Cost and Latency of Agentic Navigation: While standard single-shot RAG cannot natively navigate the relational topologies of the Retail Bank and Drug lakes, agentic loop-based methods (e.g., XMode) introduce a compounding token and temporal overhead. Table 4 shows that the zero-shot baseline uses only ~0.05–0.07 k-tokens per query (~0.06 on average). In contrast, single-shot RAG requires retrieving and processing substantially larger context, increasing token usage to ~0.34–0.49 k-tokens per query. Transitioning to iterative agentic navigation compounds this further. As reflected by the proxy token values in Tables 6 and 5, iterative navigation increases token consumption significantly—in the Retail Bank lake, to roughly 1.00–1.27 k-tokens per query (about 5–8\(\times\) over zero-shot and up to ~3\(\times\) over standard RAG). Beyond token footprint, multi-step retrieval loops incur sequential latency bottlenecks, presenting a practical constraint for real-time operational environments like front-line customer service.

7 Limitations↩︎

Domain Scope and Synthetic Data: LakeQuest evaluates text and tables, excluding dynamic logs and raw audio/visual data (e.g., vision-model caveats rely on text metadata; see App 13). Furthermore, the Retail Bank domain relies on synthetic data for privacy, which may alter real-world noise distributions.

Model Monoculture and Evaluator Bias: Using an LLM to synthesize candidate questions risks introducing distributional bias that favors the generator’s reasoning patterns. We mitigated this via strict human validation. However, evaluating baselines and rubric-based correctness with the same LLM family (GPT-5) used for synthesis introduces a risk of self-preference bias [29], which may overstate measured baseline performance. Future evaluations using independent frontier models will help separate general architectural bottlenecks from model-specific priors.

Factual Redundancy: Data lakes inherently contain redundant facts. Because LakeQuest strictly tracks single-path gold evidence (\(E^*_i\)), systems might retrieve valid alternative provenance. Consequently, our retrieval metric serves as a conservative lower bound.

8 Conclusion↩︎

We introduced LakeQuest, a comprehensive benchmark and reproducible construction framework for grounded question answering over heterogeneous data lakes. Encompassing AI/ML, retail banking, and biomedical domains, the benchmark provides a shared, human-validated evaluation environment that explicitly pairs natural language questions with precise, modality-aware evidence pointers. Through baseline evaluations, we demonstrated that high retrieval quality alone is insufficient; modern retrieve-and-synthesize architectures fail differently across domains, struggling with relation chaining, policy grounding, and multimodal composition. Ultimately, LakeQuest provides the necessary evaluation infrastructure to drive the next generation of reliable, agentic QA systems capable of rigorous discovery and faithful attribution over real-world data lakes.

9 Ethics and Reproducibility Statements↩︎

9.0.0.1 Ethics Statement.

LakeQuest includes retail banking and biomedical QA tasks where hallucinated or weakly grounded answers pose significant real-world risks. Consequently, this benchmark explicitly evaluates Faithfulness alongside accuracy, and human validation ensures that all queries are natively answerable from the retrieved context. However, strong performance on LakeQuest does not constitute an endorsement for the autonomous deployment of these systems in high-stakes, real-world decision pipelines. LakeQuest is intended as a rigorous evaluation resource, not a substitute for domain expertise or human oversight.

For the human validation, we recruited and paid data annotators using the Prolific platform [30], ensuring that research ethics were upheld. We acknowledge that ethical considerations are extremely important for any research involving human subjects, and we have ensured that our research meets the Code of Ethics for COLM 2026. Our research plan was reviewed and approved by the ethics review board at our institution. In addition, we took steps to ensure research ethics including: reviewing our institution’s protocols for ethical research; only showing suitable content to participants; collecting consent from annotators regarding how the data would be collected, used, and retained; and paying all recruits a fair wage above the minimum wage in the jurisdiction of the recruitment platform.

9.0.0.2 Reproducibility Statement

To ensure the full reproducibility of our five-stage dataset construction pipeline and evaluation framework, we release all artifacts associated with this work. The LakeQuest benchmark, including human validation annotations, explicit evidence pointers, and the underlying data lake corpora (subject to their respective licensing), is hosted on Hugging Face Datasets at anonymousurl. Furthermore, our complete codebase—containing the automated generation pipeline, human-annotation UI, and all baseline implementations (RAG, XMode, ReSP) with exact prompts—is detailed in App 11 and will be open-sourced upon publication.

Acknowledgments↩︎

We are grateful to Layer 6 AI at TD and the Vector Institute for their support of this work. This research was also supported in part by the Natural Sciences and Engineering Research Council of Canada (NSERC) through Discovery Grant.

10 Detailed Dataset Statistics↩︎

This section provides a granular breakdown of the human validation results across all three data lakes. As described in Sec 3.3, all candidate question-answer pairs generated by the synthesis pipeline underwent rigorous human review. The tables below report the total number of synthesized candidates (Questions) alongside the number of instances that successfully passed three key human-annotation criteria:

• Correct: The reference answer is factually accurate and logically sound.

• AIS (Attributable to Identified Sources): The reference answer is strictly entailed by the provided evidence pointers, with no hallucinations or reliance on external knowledge.

• Answerable: The question provides sufficient context to be answered exclusively using the retrieved data lake artifacts.

Only instances that passed all criteria were included in the final benchmark.

10.1 AI/ML Data Lake Statistics↩︎

Table 7 presents the validation breakdown for the AI/ML repository. In this domain, questions are structurally categorized based on the underlying metadata entities (Model Cards vs. Dataset Cards) and whether the reasoning requires multi-hop entity resolution across the AI/ML graph.

10.2 Retail Bank Data Lake Statistics↩︎

Table 8 outlines the validation results for the synthetic Retail Bank data lake. Unlike the other domains, Retail Bank QA instances are categorized by operational personas, testing how well models can ground their reasoning in strict policy constraints under different institutional workflows.

10.3 Drug Data Lake Statistics↩︎

Table 9 details the validation outcomes for the Drug data lake. The task families here stress cross-modal synthesis, dividing questions based on whether the required evidence resides in unstructured scientific text (Passages), structured biochemical records (Tables), or a combination of both.

11 Prompt Templates↩︎

This section details the exact prompt templates used for dataset synthesis, LLM-as-a-judge filtering, and baseline evaluation.

11.1 Candidate Synthesis Prompts↩︎

To generate candidate QA pairs, we provided the language model with a sampled evidence subgraph and the following instructions. The system prompt was shared across synthesis settings, while the task prompt was instantiated with minor wording changes depending on whether the evidence subgraph contained a model–dataset pair, multiple related model cards, multiple related dataset cards, or an anchor-plus-positive-plus-distractor setup.

<question>
<direct_answer>
<explained_answer>

11.2 Baseline Evaluation Prompts: Agentic Tool Use (XMode)↩︎

For our agentic baselines, we framed the data lake navigation as an iterative tool-use task.

12 Human Annotation Guidelines and Interface↩︎

To ensure high data quality, annotators were provided with a custom web interface that displayed the generated question, the synthesized answer, and the exact supporting evidence chunks. Annotators were tasked with independently verifying two criteria: Answerability (can the question be answered exclusively using the cited evidence?) and Attribution (is the generated answer completely faithful to the cited text/table?). We emphasize that this task evaluates strict textual entailment and reading comprehension based entirely on the provided evidence, rather than requiring annotators to verify external scientific or financial validity.

• Is the answer correct? Mark “Yes” only when the answer matches facts stated in the provided evidence.

• Does the evidence support the answer? Mark “Yes” only when specific sentences/rows in the provided evidence explicitly support the answer.

• Any additional comments/thoughts. Add a short note explaining the decision, especially for “No” labels, and reference the relevant evidence span.

• Use only provided evidence. Do not use outside knowledge; if the answer contradicts the provided evidence, mark it incorrect and unsupported.

13 Extended Case Studies and Error Analysis↩︎

To qualitatively illustrate the quantitative failure modes discussed in Sec 6, we provide representative trace-level examples from the AI/ML and Drug data lakes. The cases below show three recurring error patterns: schema-confusable retrieval, relation chaining failure across closely related model cards, and multimodal metadata composition failure in vision–language datasets.

14 Controlled Context-Bundle Ablation↩︎

We ran a fixed context-bundle ablation on the AI/ML, Retail Bank, and Drug data lakes to reduce retrieval variance and better isolate answer synthesis. In each case, the model was given a fixed evidence bundle containing the gold evidence plus relevant distractors, and we evaluated both a direct answer and a reasoning answer using the same LLM-as-a-judge pipeline as in the main experiments.

4pt

The results show a sharp cross-domain difference. On AI/ML, performance is nearly saturated once gold evidence is present; Retail Bank is materially higher but not saturated, while Drug remains lowest under the same setup, indicating that synthesis over heterogeneous evidence remains a substantial challenge; reasoning is equal to or stronger than direct answers across lakes.

15 LLM-as-a-Judge Consistency Analysis↩︎

To assess the stability of our rubric-based LLM-as-a-judge evaluation, we repeated the same AI/ML data lake evaluation three times under an identical setup and report per-question-family correctness in Table 15. The results are highly consistent across runs, with only minor variation across the three repeated tests, suggesting that the judge-based correctness metric is sufficiently stable for the comparative analyses reported in the main paper.

4pt

16 Retail Bank Data Lake QA Generation Example↩︎

Figure 4 illustrates the end-to-end question-answer synthesis pipeline for the Retail Bank data lake. The process begins by sampling a dense evidence subgraph consisting of a client profile, a transaction ledger, and a governing institutional policy. The generator model is instructed to synthesize a realistic operational query (e.g., a fraud analyst checking a transaction limit) that strictly requires joining facts across all three sampled artifacts. Following generation, the pipeline programmatically filters deictic or ill-formed queries, and an LLM-as-a-judge verifies that the question cannot be answered via parametric memory alone. Finally, the instance undergoes human validation to ensure strict attribution to the highlighted evidence spans.

17 Open Source Model Results Ablation↩︎

References↩︎