1 Introduction↩︎

In emerging systems (e.g., social media and e-commerce platforms), natural-language interfaces have been increasingly employed to retrieve data records from heterogeneous sources, such as relational tables, textual documents, image repositories, spatial databases, and knowledge graphs (KGs). For example, a user may ask for chicken wing restaurants that are highly rated, commented with positive words, close to the current location, and visually similar to a referenced venue. To identify target records for such question-answering (QA) tasks, we have to jointly consider multiple heterogeneous sources, rather than individual sources which suffer from incomplete coverage and reliability.

Formally, in the record retrieval of heterogeneous sources, it often assumes there is a collection of records \(\mathcal{R}\), and each record \(r \in \mathcal{R}\) is drawn from multiple heterogeneous sources. Given a natural language question \(q\), it aims to return a ranked list of records matching with \(q\). A returned record is correct only when its source bundle satisfies all constraints expressed by \(q\); retrieving an isolated text, image, spatial, or relational clue is insufficient if the record violates another required constraint.

3.0pt

To evaluate record retrieval for QA tasks, several benchmarks have been developed. HybridQA [1] aligns questions with Wikipedia tables and linked passages, and OTT-QA [2] moves this table-text setting to open-domain QA over Wikipedia. MMQA [3] further combines text, tables, and images, but each question is built around a question-specific multimodal context rather than a full record collection for target-record retrieval. Spider2.0 [4] and BIRD [5] stress realistic database reasoning and executable text-to-SQL tasks, yet their outputs are SQL programs or database answers rather than ranked target records. STARK [6] evaluates open retrieval over semi-structured knowledge bases with textual and relational information. However, as summarized in Table 1, none of them has considered more than three sources, so they are still limited for benchmarking record retrieval for QA on multiple heterogeneous sources. Besides, they ignore the missing-value recovery: when answer sets start from a relational table, incomplete field values may leave valid target records outside the gold set. For example, in a relational table, a record may have a missing value for the field like parking, but data from other sources like reviews and photos still indicate that parking is available. Hence, it is desirable to develop a benchmark that is built from multiple heterogeneous sources with incomplete relational fields.

Our work. We introduce HeterQA, a benchmark with 857 QA pairs for record retrieval over five heterogeneous sources, and considering the missing-value recovery. We instantiate HeterQA with the Yelp business record collection \(\mathcal{R}\)¹. Particularly, each record contains 79 relational fields that are from the source of a relational table, and 4 other fields that are from the sources of text documents, image repositories, spatial databases, and KGs, respectively. The benchmark is built in an answer-driven manner. For each QA instance, we first sample a small set \(F\) of relational-field constraints and execute them over \(\mathcal{R}\) to obtain an initial answer set \(A\). Since relational-field values may be incomplete, we perform missing-value recovery: for each serialized field-value string \(f\in F\), we search the other source fields of records outside \(A\) and collect the recovered records as \(C_f\). This yields an enriched candidate set \(\mathcal{C}=A\cup(\bigcup_{f\in F}C_f)\). We then randomly select a record from \(\mathcal{C}\), choose one of its other source fields, extract a source-specific constraint, and verbalize this constraint together with \(F\) as a natural-language question \(q\). Finally, we verify which records in \(\mathcal{C}\) satisfy all constraints in \(q\); the candidate QA instance is retained only if the verified answer set is non-empty.

Further, we conduct human validation on sampled questions for naturalness, diversity, and practicality, obtaining non-negative rates of 97.0%, 87.4%, and 84.4%, respectively. Under the same evaluation metrics, we compare five kinds of retrieval methods: sparse retrieval, dense retrieval, hybrid retrieval (sparse + dense), late-interaction retrieval, and agentic retrieval. In retrieval experiments, hybrid retrieval obtains the best Recall@10 at 32.78 and Self-RAG obtains the best MRR@10 at 25.26; together with human validation, these results show that HeterQA is both high-quality and challenging for record retrieval over heterogeneous sources.

In summary, the key features of HeterQA are:

• Questions over five heterogeneous sources. HeterQA evaluates record retrieval over relational tables, text documents, image repositories, spatial databases, and KGs. Each question require methods to retrieve records that jointly satisfy all source-specific constraints.

• Answer-driven construction with missing-value recovery. HeterQA first initializes candidate answer records from relational-field constraints, then recovers additional valid records from other source fields when relational-field values are incomplete. The final answer set is verified before the natural-language question is retained. Because each additional source can be introduced as a new source-specific constraint and verified through its support set, the workflow can be extended to more source fields.

• Human-validated and challenging benchmark. Human validation supports the naturalness, diversity, and practicality of the released questions, while retrieval experiments show that current methods remain far from saturating HeterQA.

2 Related Work↩︎

2.1 Benchmarks over Individual Sources↩︎

Text-source benchmarks evaluate QA and retrieval over unstructured corpora. Natural Questions [7], TriviaQA [8], HotpotQA [9], MuSiQue [10], and MultiHop-RAG [11] cover open-domain, evidence-grounded, and multi-hop textual QA settings. BEIR [12] and KILT [13] make retrieval and provenance explicit over text corpora. These benchmarks are valuable for passage retrieval and textual grounding, but they do not require a returned record to satisfy non-textual source constraints.

Text-to-SQL benchmarks evaluate natural-language querying over relational databases. Spider [14] is a representative cross-domain text-to-SQL benchmark, and BIRD [5] extends database-grounded evaluation to larger database contents and external knowledge. However, none of the above benchmarks has considered the retrieval from multiple heterogeneous sources.

2.2 Benchmarks over Multiple Heterogeneous Sources↩︎

Several QA and retrieval benchmarks jointly consider more than one source. HybridQA [1] and OTT-QA [2] combine relational tables with linked textual evidence. MMQA [3] combines text, tables, and images, but its questions are built around question-specific multimodal contexts rather than full-collection target-record retrieval. M-BEIR [15] studies instruction-following multimodal retrieval over text, image, and interleaved text-image candidates, but it does not evaluate QA record retrieval over heterogeneous source bundles. STARK [6] evaluates open retrieval over semi-structured knowledge bases, where node entities are grounded in textual documents and relational graph structure. Spider2.0 [4] evaluates enterprise text-to-SQL workflows that require database metadata, SQL documentation, and project-level code. BIRD-INTERACT [16] evaluates dynamic multi-turn text-to-SQL interactions with metadata, user simulation, and executable test cases. However, as summarized in Table 1, none of them has considered more than three sources, so they are still limited for benchmarking record retrieval for QA on multiple heterogeneous sources. Besides, they ignore the missing-value recovery as aforementioned. Therefore, existing benchmarks are still insufficient for evaluating the record retrieval for QA over multiple hterogeneous sources.

3 Answer-driven Dataset Construction↩︎

Figure 2 depicts our answer-driven construction workflow. Each QA instance is constructed by a five-step approach: We first initialize a set \(F\) of relational-field constraints over \(\mathcal{R}\), then expand the candidate set to \(\mathcal{C}\), next instantiate source-specific constraints \(\mathcal{H}\), and finally audit the verified answer set \(\mathcal{V}_{\mathcal{H}}^\star\) for release. In the following, we introduce these steps sequentially.

Step 1: Relational Initialization and Missing-Value Recovery↩︎

For each QA instance, we sample a small set \(F\) of relational-field constraints by first selecting one Core field, then randomly selecting one or two fields from the union of the High-value and Supplementary tiers (Appendix 8.1). Each sampled field is paired with a value from \(\mathcal{R}\), and the resulting key-value constraints are compiled into a structured SQL query over the relational table. Executing this query over \(\mathcal{R}\) gives the initial answer set \(A\).

In the original relational table, since some fields’ values are incomplete, we expand \(A\) through missing-value recovery over three fields of non-relational sources, i.e., text, image, and KG. In Yelp, text consists of reviews and tips, image consists of business photos, and KG denotes a feature-centric heterogeneous graph \(\mathcal{G}\).² For each serialized field-value string \(f\in F\), we search records outside \(A\) through these source fields. For text recovery, we search \(f\) over all Yelp reviews and tips with text embeddings and rerank the matched texts; records (i.e., businesses in Yelp) above some threshold (see details in Appendix 8.2) form \(C_f^{\mathcal{T}}\). Images are processed by a multimodal embedding model and a visual reranker, producing \(C_f^{\mathcal{I}}\). For KG recovery, we search \(f\) over feature nodes in \(\mathcal{G}\) and rerank the matched features; their one-hop neighboring business records form \(C_f^{\mathcal{G}}\). We set \(C_f=C_f^{\mathcal{T}}\cup C_f^{\mathcal{I}}\cup C_f^{\mathcal{G}}\) and \(\mathcal{C}=A\cup(\bigcup_{f\in F}C_f)\). A recovered record is discarded if its relational fields explicitly contradict any constraint in \(F\).

Step 2: Source-Specific Constraint Instantiation↩︎

After forming \(\mathcal{C}\), we sample an anchor record and select one or two non-relational source fields, each yielding a source-specific constraint \(h\). For text, \(h\) is generated by sampling a review or tip and rewriting it into a concise requirement. For image, \(h\) verbalizes a visible property from a sampled business photo, such as sea view. For spatial information, we use the anchor record to define a local search area, then choose either a nearby business or a simulated user location sampled from that area as the reference point. We pair this point with a spatial operator to form \(h\), such as radius (e.g., within 2 km of a cafe), direction (e.g., north of a museum), or nearest-neighbor selection (e.g., the closest laundromat to the user location). For KG, we inspect the graph neighborhoods of candidate records in \(\mathcal{C}\) and sample two types of feature patterns as \(h\). Attribute patterns use direct business–feature links, such as requiring “friendly staff”. Collaborative patterns use shared user connections, such as venues with “quiet ambience” that are also favored by users who praise “desserts”.

Step 3: Candidate Filtering↩︎

Recall that in Step 2, the constraint \(h\) is instantiated from one specific anchor record, but it may be satisfied by many other candidate records in \(\mathcal{C}\), so we have to exclude those records that do not satisfy \(h\) from \(\mathcal{C}\). Specifically, for each selected \(h\), we define a support set \(\mathcal{V}_h\subseteq\mathcal{C}\). For non-spatial \(h\), verification is cross-source because \(h\) may be supported outside the source field that generated it. Text- and image-derived \(h\) are checked against the other source field through embedding retrieval, reranking, and LLM/VLM judgment using the relevance prompts in Figures 8 and  9; KG-derived \(h\) is checked on the graph and against text/image source fields with the same retrieval-and-judgment pipeline. A record enters \(\mathcal{V}_h\) if any applicable check passes; for spatial \(h\), \(\mathcal{V}_h\) is obtained deterministically from the reference point and geo operator. For the selected constraint set \(\mathcal{H}\), we keep records in \(\mathcal{V}_{\mathcal{H}}=\bigcap_{h\in\mathcal{H}}\mathcal{V}_h\) and carry their verification confidences forward for ranking.

Step 4: Question Generation and Contradiction Detection↩︎

After obtaining \(\mathcal{V}_{\mathcal{H}}\), we invoke an LLM to verbalize the relational constraints \(F\) and source-specific constraints \(\mathcal{H}\) into a natural-language question \(q\). However, some records in \(\mathcal{V}_{\mathcal{H}}\) may still contain contradictory evidence elsewhere in their source bundles, so we perform contradiction detection against the generated question. For each record in \(\mathcal{V}_{\mathcal{H}}\), the detector retrieves query-relevant reviews/tips and photos, then estimates text and image contradiction ratios with the prompts in Figures 5 and 6. Let \(\mathcal{V}_{\mathcal{H}}^\star\subseteq\mathcal{V}_{\mathcal{H}}\) be the records that pass contradiction detection: records above the contradiction threshold are removed, and unresolved cases are sent to manual review. The final set \(\mathcal{V}_{\mathcal{H}}^\star\) is the verified answer set and is ranked by the composite score in Appendix 8.2.

Step 5: Human Validation↩︎

Human validation is used in two stages. First, unresolved cases from contradiction detection are manually checked before finalizing the verified answer set \(\mathcal{V}_{\mathcal{H}}^\star\). Second, to assess question quality, we sample 200 questions with subset-proportional stratified sampling and ask 13 graduate annotators, including PhD and MPhil students, to rate question naturalness, diversity, and practicality. Following STARK [6], we also compute word entropy and Type-Token Ratio (TTR) to measure query diversity. Table 2 reports dataset composition, query-diversity metrics, and human-validation results. The resulting dataset contains 857 QA pairs across ten source-composition subsets. The released questions obtain a word entropy of 8.987 and a TTR of 0.160, and the non-negative rates for naturalness, diversity, and practicality are 97.0%, 87.4%, and 84.4%, respectively. These results indicate that the released questions are generally understandable, varied, and practically meaningful. Metric definitions are in Appendix 9, and construction settings are in Appendix 8.

2.5pt

4 Experiments↩︎

4.1 Experimental Setup↩︎

We evaluate all methods on the 857 QA pairs in HeterQA; each pair provides a question \(q\) and a verified answer set \(\mathcal{V}_{\mathcal{H}}^\star\). Given \(q\), a retriever searches \(\mathcal{R}\) using serialized relational fields, Yelp reviews/tips, and business photos, then returns a ranked record list. Location metadata is included in the serialized relational fields, so retrievers can access location cues without a separate spatial index.

We evaluate five method families. (a) Sparse retrieval uses BM25 [17]; it retrieves from the three sources and fuses source-specific record rankings with RRF [18]. (b) Dense retrieval uses DPR [19], ANCE [20], KALM [21], Llama-Embed-Nemotron-8B [22], Qwen3-Embedding-8B [23], and Qwen3-VL-Embedding-8B [24], following the same fusion procedure. (c) Hybrid retrieval adds BM25 as a parallel branch to each dense retriever and fuses sparse and dense record rankings with another RRF step. (d) Late-interaction retrieval includes ColBERT [25] and ColPali [26]; ColPali retrieves rendered Yelp-record pages and maps page hits back to records. (e) Agentic retrieval includes ReAct [27] and Self-RAG [28]. For photos, caption-based retrieval embeds VLM-generated captions, while direct-visual retrieval embeds the original business photos with Qwen3-VL-Embedding-8B. We report results without reranking (w/o R) and with reranking (w/ R), where w/ R uses Qwen3-VL-Reranker-8B [24]. Hit@5 measures whether the top-5 list contains at least one record in \(\mathcal{V}_{\mathcal{H}}^\star\), Recall@10 measures coverage of \(\mathcal{V}_{\mathcal{H}}^\star\), MRR@10 measures the rank of the first correct record, and latency is average end-to-end question time. All experiments are conducted on a Linux server with Intel Xeon 2.0 GHz CPUs, 1024 GB RAM, and eight NVIDIA GeForce RTX A5000 GPUs. The two agentic workflows use Qwen3.5-9B [29] as the backend model, with a 100K-token context budget and 4 concurrent workers. Vector-based retrieval is served by OceanBase HNSW indexes over three source stores: about 7.89M text entries, 0.20M photo entries, and 0.15M serialized relational-field entries. Detailed retrieval settings and agentic procedures are provided in Appendix 10 and Appendix 10.1.

4.0pt

4.2 Overall Retrieval Results↩︎

Table 3 evaluates whether methods can retrieve the verified answer set \(\mathcal{V}_{\mathcal{H}}^\star\) from the full record collection \(\mathcal{R}\). Given the natural-language question \(q\), each method returns a ranked list of records, and all metrics compare this list with \(\mathcal{V}_{\mathcal{H}}^\star\). The results reveal four main patterns.

Observation 1. Current methods recover only a limited part of the verified answer set. The best Recall@10 is 32.78, achieved by hybrid retrieval with Llama and reranking. This shows that even the strongest setting retrieves only part of \(\mathcal{V}_{\mathcal{H}}^\star\). Self-RAG obtains the best raw MRR@10 at 25.26, but its Recall@10 remains below the best hybrid retrieval result. Thus, current methods still struggle both to retrieve enough records in \(\mathcal{V}_{\mathcal{H}}^\star\) and to rank verified records near the top.

Observation 2. Hybrid retrieval contributes larger gains than reranking. Hybrid retrieval consistently improves over the corresponding base retrievers. For dense retrievers, the average Recall@10 gain from reranking is 2.73 points, while the average Recall@10 gain from hybrid retrieval under w/ R is 10.37 points. The same pattern holds for MRR@10: reranking adds 1.57 points on average, while hybrid retrieval adds 8.92 points. For Llama, reranking improves Recall@10 from 21.44 to 24.28, whereas hybrid retrieval further increases it to 32.78. These results suggest that expanding the retrieved record set through hybrid retrieval matters more than reranking the initial dense-retrieval results.

Observation 3. Agentic retrieval improves top-rank metrics but is not a recall solution. Self-RAG obtains the best raw Hit@5 and MRR@10, reaching 30.69 and 25.26 w/ R, respectively. However, its Recall@10 is 25.77, still below the best hybrid retrieval result of 32.78. The gain also comes with a large latency cost: Self-RAG takes 108.48 seconds per question w/ R, compared with 5.56 seconds for hybrid retrieval with Llama under the same setting. Figure 4 further shows that increasing the turn budget does not monotonically improve retrieval. Thus, agentic retrieval can improve early correct hits, but higher latency and more turns do not reliably recover more records in \(\mathcal{V}_{\mathcal{H}}^\star\).

Observation 4. Late-interaction retrieval does not overcome the record-level difficulty. ColBERT reaches 27.09 Recall@10 w/ R, which is close to BM25 at 27.21 but does not improve over it. This suggests that token-level interaction over text-like content is not enough to solve HeterQA, even when serialized fields, reviews/tips, and photo captions are available. ColPali performs much worse: after each Yelp business is rendered into PDF pages, it reaches only 0.95 Recall@10 w/o R. These results show that current late-interaction baselines do not directly address the target-record retrieval setting, where a method must return records satisfying constraints across heterogeneous sources.

Observation 5. Visual photo embeddings remove the need for caption conversion. Qwen3 reaches 21.64 Recall@10 w/ R on caption-based photo retrieval, comparable to Qwen3-VL (caption) at 21.71 and Qwen3-VL (photo) at 20.89. After hybrid retrieval, the Recall@10 scores remain close: 31.09 for Qwen3, 29.13 for Qwen3-VL (caption), and 29.80 for Qwen3-VL (photo). This suggests that visual photo embeddings achieve comparable retrieval quality to caption-based embeddings while avoiding the extra VLM captioning step.

4.3 Source-Combination Analysis↩︎

Figure 3 shows how retrieval behavior changes across source-combination subsets. Panel (a) reports reranking gains, and Panel (b) reports hybrid retrieval gains over the corresponding dense retriever.

Observation 6. Reranking gains vary across source combinations and methods. Reranking is not uniformly beneficial across subsets. For BM25, reranking gives large Recall@10 gains on text-bearing combinations, including +14.78 on \(Q_{S+\mathcal{T,I}}\), +12.62 on \(Q_{S+\mathcal{T,P}}\), and +12.22 on \(Q_{S+\mathcal{T,G}}\), but only +3.09 on \(Q_{S+\mathcal{P,G}}\). A similar dependence appears for ColBERT, which gains +11.59 on \(Q_{S+\mathcal{T,I}}\) and +10.23 on \(Q_{S+\mathcal{I,P}}\), but loses 1.52 points on \(Q_{S+\mathcal{G}}\). These patterns show that reranking helps when the retrieved candidates already contain compatible source cues, but it is not a uniform fix for source-combination difficulty.

Observation 7. Hybrid retrieval gives broad gains, but spatial-KG cases remain hard. Hybrid retrieval improves Recall@10 across several dense retrievers, especially on subsets involving text documents. Under w/ R, adding BM25 improves DPR by +30.00 on \(Q_{S+\mathcal{T,P}}\), +27.62 on \(Q_{S+\mathcal{T,I}}\), and +23.47 on \(Q_{S+\mathcal{T,G}}\). ANCE shows the same trend, with +27.10 on \(Q_{S+\mathcal{T,P}}\) and +18.57 on \(Q_{S+\mathcal{T,G}}\). For stronger retrievers, the gains remain visible: Qwen3 gains +13.79 on \(Q_{S+\mathcal{T,P}}\), and Qwen3-VL (photo) gains +11.75 on \(Q_{S+\mathcal{T,G}}\). However, gains on \(Q_{S+\mathcal{P,G}}\) are smaller for most retrievers, such as +4.32 for DPR, +4.94 for ANCE, +3.26 for Llama, and +0.79 for Qwen3-VL (photo). Thus, hybrid retrieval helps recover records missed by dense retrievers, but questions combining spatial constraints and KGs remain difficult.

4.4 RAG Experiments↩︎

We evaluate downstream RAG record selection with four generator backends: Qwen3.5-9B, Qwen3.5-27B [29], Gemma4-31B [30], and GLM-4.6V-Flash [31]. For each question, the retriever first produces a top-30 record list after record-level collapse; the generator LLM receives these records with associated source content and outputs the selected records. Detailed RAG evaluation setup and prompt templates are provided in Appendix 11 and Appendix 12.

4.2pt

Observation 8. Retrieval improvements only partially transfer to RAG record selection. Table 4 evaluates whether retrieved records can support downstream record selection by generators. Hybrid retrieval often improves Macro-F1, with the largest gain being +3.43 for Gemma4-31B with Llama w/ R. However, the absolute scores remain low: the strongest raw score is 13.36, and even the best hybrid total remains far below the retrieval Recall@10 in Table 3. Thus, HeterQA also exposes a downstream bottleneck: after retrieval, generators still struggle to select the correct verified record set from the returned records.

5 Conclusion↩︎

HeterQA benchmarks record retrieval for QA over five heterogeneous sources: relational tables, text documents, image repositories, spatial databases, and KGs. It contains 857 QA pairs instantiated from Yelp business records, where each question combines two or three sources. The answer-driven construction initializes relational-field constraints, expands candidates through missing-value recovery, instantiates source-specific constraints, and applies contradiction detection plus human validation before finalizing the dataset. The experiments show that HeterQA remains challenging for current retrieval methods. The best setting, hybrid retrieval with Llama and reranking, reaches only 32.78 Recall@10, while Self-RAG obtains the best MRR@10 at 25.26 but does not achieve the best Recall@10. Source-combination and RAG analyses analyses further show that current methods struggle with spatial-KG cases, source-specific constraint composition, and downstream record selection. These findings leave substantial room for methods that can retrieve and verify records across heterogeneous sources. In the future, we will study more capable methods for retrieving records from heterogeneous sources.

6 Limitations↩︎

HeterQA is instantiated from Yelp business records, so its current scope is local business search rather than all record-centric applications. Although the construction workflow can be extended by adding new source-specific constraints \(h\) and support sets \(\mathcal{V}_h\), new domains or source fields still require source-specific preprocessing and verification rules. Finally, construction relies on LLM/VLM judgments, threshold-based filtering, and manual checks for verification and contradiction detection, which introduces annotation cost, token cost, and possible residual noise.

7 KG Construction Details↩︎

This section describes the feature-centric graph \(\mathcal{G}\) used for KG source fields in HeterQA.

Yelp reviews describe subjective business features rather than clean encyclopedic facts. We therefore construct a sentiment-aware heterogeneous graph \(\mathcal{G}=(\mathcal{V},\mathcal{E})\) with three node types: users (\(U\)), businesses (\(B\)), and extracted features (\(F\)), such as “quiet ambiance” or “friendly staff.” The graph includes review edges \(U\to B\) and polarity-aware feature edges \(U\to F\) and \(B\to F\), which store confidence scores and allow KG constraints to capture review-implied requirements that are absent from relational fields. We extract typed (feature, polarity, confidence) tuples from about 900M raw review tokens with GPT-OSS-120B and a seed ontology. Because LLM extraction can split one concept into surface variants (e.g., “24-hour service” and “open 24 hours”), we canonicalize synonymous and near-duplicate feature strings before graph construction. This reduces the feature inventory from 184,009 raw strings to 53,832 canonical features.

6pt

8 Construction Models and Settings↩︎

Table 6 summarizes the model roles and thresholds used during dataset construction. These models are separate from the retrieval baselines evaluated in Section 4.

4pt

8.1 Relational-Field Sampling Tiers↩︎

Table 7 lists the relational-field tiers used to form \(F\) during relational initialization.

4pt

8.2 Contradiction Detection and Answer-Set Ranking↩︎

After support-set filtering, the construction auditor removes records whose source bundle contradicts the final question \(q\). For each retained record \(r\), we retrieve question-relevant reviews/tips and photos using the embedding of \(q\). The prompts in Figures 5 and 6 label each retrieved item as compatible or contradictory; neutral, irrelevant, or ambiguous items are treated as compatible unless they explicitly negate a requirement in \(q\). Let \(\rho_{\mathcal{T}}(r)\) and \(\rho_{\mathcal{I}}(r)\) be the contradiction ratios over the retrieved text and image items. A record is removed if either ratio exceeds the contradiction threshold \(\tau_c=0.15\) in Table 6. Surviving records are ranked using the verification confidences produced during construction. Let \(\mathcal{J}(r)\) be the available verification channels for record \(r\), including spatial, KG, text, image, and cross-source checks. The cross-source checks verify text-derived constraints with photos and image-derived constraints with reviews/tips. Given the confidence score \(c_j(r)\) from channel \(j\), we compute \[s(r)=\sum_{j\in\mathcal{J}(r)} c_j(r)-\rho_{\mathcal{T}}(r)-\rho_{\mathcal{I}}(r).\] A higher score means that a record has stronger source support and fewer detected contradictions, so it should be ranked higher. Records in the verified answer set are ranked by \(s(r)\) in descending order.

None

None

9 Human Validation and Query-Diversity Metrics↩︎

We compute the query-diversity metrics in Table [tbl:tab:quality95audit] on HeterQA following STARK [6]. Word Entropy measures lexical distributional diversity, defined as \(H=-\sum_w p(w)\log_2 p(w)\) over the word distribution of questions. Type-Token Ratio (TTR) measures lexical variety, defined as \(\mathrm{TTR}=|V|/M\), where \(V\) is the set of unique words and \(M\) is the total number of words.

For human validation, we use the naturalness, diversity, and practicality dimensions from STARK [6] as a protocol reference. HeterQA contains 857 questions across ten source-composition subsets. We sample 200 questions with subset-proportional stratified sampling and ask 13 graduate annotators, including PhD and MPhil students, to rate the anonymized questions on a 1–5 Likert scale. For each dimension, we compute subset-level rates and aggregate them with subset weights. The positive rate uses score \(\ge 4\), and the non-negative rate uses score \(\ge 3\). Table [tbl:tab:quality95audit] reports the resulting query-diversity and human-validation results.

10 Evaluation-Time Retrieval Implementation↩︎

Algorithm [alg:retrieval95system] summarizes the retrieval implementation. Each method retrieves source-level content, maps it back to records in \(\mathcal{R}\), and scores the final ranked records against \(\mathcal{V}_{\mathcal{H}}^\star\).

5pt

10.0.0.1 ColPali input.

For ColPali, each Yelp record is rendered into PDF pages containing serialized relational fields, reviews/tips, and photo thumbnails. ColPali retrieves page-level hits, which are collapsed back to records before scoring. Figure 7 shows the rendered input.

10.1 Agentic Retrieval↩︎

ReAct. We instantiate ReAct by interleaving reasoning and tool actions [27]. At each step, the model observes the current retrieval state, selects one tool action, and receives the updated record list and source-field content.

Self-RAG. We implement Self-RAG as an inference-time reflection policy inspired by Self-RAG [28], without task-specific retraining. The model emits a reflection label to decide whether to retrieve more records, reviews, or photos, critique current source-field content, or finish.

Both agentic methods use the same tool inventory in Table 9; they differ only in the policy for selecting the next tool.

4pt

11 RAG Experiment Details↩︎

The RAG experiment evaluates downstream record selection. For each question, the generator receives \(q\), an ordered top-30 record list after record-level collapse, and the available source content for each record. Reviews/tips are provided as text blocks, and photos are supplied as image inputs when available. The input is therefore a record-level list, not the final top-10 retrieval output.

The generator returns business names in JSON because target records in the Yelp instantiation are businesses. We resolve those names to unique business identifiers within the returned record list and score them against the verified answer set. Let \(P_i\) be the predicted business-id set and \(G_i\) the verified business-id set for question \(i\). The per-question set-F1 is \[\mathrm{F1}_i=\frac{2|P_i\cap G_i|}{|P_i|+|G_i|}.\] Macro-F1 averages \(\mathrm{F1}_i\) over the 857 questions. Exact-set-match accuracy assigns 1 only when \(P_i=G_i\) as unordered sets. Generation errors are counted in the 857-question denominator.

The prompt template used for the RAG record-selection task is listed with the other prompt templates in Appendix 12.

12 Prompt Templates↩︎

The following prompt templates support RAG evaluation and construction reproducibility.

None

None

None

References↩︎

---

1. https://www.yelp.com/dataset or https://business.yelp.com/data/resources/open-dataset/↩︎

2. We construct \(\mathcal{G}\) from Yelp reviews and tips by prompting an LLM to extract feature nodes and link them to users and businesses. Each link stores positive/negative sentiment to distinguish constraints such as “good service” from “poor service,” together with extraction confidence. The graph contains 237,082 nodes and 4,011,327 edges; details are in Appendix 7.↩︎