1 Introduction↩︎

Large language models (LLMs) have achieved strong performance across diverse language and knowledge-intensive tasks [1]–[3], but remain prone to hallucinations [4], [5], factual inconsistency [6], and multi-hop reasoning failures [7]. While retrieval-augmented generation (RAG) [7], [8] mitigates these challenges through external grounding, text-based RAG systems exhibit limitations with complex multi-hop reasoning as relational structure is only implicitly represented in textual contexts [9], [10]. Incorporating symbolic knowledge, such as KGs, during retrieval preserves relational semantics and improves performance on complex multi-step questions [9], [11]. Fact‑retrieval approaches [12], [13] retrieve KG subgraphs according to their relevance to an input question and perform LLM reasoning to find an answer. These approaches become inefficient when query‑relevant subgraphs exceed the LLM’s context window [14]. For example, answering the question "How many people live in cities in the vicinity of the Nile?" requires structured aggregation over many entities and relations, which is difficult to perform reliably through direct reasoning over retrieved triples alone.

In such cases, generating executable SPARQL queries provides a more scalable and interpretable alternative. However, systems fine-tuned on Question-SPARQL pairs [15]–[17] face challenges in SPARQL query generation [18] and are limited by training‑data coverage and diversity. More recent agent-oriented approaches leverage code-pretrained LLMs for SPARQL composition and graph exploration [18], [19]. These systems are vulnerable to inconsistent behavior due to their heavy reliance on LLMs, and thus, to hallucination at each step [20]. Graph-based RAG approaches [9], [11] often retrieve and reason over graph neighborhoods or raw triples. Such methods may require prior graph summarization and subgraph indexing, which introduce significant computational and storage costs in large-scale settings [21]. Moreover, as the number of entities and relations in the KG grows, the candidate retrieval space can expand rapidly, making efficient and precise retrieval increasingly difficult [4], [11], [22].

To address these limitations, we propose MARS, a SPARQL‑based KGQA approach that combines pattern-based graph retrieval with context-augmented knowledge for knowledge-driven reasoning without requiring fine-tuning. MARS constrains the retrieved context by filtering and ranking triple patterns rather than operating directly on subgraphs or raw triples. By combining retrieved pattern instances and typological schema information, MARS provides a KG‑grounded context based on a triple pattern set that enhances the LLM’s capacity for SPARQL generation. When needed, it expands this set by extracting new graph structures, thereby maintaining graph alignment throughout the multi-hop generation process. Evaluated across three benchmark datasets, MARS achieves state-of-the-art performance on the KGQA task without fine-tuning, consistently outperforming robust baselines. Notably, on the highly complex QALD-10 benchmark, with over 29% of the queries requiring multiple hops, our approach outperforms agentic alternatives across all evaluated languages. Furthermore, we address a critical reproducibility challenge in KGQA: benchmark datasets are often released without the specific KG version used during their creation [23], [24], or with versions lacking the coverage required to answer the queries. For instance, the Wikidata [25] truthy dump provided by QALD-10 [26] omits essential property statements and qualifiers for some of the benchmark queries. To ensure fair comparison and reproducibility for future work, we release the underlying Wikidata snapshot alongside the updated datasets and employ open-weight LLMs to ensure our experiments are completely reproducible. Our work makes the following key contributions:

1. A KGQA approach that curtails LLM context size with pattern‑based graph retrieval and iteratively incorporates next‑hop facts into LLM reasoning,

2. A hybrid multilingual setup that uses translation alongside the native question, covering 10 languages, including four low‑resource languages, across three KGQA benchmark datasets,

3. An in-depth ablation study on features and configurations, and an error analysis of our approach and results,

4. Reproducibility and open access for all results, code, and resources.

In the following section, we review related work before we describe our approach in detail in Section 3. Section 4 outlines our experimental setup, followed by our evaluation and result discussion alongside error analysis in Section 5. We cover our ablation study in detail in Section 6. We conclude our paper in Section 7.

2 Related Work↩︎

2.1 Knowledge Graph Question Answering↩︎

Recent work explores the integration of KGs in LLMs for KGQA [13], [27], [28]. Although LLMs show promising results in reasoning [29], integrating structured data into KGQA remains challenging [18]. Context-augmented approaches [30]–[33] with autoregressive models demonstrate improved performance in zero-shot and few-shot scenarios. Relevant evidence collected from structured data with an iterative reading-then-reasoning approach in zero-shot and few-shot scenarios achieved promising results with LLMs [31]. ChatGPT [34] and Llama‑based models [35] generate syntactically correct SPARQL queries and reduce computational cost (e. g., on DBpedia [36]) but remain below optimal performance on complex Wikidata queries, non‑readable identifiers and non‑English inputs [32], [37]. Data augmentation using KG axioms improves coverage and performance [38], while models like Mistral [1] and GPT-3.5 outperform T5 [39] on LC-QuAD1.0 and QALD-9 [40]. Recent advances include RAG-based federated SPARQL query generation [41], dynamic few-shot learning [42], modular designs with RAG layers [43], and LLM SPARQL query generation under different prompting conditions [44]. Prompt-based methods reduce labeled data dependency but are sensitive to prompt design [45]. Agentic frameworks such as mKGQAgent [46], Agentict\(^2\)s [47], SPINACH [19] and GRASP [18] are vulnerable to error propagation due to their heavy reliance on LLMs with hallucinations. However, many existing frameworks perform a lossy and unidirectional conversion of non-English queries into English prior to retrieval [13], largely because downstream KGQA components for entity linking are predominantly optimized for English [48]. While machine translation offers a viable approach [48], [49], processing queries directly in the user’s native language is preferable for preserving the original semantic intent [16].

MARS employs a hybrid strategy that combines machine translation and entity-type augmentation [50], [51] with the original native-language text. This allows MARS to preserve the query’s original structure for semantic pattern matching while simultaneously leveraging the advantages of machine translation.

2.2 Reasoning in Large Language Models↩︎

LLMs have demonstrated strong reasoning abilities over provided contexts [52]–[54]. Notable examples such as Gemini [55], Claude [56], and DeepSeek‑R1 [57] achieve state‑of‑the‑art performance on a wide variety of reasoning tasks [58]. LLM reasoning can be classified as implicit or explicit. Implicit reasoning produces the final answer directly and is computationally efficient, whereas explicit reasoning first generates an intermediate “thinking” context before generating the answer, which improves transparency and explainability [59]. Leading implicit-reasoning systems include Gemma [2], Qwen [3], and Mistral. Among explicit‑reasoning approaches, state‑of‑the‑art models include GPT‑OSS, DeepSeek‑R1, Llama, and GLM‑4.5 [60], [61].

Rather than requiring the LLM to act as an autonomous agent that selects tools at runtime  [18], which can increase hallucinations, MARS supports a deterministic, pattern-guided abstraction loop that allows the model to concentrate solely on path pruning and the final query generation. Moreover, we focus on open‑weight models for reproducibility.

3 Our Approach↩︎

3.1 Task Definition and Preliminaries↩︎

To address the problem of answering natural‑language questions over RDF knowledge graphs, we first formalize the inputs and objectives of our approach.

3.1.1 Knowledge Graph Question Answering↩︎

Given a natural language question \(q_{text}\) and an RDF knowledge graph \(\mathcal{G} \subseteq \mathcal{E} \times \mathcal{P} \times (\mathcal{E} \;\cup\;\mathcal{L} \;\cup\;\mathcal{C})\) that consists of triples \((s, p, o) \in \mathcal{G}\), where \(\mathcal{E}\) denotes a set of entities, \(\mathcal{P}\) a set of predicates (i. e., relations), \(\mathcal{L}\) a set of literals, and \(\mathcal{C}\) a set of classes, the main objective of our approach is to answer the question \(q_{text}\) based on \(\mathcal{G}\) by expressing it as a SPARQL query \(q_{sparql}\), the standard query language for RDF KGs: \[\begin{align} q_{sparql} &= \mathrm{KGQA}(q_{text}, \mathcal{G}) \label{eq:sparql95gen}. \end{align}\tag{1}\] Executing this generated query against \(\mathcal{G}\) yields the answer set \(\mathcal{A}\): \[\begin{align} \mathcal{A} &= \mathrm{Answer}(q_{sparql}, \mathcal{G}).\label{eq:approach95answer} \end{align}\tag{2}\] This answer set \(\mathcal{A}\) represents the solution to question \(q_{text}\) found within the knowledge graph \(\mathcal{G}\).

3.1.2 Knowledge Graph Schema↩︎

Our system operates on a given KG, thus we rely on a given Terminological Box (TBox) that defines the schema of the KG, including axioms \(\langle . \rangle\) that specify domain and range constraints for predicates. We denote the domain and range classes of a predicate \(p\) with \(\text{domain}(p)=\{ c \in \mathcal{C} | \langle p, \text{rdfs:domain}, c \rangle \in \text{TBox}\}\) and \(\text{range}(p)=\{ c \in \mathcal{C} | \langle p, \text{rdfs:range}, c \rangle \in \text{TBox}\}\). Furthermore, let each term \(t \in \mathcal{T} = \mathcal{E} \cup \mathcal{P} \cup \mathcal{C}\) assigned a label \(\ell:\mathcal{T} \to \mathcal{L}\) with \(\langle t , \text{rdfs:label}, l \rangle \in TBox\), where \(l \in \mathcal{L}\) is a literal value, i. e., human-readable name or description. We denote a label \(l\) of a term \(t\) with \(\ell(t)\).

3.2 Pipeline Overview↩︎

We propose a three-stage pipeline for multilingual KGQA: entity extraction, pattern extraction,¹ and a final stage that either generates the output query or expands the context. In the latter case, the newly retrieved resources enrich the context, and the pattern extraction stage is re-invoked to capture next-hop information. This loop continues until the query is fully grounded, at which point the final SPARQL query is generated.

An overview of our MARS pipeline is depicted in Figure 1, using the example question: “When was the creator of Saturday Night Live born?” (from the QALD-10 dataset). The process begins with entity extraction and moves into the iterative reasoning loop (highlighted in light purple), where the system retrieves patterns and enriches context until the query is grounded. The final output is a generated SPARQL query, with the process flow indicated by the arrows.² We provide details for each step of our approach in the following.

3.3 Entity Linking and Text Augmentation↩︎

We first identify entities \(E \subseteq \mathcal{E}\) in \(q_{text}\) that exist within \(\mathcal{G}\) using the following operation: \[E = \mathrm{ExtractEntities}(q_{text}, \mathcal{G}). \label{eq:approach95el}\tag{3}\] \(\mathrm{ExtractEntities}(\cdot)\) is implemented using an entity linking pipeline based on GRASP [18].³ To further support the graph traversal, we adopt a text augmentation strategy [50], [51] that utilizes an instruction-tuned LLM to generate \(q^{\mathrm{\small aug}}_{text}\), which enriches the query context by incorporating predicted entity type labels. To support non-English queries in both of these steps, we employ a multilingual LLM [2] for translation.

3.4 Pattern Retrieval and Filtering↩︎

The aim of this step is to retrieve a set of patterns to extract first-hop information from \(\mathcal{G}\) about the extracted entities \(E\), thus about the question \(q_{text}\).

Using a set of patterns, instead of full triples, significantly reduces the amount of context retrieved from the graph. Each extracted pattern consists of a predicate, an entity, and a placeholder variable. The entity is positioned as either subject or object depending on the edge’s direction. If the entity functions as the subject, the placeholder serves as the object, and vice versa. We impart this functionality by defining a set of patterns:

\[\mathfrak{P} = \left\{ (s, p, \_) \mid s \in E, (s, p, o) \in \mathcal{G}\, \right\} \cup \left\{ (\_, p, o) \mid o \in E, (s, p, o) \in \mathcal{G}\, \right\}. \label{eq95pattern}\tag{4}\] The retrieved patterns are then verbalized (\(\oplus\) denotes string concatenation): \[\begin{align} \mathsf{V}_{\mathfrak P} &= \{\, \mathsf{v}(\mathfrak{p}) \mid \mathfrak{p}\in \mathfrak P \,\}, \text{with} \\ \mathsf{v}(s,p,o) &= \begin{cases} \operatorname{\ell}(s) \oplus \operatorname{\ell}(p) \oplus \texttt{"?object"} & \text{if } o = \_,\\[1mm] \texttt{"?subject"} \oplus \operatorname{\ell}(p) \oplus \operatorname{\ell}(o) & \text{if } s = \_,\\[1mm] \end{cases} . \end{align} \label{eq:verbalize95patterns}\tag{5}\] We then convert each verbalized pattern in \(\mathsf{V}_{\mathfrak P}\) and the input question \(q_{\text{text}}\) (or \(q^{\mathrm{\small aug}}_{text}\) depending upon the configuration, see Section 6) into \(d\)-dimensional dense embedding vectors \(\Phi_{\mathfrak P}=\{ \phi(\mathsf{v}) |\mathsf{v} \in \mathsf{V}_{\mathfrak P} \}\) and \(\phi(q_{\text{text}})\) with the text embedding function \(\phi: \Sigma^* \to \mathbb{R}^d\). These vectors are then ranked by their semantic similarity, i. e., dot product, to the vector of the input question: \[\begin{align} \mathfrak{S}_{\mathfrak P} &= \mathrm{SemanticSimilarity}(\Phi_{\mathfrak P}, \phi(q_{\text{text}})). \end{align} \label{eq:semantic95similarity}\tag{6}\] Finally, we select the top-\(N\) patterns with \(\mathrm{\small topn} \in \mathbb{N}^+\): \[\begin{align} \mathfrak{P}_\text{top} &= \mathrm{SelectTopN}({\mathfrak P}, \mathfrak{S}_{\mathfrak P}, \mathrm{\small topn} ). \end{align} \label{eq:select95top95n}\tag{7}\]

3.5 Enrichment and Multi-Hop Reasoning↩︎

Once the top-\(N\) patterns are obtained, we extract features that capture information critical for pattern understanding. These features include retrieving instance values, frequencies, and typological information based on the pattern’s properties. Let \(\mathfrak{p}\) be an example pattern drawn from the set \(\mathfrak{P}_\text{top}\). We define these features in the following.

3.5.1 Pattern Instances↩︎

We retrieve all instances associated with variables in \(\mathfrak{p}\): \[\begin{align} \operatorname{I}_{\mathfrak{p}} = \{\, o \mid \mathfrak{p}= (s,p,\_),(s,p,o)\in\mathcal{G}, o \in \mathcal{E} \} \cup \{\, s \mid \mathfrak{p}= (\_,p,o),(s,p,o)\in\mathcal{G}\,\} . \\[2mm] \end{align} \label{eq:pattern95instances95all}\tag{8}\] We operate based on the pattern structure. The first case, \(\mathfrak{p}= (s,p,\_)\), extracts all objects (\(o\)) related to subject \(s\) and predicate \(p\). The second case, \(\mathfrak{p}= (\_,p,o)\), extracts all subjects (\(s\)) related to predicate \(p\) and object \(o\).

3.5.2 Typological Information↩︎

A verbalization function \(\mathsf{t}(p)\) for typological information of the pattern \(\mathfrak{p}\) based on the domain and range of its predicate \(p\) within the pattern: \(\mathsf{t}(p)=\texttt{"domain:"} \bigoplus_{c \in \text{domain}(p)} \ell(c) \texttt{", range:"} \bigoplus_{c \in \text{range}(p)} \ell(c)\).

3.5.3 Enriched Verbalization↩︎

We then generate an enriched verbalization incorporating these features that can be defined as:

\[\begin{align} \mathsf{V}^{\text{enriched}}_{\mathfrak{P}_\text{top}} &= \Bigl\{\,\mathsf{v}(\mathfrak{p}) \;\oplus\; n \;\oplus\; L \;\oplus\; \operatorname{t}(p) \;\Big|\; \mathfrak{p}=(s,p,o) \in\mathfrak{P}_{\text{top}} \Bigr\}, \\[2mm] \end{align} \label{eq:enrich95verbalization95compact}\tag{9}\] with \(n=|\operatorname{I}_{\mathfrak{p}}|\), the pattern frequency by simply counting the number of instances, and with \(L = \bigoplus_{j \in I_i} \ell(j)\), the concatenation of instance labels, where we focus on instance subsets to reduce computational cost: \(\operatorname{I}_i \subseteq\operatorname{I}_{\mathfrak{p}}\), with \(|\operatorname{I}_i|=i\in \mathbb{N}^+\).

3.5.4 SPARQL Query Generation and Reasoning↩︎

This enriched verbalization effectively combines schema information with concrete instances, revealing the question-relevant context of the graph. The enriched verbalization and the original question are then passed to a SPARQL Reasoner⁴, i.e., an LLM, preferably one with strong reasoning and coding capabilities:

\[\begin{align} &\operatorname{SPARQLReasoner}\Bigl(q_{\text{text}},\, \mathsf{V}^{\text{enriched}}_{\mathfrak{P}_{\text{top}}}\Bigr)\;=\; \begin{cases} \displaystyle q_{\text{sparql}} & \text{if } \operatorname{qgen}\Bigl(q_{\text{text}},\, \mathsf{V}^{\text{enriched}}_{\mathfrak{P}_{\text{top}}}\Bigr), \\[2.2ex] \displaystyle \mathfrak{P}_{\text{sel}} & \text{otherwise}. \end{cases} \end{align} \label{eq:sparql95reasoner95conditional}\tag{10}\]

The reasoner determines whether it has sufficient context to generate the final SPARQL query \(q_{\text{sparql}}\), which we represent as a boolean function \(\operatorname{qgen}(\cdot)\). If the reasoner cannot generate a query, it selects the specific patterns \(\mathfrak{P}_{\text{sel}} \subseteq \mathfrak{P}_{\text{top}}\) that are then expanded in the next round of pattern extraction.

We define the expanded node set \(E_{\text{vis}} \subseteq \mathcal{E}\) from an entity set \(E\) and top patterns \(\mathfrak{P}_{\text{top}}\) with \(E_{\text{vis}} = \bigcup_{ \mathfrak{p} \in \mathfrak{P}_{\text{top}}}^{}{ \operatorname{I}_{\mathfrak{p}}} \cup E\). The non-visited nodes are then extracted as \(E_{\text{next}} = E_{\text{sel}} \setminus E_{\text{vis}}\), where \(E_{\text{sel}} = \{ s | (s,p,\_) \in \mathfrak{P}_{\text{sel}} \} \cup \{ o | (\_,p,o) \in \mathfrak{P}_{\text{sel}}\}\) is a set of all entities inside \(\mathfrak{P}_{\text{sel}}\). Then, we set \(E:= E_{\text{next}}\) and repeat the pattern extraction process to extract next-hop information \(\mathfrak{P}^{next}\) with 4 . Afterward, we select the top-\(N\) from these \(\mathfrak{P}_{top}^{next}\) with 7 .

Finally, we repeat the process in this section by adding these new patterns to the accumulated list and re-attempting SPARQL query generation with 10 : \[\begin{align} \mathfrak{P}_{top} = \mathfrak{P}_{top} \cup \mathfrak{P}_{top}^{next}, \end{align} \label{eq:repeat95loop}\tag{11}\] until a SPARQL query is successfully generated or a pre-defined multi-hop limit, i. e., mhop, is reached.

The final values for variables \(d\) and \(i\), along with other configuration details, are presented in Section 4.2. Further investigation into optimal values for topn and mhop is conducted as part of our ablation study in Section 6.

Our prompting format draws inspiration from earlier retrieval and reasoning approaches [13], [18], yet it has been adapted to suit our specific use case. While the instructions are written in English, the question itself is always retained in its original language.

4 Experimental Setup↩︎

4.1 Datasets↩︎

We evaluate our approach on three widely used Wikidata-based KGQA benchmark datasets, each addressing distinct evaluation dimensions. LCQuAD2.0 [23] provides a large-scale English-only benchmark (30k training, 6k test), establishing baseline performance for standard queries. QALD-9-plus [24] introduces multilingual evaluation (9 languages, 371 training and 136 test) to test cross-lingual capabilities across diverse language families. QALD-10 [26] challenges systems with 394 complex test queries (4 languages) featuring advanced SPARQL constructs (aggregations, subqueries), testing reasoning robustness beyond simple fact retrieval. The questions in LCQuAD2.0 are produced from system‑generated templates, whereas the QALD benchmarks consist of queries that reflect real-world information needs of proficient English users and are translated into the other languages by native speakers. Collectively, these datasets enable comprehensive assessment of scalability, multilingual adaptability, and complex reasoning capabilities. Additionally, we share our analysis on the multi-hop queries on the test splits of these datasets in Table 1.

5pt

For consistent evaluation, we use a specific Wikidata-dump⁵ hosted locally with Tentris [62]. We chose Tentris due to its faster query response times and stable scaling for queries requiring extensive join operations. We serve this KG (\(\approx\)​11B triples) as an HTTP endpoint for SPARQL query evaluation. To ensure fair comparison, we filtered reference queries yielding empty/failing results, resulting in test sets of 4624 (LCQuAD2.0), 127 (QALD-9-plus) and 382 (QALD10) pairs.

4.2 Configuration and Hardware↩︎

For the text embedding function \(\phi\), we use a trained sparse mixture-of-experts text embedding model [63] with \(d=768\) dimensions. For the semantic similarity, we applied the dot product, a standard and widely used NLP metric. For SPARQL query generation and reasoning, we limit retrieved instances to \(i=10\) per pattern.

To promote open-source development, we exclusively utilize open models and have tested MARS using several state-of-the-art open-weight LLMs⁶ in our experiments. We utilize compute resources with 256 gigabytes of memory, 16 CPU cores of an AMD EPYC 7763 processor, and 4 Nvidia A100 GPU with 40GB of memory for hosting and experimenting with these LLMs.

4.3 Baselines↩︎

We compare our approach against four SPARQL query generation baselines spanning diverse paradigms: GRASP⁷ [18], the current state-of-the-art, an agentic system that iteratively builds and executes SPARQL queries using a set of predefined tools; DeepPavlov [15], [64], a chat-based system combining templates with fine-tuned models for query prediction, entity recognition, and path ranking; UniQ-Gen [17], a recent T5-based pipeline integrating entity linking, relation extraction, and query construction, designed to generalize across multiple knowledge graphs; and MST5 [16], a multilingual approach based on a fine-tuned mT5 that augments input questions with linguistic and entity information. For systems other than MST5, we use translated questions following prior work [48], [49]. Together, these baselines combine state-of-the-art fine-tuned models with newer agentic systems.

4.4 Metrics↩︎

We evaluate the performance of our system using four standard key performance indicators. Precision and Recall were calculated to compare system-generated answers against ground-truth results. Precision quantifies the correctness of answers retrieved by the generated SPARQL query against the ground truth. Recall measures the system’s coverage of all valid answers present in the knowledge graph. The F1-score balances precision and recall as a harmonic mean, reflecting overall accuracy in answer retrieval through the generated SPARQL query. In the context of this paper, we evaluate only with macro‑averaged metrics to avoid bias caused by class imbalance. Macro F1 combines macro precision and macro recall across test questions to assess SPARQL query’s ability to retrieve the correct answer-set. Macro F1 QALD⁸ is an adaption of Macro F1 which only considers samples where the QA-System predicted an answer for the metric computation. All of the mentioned metrics are computed with the help of GERBIL-QA⁹ [65].

5 Results and Discussion↩︎

5.1 Performance Comparison↩︎

We evaluate MARS on three benchmark datasets and report full details, including all key performance indicators, in the Appendix.¹⁰ Table 2 shows Macro F1 scores per dataset and language. MARS produces competitive or top scores in each setting, and across the three datasets it beats most baselines and outperforms GRASP in 8 of 14 head-to-head comparisons. The weaker numbers of DeepPavlov, MST5, and UniQ-Gen suggest that fine-tuning on Question-SPARQL pairs gives limited returns on heterogeneous benchmarks such as QALD. The performance gap between MARS and GRASP indicates that, at least in our multi-hop question setting, a fixed pipeline can be more effective than an agent that selects tools dynamically at runtime.

3pt

5.1.1 QALD-9-plus↩︎

On this nine-language benchmark, MARS and GRASP are well ahead of the other systems and split the per-language wins. MARS takes French, Bashkir, and Armenian; GRASP takes English, German, Spanish, Russian, Belarusian, and Ukrainian. The large gaps in favor of MARS on French (\(71.93\) vs.\(12.38\)) and Armenian (\(65.96\) vs.\(10.14\)) reflect uneven language coverage in QALD-9-plus: only 23 French and 18 Armenian questions are available, against 127 for the fully covered languages, so per-language scores are sensitive to a handful of correct answers. MARS adapts to both high- and low-resource languages, whereas GRASP performs better on fully covered languages, likely because of its optimizations for common KG patterns.

5.1.2 QALD-10↩︎

This dataset contains queries with aggregations and subqueries [26], and requires more hops than the other benchmarks in our work. MARS achieves the best F1 on all four languages in this dataset (English, German, Russian, and Chinese), outperforming every baseline, including GRASP. Comprising roughly 30% multi-hop and nested queries, in our evaluation, the dataset structure appears to align well with pipelines that explicitly plan traversal.

5.1.3 LC-QuAD2.0↩︎

Reproducing a large number of baselines is computationally expensive, so we restricted the comparison to the strongest systems reported in prior benchmarks. MARS achieves the best F1, ahead of both UniQ-Gen and GRASP, without dataset-specific fine-tuning. LC-QuAD2.0 is built from a fixed set of query templates, which usually favors fine-tuned methods like UniQ-Gen.

5.2 Error Analysis↩︎

While MARS is evaluated across multiple languages, we focus our qualitative error analysis on the English subset, as these cases are representative of the general failure modes observed across the entire multilingual evaluation. We inspect the cases where MARS answers incorrectly on QALD-9-plus and QALD-10. Of the 99 answered questions in QALD-9-plus, 60 are correct; for QALD-10, 202 of 289. Of these correct answers, 26 and 95 respectively come from a SPARQL query that differs from the gold, showing that the system often finds an alternative formulation with the same answer set, a property SPARQL exact-match metrics miss. We refer to questions below by their dataset ID prefixed with #.

5.2.1 Result Truncation↩︎

MARS caps every generated query at 1,000 results to mitigate the risk of malformed queries returning excessively large result sets. For genuinely open-ended questions this may reduce recall: QALD-9-plus #177 “Which bridges are of the same type as the Manhattan Bridge?” (gold \(24{,}512\), pred \(1{,}000\)) and #213 “Show me all Czech movies” (gold \(8{,}376\), pred \(1{,}000\)) are representative examples. Some of these gold counts are likely inflated by Wikidata growth since the benchmark was released, but the cap remains the dominant factor.

5.2.2 Boolean Questions Answered with Evidence↩︎

On QALD-10, the system often returns the underlying numeric or entity evidence instead of a yes/no answer. #170 “Do more than 100,000,000 people speak Japanese?” returns \(128{,}000{,}000\) rather than true, and #223 “Is Germany bigger than Poland?” returns a comparison result instead of producing an ASK query.

5.2.3 Aggregation and Projection↩︎

Comparative “which is more Y, A or B?” questions return both candidates and comparative values rather than only the winner (QALD-10 #361 “who is older, Messi or Ronaldo?”, gold: wd:Q11571, pred: {(wd:Q11571, 1985-02-05), (wd:Q615, 1987-06-24)}), and some “how many X” questions are not able to return the correct value as the system does not consistently produce the expected structure. The system retrieves the right facts but does not consistently project them through COUNT.

5.2.4 Conservative or Imprecise Selection↩︎

“When did X?” and “What year was X?” questions often yield empty predictions because the generation step abstains rather than emit an incorrect literal (QALD-10 #238 “What year did the Berlin Wall fall?”, gold \(1989\)), trading recall for precision. For broad “Give me all X” queries, the chosen class is occasionally too narrow: QALD-9-plus #199 “Give me all Argentine films” returns one film against a gold set of \(4{,}088\).

Overall, MARS performs well across diverse benchmarks, languages, and query types without dataset-specific fine-tuning, with a clear advantage on the multi-hop and nested queries of QALD-10. The errors we observe, namely truncated results, missing ASK and COUNT projections, and overly cautious literal generation, can be resolved through targeted engineering and do not reflect deeper architectural flaws. These results suggest that fixed, planning-oriented pipelines may offer stronger generalization in our multi-hop questions setting than fine-tuned text-to-SPARQL models or agents that select tools at runtime.

6 Ablation Experiments↩︎

We run ablation experiments on the QALD-10 [26] train set, supplying the gold entities (those extracted from the reference SPARQL query) during generation, Table 3 lists evaluated features. A full factorial design is impractical, so we restrict the experiments to English queries and proceed in phases: each phase fixes one or two settings for the subsequent phases while still exploring the remaining feature combinations against those fixed values. In Table 4, the configuration carried forward to the next phase appears in bold, and \(^\dagger\) marks the locally best result in the two phases where it differs from the selection (Phases 1b and 4).

4pt

5pt 0pt 0pt

6.1 Phase 1: Model Selection and Retrieval Grid↩︎

We evaluate three open-weight LLMs as \(\operatorname{SPARQLReasoner}\) (Qwen3 [3], Gemma3, GPT-OSS [60]) at two operating points (topn/mhop = 5/1, 20/2), then sweep topn \(\in\) {5,10,20,50,100,500} against mhop \(\in\) {1,2,5,10,20,50} for the top-scoring model. GPT-OSS leads at both operating points and, being a sparse Mixture-of-Experts model, runs at lower inference cost than comparable dense models, so we fix mod=GPT-OSS for the remaining phases. The largest grid cell (topn=500, mhop=1) reaches the highest F1 (\(65.14\)) but exceeds the SPARQLReasoner’s context window once additional features are enabled; we therefore use topn=50, mhop=10 for the feature ablations, which sits within 2 F1 points of the unconstrained maximum and remains tractable.

6.2 Phase 2: Augmented Text, Schema, and Examples↩︎

A \(2^3\) factorial analysis over aug, cls, and conc at the Phase 1 operating point shows that only configurations with conc active beat the no-feature baseline (\(63.50\)). conc alone yields the largest single-feature gain; pairwise and three-way combinations with aug and cls all degrade performance, which we attribute to context dilution at topn=50, where the prompt is already information-dense. We carry conc forward to Phase 3.

6.3 Phase 3: Pattern Frequency and Verification↩︎

A \(2^2\) factorial analysis over pat and ver on top of the Phase 2 setting shows that ver yields a sizeable gain (from \(65.47\) to \(70.94\)), pat alone is neutral-to-slightly-negative, and their combination falls between the two. Verification corrects systematic SPARQL errors from the first generation pass.

6.4 Phase 4: Full-Feature Scaling↩︎

We enable the full feature set (aug+cls+conc+pat+ver) and re-sweep topn against mhop to check whether the retrieval–depth trade-off shifts. Two findings stand out. topn=100 with mhop=1 gives the highest train-set F1 (\(72.10\)) but is the most expensive (avg.\(12.3\) k tokens per request) and relies on shallow retrieval, which is unlikely to generalise to multi-hop queries. Raising mhop beyond 10 has no measurable effect (topn=50 yields the same F1 at mhop=10 and mhop=20), confirming that the relevant subgraph is fully covered before the limit.

6.5 Final Configuration↩︎

We adopt GPT-OSS with topn=20, mhop=10, and all features enabled. We use topn=20 rather than 100 despite the latter’s higher train-set F1 (\(72.10\) vs.\(67.92\)): the 20-pattern setting consumes a third of the tokens per request (\(4.3\) k vs.\(12.3\) k) at a cost of about 4 F1 points on the train set, which we prefer to keep test-time inference cost bounded and to avoid overfitting. We also keep aug and cls enabled despite their negative isolated contribution in Phase 2, since both inject grounding signals (entity-augmented text, predicate domain/range) that the model can fall back on when surface forms diverge from the schema, which we expect on the multilingual test splits where lexical overlap with the English-trained pattern index is weaker.

As a pipeline sanity check, we compare against SimpleSparqlGenerator (SSG), which asks GPT-OSS to generate SPARQL directly from the same entity and relation links. SSG reaches Macro F1 \(52.34\) against \(67.92\) for our final configuration, isolating the pipeline’s contribution beyond the underlying LLM.

7 Conclusion and Limitations↩︎

In this work, we introduced MARS, a retrieval and reasoning pipeline for KGQA over Wikidata that operates without dataset-specific fine-tuning. The system combines iterative guided context enrichment with the KG schema and instance facts, and supports multilingual input through augmented translation. Across the three benchmarks, MARS achieves the best Macro F1 on QALD-10 (all four languages) and on LC-QuAD2.0, and splits the per-language wins with the strongest baseline (GRASP) on QALD-9-plus. The advantage of MARS is most evident on multi-hop and aggregation-heavy queries, consistent with our use of an explicit pipeline rather than a tool-using agent. The shortcomings are primarily centered around technical constraints and output errors identified in Section 5.2, pointing toward concrete directions for further improvement. MARS is built on open-weight LLMs and the code is publicly available to support reproducibility.

MARS’s limitations fall into two groups: those internal to the system, and those inherited from the current state of KGQA evaluation. For the latter, we highlight contributions that aim to address them for any future system. F1 is not directly comparable across papers. KGQA systems report Macro F1 with different denominators: some over predicted answer sets, others over SPARQL exact-match, others over SPARQL components such as entities and relations. Headline numbers from other papers therefore cannot be ranked directly. To ensure consistent comparisons, we re-run every baseline through the same answer-set protocol rather than quoting numbers from respective papers.

Benchmark–KG version drift. As noted in Section 1, benchmarks often lack the specific KG snapshots used during construction, as seen with QALD-10’s incomplete Wikidata dump. To ensure reproducibility, we release the exact Wikidata snapshot used alongside our updated datasets.

Pattern retrieval on high-degree nodes. Pattern retrieval over high-degree nodes in Wikidata-scale KGs requires joins that most triple-stores cannot serve in reasonable time. We rely on Tentris to make this tractable, while broader triple-store compatibility remains an engineering target.

Benchmark coverage. Resource constraints limited our evaluation to Wikidata-based benchmarks; extending MARS to other KGs such as Freebase and DBpedia is future work. We also attempted to include SPINACH [19] for its higher query complexity, but its gold queries depend on Wikidata-specific services such as wikibase:label that cannot be deployed locally. Rewriting them to use rdfs:label partially mitigated the issue but introduced others, and SPINACH’s answer-set conventions differ enough from QALD and LC-QuAD2.0 that a fair evaluation would require additional prompt engineering.

Acknowledgements↩︎

This work has been funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – TRR 318/3 2026 – 438445824, by the Federal Ministry for Economic Affairs and Energy (BMWE) on the basis of a decision by the German Bundestag under the project ikDS (FKZ KK5175206LO4), by the German Federal Ministry of Research, Technology and Space (BMFTR) within the projects Learn2RAG (01MK250104) and KI-Akademie OWL (16IS24057B), and by the Ministry of Culture and Science of North Rhine-Westphalia (MKW NRW) within the project SAIL (NW21-059D).

Generative AI tools (ChatGPT, Claude, Gemma, Qwen) were used for language refinement and code assistance. All scientific contributions and conclusions are solely those of the authors, who reviewed and take full responsibility for the final manuscript.

References↩︎

---

1. Our terminology for patterns is synonymous with triple patterns in SPARQL: https://www.w3.org/TR/2013/REC-sparql11-query-20130321/#sparqlTriplePatterns↩︎

2. Detailed image: mars-kgqa/data_dir/figures/mars-kgqa-overview.png↩︎

3. Our modified version: grasp_el/ANNOTATION_PIPELINE.md↩︎

4. mars-kgqa/supplementary_material/appendix.md#sparqlreasoner-prompt↩︎

5. mars-kgqa/supplementary_material/resources.md#datasets↩︎

6. mars-kgqa/supplementary_material/appendix.md#models-tested-with-mars↩︎

7. We follow the original paper’s configuration, but replace GPT-4.1 with the open-weight GPT-OSS to align with this work’s focus on open-weight models. The two models achieve broadly comparable performance on general benchmarks (https://artificialanalysis.ai/models/comparisons/gpt-oss-120b-vs-gpt-4-1).↩︎

8. https://github.com/ag-sc/QALD↩︎

9. Note that the GERBIL-QA scoring implementation was updated in 2024 and 2026. Results may differ from earlier reports. https://dice-research.org/GerbilQA↩︎

10. mars-kgqa/supplementary_material/appendix.md#performance-tables↩︎