[orcid=0000-0002-3434-2717, email=dkelesis@iit.demokritos.gr, ]

[orcid=0000-0002-1910-2758, ]

[]

Relational Reasoning , Small Language Models (SLMs) , Graph Neural Networks (GNNs)

1 Introduction↩︎

Large Language Models (LLMs) have emerged as a standard for zero-shot reasoning across diverse domains [1]–[3]. However, the deployment of frontier models (i.e., models with hundreds of billions of parameters) remains cost-prohibitive and has negative environmental impact [4], [5]. Additionally, such models often require massive computational clusters that are inaccessible to the average user. Consequently, there is a growing shift toward Small Language Models (SLMs), which offer a more sustainable and budget-friendly alternative for low-resource environments [6]–[9].

Despite their efficiency, SLMs often struggle with complex, multi-hop relational reasoning where logical consistency is very important [10]–[12]. A notable example is the CLUTRR benchmark [13], which requires models to induce kinship relations from narrative text through varying lengths of reasoning chains. In such scenarios, the limited parameter count of SLMs often leads to “hallucinated" relations or a failure to maintain the symbolic state of the story [14], [15]. The vulnerability of SLMs in these tasks stems from sequential deductive fragility: as the reasoning depth increases, minor errors in early fact extraction act as false premises, compounding through the chain and leading to catastrophic logical failures [12], [16].

To address these limitations, we propose a neuro-symbolic framework that transforms the SLM from a standalone reasoner into a minimalist agent. Our approach augments the SLM with two specialized tool calls: (i) a fact extractor that generates relational triplets from the narrative (i.e., story), and (ii) an expert Relational Graph Convolutional Network (RGCN) [17] that reasons over the resulting knowledge graph to provide a “hint" (i.e., a predicted relation and a confidence score) given a pair of query entities. By decoupling the text understanding from the relational logic, we provide a mechanism for causal grounding and structured explanation.

Our main contributions are as follows:

• We introduce a framework for enhancing SLM performance in zero-shot relational learning. In realistic settings where the SLM extracts its own facts, our method yields a 1.5 - 2x performance gain over story-only baselines.

• We provide a comparative study across different open-source model architectures (Gemma 3 1B/4B [18], Llama 3.2 3B [19]), analyzing how model size influences the quality of symbolic extraction.

• We conduct an extensive practical analysis, comparing our realistic pipeline against an Oracle configuration to isolate the failure modes inherent in noisy, self-generated knowledge graphs.

2 Related Work↩︎

2.1 Relational Reasoning and CLUTRR↩︎

Relational reasoning remains a cornerstone of artificial intelligence, requiring models to induce logical rules from discrete facts. The CLUTRR benchmark [13] was specifically designed to evaluate the systematic generalization of Natural Language Understanding (NLU) systems by requiring multi-hop kinship inference from story text. While graph-based models [17], [20] operating on symbolic inputs often achieve high accuracy, standard language models exhibit a sharp performance decline as reasoning depth (i.e., hop count betweeen the query entities) increases, often due to an inability to maintain consistent symbolic state.

2.2 Reasoning in Small Language Models (SLMs)↩︎

Recent advancements have focused on narrowing the “intelligence gap" between frontier models and SLMs through specialized training and high-quality synthetic data. Works such as [21], [22] demonstrate that models under 4 billion parameters can perform competent single-hop tasks but struggle with the”lost-in-the-middle" phenomenon [23] and logical chaining in zero-shot settings. Efforts to enhance SLMs often involve parameter-efficient fine-tuning (LoRA) or knowledge distillation from larger “teacher" models to improve their logical grounding.
Recent research has identified a”quantization trap" in small models, where the accumulation of noise across logical chains violates standard linear scaling laws, making multi-hop tasks uniquely difficult for compressed architectures [22], [24]. While multi-agent architectures have shown promise in maintaining stability across 4+ hops [25], these solutions often re-introduce the computational costs we aim to avoid. Our work specifically targets the extraction phase, investigating whether specialized graph-based models can mitigate this “sequential fragility" in a single-agent, tool-augmented setup.

2.3 Neuro-Symbolic and Agentic Integration↩︎

Neuro-symbolic AI seeks to combine the robust pattern matching of neural networks with the precision of symbolic logic. Recent “agentic" frameworks have moved toward tool-augmented reasoning, where LLMs assign complex tasks, such as mathematical solving or structured planning, to specialized external modules [26], [27]. Our work aligns with recent pipelines that utilize LLMs for triplet extraction to build dynamic knowledge graphs [28], though we specifically focus on the performance degradation introduced when SLMs are used as the primary extractors in noisy, low-resource scenarios.

3 Methodology↩︎

Our framework (shown in Figure 1) adopts an agentic neuro-symbolic architecture where an SLM serves as the reasoning controller, assigning specialized tasks to symbolic and neural tools. This decoupling allows the model to ground its story understanding in explicit logical structures.

3.1 Agentic Tool Calling Mechanism↩︎

We define two primary tools that the SLM can invoke to enhance its zero-shot reasoning:

• extract_facts(story): The SLM parses the story to extract kinship triplets \((u, rel, v)\). In our implementation, we follow a reversed edge direction where \((object, rel, subject)\), following the dataset’s logical schema.

• get_hint(K_graph, target_pair): This tool queries a pre-trained Relational Graph Convolutional Network (RGCN), with the query-based Knowledge Graph \(K\_graph\) (generated by the extracted facts) as input. The expert (i.e., the RGCN) returns the most probable kinship relation and a confidence score \(\sigma\), which is then injected back into the SLM’s context.

The extract_facts tool utilizes a structured, zero-shot prompting strategy to distill story text into symbolic triplets. Given the constrained parameter count of our models, we enforce a strict output format to minimize parsing errors. The exact prompt template, including the kinship ontology and formatting constraints used to ensure consistent extraction across Gemma 3 and Llama 3.2, is provided in Appendix 6.
Moreover, adhearing to the dataset’s logical schema, we adopt a target-centric representation where a triplet \((u, r, v)\) indicates that node \(v\) is the \([r]\) of node \(u\). For instance, in a father-daughter relationship, an edge from the father to the daughter is labeled as “daughter", representing the relationship from the perspective of the target node. This reversed directionality is consistently maintained during the RGCN’s training phase. By grounding the get_hint tool in this specific directional logic, we ensure that the expert model’s message-passing layers correctly aggregate the hierarchical context needed to resolve multi-hop queries (e.g., traversing multiple”child" edges to infer a “grandchild" relation). This precision prevents the”logical inversion" errors common in standard LLM/SLM reasoning over complex family trees.

3.2 Knowledge Graph Construction and Extraction Noise↩︎

A core focus of this study is the realistic scenario, where triplets are extracted via zero-shot prompting rather than provided by an Oracle. This introduces extraction noise in the form of hallucinated edges or missing relations. We characterize this as a challenge of sequential deductive fragility: in a multi-hop reasoning task, a single error in the early extraction of a kinship link (e.g., mistaking a “sister" for a”cousin") propagates through the graph, leading to a catastrophic failure in the RGCN’s path-finding for high-hop queries. Additionally, missing a single kinship (i.e., a single relationship) might lead to disconnected nodes, hence making the subsequent task of kinship prediction by the RGCN more difficult (or in some cases infeasible).
To mitigate this phenomenon, we augment the extracted facts in a post-hoc manner by incorporating their corresponding inverse triplets. This approach ensures bidirectional information flow along the semantic chain, regardless of the original directionality of the facts as extracted by the SLM. Overall, the average recall of ground-truth facts per story was estimated to be approximately 71%.

3.3 RGCN Expert Training and Generalization↩︎

The get_hint tool is powered by an RGCN trained specifically on the training partition of the CLUTRR dataset. Crucially, the model is trained on stories with reasoning lengths of \(k \in \{2, 3, 4\}\) hops. We then evaluate its performance on test sets containing up to \(10\) hops. This setup tests the model’s ability for systematic generalization, applying learned kinship rules to reasoning chains significantly longer (up to five times longer) than those encountered during training. By providing these GNN-derived hints to the SLM, we aim to compensate for the “lost-in-the-middle" and context-window limitations typical of models in the 1B - 4B parameter range.

4 Experiments and Discussion↩︎

4.1 Experimental Setup↩︎

To evaluate the efficacy of our neuro-symbolic agentic framework, we standardize the training and inference parameters across all configurations. Our study utilizes three state-of-the-art open-weights models: Gemma 3 (1B, 4B) and Llama 3.2 (3B). The extract_facts tool is implemented via a zero-shot prompting strategy, as detailed in Appendix 6, while the expert reasoning is assigned to a Relational Graph Convolutional Network (RGCN).

The RGCN expert is configured with 4 layers to facilitate deep multi-hop message passing. We train the model for 20 epochs using the Adam optimizer with a learning rate of \(10^{-3}\). Following the standard CLUTRR evaluation protocol, the RGCN is trained exclusively on the training partition containing reasoning chains of \(k \in \{2, 3, 4\}\) hops. We then evaluate its zero-shot performance on test set, with queries needing 2 to 10 hops to be answered, to measure the model’s capacity for inductive logical generalization. All SLM inferences are conducted in a zero-shot setting to maintain the integrity of the agentic tool-calling paradigm without model-specific fine-tuning.

4.2 Performance in the Oracle Scenario↩︎

The quantitative results presented in Table 1 establish a strong contrast between theoretical reasoning capacity and practical zero-shot implementation. In the baseline (i.e., zero-shot prediction using only the Story as input), SLMs struggle significantly; for instance, Gemma 1B fails to exceed a 7.54% accuracy. This gap is dramatically bridged in the Oracle scenarios. When provided with a ground-truth Oracle Hint (i.e., the actual relation to be predicted), Llama 3.2 3B experiences a nearly 4x performance increase, reaching 62.79%. This increase suggests that the primary obstacle for SLMs is not a lack of linguistic comprehension, but rather an inability to maintain and query a consistent symbolic state over long, distracting contexts. Note that the last two rows in Table 1 correspond to the test accuracy of the RGCN, using Knowledge Graphs constructed by Oracle and SLM-extracted Facts respectively.

In Figure 2, we illustrate the performance of each model and configuration, at each hop-level. Interestingly, the performance at the 2-hop baseline is not uniform across all architectures. While configurations utilizing the Oracle Hint generally achieve high accuracy, only specific variants, most notably within the higher-parameter Gemma models (i.e., Gemma 4B), approach near-perfect scores at this stage. As the reasoning depth increases, a clear divergence in architectural resilience emerges. The Llama-based models exhibit better robustness; despite the training distribution for the expert reasoning being capped at 4 hops, these models maintain or even improve their accuracy as the chain extends toward 10 hops, particularly in configurations utilizing Oracle Hints. This indicates both the capabilities of the RGCN expert in extracting relationships using multiple hops, even though it was trained in only up to 4 hops, and the efficiency of the SML in combining the expert hint with the Story.

In contrast, the Gemma 3 variants, show a more noticeable and immediate performance decay as reasoning depth increases beyond the 4-hop threshold. This disparity suggests that the Llama architecture may possess a superior capacity for structured logical injection and long-range symbolic integration, whereas the Gemma models, particularly at the 1B scale, appear more susceptible to the “Lost-in-the-Middle" phenomenon and contextual distraction in high-hop kinship queries, even when provided with high-quality guidance.

4.3 Performance in the Realistic Scenario↩︎

The transition to the Realistic scenario exposes the fragile nature of the neuro-symbolic loop. The RGCN expert, which achieves 60.69% accuracy on ground-truth facts, sees its performance collapse to 24.88% when operating on facts extracted by the SLMs (i.e., Gemma 4B). This drop confirms our hypothesis regarding sequential deductive fragility: because a multi-hop reasoning chain is only as strong as its weakest link, a single missed or hallucinated relation during the initial extract_facts phase renders the resulting knowledge graph logically unrecoverable for the GNN solver.

An interesting finding in our study is that providing both facts and hints to the model, is not always helpful. As seen in Table 1, for Gemma 4B, the configuration Story + GNN Hint (19.75%) notably outperforms Story + SLM Facts + GNN Hint (14.98%). This implies that the presence of noisy triplets acts as a “logical anchor" that misleads the SLM, even when a correct expert hint is available in the prompt. Interestingly, Llama 3.2 3B exhibits greater resilience to this effect, maintaining nearly identical performance (20.32% vs 20.52%) regardless of the inclusion of noisy facts. This suggests that Llama’s attention mechanism may be more efficient at filtering out symbolic noise in favor of high-confidence expert signals.

In Figure 3, we illustrate the performance of the different configurations across varying hop levels, now using SLM-based facts/hint. A clear “Reasoning Cliff" appears in this realistic scenario. While some oracle configurations maintain high accuracy after 4 hops, the realistic counterparts exhibit a catastrophic failure, performing with less than 20% accuracy. Interestingly, most models struggle to exceed 5% accuracy even for 2-hop chains. This indicates that the extraction process is most vulnerable at the point of initial fact grounding; if one of the few (two in this case) actual relations are not captured accurately, the GNN expert cannot provide a valid inference.

Performance generally peaks between 3 and 4 hops, where the relative density of the extracted graph may provide enough context for the GNN, before declining sharply. While the RGCN-derived hints provide a consistent 1.5 - 2x improvement over the story-only baseline for smaller models like Gemma 1B (moving from \(\sim\)​7.5% to 12.6%), the gain remains insufficient to overcome the compounding noise of high-hop extractions. This suggests that the bottleneck is not the reasoning capacity of the GNN, but the quality of the symbolic grounding.

5 Conclusion↩︎

In this work, we investigated the performance of neuro-symbolic agentic systems on zero-shot relational reasoning, by evaluating the integration of Small Language Models (SLMs) with Relational Graph Convolutional Networks (RGCNs). Our investigation using the CLUTRR benchmark demonstrates that while SLMs like Llama 3.2 and Gemma 3 possess the latent capacity for complex relational reasoning when provided with high-quality symbolic guidance, they are significantly constrained by an extraction bottleneck. We observed that early errors in the information extraction phase compound over multi-hop chains, leading to a collapse in accuracy, when queries need more than 4 hops to be answered. Furthermore, our results reveal a notable “Distraction Effect", where the inclusion of noisy, SLM-extracted facts can degrade performance even when expert hints are present in the context.
Future work will focus on implementing iterative symbolic refinement cycles to prune extraction hallucinations and investigating architecture-specific optimizations to better leverage the structural resilience observed in Llama-based models. Bridging this extraction gap remains essential for developing reliable, autonomous systems capable of causal grounding in open-domain environments.

Declaration on Generative AI↩︎

During the preparation of this work, the authors used GPT-5, Gemini 3, and Grammarly for the purposes of: Drafting content, Grammar and spelling check, Paraphrase and reword, Improve writing style.
After using these tools/services, the authors reviewed and edited the content as needed and take full responsibility for the publication’s content.

6 Prompts used in Language Models↩︎

The following table contains the prompts used for the different scenarios as explained in the Methodology Section 3. The available kinship relations appearing within the dataset are the following: aunt, brother, daughter, daughter-in-law, father, father-in-law, wife, husband, sister-in-law, granddaughter, grandfather, grandmother, grandson, mother, mother-in-law, nephew, niece, sister, son, son-in-law, uncle.

Regarding the extract_facts procedure for Gemma 4B we use the following prompt:

"""    
You are an expert Information Extraction system.

Task: Extract all explicit kinship relationships from the short story below.
    
Story: "{story}"
    
    Instructions:
    1. Identify people names enclosed in brackets (e.g., [Alice]). Extract them WITHOUT brackets.
    2. Identify the relationship between them.
    3. IMPORTANT: Extract relations in an ASCENDANT fashion (Child -> Parent). E.g.: 
       - Use grandson/granddaughter instead of grandfather/grandmother.
       - Use son/daughter instead of father/mother.
     etc.
"""

References↩︎