1 Introduction↩︎

The Retrieval-Augmented Generation paradigm has proven remarkably effective for grounding LLM outputs in factual corpora. A typical RAG pipeline chunks documents, encodes them as dense vectors (or, in lighter implementations, TF-IDF features), and retrieves the top-\(K\) most similar chunks to condition generation. This approach excels when the answer to a question is locally contained within one or two text passages—entity lookups, definition queries, and single-document summarization.

Industrial knowledge, however, is rarely flat. An aerospace supply chain is a graph: suppliers connect to components through SUPPLIES edges with lead times and contract types; factories consume components through USES edges with quantities; factories produce products; products are delivered to customers. Risk events cascade through this topology. Temporal validity governs which edges are active. The fundamental data structure is not a document corpus—it is a typed, timestamped, directed multigraph.

When a supply chain analyst asks “Which customers are not affected by the Thailand flood?”, the answer requires computing the full blast radius subgraph and returning its complement. When a risk officer asks “Which components have only one supplier?”, the answer requires counting in-degree centrality on SUPPLIES edges across the entire graph. These are not retrieval problems—they are graph computation problems. No amount of vector similarity can solve them.

This paper makes five contributions. First, we formalize five categories of queries that are structurally unreachable for single-pass vector retrieval and provide graph-algorithmic solutions for each, analyzing which failure modes persist even under agentic multi-step retrieval (Section 3). Second, we present a complete, reproducible reference implementation that runs both architectures side-by-side on identical data with retrieval-focused evaluation—both engines use deterministic template output, isolating retrieval quality from LLM generation (Section 4). Third, we demonstrate a practical incremental update architecture with five atomic graph mutation operations, selective re-indexing, and a timestamped changelog—addressing a key gap identified in the literature (Section 5). Fourth, we provide comprehensive empirical benchmarking across six retrieval architectures—deterministic GraphRAG, LightRAG [1], LLM-based GraphRAG, ReAct agentic RAG, dense-embedding RAG, and standard (TF-IDF) RAG—with Claude-as-judge scoring validated by inter-annotator agreement (\(\kappa = 0.716\)), scale testing to 1,100 nodes, and per-query failure analysis (Sections 5.5–5.9). Fifth, we connect our findings to the broader research landscape—particularly KGQA systems [2], [3], Temporal GraphRAG (TG-RAG) [4], Microsoft’s incremental GraphRAG indexing [5], and agentic RAG approaches [6]–[8]—and identify the design choices that matter for industrial deployment, including a proposed hybrid dispatch architecture combining deterministic handlers with LLM-based retrieval (Section 5).

2 Background and Related Work↩︎

2.1 Standard RAG and Its Limitations↩︎

The canonical RAG pipeline, formalized by Lewis et al. [9], consists of three stages: indexing (chunk documents and encode as vectors), retrieval (find top-\(K\) chunks by cosine similarity to the query embedding), and generation (condition an LLM on the retrieved context). Our implementation uses TF-IDF vectorization with scikit-learn [10] rather than dense embeddings. We chose TF-IDF deliberately: our argument concerns the architectural limitations of flat-text retrieval, not the quality of the embedding model. The five structural failure modes we identify (Section 3) arise from the absence of graph topology in the retrieval index, not from insufficient semantic similarity—they persist regardless of whether the embedding is sparse (TF-IDF) or dense (e.g., sentence-transformers). We verified this empirically with a dense-embedding baseline using all-MiniLM-L6-v2 (Section 5.5): better embeddings improve retrieval recall on multi-hop queries but all structurally dependent categories remain Fail.

The standard RAG paradigm carries several well-documented limitations for graph-structured data. Temporal blindness: text chunks carry no validity windows; a 2023 procurement report naming ShenzenChip as a supplier is retrieved alongside a 2024 contract transition notice replacing ShenzenChip with TechChip, with no mechanism to determine which is current. Structural opacity: the supply chain topology (Supplier \(\to\) Component \(\to\) Factory \(\to\) Product \(\to\) Customer) is dissolved into unstructured text fragments, making multi-hop traversal impossible. Absence blindness: RAG can only find what matches the query—it cannot represent or reason about what is absent from the graph.

2.2 Graph-Enhanced Retrieval↩︎

Graph-enhanced RAG approaches address these limitations by introducing a knowledge graph as a structural index alongside (or in place of) the vector store. Microsoft’s GraphRAG [11] uses LLM-extracted entities and relationships organized into community clusters with hierarchical summaries. LightRAG [1] integrates a graph-based text indexing paradigm with dual-level retrieval. HippoRAG [12] converts corpora into schemaless knowledge graphs for cross-passage reasoning, with subsequent work extending this to non-parametric continual learning [13]. G-Retriever [14] applies retrieval-augmented generation over textual graphs for multi-hop question answering, and KG-RAG [15] demonstrates biomedical knowledge graph integration with LLM prompts. Our approach differs from these in that we operate on an explicit, pre-defined schema (the supply chain ontology) rather than LLM-extracted triples, which eliminates extraction noise and enables precise typed traversal. Our contribution is a systematic empirical characterization of failure modes in an industrial domain with a reproducible implementation, rather than a theoretical advance in graph retrieval algorithms.

2.3 Agentic and Multi-Step RAG↩︎

A growing body of work extends RAG beyond single-pass retrieval. ReAct [6] interleaves reasoning and action steps, allowing an LLM to decompose complex queries into sub-retrievals. Corrective RAG (CRAG) [7] evaluates retrieval quality and triggers corrective re-retrieval when initial results are insufficient. Self-RAG [16] adds reflection tokens that allow the model to decide when and what to retrieve. These agentic approaches can address some multi-hop reasoning tasks by iteratively gathering context. However, they face fundamental limitations on queries requiring global graph computation—exhaustive enumeration (SPOF detection), weighted multi-hop aggregation (risk scoring), and potentially complement computation (inverse queries)—because the agent has no mechanism to guarantee complete traversal of an implicit graph embedded across text chunks. As we show empirically (Section 5.6), complement computation can be achieved with sufficient iteration on small graphs, but SPOF enumeration and weighted propagation remain intractable.

2.4 Knowledge Graph Question Answering↩︎

The structural queries formalized in Section 3—in-degree counting (SPOF), set complement (inverse queries), weighted path aggregation (risk propagation)—are well-studied in the Knowledge Graph Question Answering (KGQA) literature, where they map to standard graph query primitives: SPARQL COUNT/GROUP BY, MINUS/NOT EXISTS, and property path expressions, respectively. Systems such as QAnswer [2], KQA Pro [3], StructGPT [17], and semantic parsing approaches translate natural language to formal graph queries (SPARQL, Cypher) that execute over graph databases with provable completeness. Our “structurally unreachable” claim applies specifically to RAG-style retrieval (vector similarity over text chunks), not to knowledge-based QA broadly—KGQA systems can solve all five categories by design.

2.5 Temporal GraphRAG (TG-RAG)↩︎

Han et al. [4] identify a critical gap in existing GraphRAG systems: the temporal dimension. Their TG-RAG framework models external corpora as a bi-level temporal graph consisting of a temporal knowledge graph with timestamped relations and a hierarchical time graph. Our work shares this temporal awareness—every edge in our knowledge graph carries an effective_date, and expired relationships are retained in the text corpus but excluded from the active graph.

TG-RAG’s evaluation on the ECT-QA benchmark demonstrates significant improvements over baselines: 0.599 Correct score versus 0.406 for LightRAG and 0.405 for GraphRAG on base queries. Our work extends beyond temporal representation to address five additional structural query categories. We note that our temporal model is deliberately simple—binary active/expired status on edges—and does not address overlapping validity intervals or bi-temporal modeling that production supply chain systems require.

2.6 Incremental Indexing↩︎

Microsoft’s GraphRAG team [5] describes a planned graphrag.append command that attempts to place new entities into existing communities without triggering full Leiden recomputation. TG-RAG achieves incremental efficiency through its time hierarchy. Our reference implementation supports incremental updates natively through five graph mutation operations—add_entity, remove_entity, add_relationship, expire_relationship, and add_risk_event—each of which mutates the in-memory NetworkX [18] graph, incrementally rebuilds the TF-IDF index, and maintains a timestamped changelog.

3 Five Categories of Structurally Unreachable Queries↩︎

We define a query as structurally unreachable for single-pass RAG if no amount of top-\(K\) chunk retrieval by vector similarity can produce a correct answer in a single retrieval step, regardless of the embedding model quality, chunk size, or \(K\) value. More precisely, a query \(Q\) is structurally unreachable when its correct answer \(A\) requires either (a) information distributed across multiple chunks with no single chunk containing all required entities (retrieval incompleteness—addressable by agentic multi-step retrieval), or (b) computation over graph topology that cannot be expressed as a similarity search (computational irreducibility—e.g., counting in-degrees, computing set complements, or aggregating weighted paths). The unreachability claim applies to RAG-style retrieval over text; KGQA systems [2], [3] that execute formal graph queries can solve all five categories by design (Section 2.4).

The five categories presented below are not claimed to be exhaustive. Each category includes a formal problem statement, graph-algorithmic solution, empirical results, and an assessment of agentic RAG viability.

3.1 What-If / Counterfactual Analysis↩︎

The canonical query is: “What if we dual-source the Flight Control Unit—which alternative suppliers could provide it?” This requires the system to (a) identify the target component’s type (Electronic Assembly), (b) find all other components of that type, (c) identify their suppliers, and (d) exclude suppliers already connected to the target. The answer is defined by what is absent from the graph.

RAG failure mode: Vector retrieval returns chunks mentioning the Flight Control Unit and TechChip Inc (the current supplier), but has no mechanism to identify which suppliers are not connected.

Agentic RAG viability: Partial. An agentic system could iteratively search for same-type components and their suppliers, but reliably identifying all same-type components and excluding already-connected suppliers requires exhaustive enumeration that scales poorly.

3.2 Single Point of Failure Detection↩︎

The canonical query: “Which components have only one supplier?” This is a whole-graph aggregation—it requires iterating over every component node, counting incoming SUPPLIES edges, filtering, ranking, and tracing downstream impact.

Empirical result: GraphRAG identifies 15 single-source components, ranks them by criticality (high/medium/low), and traces downstream product impact for each.

Agentic RAG viability: No. SPOF detection requires computing in-degree centrality for every component node across the entire graph—a global aggregation with no natural decomposition into targeted sub-retrievals.

3.3 Inverse / Negative Queries↩︎

The canonical query: “Which customers are NOT affected by the Thailand flood?” This requires computing the complete downstream blast radius (a 4-hop traversal spanning 9 nodes) and returning the complement set—everything outside the blast radius.

Empirical result: GraphRAG correctly identifies DefenseTech Corp as the sole unaffected customer, whose product (SkyPatrol-UAV Drone) traces through Avionics Hub Delta—a factory that does not use any ThaiRubber components.

Agentic RAG viability: No (revised to Partial based on empirical results). Our ReAct agent solved this in 8 steps via list_all_entities (Section 5.6). Scalability beyond small entity sets is uncertain.

3.4 Comparative Subgraph Analysis↩︎

The canonical query: “Compare supply chain depth for the WideBird-X50 vs the RegionalJet-150.” Table 1 presents the full comparison.

Agentic RAG viability: Partial. Computing parallel structural metrics requires complete subgraph enumeration for both products.

3.5 Risk Propagation Scoring↩︎

\[R(p) = \sum_{(c,s) \in \text{paths}(p)} \sum_{e \in \text{events}(s)} w_\text{sev}(e) \cdot \delta_\text{hop}(c, p) \label{eq:risk}\tag{1}\]

where \(\text{paths}(p)\) is the set of (component, supplier) pairs reachable via reverse traversal from product \(p\); \(w_\text{sev}\) maps severity to weights (critical\(\,{=}\,\)​1.0, high\(\,{=}\,\)​0.7, medium\(\,{=}\,\)​0.4, low\(\,{=}\,\)​0.1); and \(\delta_\text{hop}\) decays with hop distance (1-hop\(\,{=}\,\)​1.0, 2-hop\(\,{=}\,\)​0.6, 3-hop\(\,{=}\,\)​0.35, 4-hop\(\,{=}\,\)​0.2).

Fig. 2 shows the resulting risk propagation heatmap.

RAG failure mode: The query “Show risk propagation scores for all products” returns zero TF-IDF matches. Even if chunks were retrieved, computing the weighted multi-hop score requires iterating over every product’s upstream subgraph—a computation that exists nowhere in the text.

Agentic RAG viability: No. Risk propagation scoring requires exhaustive traversal of the full product-supply topology and numerical computation with distance-dependent decay.

4 System Architecture and Implementation↩︎

Scope note. This comparison evaluates retrieval architecture, not end-to-end RAG system performance. Neither engine uses an LLM for answer generation; both produce deterministic template-based output from retrieved or traversed context. The deterministic GraphRAG system should be understood as an oracle upper bound. The scientifically informative result is the failure gradient across the five LLM-based architectures (\(0 \to 1 \to 1 \to 3 \to 5\) correct).

4.1 Knowledge Base Design↩︎

The knowledge base models a representative aerospace supply chain with six entity types and five relationship types (Table 2, Fig. 3), encoded as a directed multigraph with temporal metadata on every edge.

A critical design decision is the treatment of expired relationships. Two SUPPLIES edges carry status: ‘‘expired’’ with explicit expired_date fields. These edges are excluded from the active graph during construction but remain in the text corpus as stale documents. This deliberate asymmetry creates the temporal freshness test case.

4.2 Dual-Engine Architecture↩︎

The system exposes a single Flask endpoint that dispatches each query to both engines in parallel, returning side-by-side results (Fig. 4).

4.3 Query Handler Taxonomy↩︎

The GraphRAG engine implements 11 query handlers organized into three tiers (Table [tab:handlers]).

4.4 Per-Query Correctness↩︎

Table 4 presents the per-query correctness assessment. Standard RAG achieves zero fully correct answers. GraphRAG achieves 11/11 correct.

Evaluation methodology. The correctness assessments were performed by the paper’s author. To mitigate confirmation bias, we computed inter-annotator agreement using Claude Haiku 4.5 as an independent evaluator, achieving Cohen’s \(\kappa = 0.716\) (substantial agreement). The complete ground-truth answer sets are provided in Table 12 (Appendix).

4.5 Frontend Visualization Architecture↩︎

The web frontend (625 lines JavaScript, 1,168 lines CSS) provides an interactive comparison interface with a vis.js-powered knowledge graph visualization and incremental update controls. The risk propagation query produces a structured extra.risk_heatmap payload rendered as a color-coded card grid.

5 Discussion↩︎

5.1 Relation to TG-RAG and Temporal Representation↩︎

Our temporal model shares conceptual ground with TG-RAG’s timestamped relations but differs in scope and mechanism. The key insight from TG-RAG that transfers to our context is the treatment of temporal scope as a first-class retrieval dimension. Their ablation study shows that removing temporal retrieval drops the Correct score from 0.599 to 0.382—a 36% degradation.

5.2 Incremental Update Architecture↩︎

Our architecture takes a third approach, enabled by operating on an explicit ontology rather than LLM-extracted communities. We implement five atomic mutation operations (Table 5).

Each operation atomically updates three subsystems: (1) the NetworkX directed graph, (2) the TF-IDF vector index, and (3) the entity name index. The expire_relationship operation removes the edge from the traversable graph but deliberately preserves a stale text chunk in the corpus—creating exactly the kind of temporal trap that catches standard RAG.

5.3 The Taxonomy of RAG Failure Modes↩︎

Our five structurally impossible query categories reveal a taxonomy that generalizes beyond supply chain intelligence (Table [tab:taxonomy]).

5.4 Scope and Limitations↩︎

Scale. Our knowledge base is synthetic and small: 46 nodes, 58 edges. To characterize scaling behavior, we generated a synthetic knowledge base of 1,100 nodes and 1,850 edges (24\(\times\) and 32\(\times\) the baseline). Table 6 reports per-query latency at both scales.

RAG latency scales uniformly (\(\times\)​1.4–1.9). GraphRAG latency varies dramatically: single-entity queries remain near-millisecond (\(\times\)​1.2–2.1), while graph-global queries show superlinear growth—aggregation (\(\times\)​16.4), risk propagation (\(\times\)​33.7), path finding (\(\times\)​10.2). Even so, the worst-case P95 at 1,100-node scale is 11.01 ms (Q4).

Query dispatch. We replaced the original keyword-matching dispatcher with a TF-IDF intent classifier that matches query text against \({\sim}\)​60 prototype phrases. However, the 11 queries and handlers were co-designed—GraphRAG’s 11/11 score reflects a performance ceiling.

Embedding model. The dense-embedding baseline achieves 1/11 correct and 4/11 partial (vs./11 correct, 2/11 partial for TF-IDF), improving retrieval recall but failing identically on all six structurally dependent categories.

Handler engineering cost. Each handler required 50–200 lines of Python (median \(\sim\)​120 lines), \(\sim\)​6 intent classifier training phrases, plus ground-truth construction—approximately 2–8 hours per handler.

5.5 Empirical Comparison with LLM-Based GraphRAG↩︎

We implemented an LLM-based GraphRAG pipeline using Claude Haiku 4.5 for both graph extraction and answer generation. The LLM extracted 48 entities (vs. reference) and 68 edges (vs. reference). Table 7 presents the per-query results across six architectures.

The failure gradient (\(0 \to 1 \to 1 \to 3 \to 5\) correct) confirms that richer graph context and iterative retrieval help substantially, while better embeddings alone do not cross the structural barrier.

5.6 Empirical Agentic RAG Evaluation↩︎

We implemented a ReAct-style [6] agentic RAG baseline using Claude Haiku 4.5 with four tools: search_chunks, lookup_entity, get_neighbors, and list_all_entities, with a maximum of 20 tool calls per query. Table 8 presents the results.

Q9 is particularly notable: the agent traced the flood’s blast radius through all affected entities and correctly identified the unaffected customer—a task previously predicted as intractable. However, SPOF detection (Q8) and risk propagation (Q11) remain fundamentally intractable.

Q9 scalability caveat. The agent’s success relied on our graph having only 4 customers—the list_all_entities tool returned the complete set in a single call. Whether agentic complement computation scales beyond toy-sized entity sets remains an open question.

5.7 Existing System Comparison: LightRAG↩︎

We benchmarked LightRAG [1] (v1.4.16) on the same 11 queries. LightRAG extracted 244 entities and 362 relationships—substantially richer than our custom extraction. Table [tab:lightrag] presents the per-mode breakdown.

Despite its superior LLM-extracted graph (244 nodes vs.our 48), LightRAG still fails on inverse queries (Q9) and risk propagation (Q11). LightRAG’s success on Q7 (What-If, Correct in hybrid mode) is notable: the hybrid retrieval assembled enough supplier context for the LLM to correctly conclude that no alternative suppliers exist.

5.8 Inter-Annotator Agreement↩︎

Cohen’s kappa coefficient was \(\kappa = 0.716\), indicating substantial agreement per the Landis–Koch scale. Raw agreement was 16/22 decisions (73%). Intra-evaluator stability was 22/22 across three repeated runs, confirming deterministic evaluation at temperature 0.

Judge model circularity. The same model family (Claude) serves as both generation engine and scoring judge. We mitigate this by providing explicit ground-truth answer sets and a structured rubric.

5.9 Threats to Validity↩︎

Internal validity. The 11 queries and their handlers were co-designed by the same author, creating a ceiling effect. The core architectural argument does not depend on GraphRAG achieving a perfect score; it rests on the demonstrated failures of five independent LLM-based architectures.

External validity. The knowledge base is a single synthetic domain with 46 nodes. Scale testing confirms deterministic engine correctness at 1,100 nodes, but LLM-based architectures were only tested at 46 nodes.

Construct validity. Six construct threats merit attention: (1) Claude-as-judge circularity; (2) coarse 3-level scoring rubric; (3) single model family across all LLM architectures; (4) TF-IDF vs. dense embedding baseline; (5) template vs.LLM generation confound; (6) small evaluation set with purposive sampling.

Reproducibility. All source code, ground-truth answer sets, and benchmark harnesses are included. The deterministic core engine requires no API keys and produces identical output on every run.

6 Revision: Generalized Graph Query Planning↩︎

The original evaluation demonstrated a clear failure gradient but was limited by co-design circularity. This section presents three methodological improvements.

6.1 Architecture 7: LLM Query Planner with Typed Graph Primitives↩︎

We introduce a seventh architecture that breaks the co-design circularity by replacing all 11 bespoke handlers with a single LLM-driven query planner. The planner receives the graph schema but not the data. Given a natural language query, the LLM emits tool_use calls selecting from nine typed graph primitives (Table 9).

6.2 Hold-Out Query Set↩︎

We constructed 12 hold-out queries covering all 10 intent categories plus 2 multi-category compositions. Ground truth was computed manually and verified with 13 automated validation tests.

6.3 Entity-Level \(F_1\) Scoring↩︎

We replaced the coarse 3-level ordinal with entity-level precision, recall, and \(F_1\) computed against ground-truth entity sets. Entity IDs are extracted via regex pattern matching and fuzzy entity name matching. The ordinal is retained as a secondary metric: \(F_1 \geq 0.9 \to\) Correct, \(0.3 \leq F_1 < 0.9 \to\) Partial, \(F_1 < 0.3 \to\) Fail.

6.4 V2 Results↩︎

Table [tab:v2results] presents the complete V2 results across four architectures and 23 queries, scored by entity-level \(F_1\).

6.5 Analysis↩︎

The LLM Query Planner outperforms all architectures. At \(F_1 = 0.632\) (5C, 14P, 4F), it exceeds both the Agentic RAG (\(F_1 = 0.577\)) and the bespoke Deterministic GraphRAG (\(F_1 = 0.472\)). This inverts the V1 result.

The hold-out set reveals a generalization gap. The Deterministic GraphRAG drops from \(F_1 = 0.574\) on original queries to \(F_1 = 0.379\) on hold-out queries—a 34% relative decline—confirming co-design inflation. The LLM Planner shows the opposite: \(0.557 \to 0.700\).

Typed primitives matter more than additional LLM reasoning steps. The Planner uses an average of 4.9 tool calls vs. for the Agentic RAG, yet achieves higher \(F_1\). The difference is better tools, not more reasoning.

Remaining failure modes. Three categories remain challenging: aggregation (Q4, \(F_1 = 0.23\)), what-if (Q7, \(F_1 = 0.10\)), and subgraph comparison (Q10, \(F_1 = 0.00\)).

6.6 Revised Threats to Validity↩︎

The V2 revision addresses three of five primary threats: co-design circularity (Architecture 7 + hold-out set), construct validity (entity-level \(F_1\)), and statistical scope (11 \(\to\) 23 queries). Two remain open: external validity (single synthetic domain) and cross-model replication (only Haiku 4.5 available).

7 From Traversal to Computation: Architecture 8↩︎

Section 6 established that typed traversal primitives outperform bespoke handlers. However, three categories remained intractable. We identified a common root cause: these queries require computation over graph structure rather than targeted traversal.

7.1 Graph Computation Primitives↩︎

We decomposed the three failure modes into specific computational capabilities (Table 10).

Each computation tool encapsulates a complete graph algorithm—not a primitive step—so the LLM makes one tool call where Architecture 7 would require a multi-step loop.

7.2 Adaptive Tool Selection↩︎

Architecture 8 presents the LLM with all 15 tools (9 traversal + 6 computation) and a tool selection guide. The system prompt instructs: “For aggregation queries, use aggregate_over_type INSTEAD of manually iterating. For what-if queries, use simulate_removal. For comparison queries, use subgraph_diff.”

7.3 Results↩︎

Table [tab:arch8] presents Architecture 8 results compared with Architecture 7.

7.4 Analysis: The Measurement Gap↩︎

The headline \(F_1\) numbers are flat (\(0.635 \to 0.636\)). This conceals a qualitative breakthrough. Consider Q4 (aggregation):

• Architecture 7 calls find_nodes and get_neighbors repeatedly, running out of step budget.

• Architecture 8 calls find_nodes(Supplier) once, then simulate_removal eight times. In 2 steps, it produces a complete, correct ranking of all 8 suppliers by product impact.

The answer is correct. Yet \(F_1 = 0.22\) because the ground truth contains only 3 entity IDs while the comprehensive answer mentions 23 entities. This is a fundamental limitation of entity-level \(F_1\) for structural queries. A correct aggregation answer must mention all entities in the ranking; the scorer counts additional entities as false positives.

7.5 Tool Adoption Patterns↩︎

Haiku 4.5 reliably selects the appropriate computation tool:

• Q4: Called simulate_removal\(\times\)​8 without instruction, producing a correct supplier-by-impact ranking.

• Q10: Called subgraph_diff then supplemented with targeted traversal.

• Q7: Did not adopt simulate_removal for dual-sourcing, suggesting counterfactual phrasing is harder for small models to map to removal tools.

The tool adoption rate was selective: the LLM chose computation tools only when the query category matched, correctly ignoring them for traversal-native queries.

7.6 The Operator Vocabulary Thesis↩︎

The progression across eight architectures recapitulates programming language evolution (Table 11).

The thesis: the barrier to graph reasoning is not the LLM’s intelligence—it is the operator vocabulary. Architecture 7’s remaining failures were not failures of reasoning; the LLM correctly identified what it needed to do but lacked the tool. Architecture 8 supplies those tools, and the LLM adopts them without further instruction.

The practical implication: rather than engineering bespoke handlers or training graph-specialized models, practitioners should invest in curating the right operator vocabulary—a library of typed, composable graph operations exposed as LLM tools. When new query categories emerge, adding a tool (not a handler) extends capability.

8 Conclusion↩︎

We have presented an architectural comparison of eight retrieval systems for industrial supply chain intelligence, evaluated across 23 queries using entity-level \(F_1\) scoring. The progression from flat text retrieval (Architectures 1–2) through bespoke graph handlers (Architecture 3) to LLM-composed traversal (Architecture 7) and computation (Architecture 8) reveals: the limiting factor in graph-augmented retrieval is not the LLM’s reasoning capability but the operator vocabulary available to it.

Architecture 7 demonstrated that typed traversal primitives outperform hand-coded handlers (\(F_1 = 0.632\) vs.\(0.472\)) while generalizing to unseen queries. Architecture 8 extends this: when computation tools are added, the LLM selectively adopts them for the exact query categories where traversal fails.

A critical methodological finding: entity-level \(F_1\) systematically underscores structural queries where comprehensive answers are correct. This measurement gap suggests that structural query evaluation requires task-specific metrics—ranking accuracy for aggregation, structural completeness for comparison, causal coverage for what-if—rather than flat entity extraction.

Our taxonomy of six RAG failure modes (absence, degree, complement, topology, propagation, and temporal blindness) complements engineering-oriented failure taxonomies [19] by focusing on structural failure modes. Open challenges remain: cross-model replication, cross-domain validation, and developing evaluation metrics appropriate for structural queries. The reference implementation—including all eight architectures, 23 queries with ground truth, the \(F_1\) scoring module, and the unified benchmark harness (8,154 lines across 17 source files)—is provided as a reproducible artifact (Appendix).

9 Implementation Artifact Summary↩︎

References↩︎