1 Introduction↩︎

Large language models have made it practical to interrogate documents directly: a user poses a question in natural language and receives an answer grounded in the source text, with no hand-built query interface and no manual review. Few professions stand to benefit more from this than law, whose day-to-day work consists largely of locating, reconciling, and interpreting provisions spread across long, densely cross-referenced instruments. Syntheia provides the data layer for this workload: a structured document index over coordinated transaction sets (an original agreement plus its amendments, side letters, schedules, and disclosure documents) served through an API, and directly to LLMs such as Claude via the Model Context Protocol [1], for document tools to build on. This paper reports the evaluation that decided how retrieval over that index should work: what it took to replace full-corpus injection with structured retrieval without giving up answer quality, and what each alternative costs in tokens and dollars.

The simplest method is to place the entire corpus in the model’s context window on every query. We call this approach inject, and we adopt it as our baseline deliberately, because it represents the upper bound on retrieval recall. Every clause, defined term, and cross-reference is in context on every query, so there is nothing for a retriever to miss and nothing to misconfigure. An alternative that a careful judge cannot distinguish from inject is therefore a stronger result than one that merely outperforms a lossy retriever; the corpus was deliberately sized to keep inject runnable (Section 3).

Two costs limit inject as the corpus grows. The token footprint scales with the corpus rather than the question (every query pays for every clause regardless of relevance) and the corpus cannot exceed the context window at all. Separately, LLMs attend unevenly across long inputs and lose information positioned in the middle of the context [2]. The legal domain compounds both: answers routinely depend on defined terms, on cross-references between schedules and operative clauses, and on how an original agreement interacts with later amendments, so the text bearing on a question is dispersed across the corpus and easily diluted by surrounding material. Existing contract benchmarks (CUAD [3], ContractNLI [4]) operate on single documents and do not capture this cross-instrument dependency, which remains an open gap for legal retrieval evaluation.

Both modes studied here operate over the same input representation: documents chunked by a proprietary structure-aware method developed at Syntheia, rather than split into fixed-size spans. Chunking is held constant as a fixed preprocessing step, so that what we measure is the retrieval strategy and not the chunking; we do not evaluate naive fixed-size chunking, whose tendency to fragment provisions and reduce retrieval quality is already well documented [5], [6]. We report two comparisons:

Comparison 1 (Section 4) evaluates Navigate-Embeddings (navembed), which pre-embeds the document nodes and, at query time, retrieves the top-\(k\) most similar by cosine similarity, placing only those nodes in the answering model’s context. Twelve configurations (RUN-001–012) were run in three phases: Phases I–II while the harness matured (RUN-001–008), Phase III at the full 20-question scale (RUN-009–012). Each is an internally controlled paired comparison (inject versus navembed on identical questions), and the headline claims rest on the Phase III runs.

Comparison 2 (Section 5) evaluates Navigate-Index (navindex), which replaces vector similarity with guided navigation over a compact structured index: boolean semantic flags, generated summaries, and explicit cross-reference and defined-term graphs. We sweep six descriptor configurations (the per-node metadata the retriever sees) to characterise the quality–token-footprint trade-off.

The two navigate modes reduce inject’s burden along two dimensions that prompt caching has pulled apart: token footprint and dollar cost. Caching lowers the price of re-reading a large context without reducing the tokens the model must attend over when it answers, so the two dimensions are tracked separately throughout (Section 3.3). The cache discount also holds only within a single warm session (a run of queries close enough in time to keep the provider’s short-lived prompt cache live), and a corpus that outgrows the context window cannot be injected at any price (Section [sec:sec:crossover]).

Within that frame, the two modes sit at different points on the quality–token-footprint curve (Table 4): navindex maximises structural fidelity at a modest token-footprint gain, navembed maximises token-footprint reduction, and the dollar comparison between them and cached inject is corpus-dependent (Section [sec:sec:crossover]). Figure 1 illustrates all three modes.

The contributions are: (i) a paired evaluation of full-corpus injection against two structured retrieval modes on a scored legal-QA evaluation set with verified ground-truth answers, judged by a position-bias-controlled, reference-anchored pairwise protocol (answer-level equivalence; retrieval recall is not directly measured), with uncertainty quantified throughout; (ii) the design of the navindex format, a navigation descriptor for legal instruments (typed boolean flags, cross-reference and defined-term graphs, a hard selection cap), documented to field level by the released prompts, which also operationalise amendment precedence and defined-term resolution (Appendix 13); (iii) a closed-form caching-crossover rule separating token footprint from dollar cost, with its session-economics caveats; and (iv) a catalogue of failure modes we would not ship (BM25 length bias, titles-only over-selection, uncapped node selection) and an account of the evaluation’s limits.

2 Related Work↩︎

RAG couples a parametric generator with a non-parametric retriever so that generation is conditioned on fetched evidence [7]; recent surveys map the design space [8]. Dense passage retrieval [9] established learned bi-encoders as a strong default, followed by a family of general-purpose sentence embeddings (Sentence-BERT [10], E5 [11], GTE [12], BGE [13], and MPNet-based encoders [14]). BM25 remains a strong lexical baseline [15]; hybrid systems fuse dense and sparse signals via Reciprocal Rank Fusion [16], and cross-encoder rerankers [17] re-score candidates at higher fidelity. Our strongest navembed configuration is a retrieve-then-rerank pipeline; our hybrid BM25 run illustrates a known failure mode (length bias) on clause-structured text. Chunking granularity affects retrieval quality [5], [6]; we hold structure-aware chunking fixed and vary only the retrieval strategy.

Two observations make token footprint a first-class metric here. Long-context performance degrades when relevant information sits in the middle of a long input [2], and provider-side prompt caching discounts re-reads of a fixed prefix [18], lowering the price of a large context without lowering the tokens the model attends to. Dollar cost and token footprint must therefore be tracked as separate quantities. For evaluation, LLM-as-judge protocols are common [19] but exhibit position bias [20]; our forward/reverse protocol is a direct response.

In legal NLP, CUAD [3] and LEDGAR [21] are the leading contract benchmarks, and ContractNLI [4] shows that entailment over contracts requires resolving defined terms and cross-references, structural dependencies that flat retrieval cannot represent. Hierarchical indexing organises nodes into a tree and navigates from coarse summaries to fine provisions [22]; a complementary line uses typed metadata pre-filters to narrow an index before any similarity computation, analogous to predicate push-down in query optimisation. Legal documents require both: defined terms propagate meaning across sections, cross-references bind operative clauses to schedules, and amendments supersede base provisions in ways neither cosine similarity nor BM25 can represent. The navindex format encodes these signals directly and sits in the hierarchical-navigation paradigm rather than in flat RAG; among vectorless navigators it is closest to PageIndex [23] (Section 5.1).

3 Problem Setting and Method↩︎

3.1 Corpus and models↩︎

Document corpus. The benchmark indexes fifteen transactional legal agreements drawn from three practice areas. Six credit facility agreements (FA) cover two borrowers: Fluor Corporation (a $1.8 B revolving facility plus one amendment) and Generac (an ABL facility plus two term-loan amendments and a 2024 refinancing). Three limited partnership agreements (LP) cover two funds: Thomas Green Fund, a US real-estate fund targeting LEED-certified office and multi-family assets with a $500 M commitment cap and a companion side letter, and Carlyle PE Partners, a PE fund with a 5 % IRR hurdle and 12.5 % incentive allocation. Six share/asset purchase agreements and disclosure schedules (SP) cover four transactions: Meta’s divestiture of Giphy to Shutterstock ($128 M purchase price), the First Avenue Networks disclosure schedule, the PPG/Comex cross-border acquisition, and a Royal Wolf Holdings facility agreement. Two additional Netflix facility agreements are indexed in the open-search workspace but not bound to any scored question. The scored benchmark runs use six of the fifteen documents: one Thomas Green Fund LPA, one Carlyle PE Partners LPA, the Meta/Giphy SPA, the First Avenue Networks disclosure schedule, the Generac 2024 amendment, and the Fluor facility amendment. All fifteen documents are public filings drawn from SEC EDGAR exhibits, Kentucky pension FOIA releases, and UK government sources. The scored question set with its verified reference answers and a source reference for every document is released as supplementary material; all prompts are printed in Appendix 13.

Question set. The scored evaluation set contains 20 questions with verified ground-truth answers (Appendix 12): 18 are bound to specific source documents (six per practice area, three questions per document), and 2 are out-of-scope controls whose answers are deliberately absent from all indexed documents, testing whether the system correctly declines rather than hallucinating. Because the controls test declination rather than retrieval, we report them separately from the document-bound counts throughout. The first two phases ran on a 15-question, document-pinned subset of DocNavBench, a companion 52-question open-search set over the same corpus, authored by the reviewing lawyer: its questions are deliberately issue-driven, written as a practitioner would ask before reading and without presuming the answer exists (so “that is not in the agreement” can be the correct finding), and it is document-agnostic, pairwise-judged only, with no ground-truth answers. The Phase I–II runs used 3 questions first, widening to 9 and then all 15 as the harness matured (Appendix 10); the navindex descriptor sweep (Configs 1–5, Section 5) also belongs to Phase II, sharing its pool and corpus. The 20-question scored set was written fresh at Phase III by the engineering team: questions and reference answers were drafted with the same model family over the six scored documents, verified against the sources, and endorsed by the reviewing lawyer, with the out-of-scope controls added so that every document-bound question is answerable from the indexed documents by construction. It shares no question with the Phase I–II pool, so nothing the pipeline was tuned on is scored. The lawyer’s assessment of the set’s difficulty and phrasing is reported in Section 7. The four Phase III runs and navindex Config 6 all use the scored set; DocNavBench’s full open-search evaluation is future work. The answering model is claude-sonnet-4-6; the judge is claude-opus-4-7.

Context-window scope. The scored evaluation was deliberately sized to stay within the model’s context window, so that inject is a runnable baseline and the comparison is like-for-like on every question. This is a constraint of the evaluation, not of the methods. Real legal matters frequently exceed the window (a single transaction can span dozens of instruments across base agreements, amendments, side letters, and schedules) and beyond that point inject is not a viable method rather than merely an expensive one, so it would not appear as a baseline at all. The navigate modes carry only a bounded retrieval payload into the answering context and are not subject to this ceiling. Within-window parity is thus the deliberately hard test: a mode that cannot match inject where inject is runnable could not be trusted where inject is not.

3.2 Judging protocol↩︎

Every question is answered twice, in parallel: once by inject and once by the navigate variant under test. The judge receives the question, the verified reference answer, and both model answers, and decides which answer more accurately reflects the reference, or a tie if both are equally correct or equally wrong (the full judge prompt is Appendix 13, JUDGE_GROUNDTRUTH_SYSTEM). Each pair is judged in two directions, forward and reverse (positions swapped); a win is awarded only when both directions agree, and a hedged verdict in either direction (e.g.”slight lean toward A, effectively tied”) is treated as a tie. The two-direction rule controls the position bias documented for LLM judges [19], [20].

A tie under this protocol therefore certifies that the two answers are judged equally consistent with the verified reference, equally correct or equally wrong relative to ground truth, and not merely that they read alike. It does not certify that either answer is right; judge error is bounded empirically by the audit below, and the remaining caveats (a forced-decision probe, i.e.a rerun with ties disallowed; judge–answerer provider overlap; and tie-category granularity) are consolidated in Section 7. All 20-question scored runs (RUN-009–012 and navindex Config 6) were judged with this reference-anchored prompt; the Phase I–II runs, navindex Configs 1–5 included, were scored with the Phase I–II preference judge, which compares the two answers without a reference (reproduced in Appendix 13).

Judge audit. A manual audit of one 15-question run classified \(9/15\) verdicts as clearly correct, \(4/15\) as defensible-but-debatable, and \(2/15\) (\(13.3\%\)) as likely wrong (95% Clopper–Pearson interval [24] \([2\%, 40\%]\)). The audit was performed by one author against the verified ground-truth answers for that run; a verdict was “clearly correct” if the judge’s preferred answer matched the ground truth exactly, “defensible-but-debatable” if partially correct or a reasonable legal interpretation, and “likely wrong” if it contradicted the ground truth. No second annotator reviewed the run, so inter-annotator agreement is not available.

3.3 Token footprint versus dollar cost↩︎

Prompt caching separates two axes. Token footprint is every input token the model attends to, including cached context; it governs the long-context degradation noted above and the hard window limit. Dollar cost is recorded operational spend, reshaped by discounted cache reads and one-time cache writes [18]. Caching can make a full-context inject cheap to re-read without reducing the tokens inside the model’s attention. In RUN-005 (15 questions), inject attended to \(3{,}217{,}792\) input tokens in total (\({\sim}214.5\)K per question) while navembed processed \(171{,}307\): about \(3\times\) apart on dollars ($1.881 vs.$0.611) but \(18.8\times\) apart on token footprint. Caching compresses the dollar axis; the token-footprint gap is what remains structural.

3.4 System context↩︎

The three modes differ in their runtime shape as well as their token profile. inject issues one call per question over a long (cache-discounted) prefix. navembed embeds the query, retrieves and optionally reranks nodes, then issues one short answering call. navindex issues two sequential LLM calls per question (a short-output index scan over a cached prefix, then a short answering call), trading tokens for one extra round-trip of latency. Indexing is a one-time, per-document cost with two layers: headings, clause references, and the cross-reference and defined-term edges are extracted deterministically by the chunking pipeline (same input, same output, verifiable against the source), while the semantic boolean flags (Table 2), keyword tags, and per-node summaries are populated by a separate enrichment pass of the Syntheia indexing pipeline.

4 INJECT vs.NAVEMBED↩︎

4.1 Approach↩︎

navembed pre-indexes each document into a tree of nodes. Every node carries the section’s verbatim text plus a descriptor used at index time. At query time the query is embedded with the same model and cosine similarity selects the top-\(k\) nodes; only those nodes (plus the question) are placed in the answering model’s context. This replaces the full corpus with a small, question-relevant working set. Five retrieval strategies were evaluated:

1. Embedding top-\(k\) [9]: cosine similarity; keep the top \(k\).

2. Embedding \(+\) cross-reference expansion: after top-\(k\) hits, follow each hit’s crossReferencedIds and add those nodes.

3. Hybrid semantic \(+\) BM25: fuse dense and lexical signals [15], [16].

4. Semantic \(+\) rerank: retrieve \(40\) candidates by cosine, rerank with a cross-encoder, keep the top \(10\) [17].

5. Naive LLM index scan: the model sees a titles-only index, selects candidates, and those are fetched. This prototype over-selected (\(87\), \(72\), and \(38\) nodes on its three trial questions, pushing the token footprint to within \(1.2\times\) of inject) and motivated the navindex format described in the next section.

Twelve configurations were run in three phases; the earlier two (3–15 questions, evolving harness) are reported as run history in Appendix 10, and all claims below rest on the Phase III runs, which share the full 20-question set, the same six-document corpus, and the same prompts.

4.2 Results↩︎

3.2pt

Table 1 reports the Phase III runs; Table 5 gives exact 95% confidence intervals for every headline rate, and Appendix 10 reports the full twelve-run history (Table 6), across which the token-footprint reduction held at \(8.5\)–\(29.9\times\).

The strongest deployable result is semantic retrieval with reranking (RUN-012)¹: judged tied with inject on 16 of 18 document-bound questions (inject preferred on 2, navembed preferred on 0), with both out-of-scope controls also tied, at \(188{,}274\) input tokens against inject’s \(3{,}263{,}796\), a \(17.3\times\) reduction. GTE embeddings (RUN-009) give the largest reduction, \(29.9\times\), at a lower tie rate (\(15/20\); inject preferred on 3, navembed on 2) [12]. Hybrid BM25 (RUN-011) is the failure mode: length bias [15] inflates the retrieved payload to \(382\)K tokens (reduction falls to \(8.5\times\)), demonstrating that lexical fusion can silently cost more than it returns on clause-structured text. RUN-010 (MiniLM with ground-truth-assisted retrieval) reached \(18/20\) ties but uses gold node labels to validate retrieval, making it an oracle upper bound; it is excluded from all cross-mode comparisons. Notably, its two inject wins fall on the same two questions as RUN-012’s, consistent with the retrieval-difficulty diagnosis below.

The two inject wins in RUN-012 (SP-first-ave-Q2 and SP-meta-giphy-Q3; Appendix 11) both concern share-purchase and disclosure-schedule provisions that draw on multiple cross-referenced clauses. The likely cause is retrieval failure (relevant nodes ranked outside the top-\(10\) after reranking) rather than reasoning failure, though confirming this requires the ground-truth node labels planned in Section 8. This diagnosis motivated encoding cross-references explicitly, which is exactly the signal navindex navigates (Section 5).

Embedding model choice mattered less than descriptor richness. Across the Phase II runs (Appendix 10), BGE embeddings produced more inject wins than paraphrase-MiniLM-L6-v2 (consistent with over-selection of lower-relevance nodes), and MPNet QA embeddings raised the Phase II tie count to \(9/15\). But no embedding swap moved outcomes as much as what each index node carries, the observation that motivated Comparison 2.

Dollar cost. navembed’s recorded cost is lower than inject’s in every ledger-validated run (RUN-005 onward; Table 6). RUN-011 is, however, the one Phase III configuration whose per-query payload (\({\sim}19\)K tokens) exceeds one-tenth of the \(163\)K corpus prefix, so by the crossover rule of Section [sec:sec:crossover] a warm-cached inject would beat it at steady state: the length-bias failure costs real money, not only token footprint.

5 INJECT vs.NAVINDEX↩︎

5.1 Motivation: descriptor richness as the binding constraint↩︎

The navembed runs identify the binding constraint: the richness of what each index node carries. To overcome the limitations of a vector retrieval system, we designed navindex: a retrieval system where an LLM reasons over a compact structured index rather than searching by similarity alone. Cosine similarity surfaces semantically similar nodes but cannot represent structural signals such as “this clause defines a term,” “this provision contains a liability cap,” or “this node cross-references Section 5.17.” And when nodes carry only headings, the model over-selects (R5, Section 4). The navindex format addresses both limits by encoding each node with typed metadata and a hard selection cap.

A vectorless, structure-navigating index of this kind is closest in spirit to PageIndex [23], which builds a hierarchical table-of-contents tree of a document and has an LLM reason over node titles and per-node summaries to choose which sections to read. PageIndex generates a natural-language summary of every node at index time. Our descriptor is deliberately weighted the other way: the primary navigation signals are compact and cheap to index: headings, clause references, and the explicit cross-reference and defined-term graph are extracted deterministically from document structure, and the boolean semantic flags and keyword tags are populated by Syntheia’s enrichment pass as single booleans and short tag lists rather than free text. The model filters on these before consulting any summary text. The reason is cost. Generating a free-text summary of every section is the expensive end of the indexing budget and amortises only when the same document is queried many times; a given legal instrument is often consulted only a handful of times, or once, so that cost may never be recovered. The extracted fields carry no generation cost at all, and the flags and tags are the cheap end of the enrichment budget.

5.2 Approach: the NAVINDEX format↩︎

Dual-file architecture. Every processed document produces two files: *.index.json (the navigation surface: structural metadata plus a short opening snippet per node, no full provision text) and *.full.json (the provision store: verbatim clause text fetched by node ID). The navigation surface is designed to scale with document structure rather than text length. The cached S1 prompt used in the scored runs serialises the six scored documents’ token-optimised indices together with the navigation instructions and measures \(98{,}562\) tokens (Appendix 11); the three large instruments (the two LPAs and the SPA) account for roughly nine-tenths of it, and the two amendments and the disclosure schedule for the remainder.

Boolean pre-filter flags. Seven semantic boolean fields (Table 2) let the retrieval model narrow a \(500\)-node index to a handful of candidates in a single pass, before reading any summary text. Config 4 also tests two structural fields (textLength, isTocEntry) added on top of the seven semantic flags.

3.5pt

Three-step navigation.

1. Structured scan. The LLM receives *.index.json as its cached system prompt. It applies boolean pre-filters, confirms relevance via titles and summaries, and returns at most ten node IDs.

2. Deterministic fetch. Node IDs are looked up in *.full.json; verbatim provision texts are assembled into a <provisions> block.

3. Grounded answer. The answering model receives only the question and the fetched provisions (no index, no other documents).

The hard cap of ten nodes per question bounds S3 token cost regardless of corpus size. Prompt caching on the index block means subsequent S1 calls pay only for the question text (\(20\)–\(60\) tokens), not the full index, though the model still attends over the full cached index on every scan (Section 3.3).

4pt

^\(\dagger\)Flags: textLength, isTocEntry, hasParty, hasCondition, hasLiabilityLimit, hasPercentage, hasDate, hasMoney, isDefinition.
^\(\ddagger\)Config 6 carries all Config 1 features (cross-references, AI-generated summaries, defined-term flags, and the full Config 4 metadata flag set) but serialises the index with token-optimised JSON: boolean fields are omitted when false, array fields when empty, and string fields when null; key names are abbreviated to two characters; all non-essential whitespace is stripped. These changes reduce the cached index size by \({\sim}30\%\) without altering the navigable content, the one effect attributable to the serialisation itself. Config 6 also moves to the full 20-question set and the Phase III harness, so all cross-configuration comparisons against Configs 1–5 (ties, dollar cost, and token footprint) carry that change; Config 6’s own headline numbers are unaffected, being paired against

within the same run.

Cross-reference and defined-term resolution. Standard RAG retrieves the clause most similar to the query, but legal accuracy requires more. A liability cap may invoke a defined term whose definition appears 20 pages earlier; an operative clause may point to a schedule that contains the actual threshold; an amendment may silently supersede the provision the retriever fetched. Cosine similarity and BM25 measure lexical or semantic proximity; neither can represent these structural dependencies.

The navindex format encodes them explicitly. Every node carries a crossReferencedIds list naming the node IDs it references, and the isDefinition flag marks defined-term nodes. The deterministic fetch in S2 follows the cross-reference edges automatically: retrieving a clause also retrieves every provision it cross-references, without a second similarity search. Defined terms are handled by the flag rather than an edge: the isDefinition marker lets the navigation step (S1) pull in the definitions a question turns on, instead of leaving them to embedding proximity. The answering model therefore receives a provision’s cross-referenced context and the relevant definitions rather than only its highest-scoring neighbours in embedding space. This structural navigation is not available to flat embedding retrieval; whether it is what produces Config 6’s tie result is not established (Section 6).

Encoding legal precedence in prompts. The retrieval format is only half the domain engineering; the released prompts (Appendix 13) carry the other half. The answering prompts encode an amendment-precedence hierarchy (“a later-dated amendment, side letter, restatement, or supplement overrides the original to the extent of any inconsistency”), defined-term origin tracking across documents, and an exact-string silence rule for out-of-scope questions. The S1 navigation prompt encodes a flag-routing table (defined term \(\rightarrow\) isDefinition; amount \(\rightarrow\) hasMoney; …) and an amendment rule: when an amendment addresses the same subject as the original, fetch both. These rules operationalise how transactional lawyers actually read instruments, and they transfer to any legal retrieval system regardless of index format.

Descriptor ladder. The quality–cost trade-off maps onto a descriptor ladder of increasing richness and indexing cost: (D1) heading only; (D2) \(+\) structured flags; (D3) \(+\) extracted keywords and entities; (D4) \(+\) a capped one-line micro-summary; (D5) full section summary. The configurations evaluated here do not climb it one rung at a time (they already sit near the top), so attributing answer quality to a specific rung remains an untested ablation, and the ladder should be read as a design map rather than a result.

5.3 Results↩︎

Table 3 reports inject vs.navindex across six index descriptor configurations. Configurations 1–5 use 15-question subsets; Config 6 uses all 20 questions with the token-optimised index.

The configurations show three things. First, capping returned nodes at \(10\) (Config 2) reduced navindex cost by \({\sim}20\%\) ($1.86 vs.$2.32) with navindex wins unchanged at \(3\), though ties fell from \(11\) to \(9\); over-selection in Config 1 had inflated cost without protecting quality. Second, raising the cap to \(20\) (Config 3) gave no benefit: tie count dropped and inject wins rose, so the additional nodes introduce noise rather than signal. The node cap mattered more than any descriptor enrichment we tested. Third, the token-optimised Config 6 was judged tied with inject on all 18 document-bound questions and on both out-of-scope controls, at lower dollar cost than inject ($1.21 vs.$1.61), the highest single-run tie rate in either comparison. Because Config 6 changes the serialisation and the question set together, comparisons against Configs 1–5 are confounded (Table 3, note \(\ddagger\)); its headline numbers are paired against inject within the same run and are unaffected.

navindex’s efficiency story decomposes into three numbers, all from the per-question ledger (Appendix 11): total attended token footprint is \(1.61\times\) lower than inject (the \({\sim}98.6\)K-token cached index is re-attended on every S1 scan); the answering-model context is \({\sim}56\times\) smaller (\({\sim}2.9\)K tokens of fetched provisions per query vs.\(163\)K); and recorded dollar cost is 25% lower ($1.21 vs.$1.61). The long-context benefit accrues at S3, where the answer is actually composed; the token-footprint benefit is bounded by the index size, which is why it is \(1.61\times\) rather than navembed’s \(17.3\times\).

6 Cross-Mode Discussion↩︎

Table 4 places the headline numbers side by side, and Table 5 attaches exact uncertainty to every headline rate. All intervals are two-sided 95% Clopper–Pearson [24] computed from the printed counts (a convention used throughout the paper). We use this exact binomial method rather than the textbook normal approximation because the counts are small (\(n=15\)–\(20\)) and several rates sit at the \(0/1\) boundary, where approximate intervals become unreliable: the standard Wald interval, for instance, collapses to zero width at a perfect \(18/18\), absurdly implying no uncertainty at all. Each interval is the range of true rates consistent with the observed count, so a perfect \(18/18\) bounds the true tie rate only to \({\geq}0.81\), since any lower rate would make a clean sweep implausibly lucky (\(p^{18} < 0.025\)). navembed was exercised across twelve configurations (Appendix 10) and provides the token-footprint-reduction result; navindex reaches the highest single-run tie rate and is the only mode that navigates by structure, the signal designed for questions that hinge on defined terms, liability caps, or explicit cross-references, though whether that mechanism produces its tie result is untested (Section 8).

A direct head-to-head comparison between RUN-012 and Config 6 is not yet valid even though both use the full 20-question set: each mode was judged only pairwise against its own inject arm (the two modes’ answers were never judged against each other), and matching tie counts against a shared baseline transfer token footprint and dollar cost, not quality rank. On the present evidence the two modes occupy different points on the quality–token-footprint curve, and neither result supersedes the other.

2.6pt

3.2pt

Phase III settled the token-footprint comparison; the dollar comparison is less direct. The price model uses base input at $3.00 per million tokens (\(p_{\text{in}}\)), cache write at $3.75 (\(p_{\text{w}}\)), cache read at $0.30 (\(p_{\text{r}}\)), and output at $15 [26]. Output tokens are excluded from the cost model below because output length did not differ systematically between inject and navigate modes in these runs (median output length within \(\pm 8\%\) across paired answers); if output length diverges substantially in other corpora or question sets, the output term should be reinstated. Every aggregate token and dollar figure for the two headline runs reconciles to the per-question ledgers in Appendix 11 under this price card; dollar accounting is ledger-validated from RUN-005 onward (Phase I dollar cells were recorded under an earlier harness and are excluded; Appendix 10). For a corpus of \(C\) tokens queried repeatedly against a navigate payload of \(R\) tokens, at warm steady state: \[c_{\mathrm{\small inject}{}}^{\text{warm}} = C\,p_{\text{r}}, \qquad c_{\mathrm{\small navigate}{}} = R\,p_{\text{in}} .\] inject is cheaper per query exactly when \[C < R\,\frac{p_{\text{in}}}{p_{\text{r}}} = 10\,R . \label{eq:crossover}\tag{1}\] Practical rule for navembed: cached inject is cheaper in dollars only when the corpus stays below roughly ten times the retrieval payload. The headline RUN-012 payload averages \({\sim}9.4\)K tokens per query, putting break-even near \(94\)K tokens; across the Phase III configurations (payloads \(5.5\)K–\(19.1\)K per query) the threshold spans roughly \(55\)K–\(191\)K. The measured six-document prefix of \(163{,}147\) tokens exceeds the RUN-012 threshold, so navembed stays cheaper at steady state, except for the BM25 configuration (RUN-011), whose inflated \(19.1\)K payload pushes its threshold past the corpus size (Section 4.2).

navindex has a different cost profile (steps as defined in Section 5.2): its S1 index (\({\sim}98.6\)K tokens) is itself cached, so only the small S3 provision payload (\({\sim}2.9\)K tokens per query) is billed at full input price. Warm-state per-query cost is \(I\,p_{\text{r}} + S\,p_{\text{in}}\), where \(I\) is the cached index size and \(S\) the provision payload. The crossover condition becomes \[\begin{gather} C\,p_{\text{r}} < I\,p_{\text{r}} + S\,p_{\text{in}} \;\Longrightarrow\; C < I + S\,\frac{p_{\text{in}}}{p_{\text{r}}} \\[2pt] \approx 98{,}600 + 2{,}900\times 10 \approx 128{,}000 \text{ tokens.} \label{eq:crossover-json} \end{gather}\tag{2}\] Cached inject beats navindex only when the corpus stays below roughly \(128\)K tokens. Both thresholds are exceeded by the measured \(163\)K six-document prefix, matching the recorded outcome ($1.21 vs.$1.61); for a single \({\sim}35\)K-token document, inject is the cheaper steady-state option against either navigate mode. (The Phase II corpus was a different six-document sample of the workspace (it carried the PPG/Comex SPA in place of the disclosure schedule) at \({\sim}214.5\)K tokens per question against the scored corpus’s \(163.1\)K.)

Session economics. The warm-state equation omits the one-time cache write, which must also be amortised, and Anthropic’s prompt cache has a five-minute TTL [18]. Queries arriving more than five minutes apart invalidate the cache entry, and the next call pays the write premium again. In a synchronous review session (many questions in one sitting) the model approximates reality; in an asynchronous workflow (a lawyer asking three questions today and two tomorrow) the cache write may be paid for every session, shifting the crossover to a higher query count or removing it entirely for infrequent use. Cached inject’s cheapness below the threshold therefore assumes a usage pattern that legal review often violates. The analysis here assumes a single warm-cache session. Figure [fig:crossover] shows cumulative costs for two corpus sizes: for a single \({\sim}35\)K-token document (left), cached inject overtakes navembed after about seven repeated questions; for the \(163\)K six-document corpus (right), navembed remains cheaper at every repetition count.

7 Threats to Validity↩︎

What a tie certifies. A tie under the reference-anchored protocol (Section 3.2) certifies that the judge found both answers equally consistent with the verified ground truth, not that either answer is verified correct. Judge error is bounded by the audit (\(2/15\) verdicts likely wrong; Table 5). A forced-decision variant of the navindex Config 2 setup (ties disallowed) on the 15-question Phase II pool, whose DocNavBench questions are disjoint from the scored set (Section 3), showed a \(10\)–\(5\) inject preference, judged with the Phase I–II preference judge. That lean is statistically indistinguishable from chance (two-sided exact binomial test [24], \(p = 0.30\)), the interval (\([0.38, 0.88]\)) is wide, and it attaches to the Phase II questions rather than the scored set. Even so, it remains the only forced-decision datum available. Single-pass tie counts also carry judge-pass noise: two stored judging passes over the identical RUN-010 answers differed on one verdict in twenty (a control-question tie versus a navigate win), so tallies should be read with at least \(\pm 1\) verdict of judging noise, well inside the intervals of Table 5. The forced-decision probe has not been re-run under the final protocol or on the full 20 questions; a forced-decision run on Config 6 is the single most important follow-up, since it would establish whether the headline tie counts reflect genuine interchangeability or differences beneath the judge’s resolution.

Judge–answerer provider overlap. The judge (claude-opus-4-7) shares a provider with the answering model (claude-sonnet-4-6). Because both arms of every pair are answered by the same model, any family-level self-preference [27] applies to both answers symmetrically: the comparison is between retrieval conditions, not model families, so provider bias cannot systematically favour one arm. The residual risk is tie inflation through stylistic coherence, compounded by the reference answers being model-drafted before human verification (Section 3), which the reference-anchored judging criteria (verdicts keyed to key facts against the reference, with explicit instructions to ignore formatting, length, and tone) and the forced-decision probe partially address, and which the audit bounds empirically. A cross-provider judge remains future work.

No ground-truth node labels. Retrieval recall and precision cannot be measured. When navigate answers “not found,” it is impossible to tell whether it fetched the wrong nodes or the right nodes that lacked the answer; the diagnosis of RUN-012’s two inject wins (Section 4.2) is correspondingly conjectural.

Corpus-selection artefacts. In one Phase II run, \(11/15\) questions were unanswerable because amendments were sampled without their base documents. Both methods return “not in documents” and are scored a tie, a phantom tie that inflates apparent convergence. The artefact predates the scored set: the printed 15-question rows are inconsistent with it (their per-question records show substantive retrieval and decisive verdicts on the same questions), and the scored runs are protected by construction rather than by luck. Each document-bound question has a verified ground-truth answer authored against the indexed documents, and the per-question ledgers (Appendix 11) show substantive answers on all 18 with no mass “not found” behaviour; only the 2 out-of-scope controls are deliberately unanswerable.

Out-of-scope controls. The stored answers verify declination directly for the shared inject arm (reused across RUN-010–012): both control answers are the mandated refusal string. The stored navigate answers in the ground-truth-judged oracle run likewise open with the exact refusal; for RUN-012’s and Config 6’s navigate arms the controls are certified as judged tied against a reference that says the answer is absent, under a criterion where a fabricated answer loses.

Difficulty profile and phrasing. The internal difficulty labels (14 high, 6 medium; Appendix 12) rate retrieval dispersion (how widely the answer spreads across clauses and documents), not cognitive difficulty: the reviewing lawyer assessed only a handful of the 20 as genuinely hard, the majority being precise retrieval-and-comprehension questions a modern LLM answers reliably once the governing text is in context. That cuts both ways. It makes the comparison a clean test of the retrieval condition, since ties are the expected outcome when both arms receive the right provisions and the informative signal is the failures; but the evaluation says nothing about legal reasoning. The lawyer also noted the questions are phrased answer-aware, written as if the asker already knows what the document contains, unlike DocNavBench’s issue-driven register (Section 3), so the phrasing may flatter retrieval relative to real-world queries.

Scale. All results come from one answering model, one judge, and one run per configuration, on a 20-question set over six documents; Table 5 quantifies the resulting uncertainty, and the wide intervals are the honest reading of every rate in this paper. The navembed and navindex comparisons are asymmetric in configuration count (twelve vs.six), and only their strongest configurations share the scored question set, so the cross-mode reading (Section 6) is indicative, not definitive.

8 Lessons Learned and Future Work↩︎

Four operational lessons transfer beyond this system. (1) Cap the selection. The ten-node cap cut navindex cost by \({\sim}20\%\) with wins unchanged (ties dipped \(11{\to}9\); Table 3), and every uncapped or weakly-signalled variant (titles-only scan, 20-node cap, all-related-nodes) over-selected; a hard cap is the single highest-leverage retrieval control we found. (2) Lexical fusion can silently cost money. BM25 length bias inflated the hybrid payload \(2\times\) over pure semantic retrieval and pushed the configuration past the caching crossover, worse than inject at steady state. (3) What a node carries matters more than which embedding scores it. Embedding swaps moved outcomes less than descriptor enrichment across Phases I–II (Appendix 10), which is what motivated the navindex format. (4) Track token footprint and dollar cost separately. Prompt caching makes dollar comparisons corpus- and session-dependent (Section [sec:sec:crossover]); token-footprint reduction is the durable quantity.

The evaluation gaps, in order of urgency: a forced-decision judging run on Config 6 (Section 7); ground-truth node labels to measure retrieval recall directly and to diagnose the two RUN-012 losses; a head-to-head judging run in which the two navigate modes’ answers are compared directly rather than each against inject; evaluation on the 52-question open-search DocNavBench set; substring rubrics (must_contain) as a judge-free scoring dimension; easy single-fact questions to establish a retrieval floor; corpus pairing that always includes base agreements with amendments; questions that require reconciling an original agreement against its amendments across documents (a cross-instrument dependency the present question set does not isolate); a second-annotator pass over the headline runs’ judge verdicts; a cross-provider judge; and integrating a LEDGAR-style clause-type classifier [21] as an index field.

9 Conclusion↩︎

inject (placing the entire corpus in context) requires no retriever, and on a small corpus queried repeatedly within a warm cache session it is the cheaper option in dollars by the crossover rule of Section [sec:sec:crossover]. Outside that regime, both structured alternatives held answer quality at a fraction of the token footprint on this evaluation, with the caveats of Section 7.

Against navembed, semantic retrieval with reranking (RUN-012) was judged tied with inject on 16 of 18 document-bound questions (inject preferred on 2) while attending to \(17.3\times\) fewer input tokens (\(188\)K vs.\(3.26\)M); GTE retrieval pushed the reduction to \(29.9\times\) at a lower tie rate (\(15/20\)). Across all twelve configurations the reduction never fell below \(8.5\times\), and the binding constraint on quality was descriptor richness, not embedding choice.

Against navindex, its index format encodes the defined-term and cross-reference relationships a legal answer depends on as traversable edges, and the deterministic fetch follows them without a second similarity search. Its token-optimised configuration was judged tied with inject on all 18 document-bound questions (the highest single-run tie rate in either comparison) at \(1.61\times\) lower total token footprint, a \({\sim}56\times\) smaller answering context, and 25% lower recorded dollar cost ($1.21 vs.$1.61).

For practitioners the decision rule is short: below \({\sim}10\times\) the retrieval payload (a constant set by the cache-read discount \(p_{\text{in}}/p_{\text{r}}\) under the Section [sec:sec:crossover] price card), inject and cache; above it, retrieve, with navembed when maximal token-footprint reduction matters and navindex when structural fidelity does. Real transaction sets outgrow the context window, where inject stops being an option at all, and asynchronous legal workflows re-pay the cache write per session, eroding cached inject’s advantage even below the threshold. Structured retrieval is how the system holds quality when the corpus keeps growing.

Acknowledgements↩︎

We are grateful to Horace Wu, who authored the 52-question DocNavBench open-search set and verified the scored benchmark’s reference answers against the source documents, and to Mahmoud Abdallah for his work on the document-index API and surrounding features.

10 Run History and Lessons↩︎

The twelve navembed configurations were run in three phases as the evaluation harness matured. Each run is an internally controlled paired comparison (inject vs.navembed on identical questions), but five things changed between phases: the question pool (the DocNavBench subset in Phases I–II vs.the scored set in Phase III; Section 3), the corpus sample (Phase II carried the PPG/Comex SPA in place of the disclosure schedule; Section [sec:sec:crossover]), the prompts, the caching setup (integrated at RUN-005), and the judge (the Phase I–II preference judge, Appendix 13.5, vs.the reference-anchored judge, Appendix 13.4; Section 3.2). Cross-run tie counts are therefore run history rather than a controlled sweep; the paper’s claims rest on the Phase III runs (Section 4.2). Footprint reduction (the deterministic quantity) held at \(8.5\)–\(29.9\times\) across all twelve runs.

4pt

^\(\ddagger\)Oracle configuration: uses ground-truth node labels to validate retrieval at indexing time; excluded from cross-mode comparison (§[sec:sec:compare]

).

Phase I (RUN-001–004): harness construction. The first runs built and validated the harness on 3–9 questions from the Phase I–II pool (a 15-question DocNavBench subset; Section 3): filtering irrelevant nodes, stripping rationale from model output, and adding cross-reference expansion. navembed already used \(10\)–\(15\times\) fewer tokens than inject; RUN-003 produced the first outright navembed win, and RUN-004 closed the phase with three navembed wins and six ties.

Phase II (RUN-005–008): caching and embeddings. Moving to the full 15-question pool with prompt caching integrated, the token footprint held \(14\)–\(19\times\) below inject; embedding swaps were tried here (Section 4.2): paraphrase-MiniLM-L6-v2 (the incumbent default), intfloat/e5-base-v2, and BAAI/bge-base-en-v1.5 were compared on the same pool, and MPNet QA embeddings (multi-qa-mpnet-base-cos-v1, RUN-008) gave the best Phase II tie count (\(9/15\)).

Phase III (RUN-009–012): the scored comparison. On the full 20-question set with the final harness, outcomes are reported in Section 4.2. The inject wins concentrated in Phase II, while the harness was still being tuned (Table 6, Inj. w.column); this trend is an engineering observation, not a measured result, since the question sets differ across phases.

11 Per-Question Detail for Two Representative Runs↩︎

Tables ¿tbl:tab:cfg6? and [tbl:tab:exp012] give per-question token and cost breakdowns. Column key: out = output tokens; CW = cache-write (Q1 only); CR = cache-read (Q2–Q20); in = non-cached input; Inj/Nav = cost in USD. Every aggregate token and dollar figure reported for these two runs in the body reconciles to these ledgers under the Section [sec:sec:crossover] price card. Two bookkeeping notes: the inject token totals in Table 6 and Table 3 additionally include \(856\) non-cached question-text input tokens not broken out as a column here; and the Nav rows likewise include \(20\)–\(60\) tokens of non-cached S1 question-text input per query not broken out as a column.

2.5pt

2.5pt

12 Scored Question Set↩︎

Table 7 lists all 20 scored questions with verified ground-truth answers; 18 are document-bound and 2 are out-of-scope controls that test correct declination. All are high difficulty except those marked \(\dagger\) (medium).

4pt

13 System Prompts↩︎

All prompts are reproduced verbatim from prompts.py. Runtime substitutions are shown as {placeholder}.

13.1 INJECT Answer Prompt (INJECT_ANSWER_SYSTEM)↩︎

You are a senior legal analyst preparing a precise, citation-anchored answer
for a partner working across several related documents.

# Source of truth
The only authoritative source is the documents delimited by
<document doc_id="..."> tags below.
- Each <document> is a separate instrument; treat them as a coordinated
  transaction set.
- Do not import general legal knowledge or assumptions from outside the
  documents.
- Defined terms have ONLY the meaning given in the documents. A term may be
  defined in one document and used in another -- track defined-term origin.

# Reasoning method (internal -- do not output your scratch work)
1. Decompose the question into sub-questions, conditions, thresholds, parties.
2. For each sub-question, identify which document(s) contain relevant text.
3. Resolve cross-references between documents (e.g. an amendment that modifies
   a clause of the original).
4. Reconcile precedence:
   - A later-dated amendment, side letter, restatement, or supplement OVERRIDES
     the original to the extent of any inconsistency.
   - A more specific provision usually overrides a more general one.
   - When in doubt, state which document you treated as controlling and why.
5. Compose a single integrated answer.

# Answer format
- Lead with the direct answer (1-2 sentences). Then a short supporting analysis.
- Cite every load-bearing fact as: "[doc_id] Clause X".
- Quote short, decisive phrases verbatim where wording matters.
- Preserve exact numeric values, currencies, dates, and party names.
- If documents conflict, name the controlling document and explain why.
- If a defined term is used: identify which document defines it.

# When the documents are silent
If the documents do not address the question, respond EXACTLY:
"The provided documents do not contain the answer."

{documents_block}

13.2 NAVINDEX S1 Prompt (NAVINDEX_SCAN_SYSTEM)↩︎

The model sees only the compact index JSON (no provision text) and returns a list of (doc_id, node_id) pairs to fetch.

You are a legal retrieval planner working across several related documents in a
transaction set (e.g. an original agreement plus amendments, side letters,
schedules). Your job is to decide which provisions across all documents a human
lawyer would need to read end-to-end to answer the user's question.

# What you can and cannot see
You see ONLY the indices -- one per document -- containing clause references,
headings, text snippets, summaries, keyword tags, and structural metadata.
You do NOT see the full provision text, only the opening snippet of each node.
You must decide what to fetch from STRUCTURAL SIGNALS ALONE.

# Index field guide (same structure per document)
Each <index doc_id="..."> block has:
- documentTitle         -- the document's title.
- documentTags          -- Jurisdiction (governing law) and Topic (key
                           subject-matter terms). Use to identify which
                           documents are likely relevant before scanning nodes.
- documentIndex         -- a recursive tree of section nodes. Each node has:
  - nodeId              -- the identifier you return. Always use this, never
                           clauseReference.
  - clauseReference     -- printed clause reference. Use for orientation only.
  - title               -- clause heading; primary topical signal.
  - summary             -- AI-generated summary. Use to confirm relevance when
                           title and snippet are insufficient.
  - keywords            -- topic tags applied to this clause.
  - crossReferencedIds  -- nodeIds this clause explicitly cross-references.
                           Follow selectively for definitions/conditions.
  - children            -- nested sub-clauses, recursively structured.
  - isDefinition        -- true if this node defines one or more terms.
  - hasMoney            -- true if the provision contains a monetary amount.
  - hasDate             -- true if the provision contains a date or deadline.
  - hasPercentage       -- true if the provision contains a percentage or rate.
  - hasLiabilityLimit   -- true if the provision contains a liability cap.
  - hasCondition        -- true if the provision contains a condition precedent.
  - hasParty            -- true if the provision references a named party.
  - isTocEntry          -- true if table-of-contents entry. NEVER include these.
  - textLength          -- character count. Nodes under ~50 chars are stubs.

# Retrieval strategy (think before you answer)
1. Decompose the question: sub-questions, concepts, thresholds, conditions,
   parties.
2. Scan documentTags.Topic across all documents to identify relevant documents.
3. Discard any node where isTocEntry is true.
4. Treat EVERY document as in scope until you have a structural reason to rule
   it out. The answer often lives in more than one document.
5. Use boolean flags as fast filters before reading summaries:
   - defined term in question  -> isDefinition: true nodes across all docs
   - amount or fee             -> hasMoney: true nodes
   - date or deadline          -> hasDate: true nodes
   - rate or percentage        -> hasPercentage: true nodes
   - liability cap             -> hasLiabilityLimit: true nodes
   - condition or trigger      -> hasCondition: true nodes
   - specific party            -> hasParty: true nodes
6. Confirm each candidate via title, snippet, summary, keywords.
7. When an amendment addresses the same subject as the original, include BOTH.
8. Follow crossReferencedIds only when the node supplies a definition or
   condition the selected clause explicitly depends on.

# Hard rules
- Return AT MOST 10 (doc_id, node_id) pairs.
- doc_id MUST exactly match a <index doc_id="..."> value in the block below.
- Do not invent doc_ids or node_ids.
- Decide from structural signals only -- do not speculate about provision text.

{indices_block}

13.3 NAVINDEX / NAVEMBED Answer Prompt (NAVIGATE_ANSWER_SYSTEM)↩︎

The model sees only the fetched provision texts and composes the final answer.

You are a senior legal analyst answering a question from a CURATED EXTRACT
spanning several related documents. The provisions below were retrieved because
they were judged likely to be relevant; you must answer from this extract alone.

# Source of truth
The only authoritative source is the text inside <provisions>...</provisions>.
- Each provision is tagged "[doc=..., clause_ref=..., node_id=...]". Always cite
  BOTH the doc and the clause.
- An amendment, side letter, or restatement OVERRIDES the original to the
  extent of any inconsistency.
- Do not invoke general legal knowledge from outside the extract.
- Defined terms have ONLY the meaning given in the extract. If a term is used
  but its definition is NOT in the extract, say so.

# Relevance filtering -- DO NOT output any part of this step.
Before reasoning, screen every provision against the question:
- Mark each provision RELEVANT or IRRELEVANT.
- Answer EXCLUSIVELY from the RELEVANT provisions.
- If uncertain, err on the side of INCLUDING -- over-exclusion produces a false
  "no answer" finding; over-inclusion is a minor inefficiency.
- Only mark IRRELEVANT if the provision clearly addresses a different subject
  with no bearing on any sub-question.

# Reasoning method -- DO NOT output this step.
1. Decompose the question into sub-questions, conditions, thresholds, parties.
2. For each sub-question, identify which document(s) bear on it.
3. Resolve precedence:
   - A later-in-time amendment overrides the original to the extent of
     inconsistency.
   - A more specific provision usually overrides a more general one.
4. Resolve cross-references within the extract. Flag pointers outside it.
5. Compose a single integrated answer.

# Answer format
- Lead with the direct answer (1-2 sentences). Then a brief supporting analysis.
- Cite every load-bearing fact as: "[doc_id] Clause X".
- Quote short, decisive phrases verbatim where wording matters.
- Preserve exact numeric values, currencies, dates, and party names.
- When documents conflict, name the controlling document and explain why.

# Partial-information rule
If the extract is missing a key linked clause: state what the extract DOES
establish, identify what is missing, and do not fabricate the missing content.

# Silence rule
If the extract does not address the question at all, respond EXACTLY:
"The provided text does not contain the answer."
Only invoke this after confirming every provision is genuinely irrelevant.

<provisions>
{provisions}
</provisions>

13.4 Ground-Truth Judge Prompt (JUDGE_GROUNDTRUTH_SYSTEM)↩︎

Used for the scored benchmark (20 questions with verified reference answers). The forward and reverse calls share an identical system prompt; only the user message (which places Answer A and Answer B) changes between orderings.

You are evaluating two model answers to a legal question. A verified reference
answer is provided -- treat it as the authoritative ground truth.

# Your task
Decide which model answer (Answer A or Answer B) more accurately reflects the
reference answer, or declare a tie if both are equally correct or equally wrong.

# Judging criteria (in order)
1. Key facts -- does the answer include the specific facts stated in the
   reference (exact numbers, dates, party names, amounts, percentages, clause
   citations)? Missing or wrong key facts are the primary basis for a loss.
2. No hallucination -- does the answer avoid asserting facts that contradict
   the reference or that are not supported by the reference?
3. Completeness -- if the reference has multiple parts, does the answer address
   all of them?
4. No-answer correctness -- if the reference says the answer is not in the
   documents, an answer that correctly says "not found" is correct; an answer
   that fabricates a plausible-sounding fact is wrong.

# Verdict rules
- Return "A" if Answer A is closer to the reference.
- Return "B" if Answer B is closer.
- Return "tie" if both are equally correct, equally wrong, or differ only in
  style while containing the same key facts.
- A partial answer beats a "not found" when the reference confirms a fact
  exists -- and loses to it when the reference confirms the answer is absent.

# Anti-bias rules
- Ignore formatting, length, and tone.
- Do not prefer more verbose or more confident answers.
- The reference may be terse; do not penalise extra detail that is not
  contradictory.

The question, reference answer, and two model answers are in the user message.

13.5 Phase I–II Preference Judge (JUDGE_PREFERENCE_SYSTEM)↩︎

Used to score the Phase I–II runs, navindex Configs 1–5 included: a pairwise comparison of two anonymised answers with no reference answer. The 20-question scored runs use the reference-anchored judge of Appendix 13.4 instead. The criteria block is static across every judge call in a run, so it forms a cacheable prefix shared by the forward and reverse calls.

You are comparing two anonymised answers to the same legal question. Decide which answer is better, or whether they are equivalent.

# Judging criteria (in order of importance)
1. Factual correctness -- does the answer state the right facts? An answer with one wrong fact loses to an answer with all right facts, even if the right-fact answer is shorter.
2. Completeness -- does it cover everything the question asks? A multi-part question requires every part addressed.
3. Specificity -- does it cite the controlling clause? does it preserve exact numeric values, currencies, dates, parties?
4. Honesty about uncertainty -- an answer that correctly flags missing context beats one that hallucinates a confident wrong answer.
5. Clarity -- is it unambiguous and easy to follow? (least important tiebreaker)

# Verdict rules
- Return "A" if A is clearly better on the criteria above.
- Return "B" if B is clearly better.
- Return "tie" if both are equally good, equally bad, or differ only in style.
- Do not break ties artificially. If neither is meaningfully better, "tie" is the correct answer.

# Anti-bias rules
- Ignore length unless one answer is so terse it omits required information.
- Ignore formatting (bullet points vs. prose).
- Ignore tone and confidence wording.

The question and the two anonymised answers will be provided in the user message tagged <question>, <answer_a>, and <answer_b>. Return your verdict using the JSON schema you have been given.

References↩︎