1 Introduction↩︎

The ability of large language models (LLMs) to solve complex reasoning tasks has grown substantially, yet a persistent gap remains between peak performance—achievable under carefully crafted prompts—and robust performance across distributional variation. Understanding the nature of this gap is central to deploying LLMs reliably in high-stakes domains such as automated software security analysis.

1.1 The Router Hypothesis↩︎

Cázares [1] introduced the router hypothesis in the context of the SAIR Equational Theories Stage 1 competition: language models route problems to cached structural patterns rather than deriving answers compositionally. Injecting a structural prior—a compact cheatsheet enumerating canonical solution routes—can dramatically raise in-distribution (ID) performance. However, the same cheatsheet creates a distribution-shift hazard: when the test distribution shifts even modestly, the injected routing prior misaligns with the new patterns, systematically directing the model toward incorrect solution paths. Prior work further found that a more heavily engineered variant (AN45c), which peaks on a local hard split, performs worse than its simpler predecessor (AN38) on the official evaluation set, suggesting that the collapse reflects a structural property rather than a calibration artifact.

1.2 Extending the Hypothesis to Code Security↩︎

Software vulnerability detection is a natural and practically important stress test for the router hypothesis. Like formal mathematics, it requires precise pattern-matching against structured artifacts (source code), hierarchical categorization (CWE taxonomies), and reasoning under distributional shift—since vulnerabilities in production code look substantially different from textbook examples or synthetically generated code snippets.

Prior LLM-based vulnerability detection work has demonstrated strong performance on curated benchmarks [2], [3], but systematic analysis of the cheatsheet-induced trade-off between ID and OOD performance has not been undertaken. We fill this gap.

1.3 Contributions↩︎

This paper makes the following contributions:

1. We replicate the experimental protocol of [1] in the code security domain, establishing a cross-domain baseline for the router hypothesis (§3).

2. We demonstrate that structural priors saturate synthetic vulnerability detection (up to 100 % F1) across three vulnerability categories (two CWEs — CWE-798, CWE-284 — plus one non-CWE anti-pattern, N+1) spanning syntactic, contextual, and semantic complexity, providing evidence for the router hypothesis at scale (§4).

3. We characterize the semantic complexity gradient: zero-shot LLM performance on vulnerability detection degrades predictably for GPT-OSS-120B and partially for Gemma-4-31B as a function of the semantic distance between code pattern and vulnerability class (§4.2).

4. We document amplified distribution-shift collapse: the same structural priors that saturate synthetic performance drive real-world F1 below the zero-shot baseline by up to 58 pp, exceeding the magnitude of the analogous collapse reported by [1] (§5).

5. We show that iterative recalibration amplifies rather than corrects the collapse, with the v2 cheatsheet calibrated against real error patterns underperforming v1 on the same real distribution—an exact structural parallel to the AN45c finding of [1] (§6).

6. We characterize zero-shot misrouting as a genuine but correctable failure mode: models frequently detect real vulnerabilities but assign incorrect CWE labels under zero-shot conditions (13 synthetic and 19 real-world events), a pattern structural priors eliminate entirely—though we note this correction is partly attributable to the prior directly supplying the category label (§7.4).

1.4 Significance↩︎

We refer to this evaluation framework—spanning datasets, cheatsheets, and cross-distribution evaluation protocol—as BitCoder. The practical implication is stark. Prompt-engineering-based security pipelines—checklists, few-shot exemplars, structured routing prompts—may achieve impressive numbers on synthetic or held-out benchmarks while silently degrading on the real CVE distributions they are ultimately deployed against. Our results suggest this degradation is not a tuning problem solvable by better prompt engineering, but a structural consequence of distributional mismatch between the routing priors and the target domain. This motivates an alternative path: distribution-aware training on hybrid synthetic+real datasets, which we discuss in §7.

More broadly, by reproducing the cross-distribution trade-off surface of  [1] in an entirely different domain (mathematics \(\to\) code security), with different models, datasets, and vulnerability taxonomies, we provide evidence for the cross-domain replication of the router hypothesis—suggesting it reflects a fundamental property of how LLMs activate and apply learned sub-skills rather than an artifact of any particular task or architecture.

1.5 Paper Organization↩︎

The remainder of the paper is structured as follows. §2 reviews the router hypothesis and the prior study of [1]. §3 describes our datasets, models, and experimental conditions. §4 presents results on synthetic (in-distribution) data. §5 presents results on real-world (VUDENC) data. §6 provides the v1 vs. v2 comparison and error analysis; zero-shot misrouting is analyzed in §4.3. §7 discusses implications and limitations. §8 concludes.

2 Background↩︎

2.1 The Router Hypothesis and SAIR↩︎

Cázares [1] studies prompt engineering for formal mathematical reasoning in the context of the SAIR Equational Theories Stage 1 competition hosted by The Foundation for Science and AI Research (SAIR). The task is to decide, given equational laws \(E_1\) and \(E_2\) over magmas (algebraic structures with a single binary operation), whether \(E_1\Rightarrow E_2\) holds universally (True) or a finite counterexample magma exists (False), using the public SAIRfoundation/equational-theories-selected-problems dataset. That work evaluates three models—gpt-oss-120b, Llama 3.3 70B, and Gemma 4 31B—and documents a single-prompt ceiling (equivalently, an empirical saturation region): a zone where in-distribution accuracy gains become unstable and non-generalisable across problem distributions.

Cázares [1] states the router hypothesis as: language models “route problems to cached structural patterns rather than deriving answers compositionally.” In this paper we adopt that framing and operationalise routing as the selection of an internal structural interpretation template that conditions downstream reasoning behavior on a given input. We infer routing behavior from systematic changes in error topology under fixed model weights and deterministic decoding, without claiming access to model internals. Our claims concern externally observable behavioral regularities under structural prior perturbation, not direct access to internal activation pathways. Evidence for the cross-distribution consequence comes from two findings reported by [1]:

1. Cheatsheet lift: On the local hard3 split (\(n{=}400\)), the AN45c cheatsheet lifts GPT-OSS-120B from a no-cheatsheet baseline of 59.75 % to 79.25 %—a 19.5 pp gain within the approximately 60–79 % empirical saturation region reported in that work.

2. Distribution-shift collapse: On the official SAIR competition benchmark, AN45c drops from 79.25 % local to 55.5 % official (\(-\)​23.75 pp), falling below the no-cheatsheet baseline, whereas the simpler predecessor AN38 remains robust (71.8 % local \(\to\) 65.3 % official; \(+5.6\) pp vs.the official baseline).

These findings of [1] motivated the hypothesis that the cross-distribution trade-off surface is a fundamental property of structural prior injection, not a domain-specific artefact.

2.2 LLM-Based Vulnerability Detection↩︎

Automated vulnerability detection has a long history, from static analysis [4] and symbolic execution to machine-learning classifiers trained on code representations [5]. LLM-based approaches have gained prominence recently, showing strong zero-shot performance on curated benchmarks such as BigVul [6], VUDENC [3], and Juliet [7].

However, the gap between synthetic/curated benchmark performance and real-world CVE detection remains poorly understood. Our work positions this gap as an instance of the distribution-shift collapse consistent with the router hypothesis proposed by Cázares [1], providing the first systematic cross-distribution analysis in this domain.

2.3 CWE Taxonomy and Vulnerability Complexity↩︎

The Common Weakness Enumeration (CWE) taxonomy [8] provides a hierarchical categorization of software weaknesses. We exploit the structure of this taxonomy to define a semantic complexity gradient across three vulnerability classes:

• Syntactic (CWE-798, Hardcoded Credentials): The vulnerability is detectable from local token patterns (literal strings, credential-like identifiers) without semantic reasoning.

• Contextual (CWE-284, Improper Access Control / IDOR (insecure direct object reference)): Detection requires understanding the relationship between resource identifiers and authorization checks—requiring limited inter-procedural reasoning.

• Semantic (N+1 Query Pattern): The vulnerability is defined by a runtime behavior pattern (repeated database queries inside a loop) that is invisible from any single code location and requires understanding of the ORM (object-relational mapping) execution model.

This gradient predicts that zero-shot routing difficulty increases from CWE-798 to CWE-284 to N+1, a prediction confirmed for GPT-OSS-120B and partially for Gemma-4-31B (100 % zero-shot on both CWE-798 and CWE-284, dropping to 72.7 % on N+1); see §4.2 for full per-model analysis.

3 Methodology↩︎

3.1 Models↩︎

We evaluate three instruction-tuned LLMs hosted on Together AI in bfloat16 precision—the same three-model set and local hosting provider used by [1]—to maximise comparability:

All runs use temperature \(= 0\) and seed \(= 0\), matching the SAIR competition’s official benchmark decoding settings; the local paper runs of [1] instead used seed \(= 42\) on Together AI. We host on Together AI (as in those local runs) with max_tokens 4096 for GPT-OSS-120B / Llama-3.3-70B and 8192 for Gemma-4-31B. GPT-OSS-120B uses reasoning=low; the two open-weight models use reasoning=disabled.

3.2 Datasets↩︎

3.2.0.1 Synthetic corpus.

We generated 278 labeled training pairs and 70 held-out evaluation pairs (348 pairs total, stored as 696 raw JSONL records) using DeepSeek-V4-Pro (temperature \(= 0.8\), Together AI) across three vulnerability categories. For this paper we evaluate on three categories with completed annotations (Table 2).

3.2.0.2 Real-world corpus (VUDENC).

VUDENC [3] provides Python code fragments extracted from real-world CVE-fixing commits. We evaluate on two CWE classes: SQL Injection (CWE-89, 8,646 vulnerable / 10,223 benign) and Path Traversal (CWE-22, 1,701 / 2,469). Evaluation subsets are sampled as 15 vulnerable + 15 benign = 30 samples (seed \(= 0\)) to maintain balance and reproducibility.

3.3 Experimental Conditions↩︎

We test three conditions:

• Zero-shot (A): The model receives only the code snippet and a task description requesting a binary vulnerability classification and a CWE route label.

• Cheatsheet v1 (B): The prompt is augmented with a category-specific cheatsheet enumerating canonical detection routes, structural signatures, and negative patterns. Cheatsheets were authored manually based on the synthetic training distribution.

• Cheatsheet v2 (B2): A revised cheatsheet for CWE-89 and CWE-22 only, calibrated using the false-positive and false-negative error patterns identified in the initial VUDENC evaluation.

Conditions C (unified cheatsheet) and D (progressive saturation) are reserved for future work.

3.4 Evaluation Metrics↩︎

We report vulnerable-class F1, vulnerability recall (VulnRec), benign recall (BenRec), and misroute count. Unless otherwise noted, F1 denotes vulnerable-class F1 throughout this paper. Misrouting is defined as a prediction that correctly identifies a sample as vulnerable but assigns an incorrect CWE route label. The distribution-shift delta (\(\Delta\)) is computed as \(\text{F1}_\text{real} - \text{F1}_\text{synth}\).

3.5 Reproducibility↩︎

Full source, including the evaluation harness, cheatsheets, and processed dataset splits, is available at https://github.com/bytepro-ai/bitcoder-v2-research. All experiments are reproducible via:

python scripts/evaluate_baseline.py \;
  --category [CWE-798|N+1|CWE-284|CWE-89|CWE-22] \;
  --model [gpt-oss-120b|llama-70b|gemma-31b] \;
  --condition [zero_shot|cheatsheet_isolated|both] \;
  --cheatsheet-version [v1|v2] \;
  --max-samples 30 \;
  --dataset [synthetic_eval|vudenc]

Error analysis (75 FP, 47 FN, 19 misroutes) is computed from the 12 initial VUDENC evaluation CSVs (*_vudenc.csv, excluding v1/v2 reruns). Total inference cost across all reported experiments: $0.647.

4 Results: Synthetic Distribution↩︎

4.1 Cheatsheets Saturate In-Distribution Performance (F1)↩︎

Table 3 presents the full synthetic evaluation matrix. Across all three vulnerability categories and all three models, cheatsheet-augmented prompts match or exceed zero-shot performance in 17 of 18 conditions, with several reaching the 100 % F1 ceiling (the single exception: Gemma-4-31B on CWE-798, where cheatsheet F1 is 94.7 % vs 100.0 % zero-shot).

The most dramatic gain occurs for the N+1 semantic category under GPT-OSS-120B: zero-shot F1 is 28.6 % (VulnRec \(= 20.0\,\%\)), rising to 96.8 % F1 (VulnRec \(= 100.0\,\%\)) with cheatsheet v1—a 68.2 pp improvement. All three models reach 100.0 % VulnRec on N+1 with the cheatsheet (GPT-OSS-120B achieves 96.8 % F1, Llama-3.3-70B 100.0 %, Gemma-4-31B 95.2 %), confirming that the underlying capability is present in each model and that the zero-shot gap is a routing artefact.

4.2 Semantic Complexity Gradient (F2)↩︎

Zero-shot F1 for GPT-OSS-120B follows the predicted gradient: CWE-798 (syntactic) \(= 90.9\,\%\) \(>\) CWE-284 (contextual) \(= 85.7\,\%\) \(>\) N+1 (semantic) \(= 28.6\,\%\). This monotonic gradient holds strictly for GPT-OSS-120B (90.9 % \(\to\) 85.7 % \(\to\) 28.6 %); Gemma-4-31B shows a partial gradient (100 % \(\to\) 100 % \(\to\) 72.7 %), but not for Llama-3.3-70B, which shows an inverted pattern on N+1 (87.5 % zero-shot F1), likely reflecting richer ORM pattern coverage in its training data. This 62 pp gap between syntactic and semantic categories under zero-shot conditions reflects the routing cost of semantic reasoning—the model has no reliable path to activate N+1 detection from first principles alone. The cheatsheet effectively collapses this gradient, raising all three categories to near-ceiling performance.

4.3 Zero-Shot Misrouting: A Correctable but Ambiguous Signal↩︎

After correcting the misroute-detection harness (§7.4), genuine misrouting events—samples correctly detected as vulnerable but assigned an incorrect CWE label—occur only under zero-shot conditions: 13 events on the synthetic corpus (N+1: 3 for GPT-OSS-120B, 4 for Llama-3.3-70B, 4 for Gemma-4-31B; CWE-284: 2 for Llama-3.3-70B) and 19 events on VUDENC. The real-world events are concentrated and systematic: Llama-3.3-70B routes 10 of its 15 detected CWE-22 vulnerabilities to “Arbitrary Code Execution” or “Command Injection” labels (triggered by os.popen co-occurrence), and zero-shot N+1 detections scatter across denial-of-service, broken-access-control, and performance-anti-pattern labels. By contrast, cheatsheet conditions show zero misroutes across all 17 synthetic and real-world runs.

We caution against reading this as evidence of corrected internal routing: the cheatsheet supplies the target category name directly, so the absence of misrouting under cheatsheet conditions is partly a labeling triviality rather than proof of resolved routing confusion. What the pattern does support is that zero-shot category assignment is unreliable in a way that is fully masked, but not necessarily mechanistically explained, by structural priors.

5 Results: Real-World Distribution (VUDENC)↩︎

5.1 Amplified Distribution-Shift Collapse (F3)↩︎

Table 4 presents the full synthetic-to-real transfer results. The pattern is consistent: cheatsheet conditions that saturate synthetic F1 collapse to below zero-shot performance on VUDENC.

\(^\dagger\)No synthetic CWE-89/CWE-22 corpus exists in this phase. Synthetic F1 is proxied by the CWE-798 zero-shot F1 value, the closest syntactic-category analogue, to indicate the direction of distribution shift. Because no matched synthetic CWE-89/CWE-22 corpus exists in this phase, the synthetic comparison serves as a structural proxy rather than a matched-distribution transfer pair.

The worst-case collapse is GPT-OSS-120B on CWE-89 with cheatsheet v2: 100 % synthetic F1 to 41.7 % real F1 (\(-\)​58.3 pp). Compared with the \(-\)​23.75 pp collapse reported by [1] (AN45c on the official test set), the BitCoder collapse is more than twice as severe. We attribute the amplification to the larger structural gap between synthetically generated Python code and real CVE-associated commits: while the distribution shift in [1] is between problem instances from the same formal theory, BitCoder’s shift crosses the synthetic/natural divide.

6 Analysis↩︎

6.1 Iterative Calibration Amplifies Collapse (F5)↩︎

The v2 cheatsheet was constructed by analyzing 75 false positives and 47 false negatives from the initial VUDENC evaluation and manually revising the routing priors to exclude the identified false-positive patterns (e.g., internal variable interpolation, pre-sanitized inputs) and add cues for the identified false-negative patterns (e.g., class hierarchy traversal, indirect path construction). Table 4 shows the result: for GPT-OSS-120B, v2 underperforms v1 on both CWE-89 (\(-\)​7.2 pp) and CWE-22 (\(-\)​5.8 pp).

This mirrors the AN45c finding of [1]: a more heavily engineered cheatsheet that peaks on a local evaluation split can underperform its simpler predecessor (AN38) under official distribution shift, introducing failure modes outside the local calibration distribution. The implication—reinforced here by our v2 error-pattern recalibration—is that distribution-shift collapse cannot be resolved through iterative prompt calibration; the collapse is structural, rooted in the mismatch between the routing prior’s reference distribution and the target deployment distribution.

6.2 Error Analysis: What Real CVE Code Looks Like↩︎

Post-hoc analysis of the VUDENC evaluation errors (75 FP, 47 FN, 19 misroutes) identifies three dominant patterns:

6.2.0.1 False Positives (CWE-89).

• Internal table/column names interpolated in SQL strings (PRODUCTS_TABLE, ORDER_BY): the model’s routing prior flags any dynamic SQL construction, conflating developer-controlled constants with user-controlled input.

• Pagination parameters (offset, page) used in query limits: treated as injection vectors despite being type-coerced to integers upstream.

• Pre-sanitized variables (prevented_name): the routing prior has no representation for downstream sanitization.

6.2.0.2 False Negatives (CWE-22).

• Vulnerabilities residing in class hierarchies: the vulnerable do_GET method is inherited from SimpleHTTPRequestHandler and the routing prior does not account for inherited dispatch.

• Indirect traversal without explicit open() calls: path construction that eventually feeds a file operation several frames away.

6.2.0.3 Misroutes.

The 19 misrouting events (all occurring under zero-shot conditions; see §4.3) include:

• CWE-22 \(\to\) Command Injection: triggered by co-occurrence of os.popen and a path variable.

• CWE-22 \(\to\) NoSQL Injection: triggered by mongo.find adjacent to a user-supplied path argument.

These patterns confirm that category assignment is lexically sensitive: surface token co-occurrence overrides structural vulnerability semantics in zero-shot route labeling. This lexical sensitivity is distinct from the false-positive and false-negative patterns discussed above, which occur under both zero-shot and cheatsheet conditions and are addressed separately in §5 and §6.

6.3 Cross-Domain Comparison: BitCoder vs SAIR↩︎

Table 5 summarizes the parallel between findings from  [1] and BitCoder, providing evidence for cross-domain replication of the router hypothesis.

7 Discussion↩︎

7.1 Implications for Security Tooling↩︎

Our results have direct practical implications for LLM-based security analysis pipelines. Practitioners who evaluate cheatsheet-augmented prompts on curated or synthetic benchmarks and observe near-perfect performance may be observing a routing artefact: the model has been given the answer key to a distribution that does not match production code. Deploying such a pipeline on real CVE code will produce silent degradation that is worse than the unaugmented baseline.

This failure mode is particularly dangerous because it is invisible without OOD evaluation. We recommend that any structural-prior-augmented LLM security tool be evaluated against a held-out set of real CVE commits before deployment, and that synthetic benchmark numbers be reported alongside OOD transfer deltas.

7.2 The Case for Distribution-Aware Training↩︎

The structural nature of the collapse points to a clear remedy: rather than calibrating routing priors at inference time, embed distributional knowledge at training time via fine-tuning on a hybrid dataset that includes both synthetic and real vulnerability examples.

Our synthetic corpus of 348 annotated pairs, combined with 300–500 VUDENC/CVEfixes samples, provides a starting point for this direction. A natural next step is fine-tuning a smaller open-weight model such as Qwen2.5-14B (Apache 2.0 licence) on this hybrid corpus and evaluate against CyberSecEval, comparing against frontier model baselines. The router hypothesis suggests that a fine-tuned model with a distribution-aligned internal routing prior will achieve better OOD generalization than any prompt-calibrated frontier model.

7.3 Limitations↩︎

7.3.0.1 Evaluation subset size.

Each real-world condition is evaluated on 30 samples (15 vulnerable, 15 benign). While this matches common practice in budget-constrained LLM evaluation and enables direct comparison with [1], the confidence intervals are wide. Larger evaluations are planned as part of the Juliet Java extension. Gemma-4-31B experienced intermittent API timeouts during evaluation, reducing effective sample size below the full corpus in several conditions (see Table 3 for exact \(n\) per row).

7.3.0.2 Gemma reasoning configuration.

Unlike the Gemma 4 31B configuration in [1], which used reasoning enabled via Together AI with an 8192-token budget to allow the reasoning trace to complete, our Gemma runs used reasoning disabled for consistency with our Llama-3.3-70B configuration. This is a legitimate but distinct methodological choice; our Gemma results are not directly comparable to the Gemma results in  [1] and should be interpreted independently.

7.3.0.3 Incomplete experimental matrix.

Conditions C (unified cheatsheet) and D (progressive saturation), as well as the Real\(\to\)Real and Real\(\to\)Synthetic quadrants of the cross-evaluation matrix, remain untested. These are required to fully characterize the trade-off surface and are the subject of ongoing work.

7.3.0.4 Cheatsheet authorship.

Cheatsheets were authored manually by the first author. Inter-rater reliability and the effect of cheatsheet quality on collapse magnitude are not assessed in this work.

7.3.0.5 Language and framework scope.

VUDENC covers Python only. Whether the collapse pattern holds for compiled languages (C/C++, Java) or framework-specific patterns (Spring, Django) is an open question.

7.4 Misrouting as a Cautionary Methodological Note↩︎

An earlier analysis pass in this research flagged 12 “misroutes” under GPT-OSS-120B’s N+1 cheatsheet condition, which we initially interpreted as evidence of cheatsheet-induced routing interference. Closer inspection revealed this was a detection-harness artifact: naive keyword substring matching flagged the model’s correct answer (“N+1 Query Pattern”) as a misroute because it contained the token “query”, which also appears in the CWE-89 keyword list. After correcting the detection logic to use category-normalised matching, zero misroutes remain in any cheatsheet condition; all genuine misroutes occur under zero-shot. We report this correction transparently because it materially changes the paper’s original framing: rather than cheatsheets causing routing interference, the corrected data show cheatsheets eliminate a genuine zero-shot miscategorization failure mode—though, as discussed in §4.3, this elimination may be substantially attributable to direct label provision rather than corrected internal routing behavior. We include this note as a caution about naive string-matching in LLM-output evaluation harnesses generally.

8 Conclusion↩︎

We have presented BitCoder, a systematic evaluation of structural prior injection for LLM-based source-code vulnerability detection across synthetic and real-world distributions. Our five main findings provide evidence that the router hypothesis and the cross-distribution trade-off surface documented by [1] generalize from formal mathematical reasoning to code security:

• Cheatsheets saturate in-distribution performance (up to 100 % F1) by substantially reducing routing failures, providing evidence for the router hypothesis across three models and three vulnerability categories (two CWEs — CWE-798, CWE-284 — plus one non-CWE anti-pattern, N+1).

• Zero-shot performance degrades along a semantic complexity gradient predictable from the CWE taxonomy for GPT-OSS-120B and partially for Gemma-4-31B.

• The same structural priors that saturate synthetic performance amplify distribution-shift collapse on real CVE data, reducing F1 below the zero-shot baseline by up to 58 pp.

• Zero-shot misrouting is a genuine but cheatsheet-correctable failure mode, though the correction mechanism—direct label provision versus corrected internal routing—remains underdetermined by our current evidence (§7.4).

• Iterative recalibration of the routing priors amplifies the collapse, ruling out calibration as a remedy and providing evidence for the structural nature of the phenomenon.

Together, these findings provide evidence that routing ceilings and distribution-shift collapse are cross-domain properties of structural prior injection in LLMs, with direct implications for the design and evaluation of LLM-based security analysis tools.

References↩︎