1 Introduction↩︎

Code generation from Large Language Models has achieved remarkable results on isolated programming tasks [1]–[4], driving rapid adoption, with millions of engineers using LLM-powered assistants daily. Yet a gap persists between benchmark performance and production utility [5]. Code that appears correct in isolation frequently fails when integrated into real software systems [6], and the dominant failure mode is structural rather than functional. A generated patch may compile, pass type checking, and satisfy local tests while violating invariants that span the repository. Consider a FastAPI endpoint referencing a Pydantic model with hallucinated field names, or a Django view assuming environment variables that are never declared in the project configuration. Such patches exhibit local correctness but global incoherence: they pass the checks developers rely on and fail only when exercised in the context of the full system.

Current evaluation methodologies do not surface these failures systematically. Type checking and linting often miss semantic inconsistencies that cross file boundaries. Test suites cannot cover every integration point. SAST tools typically focus on taint flows rather than structural coherence. The result is a blind spot in which generated code enters codebases carrying latent defects that remain invisible to standard toolchains. We term this the patchwork problem. LLM-generated patches may be individually well-formed yet fail to cohere into a consistent whole, particularly at repository scale, where consistency constraints span imports, dependencies, configurations, schemas, and security contracts [7].

Our approach formalizes structural coherence as consistency invariants over graph representations of repository artifacts [8]. A key design insight is that reliable detection requires matching each invariant class to an appropriate verification strategy. Categories where mature static analysis tools already capture the relevant language semantics can be delegated to those tools, while categories that require cross-graph reasoning, absent from existing toolchains, call for purpose-built detectors that target constraint violations under explicit assumptions and produce actionable evidence. This work makes three contributions: (1) We introduce a taxonomy of eight structural failure categories, each defined by graph-based consistency invariants, distinguishing failures characteristic of LLM-generated patches from issues merely amplified by them. (2) We present a multi-graph verification framework that integrates mature static analysis tools (mypy, tsc, pylint, ESLint) with purpose-built cross-graph detectors over eight repository graphs, producing localized evidence traces for each violation. (3) We provide an empirical study across 336 generations from two frontier models under four prompting conditions, along with external validation on 43 real-world AI-generated repositories.

2 Related Work↩︎

Table 1 situates our contribution relative to prior work across four research threads. Hallucination characterization work [6], [9] establishes the empirical prevalence of structural defects in generated code but characterizes them descriptively rather than as verifiable constraint violations. Repository-level benchmarks, including RepoBench [7], SWE-bench [5], EvoCodeBench [10], BaxBench [11], SecRepoBench [12], SecureVibeBench [13], and SWE-agent [14] demonstrate that snippet-level performance does not transfer reliably to repository-scale tasks, yet their evaluation criteria remain outcome-oriented (test passage, exploit success) rather than diagnosing which structural invariants are violated. Graph-based representations such as Code Property Graphs [8], CODE-MVP [15], and GALLa [16] leverage graphs as representational substrates but do not operationalize them as a constraint verification layer. Secure generation approaches, including CodeGuard+ [17] and SafeGenBench [18], target vulnerability prevention without formalizing structural coherence. Our work addresses this gap by defining structural incoherence as violated consistency constraints across graph representations and producing localized evidence traces that attribute failures to specific constraint violations.

3 Structural Failure Taxonomy↩︎

The patchwork problem manifests through structural failures, defined as violations of consistency invariants at the repository scale, verifiable via static graph analysis without execution. These failures differ from functional bugs in that individual patches appear correct yet collectively violate project contracts. Our taxonomy comprises eight categories, each defined by formal invariants, required graph artifacts, and failure characteristics specific to LLM outputs. Table 2 summarizes the classification.

Symbol Resolution Failures (SRF): A symbol resolution failure occurs when a referenced name cannot be resolved within the repository’s module graph. Formally, for symbol reference \(r\) in file \(f\) with module graph \(\mathcal{M}\), the invariant requires \(\forall r \in \text{refs}(f): \exists d \text{ s.t. } \text{resolve}(r, f, \mathcal{M}) \rightarrow d\). Detection requires the import graph and symbol table, optionally augmented with type information for generic resolution. Evidence traces record file, line, symbol, expected module, and resolution outcome. This failure class is amplified in LLM outputs because models operating with incomplete context frequently invent plausible but non-existent module names or reference deprecated APIs.

Phantom Internal API (PIA): Phantom API failures occur when generated code invokes internal functions with incorrect signatures or semantics inconsistent with declared interfaces. The invariant requires signature compatibility such that for call site \(c\) invoking symbol \(s\) with signature registry \(\Sigma\), \(\text{compatible}(\text{sig}(c), \Sigma(s)) = \text{true}\). Detection leverages the call graph and signature registry extracted from type annotations and protocol declarations. This category is strongly amplified as LLMs hallucinate method signatures based on naming conventions rather than actual declarations, particularly for internal APIs underrepresented in training data.

Dependency Hallucination (DHI): Algorithm 4 validates that all external imports reference packages declared in the project’s dependency manifests. External imports not found in the dependency graph trigger registry queries to PyPI or npm, distinguishing fully hallucinated packages (nonexistent in registries) from undeclared-but-existing packages. For npm packages, import names match registry names directly. For PyPI, the detector assumes direct correspondence between import names and package names, which holds for the majority of packages but not for cases where import and distribution names diverge (e.g., yaml vs. PyYAML, cv2 vs. opencv-python). The resulting phantom module set feeds downstream into SRF and PIA detection.

Build/Configuration Incoherence (BCI): Build configuration failures arise when generated code assumes configurations inconsistent with the repository’s declared state. The invariant requires that for each configuration assumption \(a\) implied by generated code, a satisfying declaration exists in the configuration space \(\mathcal{C}\). Detection operates over the build graph and configuration graph encompassing entrypoints, environment variables, and framework settings. Four invariant classes apply across languages, namely entrypoint existence, environment variable declaration, module system consistency, and framework configuration alignment. This category is amplified under weak retrieval context where models default to standard configurations divergent from project settings.

Resource Coherence Failures (RCF): Resource coherence failures occur when code fails to provide declared resources, encompassing both filesystem resources and computational contracts. Filesystem resource failures arise when code references files, assets, templates, or migrations that do not exist, with the existence invariant requiring \(\text{exists}(\text{resolve\_path}(r, \text{root})) = \text{true}\) for each resource reference \(r\) and additional ordering constraints for sequential resources such as database migrations. Return contract failures arise when functions with declared return types contain execution paths that do not produce a value matching the declared type, violating the contract that the function’s signature promises to callers. Schema completeness failures arise when model definitions omit fields required by consuming code. Detection operates over the resource graph mapping code references to filesystem paths, the CFG for return-path reachability analysis, and the schema graph for field completeness validation. LLMs exhibit amplified failure rates across all three sub-categories by generating plausible paths based on conventions rather than actual repository structure, omitting return statements on error-handling branches, and producing incomplete schema definitions.

Control Flow Coherence (CFC): Control flow failures manifest as CFG anomalies including unreachable blocks, contradictory conditions, exception flow misuse, and dead error handling. Invariants require full reachability from entry (\(\forall v: \text{reachable}(v_0, v)\)) and exception handler type consistency. Detection operates on intraprocedural CFGs with exception edge annotations. While unreachable code is a general bug class, LLMs exhibit amplified characteristic patterns such as overbroad exception handling, redundant null checks, and copy-paste control flow inconsistencies.

Cross-File Contract Violations (CCV): Contract violations occur when producer and consumer modules exhibit interface mismatches across file boundaries, including wrong field names, incompatible serialization, incorrect error codes, and misaligned assumptions. The invariant requires schema compatibility whereby \(\text{schema}(P.\text{output}) \supseteq \text{schema}(C.\text{input})\) with type consistency. Detection requires the call graph with edges spanning files and the schema graph extracted from OpenAPI specs, Pydantic models, or Zod schemas. LLMs amplify this failure by hallucinating response fields based on naming conventions.

Security Structural Regressions (SSR): Security structural regressions occur when application wiring violates security contracts absent classic taint flow vulnerabilities. The invariant requires guard coverage such that \(\forall r \in R: \text{guarded\_by}(r, M)\) for security-critical routes \(R\) and required guards \(M\). Detection requires routing and middleware attachment graphs specific to each framework. We scope detection to FastAPI (dependency injection guards), Django (permission decorators), Express (middleware chains), and Next.js (middleware matchers). LLMs amplify this failure by wiring middleware incorrectly, even when producing syntactically correct code.

4 Verification Framework↩︎

Architecture Overview: The verification framework employs a hybrid architecture guided by a precision-first design philosophy in which each taxonomy category is matched to the verification strategy that maximizes detection precision for its invariant class. Categories where mature static analysis tools already handle the relevant language semantics (symbol resolution, signature compatibility) delegate to those tools, inheriting their years of edge-case handling. Control flow coherence employs a hybrid approach combining custom graph reachability and pattern matching with SAST tool delegation, reflecting the observation that no single layer achieves adequate coverage alone. Categories requiring cross-graph reasoning absent from existing toolchains (configuration incoherence, dependency hallucination, security regressions, resource coherence, cross-file contracts) employ purpose-built detectors that target provable constraint violations rather than heuristic pattern matching. This division reflects the empirical observation that reimplementing established analyses from scratch produces low-precision detectors due to the long tail of language-specific edge cases (exception flows, generators, context managers, async patterns), while the novel cross-cutting invariants central to the patchwork problem have no existing tool coverage. The framework operates on each repository state after generation, constructing graph representations from the combined original and generated code and routing each to the appropriate verification backend. The framework’s output for each finding is a localized evidence trace recording the violated invariant, the implicated files and line numbers, and the constraint that would need to hold for the code to be structurally sound. All source code, graph construction scripts, detection pipelines, evaluation configurations, and external validation datasets required to reproduce our results are publicly available [19]. Figure 1 illustrates the end-to-end verification pipeline, showing how graph construction connects prompting conditions to failure detection.

Graph Construction: For each repository state after generation, the framework constructs eight graph representations spanning structural, behavioral, and configurational dimensions. Table 3 summarizes the construction method for each graph type.

Figure 2 visualizes the many-to-many mapping between graph representations and failure categories, illustrating why the framework requires multiple coordinated analyses rather than a single monolithic pass. Chords connect each graph representation (left) to the failure categories it enables detecting (right). Some categories require multiple graphs, and some graphs serve multiple categories, motivating the hybrid architecture.

Detection Algorithms: The following paragraphs formalize detection for each taxonomy category. Purpose-built detectors (Algorithms 3–5, 6, 7, and 9) target provable constraint violations, while control flow coherence (Algorithm 8) combines custom analysis with SAST tool delegation. Detection order reflects data dependencies, with DHI running first to produce the phantom module set consumed by SRF and PIA.

Configuration Incoherence Detection (BCI): Algorithm 3 detects provable runtime failures from unguarded environment variable accesses. It extracts strict access patterns that throw on missing values (os.environ["KEY"] in Python, unguarded process.env.KEY in TypeScript), eliminates accesses protected by guards (try/except, membership tests, fallback operators), and validates remaining accesses against the repository’s configuration space. Arithmetic expressions and safe access patterns with explicit defaults are excluded at extraction time. Every reported finding represents a configuration access that will produce a runtime crash if the variable is absent.

Dependency Hallucination Detection (DHI): Algorithm 4 validates that all external imports reference packages declared in the project’s dependency manifests. External imports not found in the dependency graph trigger registry queries to PyPI or npm, distinguishing fully hallucinated packages (nonexistent in registries) from undeclared-but-existing packages. The resulting phantom module set feeds downstream into SRF and PIA detection.

Symbol Resolution and Phantom API Detection (SRF, PIA): Algorithm 5 leverages the phantom module set from Algorithm 4. A LocalModulePattern filter excludes intra-generation cross-references and intentional placeholders to prevent false positives from multi-file generation tasks.

Resource Coherence Detection (RCF): Algorithm 6 targets three sub-categories. Return contract violations construct intraprocedural CFGs for functions with declared return types and identify execution paths that terminate without producing a value, excluding branches guarded by exception handlers that re-raise or call sys.exit. Filesystem resource violations check that referenced paths (templates, migrations, assets) resolve to existing files. Schema completeness violations flag consuming code that accesses fields absent from the producing model’s definition. Every finding represents a provable violation: a reachable path missing a declared return, a path literal that does not resolve, or a field access targeting an undefined name.

Cross-File Contract Violation Detection (CCV): Algorithm 7 detects interface mismatches across module boundaries via call graph and schema graph analysis. It targets four patterns: disconnected middleware (registered but never imported in route modules), unused decorators referencing nonexistent permission classes, duplicate middleware registrations causing double execution, and field naming mismatches between producer response schemas and consumer access patterns (e.g., user_name vs. username). Each finding identifies a statically verifiable disconnect between two code locations that must agree for correct execution.

Control Flow Coherence Detection (CFC): Algorithm 8 employs a hybrid three-layer approach with findings deduplicated by line number. Layer 1 performs BFS reachability from function entry nodes, flagging only entirely dead functions. Layer 2 applies pattern matching for dead code after terminators, tautological conditions, duplicate handlers, and infinite loops. Layer 3 delegates to pylint/ESLint with post-processing filters suppressing known false positives from context managers, generators, and heavy exception scaffolding.

Security Structural Regression Detection (SSR): Algorithm 9 identifies endpoints lacking authentication guards present on sibling routes. Routes are clustered by resource segment, public endpoints are filtered, and majority-rule analysis flags routes where a dominant guard (\(\geq\)​90% coverage) is absent on destructive HTTP methods (POST, PUT, DELETE, PATCH).

Illustrative Example: We trace a real-world AI-generated repository through the verification pipeline to illustrate how structural failures manifest and evade standard toolchains.

5 Evaluation and Results↩︎

5.1 Experimental Setup↩︎

We evaluate structural failure detection across 336 code generations from two frontier models, GPT-4o (2024-08-06, 128K context) and Claude 3.5 Sonnet (2024-10-22, 200K context), both at temperature zero. The evaluation corpus consists of 10 curated open-source production repositories spanning Python (Django [21], FastAPI [22]) and TypeScript (Express [23], Next.js [24]), selected for active maintenance, with a minimum 50 files and 10K LOC, type annotation coverage exceeding 50%, test coverage above 60%, and explicit schema definitions.¹ From these repositories, we extract 60 tasks derived from merged pull requests and closed issues at three complexity levels, namely L1 single-file (30 tasks), L2 multi-file (20 tasks), and L3 cross-cutting (10 tasks). Four prompting strategies control context richness ranging from P1 (minimal, task description only) through P2 (local, 2–5 same-directory files) and P3 (retrieved, 10 similarity-ranked files) to P4 (oracle, 5–15 ground-truth files). The evaluation has a partially unbalanced design in which 24 early tasks were evaluated with GPT-4o under P1 and P2 only, while the remaining 36 tasks received both models across all four strategies, yielding 192 GPT-4o generations (60 each for P1/P2, 36 each for P3/P4) and 144 Claude generations (36 per strategy). P1 and P2 therefore contain 96 generations each and P3 and P4 contain 72 each; all analyses use appropriate denominators to account for this asymmetry.

We compare our framework against four baselines, representing static CI checks: type checking (mypy [25] and tsc, test execution, SAST via bandit [26] and semgrep [27], and regex heuristics. Dependency installation is excluded as the evaluation operates on generated code before environment builds; it would catch at most the 3 DHI findings but none of the remaining 64. Detection metrics include per-category precision against ground truth and evasion rates quantifying findings that pass each baseline undetected. Ground truth labels were established through two approaches. For categories with small finding counts (BCI, DHI, PIA, SRF), every finding was manually reviewed and verified as a provable constraint violation. For categories with larger counts (RCF, CCV), precision was established through iterative pipeline refinement that systematically eliminated false positive patterns, with boundary cases resolved by consulting the formal invariants for each category.

5.2 Results↩︎

Detection Performance: Table 4 reports detection results across 336 generations (192 GPT-4o, 144 Claude 3.5 Sonnet) under four prompting conditions. Our framework identifies 67 structural failures across eight active categories, with 65 (97.0%) invisible to all baseline methods. Compilation and type checking detect only 2 findings independently (both RCF return-type violations), while test execution, SAST, and regex heuristics detect none. By category, RCF accounts for the most findings (29), followed by CCV (18), BCI (12), DHI (3), PIA (3), and SRF (2). Manual validation confirms 100% precision for BCI, DHI, PIA, and SRF (20 of 20 verified as provable constraint violations), while RCF and CCV precision was established through iterative pipeline refinement. No baseline method detects any CCV, BCI, DHI, PIA, or SRF finding; type checkers catch only 2 of 29 RCF findings. Evasion rates reinforce this gap: 97.0% of findings evade compilation (mypy --strict/tsc --strict), and 100% evade test suites and SAST tools.

Two categories, CFC and SSR, produced zero findings in the controlled evaluation despite having active detectors. To determine whether these detectors function correctly or whether controlled generation simply does not elicit these failure modes, we applied the full pipeline to 43 real-world AI-generated repositories spanning vibe-coded projects (Cursor AI, Google Gemini, GitHub Copilot), GPT-Engineer/Lovable applications, and self-declared fully AI-generated projects.² Across 1,581 analyzed files the pipeline detected 1,152 findings in 35 of 43 repositories (81.4% repo-level incidence), with 474 DHI findings, 270 RCF findings, 177 PIA findings, 148 BCI findings, 62 SRF findings, 16 CFC findings (6 duplicate switch cases, 2 dead-code-after-return, 2 infinite loops), and 5 CCV findings. The CFC findings confirm that the hybrid three-layer detector functions correctly; controlled frontier generation with explicit task specifications simply does not produce the unstructured code patterns that trigger control flow failures. SSR remained at zero across all evaluations. This is consistent with two properties of the evaluation corpus. The detector requires route clusters with at least 4 endpoints exhibiting a dominant per-route guard pattern, and most vibe-coded projects apply authentication at the framework level (e.g., global middleware, app-level decorators) rather than per-route, leaving no inconsistency to detect. The two highest-finding repositories, hypertropher-app and VoiceTradeWithSchwab (detailed in the Illustrative Example above), exemplify how structural failures cluster and compound in real-world AI-generated code.

Model Comparison: Failure distributions diverge qualitatively between GPT-4o and Claude 3.5 Sonnet as visualized in Figure 10. Overall failure rates are comparable (GPT-4o: 39 findings in 25/192 generations, 13.0%; Claude: 28 findings in 17/144 generations, 11.8%), but the models exhibit distinct failure profiles rather than simply differing in magnitude. GPT-4o produces all 18 CCV findings and all import-related failures (DHI, PIA, SRF) exclusively, while Claude generates 22 of 29 RCF findings. Both contribute equally to BCI (6 each). Chi-squared testing confirms distributional independence (\(\chi^2 = 25.1\), \(p = 2.73 \times 10^{-7}\)). With 67 total findings, these patterns are descriptive observations warranting replication rather than definitive model characterizations.

Prompt Sensitivity: Table 6 reports failure counts by prompting strategy. P3 (retrieved context) exhibits the highest count (24) and P1 (minimal) the lowest (8), indicating that richer context reshapes rather than uniformly reduces failure distributions. Notably, BCI appears in P1 through P3 but not P4 (oracle), consistent with ground-truth files helping models identify correct configuration variables. L3 (cross-cutting) tasks exhibit 44.6% finding incidence compared to 16.1% for L1 and 13.4% for L2, confirming that tasks spanning configuration, middleware, and schema boundaries are substantially more failure-prone. Figure 11 shows how failure category composition shifts across complexity levels, with BCI and CCV concentrated in L3 tasks that require cross-layer reasoning.

Runtime: The pipeline analyzes each file in a median of 47 ms end-to-end (graph construction through all seven detectors), with a 120-second timeout per external analysis script. Registry validation queries (PyPI/npm) are cached across runs. The largest repository in Track D (435 files, 70K LOC) completes in 233 seconds. Runtime is dominated by graph construction (99.8% of per-file time); all detectors combined complete in under 1ms per file. These times indicate the framework is practical as a CI integration for repositories of moderate size.

5.3 Practical Implications↩︎

These findings have direct consequences for teams adopting LLM-generated code. First, standard CI pipelines (type checking, testing, SAST) are insufficient as quality gates for generated code; 97% of detected failures pass all four baselines, meaning structurally broken code can merge undetected. Teams relying solely on existing toolchains face a growing blind spot as LLM-generated code volume increases. Second, the qualitative divergence between models suggests that switching or combining models does not uniformly reduce risk; different models produce different failure profiles, and mitigation strategies should be model-aware. Third, the concentration of failures in L3 cross-cutting tasks (44.6% incidence versus 13–16% for simpler tasks) indicates that structural verification is most critical for tasks spanning configuration, routing, and schema boundaries, precisely the tasks where LLMs are increasingly deployed. The framework’s localized evidence traces are designed to support both automated CI integration and developer review workflows, providing actionable diagnostics rather than opaque pass/fail signals.

6 Conclusion↩︎

This work formalized the patchwork problem in LLM-generated code through a graph-based failure taxonomy and a hybrid verification framework combining mature static analysis tools with purpose-built detectors. Across controlled generations and real-world repositories, the overwhelming majority of detected structural failures evade type checking, testing, and SAST entirely, and failure patterns diverge qualitatively between models. Our future work will focus on extending the framework to additional models and programming languages and on subjecting the categories currently validated through iterative refinement to independent precision audits. Further directions include repair mechanisms that leverage detected constraint violations to prompt for missing declarations, and integration into agentic coding workflows and continuous integration pipelines where incremental patch review replaces complete-file generation.

References↩︎