1 Introduction↩︎

Large Language Models (LLMs) have evolved from passive autocomplete utilities (e.g., GitHub Copilot) into autonomous coding agents (e.g., Claude Code, SWE-agent) capable of executing multi-step software engineering workflows. These agents function in closed loops where they inspect file hierarchies, edit source code, invoke compilers or test runners, and dynamically self-correct based on error outputs. The outer loop execution speed and accuracy of such agents are heavily constrained by the reasoning capability, output speed, and token cost structure of their core LLMs.

In enterprise software engineering, two primary archetypes of LLM provision exist:

1. Commercial Public APIs: Utilizing closed-source frontier models, such as Claude Opus, hosted by vendors. These models represent the state-of-the-art in reasoning and code generation. However, they require sending code to third-party endpoints and incur token-based pricing.

2. On-Premise Private Clusters: Deploying open-weights models, such as GLM-5.1/5.2, on internal compute hardware, such as an NVIDIA GB200 NVL72 cluster. These deployments often leverage aggressive quantization techniques (e.g., NVFP4) to achieve high throughput and low latency while keeping code and data within the corporate network [1].

To date, few empirical studies have quantified the developer productivity, financial efficiency, and code quality differences between these two paradigms in real-world, large-scale software engineering projects. To bridge this gap, this paper conducts a comparative study of development activities on the production repository of a corporate AI Platform-as-a-Service (PaaS) over two contiguous 28-day periods. During the first period (Period A), the developer utilized Claude Code driven by API-based Claude Opus 4.7/4.8. During the second period (Period B), the developer utilized Opencode driven by GLM-5.1/5.2 NVFP4 running on an internal NVL72 cluster. Methodologically, this is a longitudinal industrial case study: it trades the breadth of a controlled, multi-subject experiment for the depth and ecological validity of continuous production telemetry from a live enterprise codebase; we scope our claims accordingly, deferring generalization to the replication protocol of Section VII-B.

Through database queries on LLM telemetry and mining of Git commit logs, we provide a detailed analysis of the quantitative token usage, financial expenditure, and software production metrics. Concretely, we investigate four research questions:

• RQ1 (Cost economics): How do the realized token economics of a cache-optimized frontier API compare to a self-hosted quantized cluster, both per token and in total? (Sections V-A and V-D)

• RQ2 (Code quality): How do the two deployments differ in defect-repair burden—Fix Commit Ratio and defect composition—once task difficulty is controlled for? (Sections V-B and V-C)

• RQ3 (True TCO): Under realistic Taiwan-market operating parameters and a data-grounded labor model, which deployment minimizes the true total cost of ownership (TCO), and how robust is that ordering to parameter choice? (Section VI-A)

• RQ4 (Developer experience): Beyond direct dollars, what objective developer-experience cost does the local configuration impose? (Section V-G)

In answering these, this paper makes the following contributions:

1. A ground-truth measurement methodology that triangulates deduplicated LLM telemetry with Git repository mining across two contiguous 28-day production periods, specified in full so it can be reproduced on other repositories and telemetry.

2. The empirical finding that prompt caching (99.3% hit rate) inverts the per-token cost comparison between a frontier API and a self-hosted cluster—an effective $0.57 per million tokens, below even the shared on-premise amortization ($2.83/M)—a utilization-dependent inversion that reframes the conventional cloud-versus-on-premise cost narrative.

3. A difficulty-stratified defect analysis (\(\mathrm{OR}_{\mathrm{MH}} = 3.61\)) demonstrating that the local configuration’s defect-repair premium holds within every difficulty tier, independent of a strict output-volume match.

4. A data-grounded total-cost-of-ownership model under Taiwan-market parameters with a symmetric, timestamp-grounded labor basis and a four-axis sensitivity analysis.

5. A set of objective, reproducible behavioral workload indicators mined from commit timestamps that quantify the developer-experience cost without subjective self-report.

6. A cost–quality frontier analysis of hybrid routing gateways via offline counterfactual replay, showing that no routing policy dominates the pure-API baseline; shifting work to the local model trades defect rate for infrastructure savings.

The rest of this paper is organized as follows: Section II reviews the related work. Section III contrasts the architectural frameworks of Claude Code and Opencode. Section IV details the empirical methodology. Section V presents our quantitative evaluation. Section VI discusses the strategic business and opportunity costs. Section VII analyzes threats to validity, and Section VIII concludes the paper.

2 Related Work↩︎

2.1 Autonomous Software Engineering Agents↩︎

Recent work in software engineering has shifted from single-line code prediction toward end-to-end task execution by autonomous agents. Frameworks like SWE-agent [2] demonstrate that combining LLMs with a sandboxed shell environment enables agents to locate bugs, modify multiple files, and verify changes via test suites—though simpler non-agentic pipelines remain competitive on the same tasks [3]. Public benchmarks like SWE-bench [4] evaluate this capacity by testing models on real GitHub issues. However, these agents execute in multi-turn loops, often requiring dozens of sequential model calls to complete a single task. This recursive loop behavior amplifies the impact of underlying model inaccuracies, as minor errors in earlier steps can lead to compounding failures later in the execution trace.

2.2 Empirical Studies of AI Coding Assistants↩︎

A parallel line of empirical work measures the productivity and quality impact of AI coding assistants on developers. Controlled and field studies of autocomplete-style assistants such as GitHub Copilot report task-completion speedups (e.g., a randomized trial finding a 55.8% faster completion of a scoped task [5]), alongside more nuanced effects on acceptance behavior and cognitive load. These studies predominantly evaluate single-suggestion completion in short lab tasks or self-report surveys; a more recent field randomized controlled trial (RCT) of agentic assistants on experienced open-source maintainers found a measured slowdown despite perceived speedup [6], but reported no cost or deployment dimension. The agentic frameworks above [2], [4] are in turn evaluated mainly on public issue-resolution benchmarks rather than longitudinal production telemetry. Our study complements both lines by mining 56 days of production agent telemetry and Git history, and by contrasting a hosted frontier API against an on-premise quantized deployment—a cost-and-quality dimension absent from prior developer-productivity studies.

2.3 On-Premise Deployment and Quantization↩︎

For enterprise adoption, on-premise hosting of LLMs has emerged as a viable alternative to commercial APIs. Running models locally requires substantial compute infrastructure. Serving frameworks like vLLM [7] and Orca [8] optimize throughput through PagedAttention and iteration-level batching. To optimize hardware efficiency, post-training quantization techniques compress model parameters from 16-bit floating point representations to 8-bit (FP8) or 4-bit (FP4) formats; calibration-based methods such as AWQ [9], GPTQ [10], and SmoothQuant [11] reduce the accuracy loss of aggressive quantization. The release of NVIDIA Blackwell GPUs introduced native hardware acceleration for 4-bit formats (NVFP4 and MXFP4) [1], and large open-weights Mixture-of-Experts (MoE) models make quantized private serving increasingly practical. While quantization significantly increases token throughput and reduces GPU memory (VRAM) requirements, it can introduce degradation in complex reasoning tasks, such as structural code refactoring and mathematical logic. Vendor-published calibration reports claim near-lossless accuracy for NVFP4 on standard benchmarks [1], [12], and benchmark-level studies of quantized code LLMs report mixed degradation [13]; the cost on production agentic workloads, however, remains uncharacterized, motivating the deployment-level measurement this study provides. The multi-turn loops of Section II-A then compound even small single-step accuracy losses.

2.4 Model Selection Matrix and Rationale↩︎

To justify the selection of the local open-weights model and outline the design space, we establish a model selection matrix comparing state-of-the-art open-weights and API-based frontier models available during the study period (leaderboard values accessed 2026-07-07). The evaluation criteria prioritize: (1) context window capacity to prevent retrieval truncation in deep monorepos, (2) on-premise deployability to satisfy enterprise data governance, (3) systems-HPC optimization on Blackwell architectures, and (4) coding intelligence benchmarks.

Table ¿tbl:tab:model95selection? organizes the design space along the axis that governs on-premise adoption—whether a model can be self-hosted at all. Closed-source frontier models (Claude Opus, OpenAI’s GPT-5.5) are reachable only through a vendor’s cloud API and cannot run on internal hardware; open-weights models (Zhipu’s GLM, DeepSeek, Moonshot AI’s Kimi, MiniMax) can. This split makes the endpoints of our study the natural choices on each side. Within the closed-source group, Claude Opus 4.8 leads the compared frontier models on the Artificial Analysis Intelligence Index (56, ahead of GPT-5.5 at its xhigh reasoning setting at 55, and its own predecessor Opus 4.7 at 54) [14], and it is the model that drives Claude Code. Within the open-weights group, GLM-5.2 is the highest-ranked open-weights model on the same index [14] and the top open-weights model on SWE-bench Pro [15]. Selecting these two therefore pits the strongest self-hostable model against the frontier cloud model our agent already used. All Intelligence Index scores are reported at each model’s maximum reasoning effort, matching our own deployment, which ran both Claude Opus and GLM-5.2 at maximum reasoning effort.

\(^{\dagger}\)AA = Artificial Analysis Intelligence Index [14], reported at each model’s maximum reasoning effort (e.g., “max effort”/“xhigh”); SWE-bench Pro scores from the leaderboard aggregation [15]. Each group lists the two model versions used in this study plus the strongest same-class competitor by index; bold marks the primary configuration on each side, both run at maximum reasoning effort. GPT-5.5 price and context are from the vendor’s official API documentation [16].

The model selection matrix reveals key architectural trade-offs:

• Coding Capability and Openness: At the time of deployment, GLM-5.2 held the highest SWE-bench Pro score among open-weights models (62.1%, versus 59.0% for MiniMax M3) [15], and ranked as the top open-weights model on the Artificial Analysis Intelligence Index (51, versus 56 for Claude Opus 4.8) [14]. Combined with its MIT license and a native 1M-token context window, this made it the strongest candidate for private deployment.

• Serving and Memory Efficiency: GLM-5.2 pairs a 753B total-parameter MoE structure with only 40B active parameters per token, and NVIDIA publishes an official NVFP4 quantization checkpoint with an FP8 Key-Value (KV) cache [12]. This permits serving on a four-GPU slice—one GB200 NVL72 compute tray—rather than the \(\ge\)​8-GPU allocation required for unquantized checkpoints, yielding a cost-efficient private deployment. Larger open-weights alternatives such as DeepSeek-V4 Pro impose materially higher VRAM floors.

• API Cost Optimization and Prompt Caching: Claude Opus 4.8 carries the highest nominal pricing among the models deployed in this study ($5.00 input, $25.00 output per million tokens). However, Anthropic’s prompt caching discounts cached prefix reads by 90% (to $0.50 per million tokens), and cache-heavy agentic workloads realize most input tokens as cache reads (99.3% in our telemetry, Section V). This caching mechanism renders Claude far more price-competitive in repetitive multi-turn development workflows than its nominal rates suggest, while the on-premise cost basis remains hardware-hours rather than tokens.

This architectural decision matrix justifies the selection of GLM-5.2 (and its predecessor GLM-5.1 prior to the 5.2 release on 2026-06-13) as the local benchmark model, balancing context capacity, hardware overhead, and coding intelligence.

2.5 Prompt Caching in Agentic Workflows↩︎

Autonomous coding agents are highly context-intensive, as they frequently send substantial parts of the codebase, JSDocs, database schemas, and tool definitions with every request. Prompt caching (e.g., Anthropic’s Prompt Caching) allows the inference server to reuse the KV cache of static prompt prefixes. This skips the redundant prefill computation, yielding significant latency reductions (up to 85%) and input cost discounts (up to 90%) [17]. In agentic frameworks, the prefix remains highly static (containing system instructions and repository index), making such workloads ideal candidates for prompt caching.

3 System Architectures: Claude Code vs.Opencode↩︎

A critical aspect of this study is comparing the software architectures of the two agent frameworks: Claude Code and Opencode. Although both serve as terminal-based autonomous agents, they are built on fundamentally different engineering paradigms, contrasted at a glance in Fig. 1.

3.1 Claude Code Execution Model↩︎

Claude Code is a commercial, Node.js-based terminal agent developed by Anthropic [18]. It operates locally, interacting directly with the filesystem and local shell commands, and handles task execution by issuing API calls to the hosted Anthropic service. The local client parses the repository directory and dynamically updates the context window with file contents, terminal execution output, and history of past agent steps, which is then submitted to the remote inference models.

3.2 Opencode Framework Architecture↩︎

Opencode is an open-source (MIT-licensed) terminal coding agent [19]. It follows a client–server design: a background server process manages session state, conversation history, and filesystem operations, while a terminal-UI (TUI) client connects to it, so a session persists across terminal restarts. The server core is written in TypeScript and executed on the Bun runtime, whereas the TUI client is implemented in Go; model access is mediated through a provider-abstraction layer (the Vercel AI SDK), making the agent provider-agnostic. In our deployment, Opencode drove the on-premise Blackwell cluster serving the quantized GLM models via an OpenAI-compatible endpoint, with every session in its full-tool-access Build Mode.

Unlike the single-vendor architecture of Claude Code, Opencode’s design centers on provider neutrality:

• Bun Runtime Core: The server core runs on Bun rather than Node.js, providing rapid cold-start times and native TypeScript execution; the interactive TUI client is a separate Go process.

• Local Session Persistence: Sessions, message history, and token-usage metrics are persisted in a local SQLite database managed through the Drizzle ORM, independent of any vendor service.

• Provider-Agnostic Model Access: Through the Vercel AI SDK provider abstraction, the framework binds provider-specific system-prompt profiles and tool-schema encodings at session start (e.g., an Anthropic-tuned profile versus generic profiles for open-weights backends), rather than assuming a single vendor’s prompt conventions.

3.3 Architectural Trade-offs↩︎

Table ¿tbl:tab:arch95comparison? highlights the design contrasts between the two agent frameworks.

Crucially, these engineering dimensions act as capability multipliers for the underlying LLMs, explaining a portion of the performance differences observed in practice:

• Vendor Co-design: Claude Code and the Claude Opus models are developed by the same vendor, and the harness (tool schemas, system prompts, edit formats) is co-tuned with the model’s training. Opencode, as a provider-agnostic framework, must adapt heterogeneous models through generic prompt profiles, which cannot exploit model-specific alignment to the same degree.

• Prompt Caching Integration: Claude Code automatically inserts cache breakpoints so that static prefixes (system rules, tool schemas, conversation history) are reused server-side. Our local vLLM deployment similarly benefits from automatic prefix caching (86.4% of Period B prompt tokens served as cache hits, Section V-A).

These engineering differences represent a construct confound, discussed further in Section VII-C.

3.4 Base Model Performance Benchmarks↩︎

To understand the underlying capabilities of the models driving the autonomous coding agents in our evaluation, we compile independently aggregated benchmark and market data. Table ¿tbl:tab:model95benchmarks? compares the commercial cloud models (Claude Opus 4.8 and 4.7) against the local open-weights Mixture-of-Experts models (GLM-5.2 and 5.1) using the Artificial Analysis Intelligence Index [14], the SWE-bench Pro leaderboard [15], and measured serving throughput of the vendors’ hosted endpoints.

The aggregated scores show that GLM-5.2 was the strongest open-weights coding model at the time of the study—ranked first among open-weights models on both the Intelligence Index and SWE-bench Pro—yet a persistent gap remains against the frontier API models (Intelligence Index 51 vs.; SWE-bench Pro 62.1% vs.%). Our empirical evaluation examines how this single-digit benchmark gap, compounded by NVFP4 quantization and provider-agnostic harness adaptation, is associated with a much larger gap in end-to-end defect rates (\(\mathrm{FCR}\) 74.9% vs.%, Section V).

3.5 Unified Multi-Dimensional Ecosystem Comparison↩︎

To provide a complete comparative view of the two paradigms, we analyze the multi-dimensional differences between the hosted API model (Opus) and the on-premise open-weights cluster (GLM), alongside their corresponding CLI frameworks (Claude Code and Opencode).

3.5.1 Infrastructure Footprint and Scaling Limits↩︎

Claude Code acts as a thin client requiring negligible local CPU and memory. However, it is bound by vendor-side, per-organization rate limits (request- and token-per-minute quotas). In contrast, Opencode requires an on-premise private GPU cluster (in our deployment, four NVIDIA Blackwell B200-class GPUs—one GB200 NVL72 compute tray) to serve the quantized GLM model. While this demands substantial capital expenditure and VRAM allocation (approximately 465 GB of NVFP4 weights plus an FP8 KV cache), it removes external rate limits, allowing parallel requests across the corporate intranet up to the cluster’s serving capacity.

3.5.2 Context Window Dynamics and Latency Profiles↩︎

The core models exhibit distinct context capabilities. Claude Opus 4.7 and 4.8 feature a 1,000,000-token context window, allowing the Claude Code agent to ingest large directory structures and multi-turn conversation history in a single prompt, with prefill latency amortized via server-side prompt caching. On the local side, the deployed checkpoints differ by generation: the GLM-5.1 NVFP4 checkpoint is configured for a 202,752-token context, whereas the GLM-5.2 NVFP4 checkpoint supports 1,048,576 tokens (verified from the deployed Hugging Face model configurations [12], [20]). Local serving via vLLM likewise enabled automatic prefix caching (86.4% hit, Section V-A); because the local cost basis is hardware-hours (Section V-D), the TCO and defect-rate results are unaffected by the caching configuration.

3.5.3 Data Governance and Network Compliance↩︎

Data residency is a critical distinction. Operating Claude Code involves transmitting corporate intellectual property (source code, database schema, logs) to Anthropic’s cloud endpoints. This requires compliance audits under data protection regulations (such as GDPR). Opencode, operating within a private Kubernetes namespace on an internal network, keeps inference traffic inside the corporate intranet, which substantially simplifies compliance for proprietary technology assets relative to third-party API transmission.

4 Methodology↩︎

4.1 Development Environment and Tools↩︎

This study was conducted on a production codebase of a corporate AI PaaS monorepo consisting of 12 distinct packages (including frontends in TypeScript/React and backends in Python/FastAPI).

• Period A (Claude Code + Claude Opus): From 2026-05-11 to 2026-06-07. The developer utilized the Claude Code terminal agent, interacting with commercial API endpoints of Claude Opus 4.7 (released 2026-04-16) and, after its release on 2026-05-28, Claude Opus 4.8. Telemetry attributes 28,615 requests to Opus 4.7 and 3,286 to Opus 4.8.

• Period B (Opencode + GLM): From 2026-06-08 to 2026-07-05. The developer transitioned to Opencode, backed by private NVFP4-quantized deployments on a shared NVIDIA Blackwell cluster: GLM-5.1 for the initial days (GLM-5.2 was released on 2026-06-13), followed by GLM-5.2 served first from a community NVFP4 checkpoint and later from NVIDIA’s official NVFP4 checkpoint. Both periods therefore span a mid-period upgrade to the vendor’s latest generation, keeping the “rolling frontier” condition symmetric. Although GLM-5.1 and GLM-5.2 differ more in capability (Intelligence Index 40 vs.) than Opus 4.7 and 4.8 (54 vs.), the weekly defect series (Fig. 12) shows no structural break at these checkpoint transitions, so Period B is analyzed as a single configuration, symmetric with Period A’s pooling of Opus 4.7/4.8.

4.2 Data Collection↩︎

We gathered data from two primary channels:

1. Telemetry Logs: Period A’s hosted Claude API telemetry was recorded in a Langfuse server deployed in a private namespace on Kubernetes, and extracted directly from its ClickHouse database backend [21] on 2026-07-07 by aggregating observation-level usage records (input, output, cache_creation_input_tokens, cache_read_input_tokens) grouped by model and filtered to the developer’s user identifier and to the coding-agent models (claude-opus-4-7 and claude-opus-4-8); auxiliary model traffic from unrelated pipelines is excluded. Two extraction details are essential for correctness: (i) because Langfuse’s ClickHouse tables use a ReplacingMergeTree engine, queries deduplicate multi-version rows via FINAL with is_deleted = 0 on both the observations and traces tables (omitting this inflates request counts by roughly 3% in our data); and (ii) period boundaries are evaluated in the developer’s local time zone (Asia/Taipei, UTC+8), matching the time zone semantics of the Git mining, rather than the server’s native UTC. These two conditions apply to the Langfuse-recorded Period A telemetry. Period B’s local GLM traffic, by contrast, was served by an on-premise vLLM stack, whose engine-level Prometheus counters (prompt, cached, and recomputed tokens) provide the Period B token and cache accounting, with prefix-cache hits reported directly (86.4%); these source the Period B column of Table ¿tbl:tab:tokens?. The two periods thus use the backend-appropriate telemetry surface—Langfuse for the hosted Claude API, vLLM engine metrics for the local GLM cluster—not a single shared pipeline.

2. Git Repository Mining: Git logs from the PaaS repository were parsed to isolate the development commits under study via their Git author identity, from which we extracted line additions, deletions, files changed, and keyword-classified bug-fix commits [22]; all commit-level metrics exclude merge commits (--no-merges). To capture the developer’s authored work completely—including commits developed on feature branches or rebased prior to merging—we enumerate every non-merge commit authored by the developer over each period across all branches and pull-request refs. Two safeguards keep this from over-counting: git patch-id collapses rebased or cherry-picked copies of the same change to a single instance (retaining the earliest authoring timestamp), and each commit is assigned to a period solely by its author timestamp, so only in-window work under the developer’s identity is counted. As an additional attribution check, the mined commits were cross-validated against the team’s Redmine issue tracker (Table 1), corroborating—without adjusting—the author-and-timestamp filter; every metric is computed directly from Git. We further verified that every mined work stream was ultimately merged into the mainline through the reviewed pull-request flow (Section IV-G): the commit series measures accepted, review-approved output—including the repair iterations that precede acceptance—rather than abandoned drafts. Figure 2 summarizes the end-to-end extraction and reconstruction pipeline across both channels.

4.3 Defect Metric Formulation↩︎

To evaluate code quality, we define the Fix Commit Ratio (\(\mathrm{FCR}\)) as: \[\mathrm{FCR}= \frac{C_{\text{fix}}}{C_{\text{total}}} \times 100\%,\] where \(C_{\text{fix}}\) represents the count of commits whose message matches the regular expression pattern /(fix|bug|issue|patch|correct)/i, indicating that the commit was dedicated to repairing defects, resolving compile/lint errors, or fixing broken tests [22]. \(C_{\text{total}}\) is the total number of commits by the developer during the period.

4.3.1 Classification Reproducibility↩︎

Both the fix/non-fix decision and the defect taxonomy assignment (Section V-B) are produced by deterministic keyword rules applied to commit subjects, rather than ad-hoc human judgment: the fix decision uses the regular expression above, and taxonomy assignment applies a fixed priority order of category-specific keyword sets (test-related, then syntax/type-related, then API/dependency-related, with the remainder classified as logical defects). The complete rule set is specified in this paper, making every classification decision reproducible from a repository’s own commit history. The first author manually reviewed every classified commit and confirmed its rule assignment; because this rater is not blind to the hypothesis, we recommend a blind second-rater protocol reporting inter-rater agreement (Cohen’s \(\kappa\)) as a replication check. The residual risk is construct-level—which changes should count as defect repairs at all—and is discussed under construct validity (Section VII-C).

4.4 Economic and Feedback Loop Model↩︎

To evaluate the true cost of using autonomous coding agents, we construct an economic model adapted from classical software engineering economics [23], modeling both compute expenses and developer hourly costs. Let the total corporate cost of development (\(TC\)) for \(M\) tasks be represented as: \[TC = M \times \left( C_{\text{compute}} + C_{\text{ops}} + T_{\text{iter}} \times R_{\text{dev}} \right),\] where \(C_{\text{compute}}\) is the compute/inference cost per task, \(C_{\text{ops}}\) is the operational infrastructure overhead per task (inclusive of electricity, cooling, facilities, and personnel support), \(R_{\text{dev}}\) is the developer’s hourly wage rate, and \(T_{\text{iter}}\) is the feedback loop iteration time required to successfully complete a single coding task.

We formalize \(T_{\text{iter}}\) as a function of developer time and model performance: \[T_{\text{iter}} = T_{\text{coding}} + N_{\text{round}} \times \left( T_{\text{wait}} + T_{\text{review}} \right),\] where \(T_{\text{coding}}\) is the developer’s initial scoping and coding time, \(T_{\text{wait}}\) is the average inference wait time per request (determined by model throughput and network latency), \(T_{\text{review}}\) is the developer’s cognitive review and debugging time per iteration, and \(N_{\text{round}}\) is the average number of interaction rounds. A local on-premise model achieves a net financial advantage over a commercial API model if and only if: \[\Delta C_{\text{compute}} > R_{\text{dev}} \times \Delta T_{\text{iter}},\] where \(\Delta C_{\text{compute}} = C_{\text{compute, api}} - C_{\text{compute, local}}\) and \(\Delta T_{\text{iter}} = T_{\text{iter, local}} - T_{\text{iter, api}}\).

4.5 Task Difficulty and Category Mapping↩︎

To assess task selection effects, we classify every non-merge commit in both periods by category and difficulty using deterministic rules computed from the Git metadata itself. Category follows the conventional-commit prefix (feat, fix, refactor/perf/style, test/ci, docs/chore). Difficulty is derived from the change footprint: High (touching \(\ge 3\) packages, modifying \(\ge 5\) files, or \(>200\) net lines), Medium (touching \(2\) packages or modifying \(2\)–\(4\) files), and Low (single-file modifications). Both rules are fully reproducible from the repository history.

Table ¿tbl:tab:difficulty? displays the resulting distribution. Unlike an idealized controlled experiment, the two periods are not homogeneous: Period A contains a larger share of High-difficulty commits (35.0% vs.%; \(\chi^2(2) = 13.14\), \(p = 0.0014\), Cramér’s \(V = 0.15\)), reflecting that Period A included heavier feature and infrastructure work while Period B skewed toward smaller changes. We address this confound in two ways. First, all defect comparisons in Section V are additionally reported stratified by difficulty tier, and the local configuration exhibits a higher fix-commit share within every tier. Second, the direction of the imbalance is conservative with respect to our conclusion: the period that handled harder tasks (Period A, Claude) produced the lower defect-repair share, so equalizing difficulty would widen, not narrow, the observed gap.

\(^{\ddagger}\)The Fix category here partitions commits by their conventional-commit prefix (fix:) and is disjoint from the other categories; it is distinct from the Fix Commit Ratio count of Section IV-C (113 for Period A, 275 for Period B), which applies the regex /(fix|bug|issue|patch|correct)/i to the full commit message and therefore also captures repair commits filed under other prefixes. The per-tier \(\FCR\) analysis (Section V-C) uses the latter (regex) count. Percentages may not sum to 100 due to rounding.

4.6 Specification-Driven Development (SDD) Protocol↩︎

Both evaluation periods utilized a Specification-Driven Development (SDD) methodology to guide the autonomous agents. In the SDD workflow, the developer first creates a detailed markdown specification file (e.g., feature_spec.md or OpenAPI schemas) outlining the required functions, database schemas, and type definitions. The agent is then executed with the specification file injected directly into its context.

The agent’s task execution loop follows a strict specification-compliance protocol: (1) parsing the specifications, (2) generating or updating corresponding TypeScript interfaces and backend schemas, (3) creating unit tests, and (4) implementing the core logic. While the Opus configuration generally produced type-safe implementations matching the specification in a few iterations, the GLM configuration more frequently violated the specified type constraints and API boundaries, yielding type-check and runtime integration failures that required developer intervention—a difference we quantify in Section V-B.

4.7 Software Quality-Assurance and Delivery Pipeline↩︎

Both evaluation periods were conducted not on a throwaway prototype but inside the enterprise software development lifecycle (SDLC) of the production PaaS monorepo, under a single, uniformly enforced quality-assurance and delivery pipeline. This property is central to the study’s internal and construct validity: every commit we mine—in either period—is a gated artifact admitted through the same multi-stage quality gate and, once merged, promoted along the same multi-environment path. Because the pipeline is held fixed while the coding agent, model, quantization, and serving stack vary together, engineering-process maturity is controlled by design and cannot manufacture a between-period difference; the \(\mathrm{FCR}\) and defect deltas of Section V therefore isolate the deployable configuration rather than the surrounding engineering discipline. Table 1 summarizes the controls.

The quality gate is a single source of truth—one command, invoked identically by every runner (workstation, mandatory pre-push hook, nightly full run)—spanning linting, strict type checking, layered tests behind a per-file coverage gate, migration-chain checks, a full software-supply-chain security suite, and load/benchmark/chaos testing (detailed in Table 1). Merges require human review under CODEOWNERS and Conventional-Commit enforcement plus—notably for this study—two AI review agents served by an internally hosted model, so the team dogfoods on-premise agentic inference within its own review loop. Releases follow near-daily semantic-versioned delivery, promoting one Helm chart across six environment overlays with rolling updates and automatic rollback. We report this pipeline not as a contribution in itself but as evidence that the mined commits reflect production-grade, quality-gated engineering under both configurations.

5 Empirical Evaluation↩︎

5.1 Token Usage and Caching Characteristics↩︎

Table ¿tbl:tab:tokens? presents the token telemetry for both periods, restricted to the coding-agent models (Period A from the Langfuse ClickHouse backend per Section IV-B; Period B from the on-premise vLLM engine’s Prometheus counters, whose request counts are not directly comparable to Langfuse’s).

Under the agentic loop of Claude Code, every request re-transmits the accumulated session context. This results in an enormous raw prompt volume of 15.31 billion input-side tokens (approximately 480k prompt tokens per request) for Opus. However, due to Anthropic’s prompt caching mechanism, 99.3% of these prompt tokens were served as cache reads; only 0.7% were billed at uncached rates. The two output volumes also differ sharply: the Opus configuration generated 20.2M output tokens versus 5.06M for GLM—a 4.0\(\times\) difference. Since the two periods delivered comparable code volume (Section V-B), this reflects Claude Code’s more verbose agentic loop (extensive reasoning traces and repeated full-file rewrites) rather than a code-output gap.

The GLM deployment exhibits the same reliance on prefix caching: with vLLM’s automatic prefix caching enabled, Period B served 86.4% of its prompt tokens (778M of 900M) as KV-cache hits, with the recomputed-prefill counter effectively zero; the agentic loop’s prompt prefixes are equally repetitive. In our cost model (Section V-D), caching reduces Claude’s realized API cost by 88.6% relative to the same token volume at nominal input rates, consistent with the up-to-90% discount bound documented for prefix caching in agentic workflows [17].

5.2 Software Output and Code Quality Metrics↩︎

Table ¿tbl:tab:git? summarizes the Git mining results from the PaaS repository.

\(^{\dagger}\)Excluding Markdown documentation, dependency lockfiles, and extension-less files.

\(^{\S}\).json, .sh, .yaml, .toml. Percentages may not sum to 100 due to rounding.

Commit counts were higher in Period B (367 vs. non-merge commits). Total inserted volume was higher in Period A (129,556 vs.,281 lines), but this gap is largely an artifact of documentation: 36.6% of Period A’s insertions were Markdown specification files produced under the SDD protocol (Section IV-F), versus only 10.1% in Period B (Table ¿tbl:tab:languages?). Restricted to code-only insertions, the two periods are comparable—indeed Period B is marginally higher (80,258 vs.,738 lines). We take code-only gross insertions as the volume-invariance basis. The all-file net growth of Table ¿tbl:tab:git? runs the other way (90,718 vs.,673, favoring Period A), again a documentation artifact compounded by Period A’s larger deletion churn—which only strengthens the quality reading: Period A produced more retained net lines and carried the lower defect-repair share. The two configurations therefore delivered a similar gross code churn (Fig. 3 traces the cumulative net-line trajectory of each period); because the all-file net growth diverges, we do not claim strict output-volume invariance and instead rest the quality comparison on the difficulty-stratified within-tier analysis (\(\mathrm{OR}_{\mathrm{MH}} = 3.61\), Section V-C), which does not depend on equal volume. However, raw LOC is a flawed proxy for developer productivity when accompanied by high defect rates. To measure the actual utility of the generated code, we define the Effective Productivity Ratio as \(E_{\text{net}} = 1 - \mathrm{FCR}\), the proportion of commits that deliver new logic rather than defect corrections. With an \(\mathrm{FCR}\) of \(45.93\%\), Period A retains \(E_{\text{net}}^A = 54.07\%\) of its commits as net-new logic; with an \(\mathrm{FCR}\) of \(74.93\%\) (roughly three out of four commits were defect corrections), Period B retains only \(E_{\text{net}}^B = 25.07\%\).

This effective-yield gap restates the measured \(\mathrm{FCR}\) difference: under the local GLM setup, the developer’s activity was dominated by repairing agent-generated defects rather than writing new logic. \(E_{\text{net}}\) is a coarse discount—treating every fix commit as zero-value—but it bounds the direction and rough magnitude of the quality adjustment (Section V-G).

To further analyze the quality gap, Table ¿tbl:tab:defects? provides a defect taxonomy classifying all non-merge fix commits during both periods, using the deterministic keyword rules of Section IV-C. The distributions differ, but the difference is concentrated in the Test/CI category—a Period-A infrastructure confound (Fig. 4); we test its statistical significance in Section V-C.

5.2.1 Syntax and Type Discipline↩︎

The GLM configuration produced 20 syntax/type-repair commits versus 2 under Opus (7.3% vs.% of fix commits—a \(10\times\) absolute inflation). A representative Period B example is a static-typing violation in which the agent assigned an endpoint response to a variable without the required dict | list union annotation, failing mypy on the exception branch (Listing 1). We note that message-level mining underestimates this category for both configurations: type errors that the agent repairs within the same working session never surface as separate fix commits, so these counts reflect only the residual type errors that escaped the agent’s own iteration loop and required dedicated repair commits.

5.2.2 Dependency Mapping and API Boundary Compliance↩︎

API, import, and dependency misuse accounted for 10.5% of GLM’s fix commits (29 commits) versus 2.7% for Opus (3 commits)—a nearly \(10\times\) absolute inflation. Notably, this gap cannot be attributed to context capacity: the deployed GLM-5.2 checkpoint supports the same 1M-token context class as Claude Opus (Section III-E). The failure modes instead concern the correct use of repository-internal and library APIs: hallucinated import paths, stale method signatures, migration-revision collisions, and environment-behavior misuse such as an HTTP client inheriting proxy environment variables inside the cluster network (Listing 2). This suggests that long-context capacity alone does not confer accurate API grounding in a large monorepo.

5.2.3 Logical Defects Dominate the GLM Repair Load↩︎

Logical defects are the largest category in both periods but dominate under GLM: 196 commits (71.3% of fix commits) versus 61 (54.0%) under Opus—a \(3.2\times\) absolute inflation. Period B logical repairs include broken control flow around failure paths (e.g., aborting swap operations on deleted resources, preserving recoverable state on post-commit failures) that required developer-directed correction after the agent’s initial implementation shipped subtly incorrect behavior.

5.2.4 Test and CI Repair: An Infrastructure Confound in Period A↩︎

The test/CI category inverts the pattern: 47 commits (41.6%) under Opus versus 30 (10.9%) under GLM. Inspection shows that much of Period A’s repair volume was test-infrastructure maintenance (raising CI memory limits, scoping coverage guards, deflaking suites) coinciding with a phase of active CI build-out, rather than repairs of model-generated logic. This has an important interpretive consequence: Period A’s \(\mathrm{FCR}\) of 45.9% partially reflects infrastructure churn unrelated to model output quality, meaning the raw \(\mathrm{FCR}\) comparison likely understates the model-attributable quality gap between the two configurations. We revisit this as a construct-validity consideration in Section VII-C.

Percentages may not sum to 100 due to rounding.

5.3 Statistical Hypothesis Testing↩︎

To validate the productivity and quality differences observed between the two periods, we formulate three statistical hypotheses and perform non-parametric Mann–Whitney U tests (Wilcoxon rank-sum) on the Git-derived metrics, since daily software metrics do not conform to a normal distribution. We measure effect sizes using Cliff’s Delta (\(\Delta\)) [24], a non-parametric measure recommended for small-sample software-engineering experiments, classified as negligible (\(|\Delta| < 0.147\)), small (\(|\Delta| < 0.33\)), medium (\(|\Delta| < 0.474\)), or large (\(|\Delta| \geq 0.474\)).

The three hypotheses are defined as follows:

• \(H_0^{(1)}\): There is no difference in daily commit frequency between Period A and Period B.

• \(H_0^{(2)}\): There is no difference in code volume per unit of work between Period A and Period B.

• \(H_0^{(3)}\): There is no difference in daily \(\mathrm{FCR}\) between Period A and Period B.

Table ¿tbl:tab:statistical95testing? summarizes the results.

Daily metrics are means over active days (22 in Period A, 20 in Period B). Cliff’s \(\Delta > 0\) indicates Period A stochastically larger; effect sizes: N = negligible, M = medium, L = large (thresholds of Section V-C).

Three findings emerge. First, we fail to reject \(H_0^{(1)}\) at \(\alpha = 0.05\) (\(p = 0.066\)): daily commit count does not differ significantly across periods, though Period B trends toward a higher count (mean 18.4 vs., \(\Delta = -0.332\)). Second, for \(H_0^{(2)}\) we report a transparent null result: neither the day-level insertion distribution (\(p = 0.801\)) nor the per-commit insertion size (median 36 vs. lines; \(p = 0.068\)) differs significantly, with negligible effect sizes—consistent with the comparable code volume (75.7k vs.80.3k code-only lines) that holds output roughly constant. Third, \(H_0^{(3)}\) is rejected decisively (\(p = 0.0002\)) with a large effect size (\(\Delta = -0.66\)): on comparable working days, the share of repair work under the local GLM configuration is dramatically higher.

To quantify the association on the aggregated commit contingency table (Period A: 113 fix vs. non-fix; Period B: 275 fix vs. non-fix), a Pearson \(\chi^2\) test of independence rejects the null hypothesis with high significance: \(\chi^2(1) = 53.30\), \(p < 0.0001\), \(\phi = 0.295\). The unadjusted Odds Ratio is \(3.52\) (95% CI \([2.49, 4.96]\)). Because the two periods differ in difficulty composition (Section IV-E), we additionally compute the Mantel–Haenszel odds ratio stratified by difficulty tier, obtaining \(\mathrm{OR}_{\mathrm{MH}} = 3.61\), with per-tier odds ratios of \(4.88\) (High: 46.5% vs.%), \(2.63\) (Medium: 80.6% vs.%), and \(3.81\) (Low: 86.9% vs.%). The stratified and unadjusted estimates are close, demonstrating that the defect-repair premium of the local configuration is not an artifact of task mix: within every difficulty tier, the odds of a commit being a defect repair are \(2.6\)–\(4.9\times\) higher under GLM (Fig. 5). We flag one construct nuance: because difficulty is derived from change footprint and fix commits are systematically smaller (median 25 vs. lines, fix vs.non-fix), the Low tier is mechanically enriched with fixes (hence Claude’s inverted ordering: Low 63.4% \(>\) Medium 61.2% \(>\) High 15.1%). This couples the stratifier to the outcome base rate, not to the between-period contrast; applied identically to both periods, it cannot manufacture the odds-ratio gap—so the tiers read as change-size strata rather than intrinsic-difficulty measures.

Finally, the defect-taxonomy distributions of Table ¿tbl:tab:defects? differ significantly between deployments (\(\chi^2(3) = 51.93\), \(p < 0.0001\), Cramér’s \(V = 0.37\)); however, this difference is driven by the Test/CI category, whose Period-A excess is infrastructure churn unrelated to model output (Section V-B, “Test and CI Repair”). Excluding Test/CI, the remaining model-attributable categories do not differ significantly (\(\chi^2(2) = 5.59\), \(p = 0.06\), \(V = 0.13\)), so the two deployments differ primarily in the volume of repair work, not its structural composition.

We treat the daily-FCR difference (\(H_0^{(3)}\)) as primary and the rest as descriptive; the principal results survive multiplicity. Under a conservative Bonferroni correction across the seven tests here—the four contrasts of Table ¿tbl:tab:statistical95testing?, the aggregate and taxonomy \(\chi^2\) tests, and the Mantel–Haenszel stratified contrast (\(\alpha = 0.05/7 = 0.0071\))—the daily-FCR (\(p = 0.0002\)), aggregate-fix (\(p < 0.0001\)), and defect-taxonomy (\(p < 0.0001\)) differences all remain significant, while the non-significant tests are unaffected. Other tests reported elsewhere (per-language FCR, weekly trends) are exploratory and excluded from this confirmatory family. No conclusion depends on a \(p\)-value near threshold.

Scope of inference. These tests take the day or the individual commit as the unit of analysis. Because every observation derives from one developer on one repository, these units are not independent replicates: consecutive days and commits are autocorrelated, and the design is single-subject and two-period rather than randomized between-subjects. The reported \(p\)-values and CIs therefore quantify how strongly and consistently the difference manifests within this developer and project, and do not by themselves license inference to a broader population; we defer that to the multi-developer cross-over protocol of Section VII-B.

5.4 Financial Cost Evaluation↩︎

We calculate the financial cost of both configurations. For the Claude API, we apply Anthropic’s published Opus 4.7/4.8 rates with prompt caching ($5.00/M base input, $6.25/M cache write, $0.50/M cache read, $25.00/M output) [25]. The on-premise workload ran on a shared NVIDIA GB200 NVL72 cluster; for cost accounting we attribute it to the minimum viable serving footprint of GLM-5.2 NVFP4—a 4\(\times\) B200 slice, one GB200 NVL72 compute tray (Section V-H), i.e., a lower bound on the hardware a team must provision. We evaluate two rental-equivalent scenarios for this footprint at $16.00/hour ($4.00 per GPU-hour, consistent with June–July 2026 market on-demand rates of $3.70–$5.90 per B200-hour across major GPU clouds [26]):

• Scenario I (Dedicated Reservation): Dedicated around-the-clock allocation for the full period (24 h \(\times\) 28 d = 672 hours) = $10,752.00 USD.

• Scenario II (Shared Allocation): Allocation charged as a standard 160-hour monthly developer seat under the shared cluster’s chargeback model = $2,560.00 USD.

Because both agent configurations ran around the clock—including unattended sessions—per-developer attribution is a billing convention rather than a measurement; Scenarios I and II bracket that attribution spectrum, from full wall-clock reservation to a standard seat.

\(^{\dagger}\)Total-bill comparison; the per-token rates below invert it—the cached API is cheaper per processed token despite the larger bill—as analyzed in Section V-D.

Two structural observations follow from Table ¿tbl:tab:cost?. First, absent caching, the same Claude token volume would have cost $77,059 at nominal input rates; prompt caching therefore reduced realized spend by 88.6%. Second, and counterintuitively, caching drives Claude’s effective unit price ($0.57 per million processed tokens) below both the shared on-premise amortization ($2.83/M) and the dedicated-reservation unit cost ($11.87/M). The API’s total bill remains higher only because the Claude configuration processed \(16.9\times\) more tokens: per processed token, the cached frontier API is the cheaper option—an inversion of what nominal price sheets suggest. The on-premise per-token figure divides a time-billed resource by one developer’s token volume, so it is inflated by low single-tenant utilization; we therefore treat total realized spend and the TCO of Section VI-A as the robust quantities, with the per-token rate as corroboration.

Extrapolating to a 100-developer organization, a pure API approach would cost approximately $879,000 USD monthly versus $256,000 USD for shared on-premise allocation. We caution that this assumes the per-developer amortization holds at fleet scale: serving 100 concurrent workloads needs proportionally more GPU capacity (one 4\(\times\) B200 tray cannot serve 100 developers), so the extrapolation compares amortization rates, not a fixed footprint.

5.5 Model Performance Benchmarking↩︎

To provide context for the empirical productivity results, Fig. 6 contrasts output throughput with the Artificial Analysis Intelligence Index for the four models, as measured on the vendors’ hosted endpoints in July 2026 [14]. There is a clear trade-off between throughput and reasoning quality: the Claude Opus 4.8 and 4.7 APIs lead on intelligence (Index 56 and 54) but deliver 66.0 and 56.6 output tok/s, whereas hosted GLM-5.2 sustains 214.9 tok/s at Index 51 (GLM-5.1: 76.4 tok/s at Index 40).

Our local NVFP4 deployment of GLM-5.2 on the shared NVL72 cluster sustained observed output speeds of roughly 185 tok/s under interactive agent load—a single-tenant deployment observation rather than a controlled benchmark, but of the same order as the hosted 214.9 tok/s figure and consistent with Blackwell’s native 4-bit hardware acceleration, and approximately \(2.8\times\) the Opus 4.8 API. Faster raw generation, however, is not expected to translate into faster end-to-end task completion in Period B: the elevated defect rate multiplied the number of repair iterations per task (Section V-B), so review-and-repair overhead—not raw token throughput—dominates wall-clock time. We did not instrument wall-clock completion time directly; consistent with this expectation, commit cadence slowed in Period B (median inter-commit interval 12.7 vs. min, Section V-G)—echoing the slowdown measured in the agentic-assistant field RCT [6]—though that proxy also absorbs the higher repair volume.

5.6 Qualitative Case Studies and Code Anomalies↩︎

To understand the nature of coding errors generated by the on-premise quantized GLM-5.1/5.2 configuration, we examine two representative defects drawn from Period B fix commits (code paraphrased and identifiers genericized).

5.6.1 Static Typing Violation (mypy)↩︎

Listing 1 shows a type-discipline failure. The agent implemented an internal API call whose backend endpoint returns a dict for POST requests but a list for GET requests, and bound the result without the required union annotation. The un-annotated assignment fails mypy on the exception-handling branch, and a dedicated repair commit adding the dict | list annotation was required.

Listing 1. GLM omits the union type (POST returns dict, GET returns list):

result = await client.request(token, payload)
# mypy: incompatible assignment on the
# except-branch. Fix:
result: dict | list = \;
    await client.request(token, payload)

5.6.2 Environment-Behavior API Misuse (httpx proxy inheritance)↩︎

In the second case, the agent constructed an httpx client for a cluster-internal service without disabling environment-variable trust. The client silently inherited the pod’s HTTP_PROXY settings, routing in-cluster traffic through an external proxy and breaking the integration—a defect invisible at compile time that surfaced only at runtime inside the Kubernetes network.

Listing 2. GLM output inherits the proxy environment:

client = httpx.AsyncClient(base_url=URL)
# -> in-cluster calls routed via HTTP_PROXY
# Fix:
client = httpx.AsyncClient(
    base_url=URL,
    trust_env=False)  # bypass proxy for
                      # cluster-internal calls

Both anomalies illustrate the pattern quantified in Table ¿tbl:tab:defects?: the local configuration’s failures concentrate in typing discipline and in the correct use of API and environment semantics—categories where the frontier API model rarely required dedicated repair commits.

5.7 Objective Workload and Developer-Experience Indicators↩︎

Rather than rely on subjective self-report to characterize the developer’s day-to-day experience, we derive a set of objective behavioral indicators directly from the timestamped Git history. Each indicator is computed deterministically from commit metadata and is fully reproducible; together they operationalize the workload dimensions that instruments such as the NASA-TLX [27] capture subjectively, but without recall bias or self-report subjectivity. We define: (i) the debugging-spiral share, the fraction of commits occurring inside a maximal run of three or more consecutive fix commits; (ii) the longest uninterrupted repair run; (iii) the median inter-commit interval within a working stretch (gaps under 8 hours), a proxy for the cadence of visible progress; (iv) the median daily active span, the wall-clock hours between the first and last commit on active days, a proxy for the workflow’s temporal footprint (sessions frequently ran unattended) rather than continuous human presence; (v) the mean rework multiplicity, the average number of distinct commits touching each file. Table ¿tbl:tab:behavioral? reports these indicators for both periods.

The indicators consistently show a heavier, more repair-dominated workload under the local GLM configuration: 69.8% of Period B commits fell inside consecutive-fix runs (versus 35.0% in Period A), the longest such run growing from 18 to 58 repair commits, with progress cadence, active-day span, and rework multiplicity all degrading in step (Table ¿tbl:tab:behavioral?; Fig. 7 plots each indicator’s Period-B-to-Period-A ratio, every axis worsening with no offsetting improvement). These behavioral signals corroborate the quality findings of Section V-B along a complementary axis: the cadence and span indicators derive purely from timestamps, and the run-structure indicators add temporal patterning to the keyword classification. Unlike a subjective questionnaire, they are free of recall bias and self-report framing—though not of a hypothesis-aware subject’s labeling behavior, bounded by the invariance arguments of Section VII-C.

5.8 Hardware Serving Configuration and Quantization↩︎

The local serving configuration follows the deployed Hugging Face checkpoint configurations [12], [20]: both generations were served via the vLLM engine with PagedAttention [7] on a shared NVIDIA GB200 NVL72 cluster. The smallest allocation on which the NVFP4 checkpoint can be served—a 4\(\times\) B200 slice, one GB200 NVL72 compute tray—is the cost-accounting basis rather than the full physical cluster.

The arithmetic of 4-bit compression is what makes the one-tray footprint possible: a 753B-parameter model occupies roughly 1.5 TB in BF16—beyond even an eight-GPU allocation (\(\sim\)​1.5 TB HBM) once KV cache and activations are included—whereas the NVFP4 checkpoint stores quantized weights at 4.5 bits per parameter (4-bit values plus one FP8 scale per 16-element block); with embeddings, router, and normalization tensors retained at higher precision, the deployed checkpoint measures \(\approx\)​465 GB, fitting a 4\(\times\) B200 tray (\(\sim\)​750 GB HBM) with an FP8 KV cache. As noted in Section II-C, vendor calibration reports claim near-lossless NVFP4 accuracy on standard benchmarks; Blackwell’s native FP4 tensor-core path underlies the high observed generation throughput. This one-tray footprint is what enables the low amortized cost of $2,560.00 USD in our economic model.

We emphasize a scoping limitation: we did not deploy FP16 or FP8 baselines of GLM-5.2 for a controlled quantization ablation (an FP16 baseline alone would occupy more than a full eight-GPU allocation). Consequently, our defect analysis cannot empirically separate quantization-induced degradation from the base model’s intrinsic capability gap relative to Claude Opus. We instead treat the NVFP4 checkpoint as the unit of analysis—it is the artifact an enterprise would actually deploy—and flag the attribution question as a threat to construct validity in Section VII-C.

6 Discussion↩︎

Before interpreting these results, we state their causal scope. The gap we measure—most sharply the \(\approx\) \(3.6\times\) difference in defect-repair odds—is the joint effect of a deployable bundle: base-model capability, NVFP4 quantization, vLLM serving, the agent harness (Claude Code vs.Opencode), and the fixed period ordering of a naturalistic study. It cannot be attributed to any single factor—least of all “the model” in isolation—so every claim below should be read at the level of the deployable configuration an enterprise would adopt; factor decomposition is deferred to Section VII-C.

To synthesize the empirical findings, Table ¿tbl:tab:overall? presents a multi-dimensional comparative summary of the commercial public API model and the on-premise open-weights cluster configuration.

6.1 The Cost–Quality Trade-off and Developer Churn↩︎

Our results highlight a classic software quality trade-off. While the local infrastructure reduces direct compute expenditure by up to 71% (shared allocation), it introduces substantial developer churn: the high \(\mathrm{FCR}\) (74.93%) of the GLM-5.1/5.2 configuration implies that the developer spent considerable work hours correcting suboptimal AI outputs.

We can model the total cost of development (\(TC\)) as: \[TC = C_{\text{compute}} + C_{\text{ops}} + (T_{\text{dev}} \times R_{\text{dev}}).\] To parameterize this model under realistic industrial conditions in Taipei, Taiwan (all conversions at 32 NTD/USD), we set the developer wage rate to \(R_{\text{dev}} = \$35.00\) USD per hour (NT$1,120/hour). This approximates the total employer compensation cost of a senior software engineer in Taiwan—official earnings statistics and market data (cf. [28]) place senior engineer base pay near NT$90k–130k per month, which grosses to roughly NT$170k–190k per month after statutory labor/health insurance, pension contributions, and bonus provisions, i.e., about NT$1,120 per working hour.

For the cloud API configuration (Period A), \(C_{\text{ops}} = 0\) as infrastructure maintenance, electricity, and cooling are fully absorbed by the provider. For the private on-premise B200 allocation (Period B), we define \(C_{\text{ops}}\) from Taiwan operating parameters:

1. Power and Cooling: A 4\(\times\) B200 serving tray draws approximately 6.0 kW under load, accounting for the roughly 1.2 kW per-GPU board power plus the amortized host system share. Over a 28-day cycle (672 hours) at Taipower’s high-voltage industrial summer rate of approximately NT$4.3 per kWh [29] ($0.134 USD/kWh), direct electricity is $542; applying a Power Usage Effectiveness (PUE) multiplier of 1.5 for sub-tropical cooling overhead yields a total power cost of \(\approx\$813\); inference-time carbon emissions [30] are a further dimension our dollar-denominated model does not price.

2. MLOps Personnel Amortization: A local site-reliability or MLOps engineer at a monthly employer cost of NT$100,000 ($3,125 USD/month), amortized across a shared cohort of \({\sim}100\) developers, contributes \(\approx\$30\) USD per developer.

3. Facilities and Maintenance: Space and data center bandwidth amortized at \(\$50.00\) USD per developer.

This yields a total operational overhead of \(C_{\text{ops}} \approx \$893\) USD (NT$28,576) per developer for Period B. The three components deliberately use different bases—power at the full-tray 672-hour draw, MLOps and facilities amortized across a 100-developer cohort. Charging the whole tray’s power to one developer over-attributes idle-time energy to Period B, and adding local ops on top of a rental rate that already embeds facility overheads double-counts in the same direction; both are conservative against our savings conclusion, so the true saving is if anything larger. We evaluate the developer’s defect-repair time (\(T_{\text{dev}}\)) as the number of fix commits times a per-fix review time. This directly instantiates the developer-time term of the feedback-loop model (Section IV-D): the timestamped commit history measures realized repair effort directly, so the \(T_{\text{iter}}\) decomposition serves as the conceptual frame while the observed fix-commit volume supplies the quantity. Rather than assume an asymmetric per-fix duration, we apply a single symmetric review time of 15 minutes (0.25 hours) to both configurations, so that the labor difference is driven entirely by the objectively counted difference in fix-commit volume rather than by a subjective per-fix penalty. The symmetric value is empirically grounded: the median inter-commit interval immediately preceding a fix commit is nearly identical across periods (14.1 min in Period A, 13.2 min in Period B), so the per-fix repair cadence itself shows no asymmetry—the overall cadence gap (5.9 vs. min, Section V-G) stems from the surrounding non-repair workflow. Initial feature-development labor, common to both periods and of comparable magnitude, cancels in this between-configuration comparison, so the “True TCO” below is compute plus differential repair labor rather than an absolute lifecycle cost.

• Period A (Opus): 113 fix commits \(\times\) 0.25 h \(\times\) $35 \(= \$988.75\) (\(T_{\text{dev}}^A = 28.25\) h). The True TCO (Compute + Labor) is \(\$9{,}773.96\) USD.

• Period B (GLM): 275 fix commits \(\times\) 0.25 h \(\times\) $35 \(= \$2{,}406.25\) (\(T_{\text{dev}}^B = 68.75\) h). Adding \(C_{\text{ops}}\), the True TCO under shared allocation is \(\$2{,}560.00 + \$893.00 + \$2{,}406.25 = \$5{,}859.25\) USD (NT$187,496); under dedicated reservation it is \(\$10{,}752.00 + \$893.00 + \$2{,}406.25 = \$14{,}051.25\) USD.

We caution that commit cadence is an upper-bound proxy for developer effort (it includes agent and test-runner wait time); we therefore report TCO to indicate magnitude, not precision. Comparing the two periods, the allocation regime decides the outcome: under shared allocation, local deployment saves 40.1% of True TCO ($5,859.25 vs.$9,773.96, a net $3,914.71 saving per 28 days), whereas under dedicated reservation it costs 43.8% more than the cached API ($14,051.25 vs.$9,773.96)—the 512 of 672 tray-hours billed beyond the standard seat erase the unit-cost advantage. In the shared regime, the dominant driver is that the cached-API compute bill ($8,785) still dwarfs the tray amortization ($2,560) plus the incremental repair labor ($2,406 vs.$989), so even a \(2.4\times\) defect-repair burden does not erase the compute advantage. Instantiating the break-even criterion of Section IV-D: \(\Delta C_{\text{compute}} = \$8{,}785 - \$3{,}453 = \$5{,}332\) against \(R_{\text{dev}} \times \Delta T_{\text{dev}} = \$35 \times 40.5\,\text{h} = \$1{,}418\)—the shared-allocation advantage clears the criterion by \(3.8\times\). The genuine cost of the local configuration is therefore not dollars but the developer-experience burden quantified objectively in Section V-G.

Parameter sensitivity. Because the cost model rests on several point estimates, we sweep the three most consequential: the wage rate ($27–45/hr), the GPU lease rate ($3–6 per GPU-hour), and the per-fix review time (10–25 min). The shared-allocation conclusion is robust—on-premise True TCO savings stay positive across the entire sweep, ranging from about 9% (all-adverse corner) to 56% (all-favorable). The dedicated-reservation option, by contrast, is dominated throughout: with the tray reserved around the clock, its True TCO exceeds the cached API’s at every point of the sweep (from roughly 10% above in the most favorable corner to more than double in the most adverse), so dedicated reservation only pays off when the tray serves substantially more than a single developer’s workload. Attribution hours are a fourth axis: at the point estimates, the shared-allocation saving survives any per-developer attribution up to \(\approx\)​405 of the 672 tray-hours (\(\approx\)​2.5 standard seats, 60% of the period’s wall clock); only beyond that does the on-premise True TCO exceed the cached API’s. The qualitative ordering—shared on-premise cheapest, then the pure API, then dedicated on-premise—holds throughout the sweep. One axis is held fixed: the Claude bill uses public list rates, though Anthropic also offers negotiated volume and committed-use discounts [25]; a committed-use reduction of \(\approx\)​45% in the API compute bill would by itself equalize the shared-allocation TCO, so this advantage is scoped to list-price API economics.

6.2 Information Security and Governance Context↩︎

Beyond mitigating the third-party data-governance risk already noted (Section III-E), local NVL72 serving keeps in-scope inference data inside the corporate network.

However, operating a shared local GPU cluster introduces internal multi-tenancy risk: proprietary code, schemas, and request contexts from multiple departments co-reside in shared GPU memory and host logs. Our infrastructure therefore layers hardware-level isolation (NVIDIA Multi-Instance GPU partitions with dedicated compute and memory outside the tensor-parallel serving trays), network and container isolation (dedicated Kubernetes namespaces under strict network policies, mutual TLS in transit), and token-based request isolation (role-based access control via developer-specific tokens). Because prefix caching is enabled, KV prefixes may be reused across requests, so cross-tenant isolation is enforced at these hardware, network, and access-control layers—maintaining data sovereignty not only against external third parties but also internally across multi-tenant development cohorts.

6.3 Cost–Quality Frontier of a Hybrid Routing Gateway↩︎

To evaluate the proposed hybrid routing gateway, we perform an offline counterfactual replay over the pooled set of 613 real non-merge commits from both periods, using the empirically measured per-tier fix rates of each configuration (Section V-C): Claude 15.1%/61.2%/63.4% and GLM 46.5%/80.6%/86.9% for High/Medium/Low tiers respectively. Each routing policy assigns every difficulty tier to one backend; infrastructure cost is amortized per commit by dividing each backend’s measured period total (API $8,785.21; shared GPU + ops $3,453) by the pooled 613-commit denominator—so routing 100% of the pooled workload to a backend reproduces its measured bill—and repair labor is priced per fix commit at the symmetric per-fix review time of Section VI-A. This assumes each backend’s infrastructure cost scales linearly with commit count and that the pooled workload’s footprint matches that backend’s own period—first-order approximations, not measured quantities. The Pure Local GLM TCO reported here ($7,388) is deliberately not the Period B actual of Section VI-A ($5,859): the replay reprices the pooled 613-commit workload—not Period B’s own 367 commits—at each backend’s per-tier fix rates, so that all four policies are scored on an identical workload and are therefore mutually comparable, at the cost of not being directly comparable to the single-period figures.

Table ¿tbl:tab:hybrid95simulation? yields a sobering correction to the intuitive “best of both worlds” narrative. Because the local configuration’s fix rate exceeds Opus’s within every difficulty tier—including 86.9% on Low-difficulty commits—no routing policy that sends any tier to the local model can match the pure-API defect profile. Routing decisions therefore trace a monotonic cost–quality frontier rather than finding a dominating optimum: each increment of workload shifted to the local backend buys infrastructure savings at the price of a higher overall FCR and more repair labor (Fig. 8 decomposes each policy’s TCO into compute/infrastructure and repair-labor components). On this frontier, the High\(\rightarrow\)Claude hybrid is a defensible compromise (26.3% TCO saving while protecting the highest-stakes work with the stronger model), and pure local remains the TCO minimum (35.3% saving) for organizations that can absorb the defect churn. Notably, the High+Med hybrid still saves 13.4% but delivers an FCR 9 points worse than pure Claude—illustrating that partial-quality strategies must be checked against the frontier, not assumed beneficial. A dynamic escalation policy (start locally, escalate on repeated self-healing failure) would require modeling escalation latency and retry token burn that our commit-level replay cannot resolve; we leave its evaluation to the gateway prototype in future work. Figure 9 plots the four policies in cost–quality space, making the monotonic frontier explicit: moving left (cheaper) strictly moves up (worse FCR). Cost–quality routing has been explored on public benchmarks—cascades [31] and learned routers [32]—and cost-controlled agent evaluation is an emerging norm [33]; what this frontier adds is per-tier defect-repair rates and repair labor measured on a production enterprise workload.

6.3.1 Developer-Adaptive Thresholding (Design Sketch)↩︎

The static routing policies above assign whole difficulty tiers uniformly, but the automation-bias risk of accepting a defective suggestion depends on the reviewing developer. We therefore sketch—as an explicitly unevaluated design direction—an adaptive gateway that routes on the product of a normalized task complexity \(C(T) \in [0, 1]\) (aggregating change size, module dependency fan-out, and target-file cyclomatic complexity) and a developer-experience coefficient \(E(P) \in [0, 1]\) (estimated from the developer’s historical Git-mined success rate): a task is routed to the local model only when \(C(T)\,(1 - E(P)) \le \theta\) and to the cloud API otherwise, where \(\theta\) is an enterprise safety tolerance. The intent is that harder tasks and less-experienced reviewers both bias routing toward the stronger cloud model, shrinking the low-yield manual-repair loops most likely to trap junior developers. We deliberately report no numeric instantiation: the present study does not vary developer profiles, so calibrating \(\theta\), the complexity weighting, and \(E(P)\) against measured per-developer outcomes requires the multi-developer replication we leave to future work.

6.3.2 Uncertainty Quantification via Bootstrap Resampling↩︎

Because both the per-commit outcomes and the per-cell fix rates are estimated from finite samples, presenting the simulated TCO and FCR as point estimates ignores sampling variance. We therefore quantify uncertainty by a two-stage bootstrap: in each of \(N = 10{,}000\) iterations, we first resample every backend\(\times\)tier cell at its observed period size to redraw its fix rate (propagating rate-estimation error), then resample the 613 pooled commits with replacement and redraw each commit’s fix outcome as a Bernoulli trial from the redrawn rate of its tier and assigned backend. The resulting 95% confidence intervals are reported in Table ¿tbl:tab:hybrid95simulation? and visualized in Fig. 10. The four TCO intervals are pairwise disjoint, and the full TCO ordering (Pure GLM \(<\) High\(\rightarrow\)Claude \(<\) High+Med \(<\) Pure Claude) persists in all 10,000 paired resamples; the inverse FCR ordering persists in 99.8% of resamples for the three-policy comparison (97.3% across all four, whose adjacent FCR intervals overlap). The cost–quality frontier’s ordering is thus statistically stable under workload and rate resampling—the trade-off itself, not merely its point estimate, is robust.

We stress that the bootstrap quantifies sampling variance only. The replay rests on a stronger structural assumption: that each backend’s per-tier fix rate is a transportable property of the (difficulty tier, backend) pair, independent of which tasks are routed. Because the two periods carried different task streams, this treats the tiers as exchangeable across periods—for instance, it assumes a High-difficulty task rerouted to Claude would inherit Claude’s observed High-tier rate rather than a rate specific to that task’s content. The simulation is therefore a first-order projection, not a deployed-system measurement; a routing gateway can also change outcomes through mechanisms this static replay cannot capture (e.g., context handoff on escalation, or selection effects in which tasks reach which backend). We accordingly present the frontier as an illustrative decision aid whose ordering is robust, and defer measured validation to the gateway prototype of Section VIII (Future Work).

6.4 Generalizability across Programming Language Paradigms↩︎

Our empirical evaluation was conducted within the production monorepo under study, a hybrid codebase mixing dynamically typed Python (37.6% of Period A insertions) with statically typed TypeScript (17.1%), plus specification documents and configuration (Table ¿tbl:tab:languages?). To analyze how the cost–quality trade-offs vary by language paradigm, we assign each non-merge commit a dominant language by the file types it touches, and compute per-language FCR for both periods:

• Statically Typed (TypeScript): In Period A (Opus), TypeScript-dominant work yielded 48 commits, of which 20 were fixes (\(\mathrm{FCR}_{\text{TS}} = 41.7\%\)). In Period B (GLM), 79 commits with 50 fixes (\(\mathrm{FCR}_{\text{TS}} = 63.3\%\)). TypeScript’s compile-time checks surface interface and type violations immediately in the agent loop, keeping the repair cycle localized to fast compiler feedback.

• Dynamically Typed (Python): In Period A, Python-dominant work yielded 95 commits with 46 fixes (\(\mathrm{FCR}_{\text{Py}} = 48.4\%\)). In Period B, 228 commits with 198 fixes (\(\mathrm{FCR}_{\text{Py}} = 86.8\%\)). Without compile-time type enforcement, incorrect interface assumptions in the local model’s output tend to bypass initial static checks and surface only at runtime or in test suites (e.g., FastAPI dependency wiring and Pydantic validation), lengthening the repair feedback loop relative to compiler-caught defects.

In both periods, Python-dominant commits exhibit a higher FCR than TypeScript-dominant commits (\(48.4\%\) vs.\(41.7\%\) under Opus; \(86.8\%\) vs.\(63.3\%\) under GLM), and the deployment effect dwarfs the language effect: switching from Opus to the local GLM configuration sharply raises the per-language FCR in both paradigms (Fig. 11). This suggests our cost–quality findings are not an artifact of a single language ecosystem, while static typing shows a consistent directional defect-surfacing advantage (the within-period gaps are not individually significant)—in line with large-scale repository evidence associating type discipline with defect proneness [34], an association whose magnitude subsequent replication work has questioned [35].

7 Threats to Validity↩︎

In this section, we discuss the potential threats to the validity of our empirical evaluation, categorized into internal, external, and construct validity.

7.1 Internal Validity↩︎

Internal threats concern factors that could affect the observed causal relationships.

• Learning Curve (Maturation) Effect: Since Period B (local GLM deployment) succeeded Period A (cloud API deployment), the developer’s familiarity with the codebase naturally increased; any maturation effect would therefore favor Period B. Within Period B, the FCR showed no significant decline over time—first half (days 1–14): 77.6% (121/156), second half: 73.0% (154/211); \(\chi^2(1) = 1.00\), \(p = 0.32\)—indicating that four weeks of accumulated experience with the local model did not reduce its repair burden (Fig. 12). Within Period A, the FCR rose modestly from 42.2% to 50.0% (\(p = 0.22\), not significant) as the work mix shifted from feature build-out toward stabilization and CI hardening in later weeks. Neither temporal pattern supports maturation as an explanation for the between-period gap: even Period A’s most repair-heavy week (55.4%) remains below Period B’s least repair-heavy week (63.8%). Conversely, Period B inherited a codebase enlarged by Period A’s \(+\)​90.7k net lines, which could raise task hardness independently of the model; however, the repair premium persists within every change-size tier (Section V-C), and the repair mix shifts toward local syntax/type defects (Table ¿tbl:tab:defects?)—a pattern repository growth alone does not explain.

  

• Task Selection Bias: The difficulty imbalance runs opposite to the confound (Section IV-E), and the stratified \(\mathrm{OR}_{\mathrm{MH}} = 3.61\) confirms the GLM premium within every tier (Section V-C).

• Experimenter and Demand-Characteristic Bias: The sole subject is the hypothesis-aware first author, so the self-benchmark is exposed to expectancy effects. This is bounded structurally: the fix label is assigned by a deterministic keyword rule (Section IV-C) rather than discretionary judgment, every counted commit passed the same enforced CI/review gate (Section IV-G), and the premium holds within every difficulty tier (Section V-C) and both language paradigms (Section VI-D)—a structured pattern a diffuse expectancy effect would not produce.

7.2 External Validity↩︎

External threats concern the generalization of our findings.

• Single Project and Deployment Context: Our empirical data derives from one production corporate AI PaaS monorepo under a single organizational deployment. While this controls for codebase and organizational noise, it may not immediately generalize to other codebases or larger, more heterogeneous engineering organizations. However, the scale and density of our data—over 44,000 LLM requests and 613 non-merge commits across 56 days—pin down the aggregate token, cost, and defect measures with high precision. This sharpens the aggregates’ precision, not their generality; the inferential unit remains the day or the commit (Section V-C). To facilitate broader generalization, a two-phase cross-over protocol—randomly splitting a cohort of \(N\) developers into two groups and swapping the API and on-premise configurations between phases—would neutralize inter-subject skill variance and linear learning curves.

• Experience Level and Automation Bias: The development activity under study reflects experienced-engineer usage patterns, with deep familiarity with the codebase. If replicated in a less-experienced context, the results may differ. Less-experienced developers may suffer from automation bias, blindly accepting incorrect code changes generated by the local GLM agent, which would raise the risk of defect leakage into the production branch. They may also experience a steeper surge in the behavioral workload indicators of Section V-G (debugging-spiral share, active-hours span), owing to a reduced ability to resolve repetitive compile loops efficiently. Section VI-C outlines an adaptive routing design intended to contain this risk by adjusting complexity thresholds to developer experience; it remains unevaluated, so the threat stands for less-experienced cohorts.

• Model Specificity: The evaluation is limited to the Claude and GLM model families. While these models represent current state-of-the-art hosted and local archetypes, rapid LLM iterations may alter the absolute performance scores.

7.3 Construct Validity↩︎

Construct threats concern the relationship between the theory and the observation.

• Fix Commit Ratio (FCR) Validity: We measured quality via keyword mining of commit messages, which inherits a known limitation of label-based defect classification—misclassification can bias downstream analyses [36]. Trivial lint, typo, or formatting changes may be miscounted as repairs, and—as the taxonomy showed (Section V-B)—a large share of Period A’s fix commits were test/CI-infrastructure maintenance rather than model-generated-logic repairs. FCR is thus an upper bound on model-attributable defects for both periods, and the infrastructure share was larger in Period A (41.6%), biasing the headline comparison against our conclusion.

  Single-source and commit-granularity threats. All quality signals derive from the same Git-commit stream; we have no independent defect oracle such as field-incident reports. Two properties bound this. First, both periods ran under an identical, enforced pre-push CI gate (Table 1), so trivial churn is blocked before push and the fix commits we count predominantly reflect defects that survived a consistent gate; applied symmetrically, it cannot manufacture a between-period difference. Second, a change in commit granularity alone cannot explain the pattern: the taxonomy shifts in composition toward syntax/type and API defects under GLM (Table ¿tbl:tab:defects?), the premium persists within every difficulty tier (\(\mathrm{OR}_{\mathrm{MH}} = 3.61\)), and it holds for both statically and dynamically typed code (Section VI-D)—content-defined properties invariant to how repairs are packaged.

• Benchmark Contamination: Public base-model benchmark scores (e.g., SWE-bench) might suffer from pre-training data contamination; our study, conducted on a proprietary enterprise monorepo with no public internet exposure, provides a contamination-free complement to those leaderboards (Section III-D).

• Model Capacity vs.Agent Harness Confounding: The observed differences are a combined product of base-model capacity, serving configuration (NVFP4 quantization), and harness design (Claude Code vs.Opencode). These cannot be fully separated in a naturalistic study: Claude Code is vendor-co-designed with Claude models, whereas Opencode adapts open-weights models through generic per-provider prompt profiles. The comparison should therefore be read at the level of deployable configurations—the harness-plus-model bundles an enterprise would adopt—not as isolated base-model capability. Disentangling these factors (e.g., Claude through Opencode, or GLM through multiple harnesses) is left to controlled future experiments.

8 Conclusion and Future Work↩︎

We answer the four research questions of Section I directly:

• RQ1 (Cost economics): Prompt caching (99.3% hit rate) cuts the API’s realized cost by 88.6% to an effective $0.57 per million tokens—below even the shared on-premise amortization ($2.83/M). Per token, the cached API is the cheaper option at single-tenant on-premise utilization; its larger total bill stems only from its \(16.9\times\) higher token volume.

• RQ2 (Code quality): At comparable gross code churn, the local configuration was associated with a far higher defect-repair burden—\(\mathrm{FCR}\) 74.9% vs.%, with defect-repair odds \(2.6\)–\(4.9\times\) higher within every difficulty tier (\(\mathrm{OR}_{\mathrm{MH}} = 3.61\)).

• RQ3 (True TCO): Shared on-premise allocation minimizes True TCO—a 40.1% saving—whereas dedicated reservation costs 43.8% more than the cached API; the shared saving is robust across the swept parameters at list-price API economics (Section VI-A), while the dedicated option is dominated throughout.

• RQ4 (Developer experience): The local configuration was associated with an objective developer-experience burden—roughly \(2\times\) the debugging-spiral share, a \(3.2\times\) longer worst-case repair run, and a \(2.2\times\) slower commit cadence (a timestamp proxy that also absorbs unattended agent runtime).

For future work, we propose the automated hybrid routing gateway of Fig. 13: complexity pre-filtering routes high-stakes tasks to the cloud API; routine tasks start locally and escalate after a bounded number of failed self-healing iterations; evaluating its escalation latency and retry token burn is the primary open measurement question. We further plan parallel candidate generation with test-based filtering [37], exploiting local serving’s zero marginal cost, and the cross-over, multi-developer replication of Section VII-B.

Acknowledgments↩︎

The authors thank the PEGAVERSE platform engineering team for operating the shared NVIDIA GB200 NVL72 cluster and the enterprise quality-assurance and delivery pipeline spanning both evaluation periods.

Ethical Considerations↩︎

This work is a self-study: the single developer analyzed is the first author, and the behavioral indicators (Section V-G) derive from that author’s own commit metadata, with consent. No customer or personal data was collected and no PII or proprietary source code is disclosed (the Section V-F code is paraphrased); with no external human subjects, IRB oversight did not apply. The study targets deployable configurations, not individuals—a point-in-time snapshot, not a durable model-family ranking.

Data Availability↩︎

The proprietary enterprise codebase, raw telemetry, repository history, and analysis scripts remain confidential. Instead, the methodology is specified in full herein—telemetry queries, deterministic classification and difficulty rules, cross-branch git patch-id mining, and cost-model parameters (Sections IV–VI)—sufficient to reproduce the pipeline on other data.

References↩︎