1 Introduction↩︎

Large Language Models (LLMs) for code are increasingly integrated into modern software development environments, assisting developers with tasks such as boilerplate code generation, bug fixing, and refactoring [1]. As these systems become embedded in developer workflows, they act as programming assistants that influence how software systems are maintained and evolved. This widespread adoption raises important questions about the security implications of AI-generated code in real-world development settings [2].

Unlike functional correctness, where failures manifest as test failures or runtime errors, vulnerabilities may remain latent until exploitation. Prior work shows that LLMs can generate vulnerable code, including SQL injection, cross-site scripting (XSS), and insecure deserialization [3]–[6]. Even more concerning, models can produce such code with high confidence, encouraging developers to place false trust in the generated outputs [7], [8]. In practice, developers must quickly evaluate AI-generated suggestions and decide which ones require scrutiny. High-confidence outputs are more likely to be accepted without careful review [5], creating a systemic risk: vulnerable AI-generated code may propagate into production systems through misplaced trust in overconfident models.

Existing benchmarks such as HumanEval [9], MBPP [10], and SWE-Bench [11] primarily evaluate functional correctness using unit-test pass rates and provide little assessment of security properties. Recent security-oriented benchmarks [6], [12] evaluate whether generated code contains vulnerabilities, but do not measure whether models recognize the security risks in their own outputs. That is, they measure whether the code is vulnerable, but not whether models know, in the sense of accurately self-assessing, that it is vulnerable. We use the term knowing to denote probabilistic self-assessment of security correctness: the degree to which a model’s stated confidence in the security of its own output aligns with the output’s actual security status, as determined by executable test outcomes. This notion is distinct from vulnerability reasoning, in which a model detects or classifies known weakness patterns [6], [12], and from self-explanation, in which a model verbalizes its rationale without quantifying correctness likelihood [13].

In machine learning, calibration measures the alignment between a model’s confidence and its empirical accuracy. A well-calibrated model reporting “80% confidence” should be correct 80% of the time. Calibration is therefore critical for trustworthy AI: even accurate models become unreliable when their confidence estimates are systematically misaligned [8]. In code generation, poor calibration manifests as the False Trust problem [14], where models assign high confidence to vulnerable code. Developers who rely on such estimates may deploy insecure code, creating a risk: missed vulnerabilities can cause severe damage (e.g., data breaches or remote code execution), whereas false alarms only incur additional review cost. Consequently, we argue that calibration, not just accuracy, should be a central evaluation criterion for LLMs in security-critical settings.

Despite growing research on vulnerabilities in LLM-generated code [3], [6], [15], no prior work has systematically measured security calibration. Recent work on LLM calibration in software engineering examines functional correctness [8] or general detection rates [16], without considering security. Thus, four research gaps remain. First, existing studies report binary outcomes (secure vs.insecure) without collecting model confidence, making calibration assessment difficult and leaving the relationship between functional calibration (\(p^{\mathrm{func}}\)) and security calibration (\(p^{\mathrm{sec}}\)) unexplored. Second, it is unknown whether calibration patterns observed in self-contained programs generalize to realistic repository-level contexts with cross-file dependencies. Third, automated repair methods rarely leverage model uncertainty to guide vulnerability remediation, and it remains unclear whether calibration signals can drive effective repair. Fourth, mitigation strategies such as execution gating, prompt engineering, and few-shot prompting have not been evaluated for their impact on security calibration.

To address these gaps, we present the first large-scale empirical study of security calibration in LLM-generated code. We analyze whether models accurately estimate the security correctness of their outputs across thousands of samples, multiple models, and both controlled and repository-level development contexts. Our study provides new insights into how confidence signals from AI coding assistants can be interpreted, calibrated, and integrated into secure software development workflows. This work makes the following contributions¹:

• We conduct the first large-scale study of security calibration in LLM models (RQ1). We investigate 36,000 generated code snippets across three models (GPT-4o-mini, Gemini-2.0-Flash, Qwen3-Coder-Next), six temperature settings, and 100 security-critical tasks. We measure Expected Calibration Error (ECE) [17], Brier score [18], False Trust rate, and overconfidence gap. Using these metrics, we compare security and functional confidence estimates to analyze differences between the two calibration signals.

• We quantify the calibration degradation when moving from self-contained programs to repository-level code generation across multiple programming languages (RQ2). Our study reveals consistent calibration penalties driven by cross-file structural complexity.

• We investigate whether a closed-loop repair system using post-hoc calibrated confidence thresholds is suitable for removing vulnerabilities in LLM-generated code (RQ3). Our study shows that, even after applying isotonic and logistic calibration, models rarely revise their outputs for vulnerabilities that require replacing an insecure API call or data sink with a secure alternative.

• We evaluate mitigation strategies for reducing False Trust in secure code generation (RQ4). We study three interventions: execution gating, authorized-breaking repair, and few-shot prompting (with and without chain-of-thought). Our results show that architectural interventions (execution gating) help improve calibration, while prompt-level techniques provide limited benefit under strict harness evaluation.

2 Background and Related Work↩︎

2.1 LLMs for Code Generation↩︎

LLMs are increasingly integrated into software development workflows through tools like GitHub Copilot. These systems translate natural language into code, generating boilerplate, API calls, and configuration artifacts [9], [10], [19], [20]. Since LLMs are trained on large corpora of open-source code, they learn both secure and insecure programming practices [3]. Prior works show that LLM-generated code may contain vulnerabilities (e.g., SQL injection, Cross-Site Scripting, unsafe deserialization, weak cryptographic uses, etc.) [3], [4], [21], [22]. These flaws often occur in syntactically valid and functionally correct programs, making them difficult to detect solely through functional testing.

Security correctness differs from functional correctness. Functional correctness measures whether a program satisfies its intended input–output behavior and is commonly evaluated using benchmarks such as HumanEval and MBPP [9], [10]. Security correctness concerns properties such as confidentiality, integrity, and availability under adversarial conditions. Vulnerabilities typically arise from unsafe data flows, using insecure APIs, or missing authorization checks [23], [24].

To evaluate these risks, benchmarks and frameworks have been introduced, including scenario-based vulnerability studies [3], [25], security-oriented prompt suites [22], [26], and automated pipelines combining prompts with static analysis [6], [15]. More recent frameworks (e.g., AICGSecEval [12]) extend evaluation to multi-language and repository-level settings. However, these benchmarks typically report only binary outcomes (secure/insecure) and do not analyze the LLM’s confidence in its outputs. Our work studies whether models are calibrated about the security of the code they generate.

2.2 Calibration and Confidence Estimation↩︎

Calibration measures the alignment between a model’s predicted confidence and empirical accuracy. A perfectly calibrated model satisfies \(P(Y' = Y \mid P(Y' \mid X) = p) = p\), meaning that predictions made with confidence \(p\) are correct approximately \(p\) of the time [27]. Modern neural models, however, are often overconfident, assigning high probability to incorrect predictions [28]. In human–AI interaction, such miscalibration can produce the False Trust phenomenon: users trust incorrect predictions because the system expresses high confidence [14]. In code generation, this occurs when a model confidently produces insecure code, increasing the likelihood that developers accept vulnerable outputs.

Calibration is typically measured using metrics such as the Expected Calibration Error (ECE) and the Brier score [18], which are widely adopted in the literature [17], [29]. Post-hoc calibration techniques, including temperature scaling [28], Platt scaling [30], and isotonic regression [31], can partially correct miscalibration without retraining the model. In AI-assisted development, reliable calibration is important because developers may implicitly rely on model confidence when deciding whether to trust generated code.

Calibration requires estimating the LLM’s confidence score, which can be derived from several mechanisms. One approach aggregates token-level log probabilities to approximate sequence likelihood [32]. Another uses verbalized confidence, prompting models to explicitly report their certainty [32], [33], often via specialized prompting or fine-tuning strategies [34], [35]. Confidence can also be inferred through sampling-based consistency, where agreement across multiple outputs indicates reliability, or through reflection and self-consistency prompting that evaluates reasoning stability across sampled paths [36], [37]. Each method has trade-offs: token-based approaches require API access, sampling methods are computationally expensive, and verbalized confidence may reflect linguistic bias rather than true uncertainty. Importantly, most prior work evaluates confidence on general tasks or functional correctness rather than the security properties of generated code [8], [27], [32], [38].

2.3 Calibration for LLMs and Software Engineering↩︎

Prior research shows that LLM confidence is often poorly aligned with empirical correctness across tasks such as question answering and reasoning [27], [38]. While post-hoc methods can partially improve calibration, misalignment remains common. Some work suggests that LLMs can produce calibrated self-assessments with appropriate prompting [32], while other approaches estimate correctness probabilities through specification-based verification [39]. However, these studies focus primarily on natural language tasks or general program synthesis rather than security-critical code generation.

Within software engineering, calibration remains underexplored. Recent studies show that confidence estimates derived from token probabilities or self-evaluation often misalign with functional correctness in code generation tasks [8], [40]. These analyses focus exclusively on functionality and do not examine calibration with respect to security properties. Thus, we fill this gap by conducting the first study of security calibration in LLM-generated code, measuring the alignment between model confidence and vulnerability outcomes across both self-contained and repository-level software contexts.

3 Research Questions↩︎

• How well-calibrated are LLMs for secure code generation in self-contained programs?

We measure baseline calibration using self-contained Python programs, in which the full specification, including inputs, expected outputs, and security requirements, is provided in a single prompt with no external files or dependencies. We adopt the SALLM benchmark [6] for this setting. Calibration is assessed via Expected Calibration Error (ECE), Brier score, the overconfidence gap, and the False Trust rate; lower values indicate better calibration for all four metrics (with Gap\(\,\approx 0\) indicating neither over- nor underconfidence). We also analyze calibration across vulnerability types and compare security calibration (\(p^{\mathrm{sec}}\)) with functional calibration (\(p^{\mathrm{func}}\)) to determine whether failures arise from general uncertainty or security-specific miscalibration.

• How does repository-level context affect security, functionality, and calibration?

Real-world software development occurs within multi-file repositories that introduce dependencies such as imports, configuration, and cross-module interactions. This RQ evaluates how repository-level complexity influences security outcomes, functional correctness, and calibration across multiple languages, and whether patterns observed in self-contained programs generalize to realistic development settings.

• Can calibration-guided repair improve vulnerability reduction and calibration accuracy?

We test whether model confidence can guide automated repair by flagging low-confidence samples and applying an iterative repair loop. To explain repair outcomes, we stratify repair outcomes by vulnerability types based on the samples in our benchmark: Sink Replacement (SR) vulnerabilities that require replacing insecure APIs (e.g.,  pickle to json), and Constraint Enforcement (CE) vulnerabilities that can be fixed by adding validation logic (e.g.,  input sanitization or bounds checking).

• Can architectural gating and typed prompting mitigate the False Trust paradox?

We evaluate different mitigation strategies to reduce overconfident, vulnerable outputs. Specifically, we test (1) execution gating to filter functionally incorrect code before security analysis, (2) authorization prompts that permit structural code changes, and (3) secure exemplar prompting to reinforce safe programming patterns.

4 Methodology↩︎

To answer our RQs, we conduct a multi-stage empirical evaluation of LLM calibration in secure code generation (Fig. 1). It combines controlled self-contained benchmarks (SALLM [6]) with realistic repository-level tasks (AICGSecEval [12]) to measure calibration behavior across different contexts, models, and programming languages.

4.1 RQ1: Calibration in Self-contained Generation↩︎

4.1.1 Dataset↩︎

To examine LLMs’ calibration, we first needed a reliable dataset of prompts that we could use as input to LLMs. To this end, we retrieved the prompts and unit tests from the SALLM [6] benchmark for our work. It contains 100 security-critical Python code-generation tasks, each designed to elicit a specific vulnerability category (CWE). Each task includes a natural language prompt, a reference insecure implementation, a set of functional unit tests, and a set of dedicated vulnerability tests that passes on vulnerable code and fails on secure code. This benchmark covers 45 distinct CWE categories and enables precise, automated evaluation of both functional correctness and security.

4.1.2 Models & Parameters↩︎

Each of the 100 prompts from the SALLM benchmark is provided as input to three models that represent the current landscape of frontier and open-weight architectures [41] GPT-4o-mini [42], an optimized OpenAI model designed for low-latency reasoning with a 128k-token context window; Gemini-2.0-Flash [43], a multimodal Google DeepMind model supporting real-time and agentic workflows with up to a 1M-token context window; and Qwen3-Coder-Next [44], a state-of-the-art open-weight model from Alibaba Qwen pretrained on high-quality source code and technical documentation (we use the 80B-parameter variant via Hugging Face inference providers). Following best practices in LLM-for-SE research [45], we evaluate each model using 6 temperature settings (\({0.0, 0.2, 0.4, 0.6, 0.8, 1.0}\)) to characterize how output stochasticity interacts with calibration. At each temperature, we generate 20 independent samples per task for the SALLM dataset.

4.1.3 Confidence score estimation↩︎

We estimate each LLM’s confidence score by using four different methods:

(i) Token-probability aggregation

For the GPT and Gemini models, we use their APIs, which provide direct access to token-level logit probabilities. In contrast, the Hugging Face inference provider does not expose raw logits or logit access, so the confidence estimation for the Qwen3-Coder-Next model uses indirect measures, i.e., (ii)–(iv) described below.

(ii) Verbalized confidence

After generating code, the model is prompted in a separate API call to estimate the probability that the code is secure or functionally correct, returning JSON of the form: {"confidence": <float 0--1>, "reasoning": "<text>"}. We elicit two independent scores: functional confidence (\(p^{\mathrm{func}}\)) and security confidence (\(p^{\mathrm{sec}}\)). The security prompt is conditioned on functional correctness (“Assuming the code passes functional tests”) to reduce cross-signal anchoring. This method is model-agnostic and works across all providers without requiring access to token-level logit probabilities.

(iii) Sampling-based consistency

Using the 20 samples generated per task, ensemble confidence is estimated as the fraction that passes the security (or functional) verifier: \(\hat{p}_{\mathrm{ens}} = \frac{1}{20}\sum_{j=1}^{20} y_j^{\mathrm{sec}}\). This estimate is perfectly calibrated at the task level but provides no per-sample discrimination, as all samples from the same task share the same confidence value.

(iv) Self-consistency prompting

We apply a chain-of-thought (CoT) variant [36] in which the LLM first enumerates potential vulnerability patterns before reporting a numeric confidence estimate in the same JSON format as (ii). This allows us to examine whether explicit reasoning reduces overconfidence or reinforces heuristic-based confidence estimates.

4.1.4 Execution and Verification↩︎

All LLM-generated code executes in isolated Docker containers to mitigate unsafe behavior from potentially malicious generated code. The harness provides Python 3.10, common libraries (e.g., Flask, requests, lxml, etc.), strict network isolation (--network none), and resource limits (2 CPUs, 512MB RAM, 10 sec. timeout).

For each generated sample, we execute both functional and vulnerability tests. Functional unit tests produce a label \(y^{\mathrm{func}} \in \{0,1\}\) indicating whether all tests pass, while vulnerability tests produce \(y^{\mathrm{vuln}} \in \{0,1\}\) indicating whether at least one exploit test succeeds (i.e., the code is exploitable). The final security label is then defined as \(y^{\mathrm{sec}} = y^{\mathrm{func}} \wedge \neg y^{\mathrm{vuln}}\), meaning the code is considered secure only if it passes all functional tests and no vulnerability test succeeds. This definition ensures that a sample is labeled secure if and only if it (a) passes all functional unit tests and (b) fails the vulnerability exploit test, meaning no known attack vector succeeds. Code that is functionally broken cannot be labeled secure under this definition, which requires that deployed code be both correct and secure.

4.1.5 Calibration Metrics↩︎

After estimating confidence and executing tests, we compute four calibration metrics.

4.1.5.1 Expected Calibration Error (ECE)

It is calculated as the average absolute difference between a model’s expressed confidence and its empirical accuracy across \(B\) equal-width confidence bins [17]: \(\mathrm{ECE}= \sum_{i=1}^{B} \frac{|B_i|}{N} \left|\mathrm{acc}(B_i) - \mathrm{conf}(B_i)\right|\), where \(B_i\) is the set of samples whose confidence falls in bin \(i\), \(\mathrm{acc}(B_i)\) is the empirical accuracy in that bin, and \(\mathrm{conf}(B_i)\) is the average confidence. We use \(B=10\) following standard practice [28]. ECE scores range from \(0\) (perfect calibration) to \(1\) (complete miscalibration). A perfectly calibrated model has ECE \(=0\); a model that always predicts \(1.0\) confidence but is correct only \(20\%\) of the time has ECE \(=0.8\).

4.1.5.2 Brier Score

It is the mean squared error between confidence and actual binary outcome, providing a scoring rule that rewards accurate probabilistic predictions [18], i.e.,  \(\mathrm{Brier}= \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2\). A score of 0 indicates perfect predictions; a score of 1 indicates maximally wrong predictions. Unlike ECE, the Brier score captures both calibration and sharpness (discriminability), making it a holistic measure of probabilistic prediction quality.

4.1.5.3 Overconfidence Gap

The signed mean difference between confidence and outcome isolates the direction of miscalibration [28]: \(\mathrm{Gap} = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)\). Positive values indicate systematic overconfidence (confidence exceeds accuracy); negative values indicate underconfidence. When Gap equals ECE, all miscalibration is in the same direction (pure overconfidence).

4.1.5.4 False Trust (FT) Rate

It quantifies the prevalence of the most operationally dangerous miscalibration pattern, i.e.,  high-confidence predictions that are wrong: \(\mathrm{FT}(\tau) = \frac{|\{i : p_i \ge \tau \wedge y_i = 0\}|}{N}\). We use \(\tau = 0.8\) as the default threshold, representing the regime where developers are most likely to trust and deploy code without further review. A developer who filters on \(p \ge 0.8\) implicitly assumes that such code is correct roughly \(80\%\) of the time; our results show the actual security accuracy in that regime is far lower.

4.2 RQ2: Calibration in Repository-level Generation↩︎

4.2.1 Dataset↩︎

While RQ1 uses the self-contained prompts from the SALLM dataset, in this RQ, we examine calibration in a broader repository-level context. To do so, we retrieve 125 prompts from the AICGSecEval [12] dataset, which provides a qualitatively different evaluation context: real-world, multi-file repository settings across 4 programming languages (C, Java, PHP, and Python) and 29 CWEs. Unlike SALLM’s self-contained Python functions, which generate a single function in isolation, AICGSecEval tasks require the model to generate code that is secure within the context of an existing project, respecting external library APIs, cross-file data flows, build system conventions, and language-specific idioms. The 125 prompts span 29 distinct CWEs drawn from diverse project structures, ranging from single-file utilities to multi-module applications with build configurations and third-party dependencies. AICGSecEval was the most recent and comprehensive repository-level security benchmark available at the time of the experiment (February, 2026).

4.2.2 Experiment Setup↩︎

Each AICGSecEval instance consists of a real GitHub CVE description paired with a repository snapshot containing the vulnerable code. The model receives (i) the CVE description, (ii) up to 50k tokens of BM25-retrieved repository context (relevant files and imports), and (iii) an instruction to produce a git patch (unified diff) that fixes the vulnerability. This setup reflects a realistic developer workflow in which security fixes must be written within an existing project.

4.2.2.1 Code generation

We apply the same six temperature settings (\(\{0.0,0.2,0.4,0.6,0.8,1.0\}\)) as in RQ1 to generate patches. Each patch is applied to the repository and evaluated inside a language-specific Docker environment.

4.2.2.2 Verification

Each patched repository is evaluated in three stages: (1) Build: the repository compiles successfully; (2) Functional tests: the project test suite passes (\(y^{\mathrm{func}}=1\)); (3) PoC exploit: the benchmark exploit is executed to verify whether the vulnerability remains. A patch is considered secure if \(y^{\mathrm{sec}} = y^{\mathrm{func}} \wedge \neg y^{\mathrm{vuln}}\).

4.2.2.3 Confidence elicitation

After generation, a separate API call asks the model to report its confidence that the patch is secure using the same JSON format as in RQ1. Calibration metrics (ECE, Brier score, and False Trust) are then computed for each model and compared with the SALLM results to assess calibration degradation at the repository level.

4.3 RQ3: Calibration-Guided Repair↩︎

In RQ3, we empirically study whether calibration signals can be used constructively to drive vulnerability repair. All experiments in this section use the self-contained SALLM tasks [6]. This controlled setting ensures that the full program state is visible within a single prompt, allowing the causal impact of calibration-guided repair to be measured without confounding factors such as cross-file dependencies, incomplete repository context, or environment configuration effects.

Rather than applying vulnerability repair indiscriminately, we use the model’s own security confidence as a filter: only samples for which the model is uncertain are candidates for repair. This design reflects the intuition that low confidence may signal an underlying problem, even if absolute confidence values are miscalibrated.

To establish reliable calibration signals, we first train post-hoc calibrators on a task-level train split (50 SALLM tasks with \(seed=42\)): Isotonic Regression [46] for functional confidence and a Logistic Regression pipeline for security confidence, both fitted on the logit of \(p_{\mathrm{true}}\) augmented with 16 static sink-pattern counts (e.g.,  occurrences of pickle, eval, os.system, mined from vulnerable code patterns in the SALLM dataset). Task-level splitting prevents data leakage, ensuring the calibrator never sees confidence scores from the same task during training and evaluation. The remaining 50 tasks (6,000 samples = \(50\) Tasks \(\times\) \(20\) Samples \(\times\) \(6\) Temperatures) form the held-out test split on which all repair results are reported.

The repair loop consists of three stages (S1–S3).

4.3.0.1 S1: Filtering

Samples with calibrated \(p^{\mathrm{sec}}< 0.5\) are flagged for repair. This threshold balances coverage (flagging enough samples to measure repair success) and precision (avoiding unnecessary repair attempts on samples the model is moderately confident about).

4.3.0.2 S2: Repair

Each flagged sample is resubmitted to the model along with its own insecure output from RQ1 and the original task description, with a prompt instructing it to fix vulnerabilities while preserving functionality (no known fix included in the prompt). This way, the model conditions on its own vulnerable code and is steered toward repair. The loop iterates up to three times per sample; if a repair attempt yields a calibrated \(p^{\mathrm{sec}}\ge 0.5\), iteration stops early.

4.3.0.3 S3: Re-verification

All repaired samples are re-executed under the same Docker harness, measuring functional accuracy and security status. This provides a ground-truth evaluation of repair quality independent of the model’s self-assessment.

4.3.0.4 Measurement and Reporting

After the repair loop terminates, all repaired samples are re-evaluated under the same Docker harness as the baseline. We report several metrics relative to the pre-repair baseline. First, functional accuracy measures the proportion of samples that pass the benchmark test suite after repair. Second, the secure rate measures the fraction of samples labeled secure (\(y^{\mathrm{sec}}=1\)), reported both overall and restricted to functionally correct programs. Third, we evaluate calibration quality using Expected Calibration Error (ECE) and the False Trust rate (confidence \(\ge 0.8\) on vulnerable outputs) computed from the model’s post-repair confidence estimates. Fourth, the repair success rate measures the fraction of flagged samples (\(p^{\mathrm{sec}}< 0.5\)) for which the repaired program becomes secure. Finally, functional breakage captures the fraction of programs that were originally functional but fail the test suite after repair.

To diagnose structural repair limitations, we further stratify repair outcomes by vulnerability type using two repair categories derived from the SALLM dataset: Sink-Replacement (SR) and Constraint-Enforcement (CE). These classes capture the dominant vulnerability patterns in SALLM and represent two distinct repair strategies: replacing an unsafe API or sink with a secure alternative (SR), and enforcing input validation or security constraints without altering the underlying sink (CE). For each class, we report repair success rates and functional breakage, allowing us to determine whether certain vulnerability patterns are harder for LLMs to repair.

4.4 RQ4: Mitigation Experiments↩︎

RQ4 investigates whether the False Trust phenomenon can be mitigated through architectural or prompting interventions applied at inference time, without retraining or fine-tuning the underlying models. These mitigation experiments are also conducted on the self-contained SALLM tasks [6]. Using single-file programs ensures that security fixes can be evaluated in a controlled environment where functional tests and vulnerability checks are fully observable, enabling precise attribution of improvements to the intervention itself. We design four experiments, each targeting a different hypothesis about potential sources of False Trust.

4.4.0.1 Experiment 1: Execution Gating

This experiment evaluates whether filtering out functionally incorrect outputs affects security calibration measurements. Execution gating removes code samples that fail functional tests before eliciting security confidence from the model. Calibration metrics are recomputed on the subset of samples that pass functional tests and compared with the ungated baseline. This analysis is conducted post-hoc on the existing RQ1 generation outputs (12,000 samples per model). We apply the functional pass filter (\(y^{\mathrm{func}} = 1\)) to the already-computed verbalized confidence scores and ground-truth labels, then recompute ECE, Brier score, and False Trust on the gated subset.

4.4.0.2 Experiment 2: Authorized-Breaking Repair

This experiment evaluates whether allowing the model to violate backward compatibility constraints improves repair of structurally rigid Sink-Replacement (SR) vulnerabilities. Each SR-class sample is submitted to a single-pass repair prompt (no calibration filtering) given below, where the model is explicitly authorized to replace insecure APIs with safer alternatives:

“The following Python code is insecure because it uses a dangerous function or library (e.g.,  pickle, eval, marshal, os.system). Your task is to make the code secure by replacing the insecure primitive with a safe alternative.

AUTHORIZED BREAKING CHANGES: You may replace the insecure API or serialization format (e.g.,  pickle \(\rightarrow\) json) if necessary to eliminate the vulnerability.

Rules: remove the insecure function, preserve the function signature, and output the complete fixed Python module.”

The repaired code is extracted and re-evaluated under the same Docker harness used in RQ3 to measure vulnerability removal, functional breakage, and strict success (secure and functional). Results are compared against the baseline RQ3 repair pipeline on the same 1,560 SR-class samples.

4.4.0.3 Experiment 3: Few-Shot Prompting

This experiment evaluates whether providing secure coding examples improves vulnerability repair. For each sample, the prompt includes two handcrafted vulnerable–secure code pairs corresponding to the target CWE category. The examples illustrate minimal secure patterns (e.g.,  replacing os.system with subprocess.run, escaping HTML using html.escape, or replacing pickle/eval with safer alternatives). Example sets are selected at runtime based on the sample’s CWE tag, and no examples are drawn from the test split. These examples aim to guide the LLM toward safer implementation patterns.

4.4.0.4 Experiment 4: Few-Shot + Chain-of-Thought (CoT)

This experiment extends the few-shot setup by adding explicit reasoning instructions using Chain-of-Thought prompting [37]. Each example now includes a short reasoning section explaining why the vulnerable code is insecure and how the secure version fixes it. The task prompt also instructs the model to (1) identify the insecure pattern, (2) explain the root cause, (3) determine a secure alternative, and (4) generate the corrected implementation. Output format, CWE-based example selection, and Docker re-verification remain identical to Experiment 3.

5 Results↩︎

5.1 RQ1: Calibration in Self-Contained Generation↩︎

Table [tab:rq195combined] reports calibration metrics across six temperature settings for three models and four confidence elicitation methods. Calibration here measures the trustworthiness of the model’s confidence signal, not security accuracy itself; its operational value lies in quantifying False Trust, how often code flagged as high-confidence (\(p^{\mathrm{sec}}\ge 0.8\)) would bypass a confidence-based review gate despite being insecure. Across all models, we observe systematic overconfidence. Using verbalized confidence, GPT-4o-mini and Qwen3-Coder-Next exhibit severe miscalibration (\(\mathrm{ECE}\in [0.456,0.481]\) and \([0.408,0.421]\), respectively), while Gemini-2.0-Flash shows comparatively better calibration (\(\mathrm{ECE}\in [0.247,0.263]\)), though still far above typical well-calibrated thresholds (\(\mathrm{ECE} \le 0.01\)) [28]. The overconfidence (OC) gap closely matches ECE, confirming that miscalibration is unidirectional, meaning models overestimate the probability that generated code is secure.

Calibration quality is insensitive to sampling temperature. ECE has small variations across \(T \in [0,1.0]\), indicating that overconfidence is primarily a property of the models’ internal confidence estimation rather than sampling stochasticity. Security performance also varies across models. GPT-4o-mini produces secure outputs in only \(12.2\)–\(14.9\%\) of cases, Gemini-2.0-Flash slightly improves to \(16.4\)–\(17.3\%\), while Qwen3-Coder-Next performs worst (\(5.4\)–\(6.7\%\)) and exhibits higher False Trust rates (\(36.4\)–\(40.3\%\)).

Table 2 reports 95% bootstrap confidence intervals (\(B = 2{,}000\)) for ECE, Brier score, and False Trust, confirming consistent differences across models. All confidence intervals are non-overlapping across models for ECE and Brier, and Gemini-2.0-Flash is clearly separated from both GPT-4o-mini and Qwen3-Coder-Next on False Trust (17.5% vs.% and 38.3%). The narrow interval widths (typically \(\pm\)​0.02 for ECE and Brier, \(\pm\)​2 percentage points for False Trust) indicate that the reported miscalibration patterns are stable and not driven by sample variance.

We also compare four confidence elicitation methods: verbalized confidence, token probabilities, sampling-based consistency, and self-consistency prompting. Token-probability estimation performs poorly (\(\mathrm{ECE}\approx 0.80\)), reflecting the mismatch between next-token likelihood and program-level security outcomes. Sampling-based consistency yields near-zero ECE, but this occurs because confidence is computed at the prompt level, providing no per-sample discrimination. Self-consistency improves calibration for GPT-4o-mini and Qwen3-Coder-Next (reaching \(\mathrm{ECE}\approx 0.07\) at higher temperatures) but requires multiple generations per query, increasing inference cost. Overall, verbalized confidence offers the most practical signal for downstream analysis, as it provides per-sample estimates across models without requiring token-level access or repeated sampling.

5.1.1 Security vs.Functional Calibration↩︎

We next compare calibration when models estimate security correctness versus functional correctness. Table 3 summarizes the corresponding calibration metrics across all temperatures and models.

1pt

Across all models and temperatures, functional calibration is consistently worse than security calibration. For every configuration, \(\Delta\mathrm{ECE}= \mathrm{ECE}^{\mathrm{s}} - \mathrm{ECE}^{\mathrm{f}}\) is negative, indicating that models are more miscalibrated when assessing functional correctness than security correctness.

For GPT-4o-mini, \(\Delta\mathrm{ECE}\) ranges from \(-0.15\) to \(-0.16\). Functional ECE remains consistently high (0.55–0.59), whereas security ECE is lower (0.40–0.43), representing a 25–28% relative reduction. Gemini exhibits an even larger gap (\(\Delta\mathrm{ECE}\in [-0.26,-0.27]\)), with functional ECE between 0.42 and 0.44 compared to security ECE between 0.16 and 0.17. The disparity is most extreme for Qwen3-Coder-Next (\(\Delta\mathrm{ECE}\in [-0.52,-0.53]\)), where functional ECE approaches 0.77 while security ECE remains near 0.25.

Importantly, this pattern holds across all six temperature settings and across all models, ruling out sampling stochasticity as an explanation. The magnitude and consistency of the effect suggest a structural difference between functional and security self-assessment.

5.2 RQ2: Repository-Level Calibration Degradation↩︎

Having established in RQ1 that miscalibration exists for self-contained programs, we examine whether it worsens in realistic multi-file repository contexts. As shown in Table 4, which compares calibration on self-contained tasks from SALLM with repository-level tasks from AICGSecEval, we observe that repository-level generation has higher calibration error across all models. For GPT-4o-mini, average ECE rises from 0.411 on SALLM to 0.697 on repository tasks (\(\Delta\mathrm{ECE}=+0.286\)), while False Trust increases from 16.9% to 83.2% (\(\Delta\mathrm{FT}=+66.3\%\)). Gemini-2.0-Flash exhibits a similar pattern, with ECE increasing from 0.161 to 0.721 (\(\Delta\mathrm{ECE}= +0.560\)) and False Trust rising from 9.6% to 70.1% (\(\Delta\mathrm{FT}=+60.5\%\)).For Qwen3-Coder-Next, ECE increases from 0.241 to 0.734 (\(\Delta\mathrm{ECE}=+0.493\)) while False Trust rises from 9.0% to 73.9% (\(\Delta\mathrm{FT}=+64.9\%\)). Thus, calibration error roughly triples to quadruples when moving from isolated functions to repository-level code.

The degradation appears consistently across programming languages. For GPT-4o-mini, Python and C exhibit the highest repository-level miscalibration (ECE \(=0.773\) and \(0.763\), False Trust \(=89.7\%\) and \(90.7\%\) respectively), while Java and PHP remain slightly lower but still severe (ECE \(=0.658\) and \(0.593\)). Similar patterns appear across Gemini and Qwen, where repository contexts consistently produce ECE values above \(0.68\) and False Trust rates above \(55\%\). These results suggest that vulnerabilities involving cross-file dependencies, such as serialization flows, framework interactions, and memory management invariants, introduce uncertainty that models fail to reflect in their confidence estimates.

5.2.0.1 Decomposing the degradation

To disentangle the sources of repository-level miscalibration, we decompose the ECE increase by conditioning on whether the generated patch successfully builds within the target project. Patches that fail to build cannot be exploit-tested and are labeled insecure regardless of model confidence, conflating build fragility with genuine cross-file complexity. Separating samples by build outcome isolates two effects: the build-fragility contribution, \(\mathrm{ECE}(\text{all}) - \mathrm{ECE}(\text{build\_pass})\), and the cross-file contribution, \(\mathrm{ECE}(\text{build\_pass}) - \mathrm{ECE}\_{\text{SALLM}}\). Table 5 reports this decomposition averaged across all temperatures.

Build fragility dominates GPT-4o-mini’s degradation, accounting for 82% of the ECE increase. For Gemini and Qwen, patch-merge failures are absorbed into the build-fragility term, making their cross-file estimates upper bounds. However, genuine cross-file complexity remains evident: even among patches that build successfully, ECE stays well above SALLM levels (e.g.,  Gemini: 0.524 vs., a 27 percentage-point gap).

5.3 RQ3: Calibration-Guided Repair↩︎

Table 6 summarizes the results of the calibration-guided repair pipeline on the 6,000-sample held-out test split for each model. For GPT-4o-mini, calibration-guided repair reduces functional accuracy from 29.49% to 12.35%. While the secure rate among functionally correct samples increases (23.32% \(\rightarrow\) 30.16%), the overall secure rate declines from 6.88% to 3.73% due to functional breakage (69.09%). Calibration quality also deteriorates (ECE: 0.469 \(\rightarrow\) 0.569). The false trust rate decreases slightly (95.79% \(\rightarrow\) 89.68%), indicating a limited benefit from filtering overconfident, vulnerable outputs.

Gemini-2.0-Flash exhibits a smaller functional degradation (59.00% \(\rightarrow\) 54.72%) and a modest improvement in the secure rate among functionally correct samples (26.23% \(\rightarrow\) 28.28%). However, the overall secure rate remains unchanged (15.47%), while calibration worsens (ECE: 0.195 \(\rightarrow\) 0.487) and the false trust rate increases from 61.83% to 82.88%. Notably, the pipeline achieves no repair success on vulnerable samples. Similarly, the repair pipeline produces poor outcomes for Qwen3-Coder-Next. Functional accuracy drops from 15.41% to 6.62%, while the overall secure rate decreases from 5.78% to 2.40%. Although calibration deteriorates (ECE: 0.216 \(\rightarrow\) 0.740), false trust remains high (100% \(\rightarrow\) 97.63%).

1.2pt

We find that repair success rates are low (0%–2.39%), while functional breakage often exceeds 60%. The stratified results signal a structural barrier: sink-replacement (SR) vulnerabilities are never successfully repaired across any model (0% repair success). Instead of replacing insecure APIs, models consistently preserve the original call and introduce superficial validation logic. Constraint-enforcement (CE) vulnerabilities show slightly higher repairability (1.4%–3.2%), but these improvements are offset by high rates of functional breakage. These results reveal that the bottleneck lies in the repair mechanism, not in the calibration signal itself. Calibration-guided filtering correctly identifies high-risk generations, but current models cannot perform the structural transformations needed to eliminate the flagged vulnerabilities. For SR vulnerabilities, the rigidity barrier, models’ reluctance to replace insecure APIs, produces 0% repair success regardless of filtering quality. For CE vulnerabilities, cascading functional breakage outweighs the modest security gains. Calibration should therefore be understood as a risk-stratification signal that prioritizes samples for human review, not as a correctness amplifier that enables reliable automated repair.

5.4 RQ4: Mitigating False Trust↩︎

5.4.1 Experiment 1: Execution Gating↩︎

Table 7 shows ECE before and after functional gating for all three models, averaged across all six temperatures. All LLMs show ECE reduction through functional gating, confirming that the execution brittleness mechanism is universal. Gemini achieves the largest relative reduction (\(-39.3\%\)) at a lower rejection cost (45.8%), owing to its higher baseline functional accuracy. Qwen discards 86% of samples, consistent with its very low functional accuracy (6–7% of all samples are functional), and achieves only a 30.8% ECE reduction; its absolute ECE remains high (0.286) even after gating, because functional uncertainty is not the sole source of its miscalibration. This intervention requires no model changes, retraining, or additional API calls, meaning simply execute unit tests before eliciting security confidence. The cost is a 46–86% discard rate, acceptable in high-stakes contexts but potentially prohibitive in high-throughput settings. The consistency of the effect across all models further supports the execution brittleness hypothesis discussed in Section ¿sec:sec:exec95brittleness?.

5.4.2 Experiment 2: Authorized-Breaking Repair↩︎

Table 8 shows the cross-model results for authorized-breaking repair on SR-class vulnerable samples, comparing naive repair (RQ3 baseline) with the authorization-enabled strategy. For GPT-4o-mini, explicit authorization raises vulnerability removal from 0% to 35.35%, but functional breakage remains high (55.02%), reflecting the cost of replacing pickle serialization with the safer json format. Despite this trade-off, the model achieves a small 4.62% strict repair success rate. Calibration degrades under this intervention (ECE increases from 0.184 to 0.419), but False Trust is eliminated (0.13% \(\rightarrow\) 0.00%), meaning the model no longer expresses high confidence in vulnerable outputs. For Gemini-2.0-Flash, vulnerability removal is aggressive at 42.01%, but 29.26% of functional samples break, yielding 0% strict repair success. Although the vulnerability has been removed, the model often makes heavy structural rewrites that alter function names, imports, and interfaces, so the code no longer runs in the original harness. Calibration degrades (ECE 0.125 \(\rightarrow\) 0.401), though False Trust stays at 0%. Qwen3-Coder-Next shows almost no vulnerability removal (0.38%) even with explicit authorization, but breaks 100% of functional samples. This suggests that the model struggles to map the authorization signal to the required API replacement in this domain. Its calibration collapses (ECE: 0.088 \(\rightarrow\) 0.666), and False Trust rises to 69.23%, showing overconfidence in insecure outputs after repair attempts.

1pt

5.4.3 Experiments 3 & 4: Few-Shot Prompting and CoT↩︎

Table 9 reports results for pattern-conditioned few-shot prompting and few-shot with Chain-of-Thought reasoning, evaluated on 1,560 samples. Across all models, few-shot prompting removes 61–65% of vulnerabilities when functional correctness is ignored, indicating that the models learn the correct security pattern from the examples. However, strict secure rates remain very low because the repaired code frequently fails structural integration with the evaluation harness. Gemini has the highest strict secure rate (3.27%) with 6.79% functional accuracy, while Qwen reaches 1.22%. GPT-4o-mini achieves only 0.38%, mainly due to minor output-format mismatches. For GPT, the main failure is a trailing newline mismatch, where the harness expects b’Greetings!\n’ but the model outputs b’Greetings!’. CoT prompting degrades outcomes. Strict secure rates fall to 0% for all models, and ECE increases. Although vulnerability removal remains similar (around 61–62%), explicit reasoning tends to produce framework-specific outputs that conflict with strict harness requirements.

These results highlight a limitation of automated security benchmarks: strict evaluation cannot distinguish between true security failures and structural integration failures. Few-shot examples drawn from one framework may conflict with the target task’s framework, causing import conflicts or incompatible APIs. In such settings, vulnerability removal rate better reflects actual security gains, whereas the strict secure rate captures deployability, the ability of repaired code to integrate with the target harness and pass all functional checks.

1pt

6 Discussion↩︎

Execution Brittleness

The calibration patterns observed in RQ1 can be partly explained by execution brittleness. Functional correctness relies on runtime factors hidden from the model during generation, e.g., missing imports, framework initialization, environment-specific configuration, and test-harness assumptions. As a result, generated code may appear syntactically plausible, leading the model to express high confidence, yet fail at execution for hidden environmental constraints. In contrast, security properties are identifiable through static code patterns. Vulnerabilities such as SQL injection, unsafe deserialization (e.g.,  pickle), or insecure randomness are visible from code structure alone. This distinction suggests that functional calibration is more sensitive to environmental uncertainty, whereas security calibration depends more directly on the model’s knowledge of vulnerability patterns.

Implications for Practitioners

Our results show that models’ self-reported confidence should not be treated as a guarantee of security: across all models, verbalized security confidence is systematically inflated, and even high-confidence predictions (\(p_{\sec}\ge 0.8\)) often have much lower empirical security rates. For instance, at \(p^{\mathrm{sec}}\ge 0.8\), GPT-4o-mini’s outputs are insecure about 33% of the time, meaning a confidence-based review gate would often pass vulnerable code. Moreover, the marginal RQ3 gains show that correcting miscalibration alone does not improve security when the underlying repair capability is the binding constraint; calibration flags risk but cannot replace the structural transformations needed to remove vulnerabilities. Confidence is, therefore, best used as a relative triage signal to prioritize human review rather than deployment decisions. Functional testing should also precede any security self-assessment. We observe that models are more miscalibrated for functional correctness than for security correctness, consistent with the execution brittleness mechanism. Applying functional gating before security evaluation reduces calibration errors, implying that security assessment should occur only after the generated code passes the execution harness. Our results also show that functional and security confidence should be elicited separately. Combining both judgments into a single query (e.g.,  “Is this code correct and secure?”) produces a confidence estimate contaminated by execution uncertainty. Tools and pipelines should therefore collect \(p^{\mathrm{func}}\) and \(p^{\mathrm{sec}}\) independently and treat them as distinct signals. Calibration-guided repair should also be applied cautiously: iterative repair loops are fragile and highly model-dependent, sometimes degrading functionality while giving only small security gains. Finally, mitigation strategies should consider the structural nature of vulnerabilities. The SR/CE highlights two repair regimes: CE vulnerabilities can often be fixed through validation or enforcement logic, whereas SR vulnerabilities typically require API-level migration and may introduce interface or test incompatibilities. In practice, SR fixes represent higher-risk changes and frequently require human oversight.

Implications for Researchers

Our results have several implications for future research. First, security calibration should become a standard benchmark output. Existing benchmarks typically report secure rate and functional accuracy but omit model confidence, despite its influence on developer trust and its potential to create many high-confidence failures (False Trust). Benchmarks should therefore report calibration metrics such as Expected Calibration Error (ECE), Brier score, False Trust rate, and reliability diagrams alongside traditional performance measures. Repair evaluations should also be stratified by vulnerability structure. Aggregate secure rates can obscure severe failures within specific classes, whereas our SR/CE taxonomy reveals distinct repairability regimes: CE vulnerabilities show moderate success, whereas SR vulnerabilities encounter a rigidity barrier under standard prompting.

More broadly, our findings challenge the common assumption that “security is harder” for LLMs. We observe that models are more miscalibrated about functional correctness than about security correctness, even though absolute security accuracy remains low (Table [tab:rq195combined] and 3). This suggests that miscalibration is driven mainly by hidden runtime constraints rather than weak security knowledge. Future work should therefore explicitly model different uncertainty sources, especially the distinction between static code cues and hidden execution context. Finally, repository-level evaluation should be the default generalization test for secure code generation systems. Calibration degrades substantially when moving from self-contained tasks to repository contexts, increasing False Trust and suggesting that self-contained benchmarks may understate deployment risk. Future studies should therefore prioritize multi-file, dependency-aware evaluation settings and explicitly report contextual complexity factors.

7 Threats to Validity↩︎

Internal validity

To reduce confounds threatening this work’s internal validity, all LLMs were evaluated under the same protocol, including prompts, temperature settings, and an isolated Docker execution harness. Mitigation experiments were conducted on self-contained programs where the entire program state is visible in a single prompt, enabling precise measurement of repair outcomes and calibration shifts without cross-file dependencies interference. Moreover, security outcomes were evaluated on two independent benchmarks with different task structures and vulnerability distributions, reducing the risk that results depend on a single dataset.

Construct validity

Calibration results depend on how model confidence is measured. To mitigate measurement bias, we evaluated multiple confidence estimators, including verbalized confidence, token probabilities, sampling-based estimation, and self-consistency signals. The persistence of miscalibration patterns across these methods suggests that our findings do not arise from a particular confidence elicitation technique.

External validity

A potential threat concerns the generalizability of our findings. Our study evaluates three models from independent ecosystems (GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next), all tested under an identical evaluation protocol. Four structural phenomena: systematic overconfidence, the security-vs-functional calibration asymmetry (\(\Delta\mathrm{ECE}< 0\)), repository-level degradation, and the SR rigidity barrier, appear consistently across all three independently trained model families and six temperature settings, suggesting they reflect properties of current LLM training rather than vendor-specific artifacts. However, absolute values such as specific False Trust rates, per-language degradation magnitudes, and ECE scores are model- and version-specific and may shift with future releases. We also combine two evaluation settings: controlled, self-contained programs (SALLM) and repository-level benchmarks (AICGSecEval) that capture multi-file development contexts across multiple programming languages. This two-tier design improves ecological validity by verifying whether patterns observed in controlled tasks persist in more realistic software environments. Finally, experiments were run in February 2026 on current model versions. Although absolute calibration values will evolve as models update, the structural phenomena identified here are likely to persist across future iterations, as they arise from fundamental aspects of autoregressive training and confidence elicitation rather than from specific model weights.

8 Conclusion↩︎

This paper presents the first empirical study of security calibration in LLM-generated code across both self-contained and repository-level settings. Across multiple models, temperatures, and programming languages, we observe systematic overconfidence and high False Trust rates, which increase substantially in repository contexts. Our results show that functional calibration is consistently worse than security calibration. We also find that calibration-guided repair is unreliable: SR-class vulnerabilities face a rigidity barrier with near-zero repair success, while CE-class fixes yield only modest gains and often introduce functional regressions. Among mitigation strategies, architectural interventions, particularly functional execution gating, provide the most consistent improvements, whereas prompt-level techniques show limited effectiveness under strict evaluation.

References↩︎