1 Introduction↩︎

Large language models (LLMs) have demonstrated strong capabilities across diverse software engineering (SE) tasks, ranging from code generation [1], [2] to automated program repair (APR) [3]–[5]. In code generation, models are presented with a natural language problem statement and evaluated on whether the generated program is functionally correct. In the case of APR, they are evaluated on whether the generated patch for a given buggy program restores correctness without introducing regressions. Despite their apparent differences in problem formulation, both settings share the same success measurement criterion: a binary, instance-level pass/fail outcome determined by the associated test suite.

While a good first-approximation, such binary measures are sparse and provide a narrow view of overall performance. A model is considered successful only if its generated solution passes all tests in the associated test suite. In practice, however, test suites are often weak and rarely exhaustive [6], [7]. Thus, a model-generated solution that fixes the reported failure but introduces subtle regressions may still be counted as correct, leading to an inflated assessment of model performance [8]. Even when a model is unsuccessful, pass/fail outcomes are uninformative: a solution that fails one edge case is treated the same as one that fails all tests, nor does a partially correct solution receive any credit for progress.

The limitations of binary measures are further exposed in workflows where models operate in iterative refinement loops, progressively improving upon an initial solution through compiler feedback [9], [10], execution information [2], [11], [12], or model-generated diagnostics [10], [13]. We refer to this process of transforming an incorrect or incomplete program into a more correct one through feedback-guided refinement as code improvement. Despite its inherently incremental nature, evaluation still remains limited to the binary outcome of the final solution, ignoring the refinement trajectory altogether. Such an assessment can therefore be misleading: a stronger model may produce an initial patch that fails the current test suite but moves the program substantially closer to the correct behavior, while a weaker model may pass the available tests by simply making a shallow edit. Under a binary pass-rate metric, the weaker model would be deemed superior, though the stronger model demonstrates more effective use of feedback and better generalization. Accordingly, we position that benchmarking a model’s ability to make progress yields a more fine-grained and faithful measure of its real-world utility.

However, measuring progress under code improvement is not straightforward, as model performance is highly sensitive to the informativeness of the feedback provided [14], [15]. For instance, a hint that directly reveals the solution provides little evidence of model reasoning, while one that is too abstract to act upon conflates feedback quality with model failure. Therefore, the feedback provided to guide code improvement must itself be model-dependent and calibrated to model capabilities. In particular, a faithful progress-centric evaluation should consider the trajectory of code improvement: informativeness of feedback provided at each intermediate step, how much corresponding progress is made, and whether improvement remains monotonic across iterations?

To address these challenges, we introduce PAIR-Bench, a progressive and adaptive benchmark for evaluating code improvement. The central mechanism of PAIR-Bench is progressive hinting, a structured feedback protocol that controls hint difficulty along two dimensions. The first, failure-region control determines what the hint is about. It groups hidden failing tests into failure scenarios, where each scenario corresponds to a region of incorrect program behavior, such as a shared edge case, input pattern, or execution path. By prioritizing which failing tests are exposed to the candidate model, PAIR-Bench via its feedback model steers the candidate model’s fixing toward specific regions of incorrect behavior. The benchmark can progressively provide hints that expose different failure regions and measure how the model improves within the hidden failure space. The second, hint-depth control determines how much repair-relevant information is revealed. Inspired by Item Response Theory (IRT), it starts from coarse-grained symptoms to fine-grained implementation-level feedback. Together, both dimensions define failure sub-regions selected relative to the capabilities of the model under evaluation. Such within-instance adaptability ensures that the trajectory of code improvement better reflects model capability rather than the informativeness of an unstructured feedback oracle.

By design, progressive hinting governs how a model is allowed to improve. Accordingly, for each instance, we measure progress at each intermediate step, awarding a partial credit that reflects the proportion and significance of failures resolved rather than treating every step as a binary success or failure. For this, we have metrics measuring the progress where outcomes are continuous rather than binary. PAIR-Bench thus achieves both progressivity and adaptivity, yielding a benchmark that is more faithful to a model’s true capabilities. In this paper, we instantiate code improvement as iterative automated program repair (APR). We use “repair” to refer to each concrete code revision within this benchmark, while “code improvement” denotes the broader evaluation task.

In this paper, we make the following contributions:

(1) PAIR-Bench: A new progressive and adaptive benchmark paradigm. Data and code is available at [16].

(2) A suite of progress-centric metrics that measure different aspects of interactive code improvement.

(3) A controlled interactive feedback protocol. We propose progressive hinting, a structured feedback protocol that jointly controls failure regions and hint depth.

(4) Empirical study. We instantiate PAIR-Bench on interactive code repair and evaluate multiple state-of-the-art LLMs.

2 Motivation and Key Ideas↩︎

2.1 Binary Pass/Fail Outcomes Mask Improvement Ability↩︎

Code-generation and APR benchmarks often evaluate models through final functional correctness: a generated code or patch is executed against a test suite and marked as either passed or failed. However, this strategy compresses model capability into a single binary outcome. In a one-shot evaluation, a weaker model may achieve a higher hidden-test pass rate than a stronger one, making it appear better under a standard pass-rate metric (Fig. 1). However, after receiving targeted feedback, the stronger model may repair the solution while the weaker model still fails. Pass@k only partially addresses this. Although it samples multiple attempts, each attempt is still evaluated as an independent one-shot generation. Thus, one-shot correctness can misrepresent code-improvement ability, which depends not only on how many tests a model initially passes, but also on whether it uses feedback to correct the underlying failure. We call an evaluation progress-centric if it measures not only final correctness, but also the sequence of improvements made during refinement: which failures are fixed, whether already-correct behavior is preserved in a monotonic progress, and how much assistance is needed.

2.2 Interactive Feedback Helps, but Must Be Controlled↩︎

A natural way to address the limitation of one-shot evaluation is to make the benchmark interactive. Instead of judging only the first candidate solution, an interactive benchmark allows the model to receive feedback, and revise and improve its code. This better matches practical LLM-based interactive refinement workflows. Recent benchmarks have started to move in this direction as they provide execution outputs, compiler/test diagnostics, or natural-language feedback (Section 3). Those existing interactive feedback benchmarks do not fully address the requirements of progress-centric evaluation. A faithful evaluation of code improvement should not only ask whether feedback eventually helps, but should also characterize the repair trajectory as defined earlier. In existing settings, there are two uncontrolled factors on the feedback: 1) the underlying failure being targeted or 2) the amount of repair-relevant information being revealed.

This limitation may lead to inconsistent evaluation across runs, even with the same model on the same problem. Different feedback runs may thus produce different repair trajectories and benchmark scores. In one run, the feedback may quickly identify the decisive failure mode or provide a highly direct hint, allowing the model to repair the solution after only a few turns. In another run, the feedback may focus on a local, scattered, less informative symptom, target a different failing behavior, or remain too vague to guide the repair, causing the same model to fail even after many turns (Fig. 1). In such cases, the benchmark is no longer measuring only the model’s code-improvement ability; it is also measuring the variability and informativeness of the feedback itself.

2.3 Key Ideas↩︎

The instability illustrated in Fig. 1 suggests that interactive feedback is necessary but insufficient. To reduce such instability and support faithful progress-centric evaluation, a benchmark must make feedback comparable across runs and models. We argue that feedback should be controlled along two dimensions: the failure region being addressed and the hint depth of the feedback. These two dimensions separate the target of feedback from the strength of feedback.

2.3.1 Failure-region control↩︎

This dimension determines which part of the hidden failure space the feedback targets. Instead of generating feedback from an arbitrary failing test case, we group hidden failing tests into failure scenarios, where each scenario corresponds to a related region of incorrect behavior. To make feedback comparable across turns, each hint is anchored to a selected scenario, whose scope can be approximated by the number of covered failing test cases.

2.3.2 Hint-depth Control↩︎

This dimension determines how much repair-relevant information the hint reveals. Lower-depth hints expose only external evidence of the failure, e.g., symptoms or input patterns, while higher-depth hints provide increasingly direct repair guidance, e.g., missing state, suspicious code regions, conceptual corrections, or implementation directions. We use a six-level hint-depth scale (Table 1), ranging from symptom-only feedback to implementation direction.

2.3.3 Progressive Evaluation↩︎

A code-improvement benchmark should be progressive. A buggy solution may fail for multiple reasons. If interaction repeatedly focuses on the same unresolved failure, the benchmark may observe many repair attempts without measuring broader improvement.

The failure-region control enables progressive evaluation. The benchmark can progressively provide hints that expose different failure regions and measure how the model improves within the hidden failure space. After each hint, the benchmark can evaluate whether the model fixes the targeted scenario, how much progress the model makes after each hint, whether the repair generalizes to other unhinted failures, and whether previously passing tests remain passing. Progressive evaluation thus treats repair as a trajectory over the hidden failure space.

2.3.4 Adaptive Evaluation↩︎

A code-improvement benchmark should also be adaptive. Even when two hints target the same failure region, they may reveal very different amounts of repair-relevant information. A shallow hint may describe only the observed symptom, while a more direct hint may reveal the input pattern, missing state, suspicious code region, conceptual correction, or concrete repair direction.

Hint-depth control enables adaptive evaluation. Our benchmark treats hint depth as a way to calibrate feedback difficulty to the model’s demonstrated code-improvement capability. A model that repairs a failure from shallow evidence should receive less direct guidance, while a model that remains stuck can be given deeper hints that reveal more repair-relevant information. This allows the benchmark to estimate not only whether a model eventually fixes the program, but also how much assistance it requires to make progress.

3 Related Work↩︎

1.7pt

We compare existing code-evaluation benchmarks along six dimensions: execution grounding, multi-turn repair, natural-language feedback, hidden-test protection, progressive evaluation, and adaptive feedback. Execution grounding means that generated code is executed and evaluated against tests or an interactive environment. Multi-turn repair means that the model can revise its solution after receiving feedback. Natural-language feedback means that the benchmark can provide verbal guidance beyond raw execution output. Hidden-test protection captures whether the candidate model is shielded from hidden test cases, expected outputs, and raw hidden execution traces. This is crucial because exposing hidden failures makes repair easier but turns the task into test-specific patching. In realistic settings, the model only sees the problem requirements and public tests, while hidden tests are for evaluation.

Table ¿tbl:tab:benchmark95comparison? summarizes how PAIR-Bench differs from representative benchmarks. Static code-generation and APR benchmarks evaluate if a model can synthesize a correct program from a problem statement or fix a buggy code [1], [17], [19]–[24]. LiveCodeBench includes self-repair and execution-related tasks [2]. SWE-bench [3] and SWE-bench Verified [8] move evaluation toward realistic repo-level repair by asking models to resolve real GitHub issues. Another line of work evaluates models in settings where code execution results are returned to the model. InterCode treats code as actions and execution results as observations [18]. Other work on self-editing, self-debugging, and execution-guided code generation studies whether models can use compiler messages, runtime errors, failed tests, or execution information to revise their programs [11]–[13]. Recently, FeedbackEval studies how LLMs use different feedback types in code repair [10]. Conversational benchmarks, e.g., ConvCodeWorld and When-Benchmarks-Talk, show that model behavior can change substantially depending on the feedback setting [14], [15].

The comparison shows that the missing capability is not simply interaction, but controlled interaction.

4 PAIR-Bench: Progressive and Adaptive Interactive Refinement Benchmark↩︎

Based on the above requirements, PAIR-Bench operationalizes progress-centric evaluation via the pipeline shown in Fig. 2. The benchmark consists of four components:

(1) a scenario constructor that groups currently failing hidden tests into failure regions;

(2) a hint generator that produces feedback at a specified hint depth for a selected scenario;

(3) an adaptive policy that selects the next scenario and hint level based on observed repair behavior; and

(4) a trajectory evaluator measuring targeted repair, broader repair, preservation, hint efficiency, and final convergence.

An instance in our benchmark is defined as \(\langle x_i, c_{i,0}, \mathcal{T}*{i} \rangle\), where \(x_i\) is the problem statement, \(c*{i,0}\) is the buggy solution, and \(\mathcal{T}_{i}\) is the hidden test suite.

Given an instance, the candidate model first produces an initial repaired program from the problem requirements and buggy program (Fig. 2). Hidden tests are hidden from the candidate model but available to the benchmark controller, which evaluates the generated program on the hidden test suite, records the pass rate, and extracts the remaining failing test cases. These failing tests are grouped into a selected failure scenario S, and the controller assigns a hint depth level H.

The feedback model then receives the problem requirement, the correct reference program, the current buggy program, the selected scenario test cases, and the requested hint level. It generates only a controlled hint for that scenario and depth.

If the program still fails at least one hidden test case, PAIR-Bench enters a multi-turn refinement loop. At each turn, the benchmark selects a controlled failure scenario, asks the feedback model to generate a hint at the specified depth, asks the candidate model to revise the code, and re-evaluates the revised program on the hidden test cases. At turn (t), the candidate model receives: \(\langle x_i, c_{t-1}, h_t, \mathcal{H}*{<t} \rangle\) where \(x_i\) is the problem statement, \(c_{t-1}\) is the current code, \(h_t\) is the controlled hint, and \(\mathcal{H}_{<t}\) is the prior interaction history. It is not given hidden tests, oracle traces/outputs, or reference code.

4.1 Scenario Constructor: Scenario-Level Failure Modeling↩︎

Let \(\mathcal{T}_i=\{1,\ldots,n_i\}\) denote the hidden test cases for instance program \(i\). After turn \(t\), let \(F_{i,t}\subseteq \mathcal{T}_i\) be the set of failing test cases and \(P_{i,t}=\mathcal{T}_i\setminus F_{i,t}\) be the passing ones. PAIR-Bench groups the currently failing test cases into failure regions, where each region corresponds to a related behavior that the current program handles incorrectly.

For each failing test case, PAIR-Bench constructs a failure signature using three signals: 1) the reference execution trace, 2) the expected-output shape, and 3) the candidate failure type. The reference trace records the source lines executed by the correct solution, approximating the behavioral path exercised by the test case. The output-shape signature captures coarse output structure, such as a single token, a binary line, a grid, or a multi-line output. The candidate failure type distinguishes different failure types including timeouts, runtime errors, wrong token counts, line counts, or values. Failing test cases with the same signature are grouped into one scenario: \[G_{i,t}^{(k)} = \{j \in F_{i,t} : \sigma(j)=k\},\] where \(\sigma(j)\) is the failure signature of the test case \(j\).

The expected-output shape is computed from the oracle output and captures the structural form of the required answer, while the candidate failure type is computed from the comparison between the candidate output and the oracle output.

To avoid over-fragmenting the failure space, we use a hierarchical grouping strategy. We group failing tests into failure scenarios, each representing a region of incorrect behavior. It uses the full signature consisting of trace, output shape, and failure type. If more than \(K_{\max}\) scenarios are produced or if the median scenario size falls below \(s_{\min}\), it backs off to output-shape plus failure type, and finally to failure type alone. \(K_{\max}\) and \(s_{\min}\) are predefined. The benchmark prioritizes the active failure region by the number of current failing test cases it covers, using trace coverage as a secondary tiebreaker. Thus, each hint is anchored to a concrete and reproducible region of hidden failure behavior. The benchmark progressively provides hints that expose more failure regions and measure how the candidate model improves within the failure space. We do not claim that the signature perfectly identifies semantic bug causes. Rather, it provides a reproducible approximation of execution-grounded failure regions. The grouping is used to control feedback targets, not to infer oracle root causes.

4.2 Scaffolded Hint-Depth Design↩︎

According to the foundation on formative feedback and intelligent tutoring systems, effective feedback should be task-focused, specific enough to support progress [25], [26]. In educational scaffolding, assistance is adjusted to the learner’s current capability and gradually increased or faded as needed [27], [28]. PAIR-Bench applies this idea to interactive code improvement: a model should first be given weak evidence about the failure, and only receive more direct hinting information when weaker feedback is insufficient.

1.5pt

We organize hints along an increasing informativeness axis, using the six-level scale shown in Table 1. The levels progress from outcome-level feedback to repair-level guidance. L1 describes only the observable symptom, corresponding to minimal feedback. L2 reveals the input condition under which the failure occurs, making the feedback diagnostic but still implementation-agnostic. L3 identifies the missing state or condition the solution fails to track, moving from external behavior to internal reasoning. L4 localizes the suspicious part of the candidate implementation, similar to context-specific hints in intelligent tutoring systems. L5 states the missing invariant or conceptual correction, and L6 gives a concrete repair direction without giving code, pseudocode, or exact patch.

This hierarchy separates diagnostic evidence from repair disclosure and makes hint depth a controlled assistance variable. Lower levels test whether the model can infer the bug from observed behavior, while higher levels reveal increasingly direct guidance when weaker feedback is insufficient. Thus, repairing a scenario at a lower level indicates stronger diagnostic and refinement ability, whereas requiring higher-level hints indicates greater dependence on explicit guidance.

This design also supports adaptive evaluation: after a successful repair, the benchmark can reduce the starting hint level for the next scenario, analogous to fading support after demonstrated competence; if the model remains stuck, it escalates the hint level toward the model’s effective capability. In this way, the hint scale operationalizes scaffolded interaction for APR while keeping feedback comparable across models and runs.

4.3 Controlled Hint Generator↩︎

To preserve the meaning of the hint-depth scale, we constrain hint generation through level-specific prompt templates, evidence masks, and leakage validation. Each level is associated with an allowed information boundary that decides what evidence the feedback model may observe and what information the generated hint may reveal. E.g., an L2 hint may use failing-input features and observed output mismatches, but the feedback model is not given candidate-code locations, suspicious code regions, or repair-relevant invariants. In contrast, an L4 hint may additionally receive candidate-code context since fault localization is allowed at that level.

After generation, PAIR-Bench validates each hint against level-specific leakage criteria. A hint is rejected if it reveals information reserved for a higher hint level, such as exposing a suspicious code location in an L2 hint or suggesting a concrete repair direction in an L4 hint. Rejected hints are regenerated until they satisfy the requested information boundary or the budget is exhausted. This helps ensure that hint depth functions as a controlled assistance rather than merely a post-hoc label.

4.4 Progressive and Adaptive Hint Policy↩︎

PAIR-Bench uses a progressive policy over failure regions and adaptive over hint depth (Algorithm 3).

4.4.1 Progressive Policy↩︎

Generally, PAIR-Bench selects a failure scenario, provides feedback for that scenario, evaluates whether the model fixes it, and moves to another unresolved scenario. At turn \(t\), it re-evaluates the current program and groups the remaining failing tests into scenarios. Next, it must decide which failure region to address. We prioritize scenarios by their current failing-test coverage, since a scenario covering more failing tests represents a larger unresolved portion of the hidden failure space. This choice makes each hint target the largest remaining opportunity for measurable improvement, while completed and deferred scenarios are excluded to ensure that the interaction progresses across distinct failure regions. Formally, let \(\mathcal{U}_{i,t}\) denote the set of unresolved selectable scenarios at turn \(t\). The active scenario is selected as follows: \[G_{i,t} = \arg\max_{G\in \mathcal{U}_{i,t}} |G|\] Ties are broken using trace coverage and then a deterministic scenario identifier. The benchmark then samples up to \(m\) representative failing test cases from \(G^{*}_{i,t}\) to ground the hint. These sampled tests are used only for hint generation; scenario completion is evaluated over the full selected scenario. If all tests in \(G^{*}_{i,t}\) pass after revision, the scenario is marked as completed. Otherwise, the benchmark continues on the same scenario until either the maximum hint depth or the per-scenario turn limit is reached. A completed scenario is re-opened if any of its tests fail after a later revision, ensuring that regressions remain part of the active failure space.

To avoid repeatedly prioritizing large scenarios, PAIR-Bench excludes completed and deferred scenarios and enforces a per-scenario turn budget. This ensures that the interaction progresses across distinct failure regions rather than repeatedly focusing on one dominant failure mode. We prioritize coverage because it maximizes measurable repair opportunity at each turn, while the scenario-turn limit prevents the policy from spending the entire budget on a single unresolved region.

4.4.2 Adaptive Policy↩︎

The policy is adaptive because the next hint level depends on the model’s observed repair behavior. Inspired by adaptive testing and IRT-style assessment, PAIR-Bench treats hint depth as an ordered assistance scale with a deterministic staircase policy as a lightweight operationalization: success decreases the next starting hint level, while failure increases it. This policy does not require fitting IRT parameters, but preserves the central idea of calibrating item assistance to observed ability. Specifically, each scenario begins with the current starting level, initially L1. If the model repairs the selected tests, PAIR-Bench treats the model as capable of using the current level of feedback and starts the next scenario with a weaker or equal hint: \(\ell_{t+1}=\max(1,\ell_t-1).\) If the selected test cases still fail, PAIR-Bench increases the hint depth: \(\ell_{t+1}=\min(6,\ell_t+1).\) The interaction stops when all hidden tests pass, no remaining scenario can be selected, or the max turn budget is reached. In brief, successful repair leads to faded assistance, while failure leads to more direct guidance.

4.5 Trajectory-Level Evaluator and Evaluation Metrics↩︎

With the same notations in Section 4.1, we define our metrics. Turn \(t=0\) is the initial repair before any hint. The pass rate (PR) is defined as \(p_{i,t}=\frac{|P_{i,t}|}{|\mathcal{T}_i|}\). Let \(\mathcal{I}\) be the benchmark instances. At repair turn \(t\), let \(G_{i,t}\) \(\subseteq F_{i,t-1}\) be the failure scenario targeted by the hint, and let \(b_i\)=\(\max_{0\leq t\leq B}p_{i,t}\) be the best pass rate in the turn budget \(B\).

4.5.0.1 Initial Fix Rate

This measures unaided repair ability before feedback: \(\mathrm{InitialFix} = \frac{1}{|\mathcal{I}|} \sum_{i\in\mathcal{I}} \mathbf{1}[p_{i,0}=1].\) It separates one-shot repair capability from interactive improvement.

4.5.0.2 Targeted Repair Success (TRS)

This measures if a model fixes the failure region described by the current hint: \[\mathrm{TRS}_{i,t} = \frac{|G_{i,t}\cap P_{i,t}|}{|G_{i,t}|}.\] It captures the precise use of controlled feedback.

4.5.0.3 Broader Repair Gain (BRG)

This measures progress beyond the hinted scenario: \(\mathrm{BRG}_{i,t} = \frac{|(F_{i,t-1}\setminus G_{i,t})\cap P_{i,t}|}{|\mathcal{T}_i|}.\) It captures if a model generalizes a targeted hint to other hidden failures. We normalize by the total number of hidden tests to reflect absolute contribution to overall pass-rate improvement.

4.5.0.4 Behavior Preservation (BP)

This measures whether a revision preserves behavior that was already correct: \[\mathrm{BP}_{i,t} = 1- \frac{|P_{i,t-1}\cap F_{i,t}|}{|\mathcal{T}_i|}.\] It penalizes newly introduced regressions. We use the same total-test normalization so the regression penalty is comparable across instances and directly aligned with pass-rate change.

4.5.0.5 Progress Monotonicity Rate (PMR)

This measures whether the repair trajectory improves monotonically across interaction turns. Since \(p_{i,t}\) denotes the hidden-test pass rate after turn \(t\), PMR compares each repaired program with the program from the previous turn. Let \(\tau_i\leq B\) be the last evaluated repair turn for instance \(i\). We define: \[\mathrm{PMR}_i = \frac{1}{\tau_i} \sum_{t=1}^{\tau_i} \mathbf{1}[p_{i,t}\geq p_{i,t-1}].\] PMR measures the fraction of refinement steps that preserve or improve the previous pass rate, with higher values indicating a more stable trajectory with fewer regressions.

4.5.0.6 Hint Efficiency (HE)

This measures how much aid a model needs to close targeted failure scenarios. For each attempted scenario \(g\), let \(\ell_{i,g}\in\{1,...,6\}\) be the hint level at which the scenario is closed. We assign a scenario-level score \[h_{i,g} = \begin{cases} 7-\ell_{i,g}, & \text{if scenario } g \text{ is fully repaired},\\ 0, & \text{otherwise}. \end{cases}\] The instance-level Hint Efficiency is the average over all attempted scenarios \(\mathcal{G}_i\): \(\mathrm{HE}_i = \frac{1}{|\mathcal{G}_i|} \sum_{g\in\mathcal{G}_i} h_{i,g}.\) Higher HE indicates that a model closes failure scenarios with weaker, less explicit hints; unresolved scenarios get no credit.

4.5.0.7 Gap Closure

This measures the normalized fraction of remaining repair opportunity that the model closes: \(\mathrm{GC}_i=\frac{b_i-p_{i,0}}{1-p_{i,0}}.\) It gives credit for progress.

4.5.0.8 Final Fix Rate

This measures complete repair within the interaction budget: \(\mathrm{FinalFix} = \frac{1}{|\mathcal{I}|} \sum_{i\in\mathcal{I}} \mathbf{1}[\exists t\leq T,\;p_{i,t}=1].\) It preserves the standard end-to-end correctness criterion.

4.5.0.9 Turns to Fix

This measures convergence speed among initial failing instances that are eventually fixed. Let \(\mathcal{S} = \{i\in\mathcal{I}:p_{i,0}<1 \wedge \exists t\leq T,\;p_{i,t}=1\}.\) Then \[\mathrm{TurnsToFix} = \frac{1}{|\mathcal{S}|} \sum_{i\in\mathcal{S}} \min\{t:p_{i,t}=1\}.\] It distinguishes models that reach the same final success rate but require different numbers of refinement turns.

Turn-level metrics are averaged within each initially failing instance and then across initially failing instances. Initial Fix Rate and Final Fix Rate are computed over all instances.

3pt

5 Empirical Evaluation↩︎

For evaluation, we seek to answer the following questions:

RQ1. [Interactive Repair Performance] How effectively do LLMs repair buggy code under controlled feedback, and what aspects of the repair trajectory distinguish stronger models?

RQ2. [Robustness] Does Controlled Feedback produce more stable interactive repair results than Vanilla Feedback?

RQ3. [Hyperparameter Sensitivity] Do model rankings remain consistent when key benchmark parameters are varied?

Dataset Construction↩︎

We collected our dataset from real Codeforces Python submissions in the Hugging Face dataset [29]. We retain only submissions with the ‘Wrong Answer’ verdict, since these programs compile successfully but fail on execution behavior. We further filter for non-trivial instances by requiring each problem to have more than 100 test cases, an initial pass rate between 10% and 55%, 15–80 lines of code, and cyclomatic complexity of at least 8. For each selected buggy submission, we use GPT-OSS 120B to iteratively repair the program with failing test cases as feedback until it passes all tests, producing a validated reference solution that remains close to the original implementation logic. The final dataset contains 440 code-improvement instances, each consisting of a problem statement, a wrong-answer Python program, test cases, and a corresponding reference solution.

Model Selection↩︎

We select representative models from diverse model families to cover both closed-source and open-source systems with broadly comparable coding capabilities. Specifically, we chose DeepSeek V3.2, Gemini 2.5 Flash Lite, GPT-4o-mini, Qwen3 Coder 30B A3B Instruct, Llama 3.3 70B Instruct, Ministral 14B, Mistral Small 3.2 24B, and Gemma 3 27B. To keep feedback generation consistent across candidate models, we use GPT-OSS 120B as the feedback model, since it provides strong performance on coding and program-repair tasks. For readability, some tables and figures use the model-family name as shorthand; e.g., ‘DeepSeek’ refers to DeepSeek V3.2, ‘Gemini’ refers to Gemini 2.5 Flash Lite.

Evaluation Protocol↩︎

For each instance, the model first attempts a zero-hint repair, which is used to compute Initial Fix Rate. If hidden tests still fail, PAIR-Bench starts controlled hinting: it selects a failure scenario, provides a hint, asks the model to fix the code, and re-evaluates the result. We use two parameters: \(\alpha\)=10 feedback turns per instance and \(\beta\)=3 turns per failure scenario. Sensitivity is reported in Section 5.3.

5.1 Interactive Repair Performance (RQ1)↩︎

3pt

Table 2 reports the main progress-centric results for controlled interactive program repair, while Table ¿tbl:tab:rq195metric95ranking? summarizes the metric-wise rankings across the evaluation dimensions.

5.1.1 How much do models benefit from hints?↩︎

Final Fix Rate and Gap Closure (GC) capture complementary aspects of feedback-driven improvement. Final Fix measures whether interaction eventually produces a fully correct program, while GC measures how much of the remaining opportunity the model can close at the end of the repair trajectory. DeepSeek achieves the highest Final Fix Rate, reaching 99.31%, indicating the strongest complete-repair capability. Its GC is also the highest. While Gemini starts lower than DeepSeek, it also closes the gap quite well (97.51%), ending with the Final Fix Rate of 95.45%. Thus, high GC reflects responsiveness to feedback and partial improvement, while high Final Fix reflects the ability to convert that progress into fully correct solutions.

As a sanity check, we compare controlled feedback with a no-hint multi-turn baseline. In the no-hint setting, models can revise their code for the same number of turns but receive no scenario-level hints. Gemini 2.5 Flash Lite reaches 78.63% Final Fix Rate without hints, compared with 95.45% under controlled feedback; Qwen3 Coder 30B reaches 57.72%, compared with 90.90%. These gains suggest that PAIR-Bench’s improvements are not merely due to repeated prompting, but are driven by repair-relevant hints. This is consistent with prior interactive-feedback benchmarks showing that feedback-guided repair improves over no-feedback interaction [10], [15].

5.1.2 Do models use targeted hints effectively?↩︎

Targeted Repair Success (TRS) measures whether a model fixes the failure scenario explicitly described by the current hint. DeepSeek achieves the highest TRS, indicating that it most reliably repairs the hinted failure regions. Gemini, GPT-4o-mini, and Qwen form the next group, suggesting that they can often act on targeted hints but do so less consistently than DeepSeek. In contrast, weaker models obtain lower TRS, showing that scenario-level feedback does not automatically translate into targeted repair. These results demonstrate the value of separating targeted feedback use from final correctness.

Hint Efficiency (HE) complements TRS by measuring how much assistance is needed to close a targeted scenario. A model with high TRS but low HE can repair hinted scenarios, but only after receiving more explicit guidance. Conversely, a model with high HE can close scenarios using shallower hints, indicating stronger diagnostic and refinement ability. DeepSeek ranks first in both TRS and HE, showing that it not only repairs targeted scenarios most reliably, but also often does so with less direct feedback. This combination distinguishes strong targeted repair from hint-dependent repair. GPT-4o-mini has slightly higher HE than Qwen, but Qwen has higher Final Fix, BP, and PMR. That contrast would show that needing less assistance on closed scenarios does not necessarily imply better overall repair stability or final convergence.

5.1.3 How much assistance do models need?↩︎

Fig. 4 provides a finer-grained view of hint dependence by stratifying attempted scenarios according to the hint level at which they are closed. Percentages are computed over all attempted scenarios, including unresolved ones. DeepSeek closes the largest fraction of scenarios with shallow L1–L2 hints and leaves the smallest unresolved portion, showing that it can often infer repairs from weak diagnostic evidence. In contrast, weaker models leave larger unresolved portions or require deeper L5–L6 guidance, indicating greater dependence on explicit repair information. Thus, two models with similar final repair outcomes may still differ substantially in the amount of assistance they require. This stratification supports our key idea of adaptive hint-depth control: PAIR-Bench does not merely record whether feedback helps, but identifies the level of assistance at which each model can make progress. Models that close more scenarios at L1–L2 demonstrate stronger diagnostic repair ability, whereas models that require L5–L6 hints or remain unresolved depend more on explicit guidance.

5.1.4 Do models progressively close failure regions?↩︎

We define a new metric, HintedClosedCoverage (HCC), to measure the fraction of initially failing test cases that were both exposed through progressive hints and closed by the final solution. HCC assesses the effect of failure-region control for progressive mechanism, i.e., a model also needs to close the exposed/hinted failure regions. Formally, let \(F_0\) denote the set of tests failed by the initial solution, \(H\) denote the set of failing tests selected for hints during the interaction, and \(P_T\) denote the set of test cases passed by the final solution. We define \(\mathrm{HintedClosedCoverage} = \frac{|F_0 \cap H \cap P_T|}{|F_0|}.\) Fig. 5 shows that stronger models make more durable progress and close a larger fraction of the initially failing space through hinted regions. DeepSeek achieves the highest HCC at 20.45%, followed by Qwen, GPT, etc. The results show that failure-region control* provides a measurable way to track how much of the initial failure space is closed through controlled, scenario-level hints*.

5.1.5 Do targeted repairs generalize to other failures?↩︎

Broader Repair Gain (BRG) measures whether a repair extends beyond the hinted scenario. BRG is much smaller than TRS for all models, suggesting that most improvements are localized to the targeted failure region. Gemini ranks first on BRG, followed by Ministral and DeepSeek, showing stronger ability to generalize from controlled hints to other hidden failures. The gap between TRS and BRG supports the need for progressive evaluation: fixing one hinted scenario does not imply that the model has solved other failure regions, so the benchmark must expose and evaluate multiple regions over time.

5.1.6 Are repair trajectories stable?↩︎

Behavior Preservation (BP) and Progress Monotonicity Rate (PMR) measure whether models improve without destabilizing previously correct behavior. The two metrics are complementary. BP captures test-level regression by measuring if tests that previously passed remain passing after a repair, while PMR captures pass-rate-level stability by measuring if the overall pass rate decreases across turns. A model may maintain a non-decreasing pass rate while still changing which tests pass, so PMR alone can hide behavioral churn; BP helps expose such regressions. DeepSeek and Qwen perform better on both BP and PMR, indicating that their repairs are not only effective but also stable across turns. In contrast, models with weaker preservation or monotonicity may still benefit from feedback, but their repair trajectories are less reliable since later edits can undo previously correct behavior. This shows that progressive repair should also measure whether the improvement is preserved during the interaction.

Fig. 6 illustrates how models can follow different repair trajectories on an instance. Stronger models reach full correctness in fewer turns: Qwen and GPT-4o-mini reach 100% pass rate by turn 3. Weaker models require more turns and show less stable progress. Llama and Ministral eventually reach 100%, but only after several low-progress or regressive turns. Gemma shows substantial partial progress but then regresses to 0%. This example highlights why PAIR-Bench evaluates the full repair trajectory rather than only the final outcome.

5.1.7 How efficiently do models converge?↩︎

Because ‘Turns to Fix’ is computed only over initially failing instances that are eventually repaired, it should be interpreted together with Final Fix Rate: a low Turns to Fix indicates fast convergence only among solved instances, not necessarily stronger repair coverage overall. DeepSeek ranks first on Turns to Fix while also achieving the highest Final Fix Rate, indicating that it can translate feedback into complete repairs both reliably and quickly. Gemini and GPT-4o-mini also require relatively few turns, suggesting efficient use of feedback on instances they successfully repair. In contrast, Qwen reaches a high Final Fix Rate but requires more turns, indicating strong final repair ability but weaker convergence efficiency. This distinction matters for progressive repair workflows, where repeated feedback turns impose cost even when the final repair succeeds.

Fig. 7 reports the cumulative Final Fix Rate by turn (over 10 feedback turns), showing both convergence speed and final repair coverage: a steeper early rise shows that a model quickly converts feedback into complete repairs. DeepSeek starts from the highest Initial Fix Rate, rises sharply in the first few turns, and approaches near-complete repair by the end of the budget. Gemini follows a similar but lower trajectory, with steady early gains and strong final coverage. The other models improve more gradually. Qwen starts much lower than Gemini but steadily catches up to the third-highest final coverage, indicating strong eventual repair ability. Mistral Small and Gemma remain lower throughout, suggesting that feedback helps them improve but less often leads to complete repair.

5.1.8 Which models are strongest overall?↩︎

Table ¿tbl:tab:rq195metric95ranking? summarizes the metric-wise ranking. We do not treat the average rank as a principled aggregate score. Instead, we use it to provide a descriptive view of model repair profile. DeepSeek is the strongest all-around model, ranking first in most metrics and combining high initial correctness, effective targeted repair, strong preservation, high hint efficiency, high final repair, and fast convergence. Gemini and Qwen form the next tier with different strengths: Gemini is stronger in broader repair and net improvement, while Qwen is stronger in preservation, monotonicity, and final repair. Other models show substantial Gap Closure but lower Final Fix Rates, weaker stability, or longer convergence, suggesting that they benefit from feedback but less consistently convert partial progress into reliable full repair. Overall, this shows that good interactive repair depends on final correctness, and also on targeted repair, progressive improvement, stability, hint efficiency, and convergence.

5.2 Robustness of Controlled Feedback (RQ2)↩︎

5.2.1 Setting and Metrics↩︎

RQ2 evaluates whether controlling feedback generation makes interactive program-repair evaluation more stable. We compare two settings that use the same initial candidate programs, evaluated models, turn budget, execution environment, and feedback-generation model; the only difference is the feedback protocol. Controlled Feedback is PAIR-Bench. Vanilla Feedback is an ablated baseline inspired by prior interactive feedback evaluation [15], where the feedback model generates a hint at each turn without scenario-level testcase selection or hint-depth control.

Both settings start from the same initial candidate repairs, so the Initial Fix Rate is identical for each model. The comparison thus focuses on the interactive stage after feedback is introduced. For each setting, we run the benchmark three times and report the mean and standard deviation. Since Vanilla Feedback does not define targeted scenarios or hint levels, we use metrics shared by both settings: Repair Rate, Preservation Rate, Final Fix Rate, and Turns to Fix. We define the turn-level Repair Rate as the fraction of previously failing tests that become passing after the current repair \(\mathrm{RepairRate}_{i,t} = \frac{|F_{i,t-1}\cap P_{i,t}|}{|\mathcal{T}_i|}.\) Final Fix Rate measures the fraction of instances that are fully repaired within the budget, and Turns to Fix measures convergence speed among fully repaired instances.

1.5pt

5.2.2 Robustness Comparison↩︎

Table 3 reports a repeated-run stability analysis over three runs. While this number of runs limits strong statistical claims about variance, the pattern is consistent: Controlled Feedback yields lower run-to-run variation than Vanilla Feedback in every model-metric comparison across all eight models and four shared metrics. Averaged across models, the standard deviation of Repair Rate decreases from 1.10 to 0.24, Preservation Rate from 0.53 to 0.11, Final Fix Rate from 1.46 to 0.24, and Turns to Fix from 0.16 to 0.07 when moving from Vanilla to Controlled Feedback. These correspond to relative reductions of approximately 78%, 79%, 84%, and 56%, respectively. Thus, the repeated-run results support that controlling the targeted failure region and hint depth improves the stability of interactive repair evaluation.

5.2.3 Statistical Testing↩︎

We measure ranking stability with Kendall’s \(\tau\) by ranking models independently in each repeated run and averaging \(\tau\) over all run pairs; for Turns to Fix, lower values are ranked higher. From Table 4, Controlled Feedback produces more stable rankings than Vanilla Feedback across all shared metrics, with a higher average \(\tau\) (\(0.982\) vs. \(0.905\)). This indicates that PAIR-Bench can reduce metric variance as well as preserve model ordering across repeated evaluations.

5.2.4 Mean performance versus stability↩︎

Vanilla Feedback often gives higher mean Repair Rate, higher mean Final Fix Rate, and fewer Turns to Fix. We do not interpret this as evidence that Vanilla Feedback is a better evaluation. Because Vanilla Feedback does not control the targeted failure region or hint depth, different runs may expose different failure modes or reveal different levels of repair directness. This can make some runs easier, but also makes the measured outcome more dependent on feedback randomness. In contrast, Controlled Feedback has lower variance and clearer attribution: each repair attempt is tied to a known failure scenario and a known hint-depth level. Thus, RQ2 supports Controlled Feedback as a more stable protocol for measuring interactive APR capability.

5.3 Hyperparameter Sensitivity (RQ3)↩︎

4pt

RQ3 evaluates whether PAIR-Bench’s model comparisons are robust to varied hyperparameters. We focus on three representative models from different performance tiers: Gemini, Llama, and Mistral. We vary the total interaction budget (\(\alpha\)), the per-scenario refinement budget (\(\beta\)), and the feedback model (DeepSeek V3.2). As seen in Table ¿tbl:tab:sensitivity?, the relative or-dering is largely stable across settings: Gemini consistently ranks highest in Final Fix Rate and Hint Efficiency, followed by Llama and Mistral, yielding Kendall’s \(\tau\)=1.0. This suggests that the ranking is not an artifact of a single budget choice or feedback generator. Increasing the budget improves Final Fix Rate for all models, especially Llama and Mistral, while reducing the per-scenario budget mainly hurts these weaker models and lowers Hint Efficiency. This suggests that additional refinement opportunities help weaker models, whereas limiting scenario-level refinement makes targeted repair harder.

Using DeepSeek V3.2 as the feedback model preserves the same relative ordering, but lowers Final Fix Rate and Hint Efficiency compared with the main setting, especially for Llama and Mistral. Interestingly, Progress Monotonicity remains comparable or slightly higher. Overall, these results show that rankings are reasonably stable under the tested parameter changes, while absolute performance reflects the interaction budget and the amount and quality of the feedback source.

6 Concluding Remarks↩︎

PAIR-Bench reveals that model improvement is multidimensional. Models differ not only in whether they eventually pass all tests, but also in how precisely they use targeted hints, how much assistance they require, whether repairs generalize beyond the hinted scenario, and whether progress remains stable across turns. Our future work includes adding the adaptive process to the question level with easy, medium, and hard problems, and adaptively challenging the target model.

References↩︎