1 Introduction↩︎

Large Language Models (LLMs) have demonstrated great potential in supporting a wide range of software engineering (SE) tasks, such as code generation, code repair, and automated documentation [1]. As LLMs become increasingly integrated throughout the software development lifecycle, human-AI teaming has emerged as a predominant theme in SE. In this context, human–AI teaming refers to developers iteratively collaborating with an AI assistant toward a shared objective [1]. We consider teaming effective when developers can readily understand AI-generated outputs and reach successful outcomes with fewer iterations and stronger comprehension. Task-specific fine-tuning of LLMs for software engineering strengthens human–AI teaming by aligning models with specialized tasks and iterative developer workflows [2]. Among these applications, code repair has been studied across commonly used settings, such as programming education, open-source maintenance, and AI-assisted pair programming, and has received substantial research attention [3]–[5]. In these settings, developers commonly submit code to LLMs for correction and iterate on model suggestions. Accordingly, effective human–AI teaming depends on comprehensible feedback rather than code alone, enabling developers to understand the rationale for changes, make targeted edits, and work more efficiently [6]. However, prior work shows that developers often struggle to understand LLM-generated feedback. For example, a computer science education study revealed that CS1 students demonstrated a per-task success rate of only 32.5% in comprehending code and its corresponding explanations generated by LLMs [7]. Empirical research indicates that for a larger cohort of developers, approximately 20% of questions on forums are related to understanding content generated by models  [8]. Consequently, while many SE studies focus on improving code quality to enhance LLM reliability in repair tasks, the accompanying natural-language feedback is equally important: it helps developers locate errors, understand the rationale behind fixes, and apply repaired code effectively [9]. Moreover, since developers’ feedback needs vary by proficiency [10], profile-specific feedback is necessary to better support human–AI teaming, thereby reducing clarification cycles and improving workflow efficiency. Together, these highlight the need for alignment frameworks that tailor LLM behavior to diverse developer profiles, prioritizing developer-centered feedback that connects technical code improvements with comprehensible explanations.

Reinforcement Learning from Human Feedback (RLHF) provides a promising approach for adapting code feedback to developer preferences by optimizing language models toward human-preferred outputs [11]. However, Proximal Policy Optimization-based RLHF (PPO) is computationally and annotatively expensive, often requiring online rollouts and large-scale human preference labeling [11]. These requirements are impractical in resource-constrained settings like computer science education. By contrast, Direct Preference Optimization (DPO) emerges as a more scalable alternative, enabling direct optimization from offline preference datasets without complex online reinforcement learning, thus streamlining the training process while preserving effective alignment capabilities [12]. However, standard DPO relies on binary pair orderings and ignores the preference margin between candidates [12]. Moreover, although DPO-inspired methods have shown promise for code generation [13], its potential for generating developer-aligned feedback that integrates code edits with tailored explanation remains unexplored in SE, and particularly in code repair settings.

Thus, in this study, we take a first step toward aligning LLM-generated code-repair feedback with developer needs by proposing DPO-f+, a novel and cost-effective framework evaluated across diverse settings. By optimizing profile-specific feedback criteria for different developer groups, our approach also suggests a promising pathway for improving code comprehension with LLMs. The framework consists of four components: (1) feedback-alignment metrics for assessing alignment, (2) an automated method for constructing pairwise preference datasets to support training, (3) a fine-tuning approach that integrates Direct Preference Optimization (DPO) with a reward, and (4) an automated protocol for evaluating feedback quality. Empirical results demonstrate that DPO-f+ consistently outperforms both baseline and standard DPO. It achieves superior feedback accuracy, improving Pass@1 by 5.71 pp over the baseline and 3.30 pp over standard DPO, while also achieving the highest overall alignment scores as evaluated by both LLM-as-a-judge and 200 developers. On more complex tasks (i.e., SWE-bench Lite), it improves the resolution rate by 1.67 pp over standard DPO and 4.67 pp over the baseline. We make the following contributions:

• A novel framework for alignment of generated feedback in code repair.

• A reward-augmented DPO that optimizes for feedback under profile-specific criteria without online RL.

• A feasible approach to improve code comprehension through LLMs in software engineering.

Moving beyond accuracy alone, our framework reframes code repair as a collaborative sense-making process that helps developers better understand AI-generated outputs and reduces trial-and-error iterations through feedback tailored to different developer groups. By optimizing LLM-generated feedback, it offers a practical path to improving code comprehension and strengthening human–AI teaming in software engineering.

2 Motivation Example↩︎

In software engineering, developers increasingly rely on LLMs not only to repair code but also to explain what to fix and why. Yet the feedback accompanying LLM-generated code is often hard to follow [7], [8]. Prior work shows that such feedback is frequently generic, under-specified, and weakly justified, making it difficult to act on, especially for novices [14]. Without clear guidance on what to change, why the change resolves the bug, and how to apply edits in sequence, learners may introduce new faults. Prompting skill also strongly affects novice outcomes, highlighting the need for better alignment [15]. Similar concerns appear among professional developers, who also ask for clearer and more actionable explanations [16]. To address this gap, we augment DPO with a graded preference signal, since standard DPO captures only binary pairwise orderings and ignores preference margins [12].

Figure 1 illustrates this motivation with a buggy C++ implementation of MinStack. The snippet contains several design and safety flaws, including incorrect use of p, out-of-bounds writes from failing to grow the vector, missing precondition checks, and incorrect minimum-tracking logic. We focus on the first three issues highlighted in Figure 1. Although all three models identify these defects and generate correct repaired code, their feedback differs substantially in specificity and structure.

The baseline model identifies all three issues and suggests plausible fixes, but its feedback is verbose, repetitive, and weakly organized. It interleaves symptoms with fixes, rarely explains why the changes matter, and provides no concise correction plan, reducing actionability and increasing cognitive load.

DPO improves phrasing and adds concrete directives with a numbered correction list. However, its guidance is sometimes inconsistent and still incomplete: some recommendations conflict, and key rationales, such as why pointer misuse is unsafe or how failure cases should be handled, remain underexplained. Overall, DPO better states what to change, but still underspecifies why and how to apply the fixes safely.

In contrast, DPO-f+ pairs precise edits with clear rationale. It explains the type mismatch, replaces manual writes with push_back while clarifying safe container growth, and adds explicit empty-stack guards with defined failure behavior. Its guidance is stepwise, compact, and logically ordered, making each action concrete, verifiable, and easier to transfer to new contexts.

3 Related Work↩︎

3.1 Reinforcement Learning from Human Feedback↩︎

Reinforcement Learning from Human Feedback (RLHF) has emerged as a promising approach for aligning LLMs with human preferences [17]. Early work such as InstructGPT [11] showed that, despite remaining limitations, fine-tuning with human feedback is an effective direction for aligning language models with human intent. Traditional RLHF typically uses Proximal Policy Optimization (PPO) [18] together with a learned reward model. PPO stabilizes training by constraining policy updates, while the reward model provides the optimization signal [18]. However, this pipeline is computationally expensive and often difficult to stabilize. Direct Preference Optimization (DPO) addresses these limitations by reframing preference learning as a classification problem over preferred and dispreferred responses, enabling direct policy optimization without a separate reward model or reinforcement-learning loop [12]. As a result, DPO retains the main benefits of RLHF while reducing implementation complexity [19]. It has been applied to tasks such as code generation [13] and feedback alignment in educational settings, including math tutoring and teaching assistant-guided feedback [20], [21]. Despite these advances, the use of RLHF, particularly DPO, to generate high-quality feedback under profile-specific criteria in software engineering remains underexplored. Moreover, standard DPO captures only binary pairwise preferences and ignores preference strength, which may weaken the supervision signal and reduce data efficiency [22]. To address these gaps, we propose a lightweight reward-augmented DPO framework for aligning LLM-generated code-repair feedback with developer preferences.

3.2 LLM-based Frameworks in Software Engineering↩︎

Numerous LLM-based frameworks have been developed for various SE tasks, including code generation [23], [24], program repair [25], automated documentation [26], and code review [27], [28]. These approaches typically involve fine-tuning pre-trained language models on domain-specific corpora or task-oriented datasets to improve performance on downstream SE applications. However, these frameworks largely overlook human factors in the design and alignment of model outputs. Prior work highlights that human-centered considerations are critical in software engineering tools, as developers rely on interpretability and trust to use automated assistance effectively [29], [30]. Recent work in educational settings has applied DPO to align LLM-generated feedback with instructor preferences, resulting in feedback that is more accurate, more insightful, and preferred over state-of-the-art models (e.g., GPT-4o) [21]; however, these efforts have not targeted software engineering tasks. Therefore, a gap remains in applying preference-based alignment methods (e.g., DPO) to software engineering settings where feedback must be technically accurate, actionable, and tailored to developers with varying levels of experience. This work addresses that gap by proposing a feedback-alignment framework for code repair in SE contexts using profile-specific criteria.

3.3 Code Feedback in Software Engineering↩︎

Providing effective feedback on code is a fundamental component of software development, supporting activities such as code review [31], testing [32], software maintenance [33], and programming education [34]. Traditional feedback systems have primarily relied on static or dynamic analysis [35] and rule-based approaches [36]. With the emergence of LLMs, recent work has explored their potential for generating code feedback. For example, LLMs have been used to explain compiler errors [37], demonstrating their utility as real-time debugging aids; to produce code review comments [38], often matching or surpassing the performance of heuristic and rule-based methods; and to suggest edits for improving code quality [39] through frameworks that can effectively identify and address issues such as poor naming, code smells, and anti-patterns. Despite these advances, most systems are instruction-tuned on general-purpose datasets or optimized for surface-level metrics (e.g., BLEU, ROUGE), rather than explicitly aligned with developer needs for feedback usefulness, correctness, and clarity. Given that software development is fundamentally human-centric [40], such alignment is essential because developer expertise, workload, and emotional state affect how developers process feedback [41], while tone, trust, and team dynamics shape how it is received [42]. As human–AI teaming becomes increasingly common in software engineering [43], especially for coding, it is crucial to design AI-generated feedback frameworks that not only improve code correctness but also align with developer preferences across experience levels. Yet few alignment pipelines have been built with these requirements in mind. To address this gap, we introduce a DPO-inspired framework that aligns LLM-generated feedback with developer needs and expectations in code-repair settings.

4 Methodology↩︎

In this section, we describe the proposed DPO-f+ framework, as summarized in Figure 2. Specifically, we first define the alignment metrics and use them to construct a pairwise preference dataset, then introduce a reward-augmented DPO training procedure, and finally describe the evaluation protocol. We aim to answer the following research questions:

1. Does DPO-f+ improve the alignment of code-repair feedback for novice programming instruction compared to Baseline and DPO?

2. Does DPO-f+ maintain strong performance across more challenging and diverse code-repair tasks?

4.1 Metric Definition↩︎

To evaluate whether generated feedback matches developers’ preferences, we introduce a seven-dimensional metric for code repair guidance. Grounded in prior work, the metric operationalizes feedback qualities that developers consistently value in code reviews [44]–[46]: Conciseness (brief and free of unnecessary detail), Quality (technically correct and aligned with best practices), Explainability (clearly motivates each suggested change), Understandability (well-structured and easy to follow), Completeness (addresses required fixes and relevant edge cases), Actionability (immediately usable with minimal interpretation), and Contextual Relevance (includes enough context to clarify when and why the fix applies).

These dimensions provide a general baseline, which we adapt to reflect differences in developer experience and task demands. For novice-oriented repairs, we emphasize accessible language, step-by-step guidance, safe patterns, copy-pasteable patches with lightweight checks, and clearly stated scope. For experienced-oriented repairs, we prioritize concise prose with precise code anchors, idiomatic and composable edits, brief rationale with trade-offs, broader edge-case coverage, tool-supported verification and rollback, and consistency with the system’s architecture and deployment constraints. An example of how we tailor the description of each dimension to different profiles of developers and task needs is summarized in Table ¿tbl:tab:feedback-metrics?.

4.2 Data Preparation↩︎

We construct the training dataset through source-code collection, augmentation, feedback generation, and pair construction.

4.2.1 Source code and augmentation.↩︎

The dataset consists of human-written solutions to three introductory programming problems (TwoSum, MinStack, and TicTacToe), contributed by 53 novice programmers and made publicly available by prior work [47]. Among these, 30 participants submitted solutions for TwoSum and MinStack, and 23 for TicTacToe. To expand the original code script, we apply augmentation methods grounded in prior research on code synthesis [48]. Specifically, we randomly performed code compression (e.g., reformatting, type up-conversion, dead-code elimination), and identifier modification (e.g., systematic variable renaming, identifier mangling). Following augmentation, the number of unique scripts (deduplicated by code content) increased from 53 to 534 across the three problems. Importantly, these augmentations are designed to maintain the original failure mode while varying surface structure, ensuring that augmentation does not systematically shift the distribution of error types.

4.2.2 Feedback generation.↩︎

We collected natural-language code-repair feedback by prompting three state-of-the-art LLMs (GPT-4o, Claude 3.5 Sonnet, and DeepSeek-R1) on each code snippet. To diversify the feedback, we used three distinct prompts per snippet for each model. To support accuracy evaluation, we also instructed the models to produce a revised implementation that instantiated their recommendations. This yields, for each sample, a pool of candidate feedback-and-revision responses for downstream comparisons between higher- and lower-aligned guidance. We did not perform automated prompt tuning; the three prompts were manually drafted once and used unchanged across models and splits. We do not use external tools or function-calling in generation or judging. Unless otherwise noted, for candidate generation during dataset construction, decoding uses provider defaults (i.e., temperature or top-p are not overridden). For the final evaluation, we use fixed sampling parameters, which will be explained later.

4.2.3 Pair construction.↩︎

Each feedback candidate, comprising natural-language guidance and corresponding revised code, is assessed on two dimensions.
Feedback accuracy. To evaluate feedback accuracy, we prompted the model to generate a revised program based on its suggestions. The extracted code was evaluated via a standardized C++ execution pipeline within an isolated environment. Programs were compiled using the C++17 standard with \(O2\) optimization before being subjected to a suite of automated unit tests. Submissions with compilation errors, runtime errors, or failed tests are marked as failures; submissions with missing or unparsable code are excluded. To prevent leakage, prompts contain only the problem statement and the submitted code; the test harness is applied post-generation and is not visible to the model. All executions are isolated (fresh workspace per run) with no network access and enforced time limits. In addition, we manually evaluated the feedback text itself for whether it correctly identifies the issue and proposes a valid fix.
Feedback alignment. Using the metric in Section 4.1, we determine which generated feedback best matches a given developer profile and task. To reduce evaluation cost and latency and to mitigate bias, we use GPT-4 to score each feedback instance on all metrics, consistent with evidence that LLM judgments closely track expert ratings [49], [50]. The prompts are provided in the supplementary file¹. To further validate the automatic scores, we randomly selected 100 feedback cases for human review and obtained 95.0% inter-annotator agreement, indicating high reliability. A feedback instance is labeled accepted (i.e., preferred) if its revised code executes successfully, passes all unit tests, and achieves an average quality score of at least 4.0 across the seven metrics; otherwise, it is labeled rejected. Using these criteria, we constructed a corpus of 6,284 accepted–rejected pairs. To prevent data leakage, we split at the level of source submissions: all pairs derived from the same source script (including all its augmented variants) were assigned to the same subset. We then partitioned these source groups into training (85%), validation (5%), and test (10%) subsets.

max width=

4.3 DPO-f+ Learning↩︎

DPO learning provides a reward-free mechanism for aligning models with human preferences [12]. It operates by comparing the relative likelihoods of preferred (\(y^+\)) and rejected (\(y^-\)) responses under a policy \(\pi_\theta\) and a frozen reference model \(\pi_{\text{ref}}\). Although effective, standard DPO treats all preference pairs equally and ignores preference margins, diffusing learning across noisy signals and leading to slower training and less reliable feedback. To improve, we augmented DPO with an auxiliary reward signal and a fixed-weight integration scheme.

4.3.1 Policy Loss.↩︎

We train the policy with a DPO-style pairwise log–sigmoid objective that contrasts policy and reference margins, with an explicit KL penalty to keep the policy close to a frozen reference: \[\begin{align} \mathcal{L}_{\text{policy}} &= \mathbb{E}_{(x,y^+,y^-)}\Big[ - \log \sigma\!\Big( \underbrace{s_\pi(x,y^+)-s_\pi(x,y^-)}_{\text{policy margin}}\\ &- \underbrace{s_{\text{ref}}(x,y^+)-s_{\text{ref}}(x,y^-)}_{\text{reference margin}} \Big)\Big] \\ &\quad + \gamma\,\mathrm{KL}\!\big(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\big), \end{align}\] where \(x\) is the prompt; \(y^+\) and \(y^-\) are the preferred and rejected feedback; \(s_\pi(x,y)=\log \pi_\theta(y\mid x)\) and \(s_{\mathrm{ref}}(x,y)=\log \pi_{\mathrm{ref}}(y\mid x)\) are sequence log-scores (i.e., sum of token log-probabilities); \(\sigma(\cdot)\) denotes the logistic function; it maps a real-valued margin to \((0,1)\), which we interpret as the probability that \(y^+\) is preferred to \(y^-\); \(\beta>0\) adjusts how sharply the loss responds to margin differences; and \(\gamma>0\) sets the weight of the KL term that keeps the policy near the reference. This loss increases the policy’s preference margin relative to the reference model while the KL term regularizes drift toward the reference.

4.3.2 Reward Loss.↩︎

To model preference margins explicitly, we train a lightweight reward model \(r_{\phi}(x,y)\) with a logistic pairwise objective. The reward model maps each prompt–feedback pair \((x,y)\) to a single scalar score, is trained offline on the same preference pairs, and is frozen before policy optimization:

\[\begin{align} \mathcal{L}_{\text{reward}} &= \mathbb{E}_{(x,y^+,y^-)}\Big[ - \log \sigma\!\Big( \underbrace{r_{\phi}(x,y^+) - r_{\phi}(x,y^-)}_{\text{reward margin}} \Big)\Big] \;\; \end{align}\] where \(x\) is the prompt, \(y^+\)/\(y^-\) are the preferred/rejected feedback, with the logistic function \(\sigma\) converts the score difference between the preferred and rejected responses into a probability that encodes the preference. When the preferred response doesn’t clearly outperform the rejected one, the loss and its gradients are largest, which raises the preferred score and lowers the rejected one to make the gap clear. The learned reward provides a scalar, graded estimate of preference strength, which we next use to inform policy updates so decisive wins influence training more than near-ties.

4.3.3 DPO-f+ Loss.↩︎

We couple preference learning with an auxiliary reward by forming a reward-augmented score and applying a DPO-style margin objective against a frozen reference. Let \[\begin{align} s_c(x,y)=s_{\pi}(x,y)+\lambda\,\tilde{r}_{\phi}(x,y), \end{align}\] where \(\tilde{r}_{\phi}\) is a scaled reward score and \(\lambda\!\ge\!0\) is a fixed scalar hyperparameter controlling the contribution of the reward term. The combined training loss is: \[\begin{align} \mathcal{L}_{\text{DPO-f+}} &= \mathbb{E}_{(x,y^+,y^-)}\Big[ -\log \sigma\!\Big( \underbrace{s_c(x,y^+)-s_c(x,y^-)}_{\text{combined margin}}\\ &-\underbrace{(s_{\text{ref}}(x,y^+)-s_{\text{ref}}(x,y^-))}_{\text{reference margin}} \Big)\Big] \\ &\quad + \gamma\,\mathrm{KL}\!\big(\pi_\theta(\cdot\mid x)\,\|\,\pi_{\text{ref}}(\cdot\mid x)\big). \end{align}\] where we replace the standard DPO score with a combined score \(s_c(x,y)=s_\pi(x,y)+\lambda\,\tilde{r}_\phi(x,y)\), so the loss rewards the model when the combined margin for the preferred response exceeds the reference model’s margin. The added reward margin \(\lambda\,[\tilde{r}_\phi(x,y^+)-\tilde{r}_\phi(x,y^-)]\) provides a graded signal: clear wins trigger larger updates, while near-ties trigger smaller ones, reducing noise and focusing learning on actionable feedback. During this stage, the reward \(\tilde{r}_\phi\) is frozen, so only the policy \(\pi_\theta\) is updated; the reward simply shapes the DPO updates. This keeps DPO’s reference anchoring, improves sample efficiency by emphasizing high-confidence preferences, and avoids instability from jointly updating reward and policy.

4.3.4 Experiment Setup↩︎

We fine-tuned two base models (Qwen2.5-1.5B-Instruct, CodeLlama-7B-Instruct) on paired preference data using two NVIDIA A6000 GPUs. Pairs were split 85/5/10 into train/validation/test. We used AdamW (lr \(5{\times}10^{-6}\), betas \((0.9,0.999)\), weight decay \(0.1\)), a cosine schedule with \(3\%\) warmup, bf16 precision, gradient checkpointing, and gradient clipping at \(1.0\). LoRA (\(r{=}16,\;\alpha{=}32,\\ \text{dropout}{=}0.05\)) was applied to the attention projections. Training used an effective batch size of 64 via gradient accumulation (micro-batch size 4 \(\times\) 2 GPUs \(\times\) accumulation 8 \(=64\)), and we used a max sequence length of 1024 tokens. For DPO we used the default inverse temperature and an explicit forward-KL regularizer with \(\gamma\in\{0,\,0.02\}\). Models were trained for up to 3 epochs with early stopping and model selection based on the same validation criterion. For DPO-f+, the reward weight \(\lambda\) was treated as a fixed training hyperparameter and selected on the validation set; it is not data-dependent at training or inference time.

4.4 Framework Evaluation↩︎

4.4.1 Sampling.↩︎

The framework’s practical performance was evaluated on a newly constructed evaluation set that balanced original and augmented code. We compiled this set by first randomly drawing a 50% sample from the original test scripts. The other half was generated using the same augmentation procedures described in Section 4.2. After de-duplication and a compilability check to ensure code uniqueness and validity, the final evaluation set consisted of 200 C++ code snippets.

4.4.2 Inference Setup.↩︎

We evaluated the baseline, DPO, and our aligned DPO-f+ under identical decoding settings using a standardized C++ code-review prompt. Consistent with Section 4.2, the prompt elicits natural-language feedback on the given snippet and requests a corrected version derived from that feedback. For each snippet, each model generated \(K{=}5\) candidates from the same prompt with sampling enabled (temperature=0.7, top_p=0.95, max_new_tokens=1024), yielding \(1{,}000\) feedback instances per model. Unlike dataset construction (Section 4.2), we explicitly override provider defaults and fix these decoding parameters for all models. For a fair comparison, all models were run with the same prompt and identical decoding settings. Consequently, each output contains both the feedback and the corresponding corrected code, enabling the assessment of feedback alignment and feedback accuracy, which will be further discussed in the following.

4.4.3 Feedback Evaluation.↩︎

We evaluated the generated responses across two dimensions: (1) feedback accuracy, which measures the technical correctness of the implied code revisions; and (2) feedback alignment, which assesses the quality of the natural language feedback using the metrics detailed in Section 4.1. This dual-faceted approach ensures that the feedback is both technically effective and well-aligned with developer requirements.
Feedback Accuracy Evaluation. Following the most common practices [23], [51], we assessed the code produced from each feedback instance for executability (i.e., whether it runs without error) and Pass@k, the estimated probability that at least one of the \(k\) independent samples passes the task’s test suite. The tests are adapted from the instructional materials of our institution:

• TwoSum. A compact parameterized suite checks that the function returns distinct, in-bounds indices that sum to the target and leaves the input array unchanged. Cases include duplicates, multiple valid pairs, negatives/zeros, and minimal sizes, and infeasible instances must throw std::invalid_argument.

• MinStack. Mixed operation sequences (including duplicates, negatives, and zeros) verify the last-in, first-out behavior and that getMin maintains the running minimum across pushes and pops (including plateaus). Empty-stack operations must throw std::out_of_range.

• TicTacToe. Alternating legal moves on a \(3\times3\) board validate win detection across rows, columns, and both diagonals. Non-terminal states and full-board draws must return 0. Invalid actions (occupied cells or out-of-bounds) must throw std::invalid_
argument and leave the board unchanged.

To complement these automated metrics, we conducted a targeted human expert study to assess feedback-only correctness. We sampled 50 outputs per model (Baseline, DPO, and DPO-f+) and presented two expert annotators with the buggy code and the natural language feedback only (omitting the model-generated patch). Each item was evaluated on two criteria: diagnosis correctness (i.e., identifying the root cause) and fix validity (i.e., proposing a plausible resolution). We define the success rate as the proportion of instances satisfying both criteria. To ensure rigor, the annotators co-rated an initial calibration set of 30 samples, achieving an inter-rater agreement of 98.1%.

Feedback Alignment Evaluation. We assessed alignment using two complementary LLM-as-a-judge procedures, following established paradigms for software engineering tasks [50]. To promote consistent judgments, we standardized the judge prompts, enforced a fixed output schema with an automated retry mechanism, and randomized item order to reduce position and prompt bias. The prompts used in the judging process are shown in Figure 3. We also conducted a human study with 200 developers to evaluate their satisfaction with the generated feedback.
(1) Metric-based scoring: We employed GPT-4 to rate each feedback item against the seven metrics in Table ¿tbl:tab:feedback-metrics? using a 1–5 scale. Scoring was performed at a temperature of 0.0 across three independent runs. We report the per-criterion score as the mean of these replicates, with the overall G-Eval score representing the mean across all metrics.
(2) Direct comparison: As a robustness check, DeepSeek-V3 was used to produce deterministic pairwise judgments. Given the original code and two candidate feedback in randomized order, the judge selected A, B, or a Tie. Results are reported as win, loss, and tie rates for the Baseline and DPO against DPO-f+.
(3) Human study: We conducted a large-scale human study (\(n=200\)) to assess feedback alignment from the perspective of real developers. The study has been reviewed and approved by our local Institutional Review Board. Recruitment criteria required participants to be at least 18 years old, fluent in English, and have basic knowledge of C++. We collected preference judgments through a survey on Prolific², in which participants viewed buggy code along with three feedback responses generated by the Baseline, DPO, and DPO-f+ models. Each participant evaluated 20 randomly sampled code-feedback sets and selected the response they preferred, yielding 4,000 judgments across 200 unique items, each evaluated an average of 20 times.

4.4.4 Generalizability↩︎

To assess robustness and external validity beyond novice programming tasks, we evaluate our models on SWE-bench Lite, a curated subset of SWE-bench that comprises more challenging and diverse real-world GitHub issue–fix pairs. We maintain the experimental pipeline described previously, adapting the prompts and alignment metrics to a professional software engineering context. To construct preference pairs for training, we randomly sampled 300 issues from the full SWE-bench dataset, ensuring no overlap with SWE-bench Lite to maintain strictly disjoint training and evaluation sets. Evaluation results are reported using the official SWE-bench Lite execution-based evaluator. Because the base models lack native retrieval capabilities for repository-scale codebases, we employ a fixed retrieval-and-analysis assistant (Claude 3.5 Sonnet) across all experiments, following the methodology of prior work [52]. The assistant performs two specific roles: (i) re-ranking BM25 candidates and summarizing repository metadata, and (ii) condensing execution traces to isolate failure modes. Crucially, the assistant does not generate or edit code, nor does it participate in the evaluation of correctness. All natural-language feedback and patches are generated exclusively by the model under test. We enforce a fixed attempt budget, identical seeds, and uniform retrieval prompts to ensure that any observed performance gains are attributable to our fine-tuning approach rather than the retrieval process.

5 Results↩︎

We present an analysis of the framework’s performance, examining its efficacy on both novice-level programming tasks and complex software engineering tasks for more experienced developers.

5.1 RQ1: Does DPO-f+ improve the alignment of code-repair feedback for novice programming instruction?↩︎

5.1.1 Preference Accuracy↩︎

To evaluate the model alignment, we report preference accuracy: the fraction of test pairs for which the model assigns a higher likelihood to the preferred response than to the rejected one, given the same prompt, computed over response tokens only. Results summarized in Table 1 show consistent gains from DPO-f+ over both baseline and standard DPO. On novice programming tasks with Qwen2.5-1.5B-Instruct, the baseline preference accuracy is 0.4511. Applying standard DPO yields a modest increase to 0.4766 (+2.55 pp). DPO-f+ produces a larger gain, reaching 0.8184, which is +36.73 pp over the baseline and +34.18 pp over DPO. A similar pattern holds for CodeLlama-7B-Instruct: DPO raises the baseline from 0.5892 to 0.6212 (+3.20 pp), and DPO-f+ achieves 0.8831 (+29.39 pp over baseline; +26.19 pp over DPO). While preference accuracy effectively reflects construct-level alignment with our metric, this metric can be susceptible to overfitting to specific metric cues. To complement this evaluation, we next analyzed the quality and alignment of the actual feedback generated by the models in practice.

5.1.2 Feedback Accuracy.↩︎

We selected the CodeLlama-7B-Instruct-based DPO-f+ to evaluate the accuracy of generated feedback, given its higher preference accuracy. We evaluated the feedback accuracy through code executability and Pass@k (see Section 4.4). As summarized in Table 2, executability rises from 27.9% (baseline) to 30.1% (DPO; +2.2 pp over baseline) and to 36.9% with DPO-f+ (+9.0 pp over baseline; +6.8 pp over DPO). Moreover, we also observed substantial and consistent gains across all Pass@k metrics. Specifically, Pass@1 increases from 0.0844 (baseline) to 0.1415 (DPO-f+; +5.71 pp). Pass@3 more than doubles, from 0.1091 to 0.3250 (+21.59 pp), and Pass@5 rises from 0.2323 to 0.4151 (+18.28 pp). DPO-f+ also outperforms standard DPO across the board, with absolute gains of +3.30 pp, +10.33 pp, and +14.82 pp for Pass@1, Pass@3, and Pass@5, respectively. The larger gains at higher \(k\) suggest broader improvements across the candidate set rather than isolated wins, indicating that preference-based alignment translates into more reliable repair outcomes. In addition, to validate feedback-only correctness (see Section 4.4), two authors manually annotated the feedback and observed success rates (i.e., the proportion of feedback that both identifies the root cause and proposes a plausible fix) of 0.50 for the baseline, 0.57 for DPO, and 0.66 for DPO-f+.

5pt

5.1.3 Feedback Alignment.↩︎

(1) LLM-as-a-Judge. To quantify alignment in practice, we assessed the generated feedback using seven metrics (see Section 4.1). Table 3 reports per-metric scores (1–5; higher is better), the overall G-Eval, and the direct pairwise comparison. On the novice programming task, DPO-f+ leads every metric (ranging from 3.16 to 4.23) and attains the highest G-Eval of 3.79 on average (Baseline 3.09; DPO 3.18), yielding an absolute improvement of +0.70 (+22.7% relative improvement) over Baseline and +0.61 (+19.2% relative improvement) over DPO. The direct pairwise comparisons further corroborate these trends: against DPO-f+, Baseline wins 39.62% of pairs (loses 59.93%, ties 0.46%), and DPO wins 43.18% (loses 56.45%, ties 0.37%), indicating that DPO-f+ prevails in most direct comparisons.
(2) Human Study. We recruited 200 developers via Prolific: 155 identified as male, 43 as female, and 2 as non-binary. Participants had a mean age of 29.7 years and an average of 3.1 years of C++ experience. Although the generated feedback is primarily intended for novice programmers performing simpler tasks, we included participants with a range of experience levels, while emphasizing less experienced developers, to reduce the risk that limited novice expertise would bias the evaluation. Self-reported expertise levels were beginner (90), intermediate (64), advanced (29), and expert (17). Preference for DPO-f+ was observed across all expertise levels. In particular, expert developers preferred DPO-f+ in 73.0% of comparisons, followed by beginners (71.5%), advanced participants (64.3%), and intermediate participants (61.6%). Overall, 120 of 200 developers selected DPO-f+ as their preferred model, which was significantly above chance (exact binomial test against 50%: \(p=0.0057\)). Across all three-way comparison sets, developers selected DPO-f+ most often (54.3%), compared with DPO (23.1%) and the baseline (22.6%); this difference was significant (\(\chi^2(2)=791.43\), \(p<0.001\)).

2pt

5.2 RQ2: Does DPO-f+ maintain strong performance across more challenging and diverse code-repair tasks?↩︎

5.2.1 Preference Accuracy.↩︎

Having established strong performance on novice tasks, we next assessed generalization to more challenging settings aimed at experienced developers. We reused the same data construction, training, and evaluation pipeline, making minor adjustments to the seven metric definitions and prompts to better reflect expert needs (see Section 4.1).

Table 1 reports preference accuracy for Qwen2.5-1.5B-Instruct and CodeLlama-7B-Instruct under DPO and DPO-f+ training. On SWE-bench Lite, Qwen2.5-1.5B-Instruct rises from 0.5200 (baseline) to 0.5853 with DPO (+6.53 pp) and to 0.8055 with DPO-f+ (+28.55 pp over baseline; +22.02 pp over DPO). For CodeLlama-7B-Instruct, DPO improves the baseline’s 0.5790 to 0.6550 (+7.60 pp), while DPO-f+ attains 0.8456 (+26.66 pp over baseline; +19.06 pp over DPO). Overall, DPO-f+ outperforms both the baseline and standard DPO, with the largest absolute gains on more challenging tasks for experienced developers.

5.2.2 Feedback Accuracy.↩︎

We next assessed alignment performance in a downstream setting using SWE-bench Lite. Following the official protocol, we measured feedback accuracy via the effectiveness of the patches (i.e., whether they can resolve the corresponding issue) produced by each model. An instance is counted as Resolved only if (i) the patch applies cleanly, (ii) makes all Fail-to-Pass tests pass, and (iii) has no Pass-to-Pass tests regress. Consistent with prior work, we report the number of Resolved instances over the full evaluation split as the primary outcome, and we additionally report Completed for transparency, where Completed denotes the number of instances that successfully finished the evaluation procedure (irrespective of being Resolved). For a fair comparison, we used the same prompt across models to produce patch feedback and evaluated generated patches on the same 300 instances under the official Docker-based harness, restricting evaluation to one prediction (i.e., feedback) per instance.

Under this protocol, our DPO-f+ model resolves 6.67% of tasks (20/300), compared with 5.00% (15/300) for standard DPO and 2.00% (6/300) for the baseline, yielding absolute gains of +1.67 pp over DPO and +4.67 pp over the baseline. While the absolute rate is constrained by the 7B base model’s capacity, DPO-f+ nonetheless delivers a clear, effective improvement in patch quality over standard DPO under identical retrieval support.

5.2.3 Feedback Alignment.↩︎

We conducted a further evaluation of alignment for model-generated natural-language feedback under more challenging task conditions. Consistent with the novice-task results, DPO-f+ attains the top score on all seven metrics as reported in Table 3 and the highest G-Eval mean (3.85), a relative improvement of 2.39% over the baseline (3.76) and 1.32% over DPO (3.80). Direct pairwise comparisons corroborate these trends: against DPO-f+, the baseline only wins 13.64% of pairs (loses 56.57%, ties 29.79%), and DPO wins 35.92% (loses 52.39%, ties 11.69%), indicating that DPO-f+ prevails in most direct comparisons on this benchmark. We do not conduct human evaluation on this benchmark, as each task requires deep repository-level context and domain expertise that would make it infeasible for crowd-sourced participants to reliably assess feedback quality.

6 Limitations↩︎

6.1 Data Construction and Optimization↩︎

Our novice-task setting used a small but authentic dataset: 53 novice submissions across three introductory C++ tasks, yielding 83 original code snippets containing bugs written by novice developers. We retained real novice-written code to preserve realistic error patterns, but this dataset does not cover the full range of novice mistakes. We therefore applied code augmentation [48] to expand the source pool, while noting that these transformations mostly preserve existing bug semantics rather than introducing substantially new failure modes. To test transfer beyond this narrow instructional setting, we additionally evaluated the framework on more complex tasks.

For efficiency, we used LoRA rather than full-parameter fine-tuning and only explored 1.5B- and 7B-scale models. As a result, our novice-task and SWE-bench Lite results are not directly comparable to state-of-the-art systems built with larger datasets, stronger backbones, or more expensive training. Our findings should therefore be interpreted as evidence that the framework improves feedback alignment in a constrained educational setting, not as a claim of broad generalization.

Finally, we compared DPO-f+ mainly against a baseline and standard DPO, rather than stronger RL-based alternatives. We therefore do not claim optimality in exploration, sample efficiency, or robustness to distribution shift.

6.2 Alignment Evaluation↩︎

Beyond data and training constraints, our evaluation combines LLM-as-a-judge with a human study. We use LLM-based judging because it is practical, scalable, reproducible, and applicable to both novice tasks and the more complex SWE-Bench-Lite setting. Still, we treat it as a proxy rather than a substitute for real developer evaluation, since it may not fully capture how developers perceive, trust, or act on feedback in practice, nor does it directly measure effects on code comprehension. Prior work suggests that LLM-as-a-judge can achieve human-level performance on software engineering tasks under similar protocols [49]. To reduce model-specific bias, we use two complementary protocols (metric-based and pairwise), a cross-family judge ensemble, anonymized and randomized pairwise presentation, deterministic decoding, score aggregation across judges, and manual auditing of a subset of judgments, yielding about 95% inter-annotator agreement on 100 items. To address this limitation, we also conducted a human study on novice tasks, in which feedback could be evaluated realistically by recruited participants. We did not extend the human study to SWE-Bench-Lite, as those tasks are substantially more complex and would require a much larger pool of qualified participants to assess reliably at scale.

6.3 Generalizability↩︎

The evaluation of DPO-f+ covers novice programming tasks and the more challenging SWE-Bench-Lite, approximating novice-oriented and experienced-oriented settings in C++ and Python. We treat the novice–experienced split as a coarse first step toward richer developer profiling (e.g., role, domain familiarity, and accessibility needs), which we leave to future work. This scope, while representative, excludes several prominent ecosystems (e.g., Java and JavaScript) and domain-specific languages (e.g., SQL and shell). Future work could broaden language coverage by incorporating diverse open-source corpora and establishing cross-language benchmarks to assess the framework’s generalizability across programming paradigms.

Additionally, our assessment focuses on code repair and does not address API migration, test generation/repair, documentation updates, etc. Future work could extend the framework to these activities by developing task-specific datasets and metrics, enabling a more comprehensive evaluation of applicability across the software development lifecycle.

7 Discussion↩︎

7.1 Personalized LLMs for Code Comprehension Support↩︎

We proposed a practical framework to better align LLM-generated feedback with developers’ preferences and needs. For evaluation, we moved beyond objective accuracy and introduced customizable yet scalable quality dimensions (e.g., Conciseness, Technical Quality, Explainability, Understandability, Completeness, Actionability, and Contextual Relevance), providing a comprehensive view of natural-language feedback for code-repair tasks to help developers understand the generated content better. Framing evaluation with structured metrics (rather than open-ended preference) yields consistent, comparable scores across prompts, languages, and tasks while maintaining predictable evaluation cost. When complemented with light human validation on sampled items, we observed strong agreement between automated and human judgments.

The framework supports profile-specific objective shaping by weighting metric dimensions to match context (e.g., assigning greater weight to Explainability for novices and to Completeness in safety-critical settings). Emphasizing specific dimensions for specific goals encourages the model to produce clearer rationales, guidance, and contextual cues that align with developers’ needs, helping them reduce the cognitive load when dealing with the generated content and localize faults more quickly. Thus, this targeted shaping of feedback may yield measurable gains on code comprehension tasks. Future work could conduct human studies to further validate this.

Finally, since the metrics and data construction can be adjusted quickly and easily, the framework is well-suited to limited resource settings (e.g., CS education), enabling efficient feedback generation, rapid what-if evaluations of prompts/models, and scalable monitoring when human graders or labeled data are limited. Nevertheless, since preferences vary across teams and expertise levels, we still recommend a human-in-the-loop process with periodic audits to incorporate real-world signals and refine the alignment procedure.

7.2 Failure Analysis↩︎

While DPO-f+ improves feedback alignment and repair outcomes on average (Section 5.1), we also observe systematic cases where it underperforms relative to the baseline or standard DPO. To make these limitations transparent, we summarize the dominant failure modes in regression cases, defined as instances where DPO-f+ performs worse than the comparison model under the evaluation protocol in Section 5.1.

(1) Over-verbosity and Signal Dilution. In about 19% of regression cases, DPO-f+ produces substantially longer feedback than the baseline, often adding elaborations that do not improve actionability. This can increase cognitive load and obscure the main corrective action, especially for localized bugs where a concise diagnosis and targeted fix are sufficient. We conjecture that preference-based optimization may sometimes reward fluency and completeness at the expense of information density.

(2) Lack of Explicit Diagnostic Framing. In 98% of regression cases, the model does not explicitly frame the problem in diagnostic terms (e.g., using words such as “bug,” “issue,” “root cause,” or “this fails because”). As a result, even plausible edits may underemphasize the linguistic cues that help developers recognize what is wrong before acting.

(3) Fix-Centricity without Causal Explanation. Relatedly, the model often prescribes edits without explaining the failure mechanism. That is, it states what to change but not why the original code fails or why the proposed fix works. This weakens the pedagogical value of the feedback and may reduce transfer to future problems. One mitigation is to explicitly separate Diagnosis and Fix during training and evaluation, for example by requiring a brief causal diagnosis followed by a minimal grounded fix.

(4) Absence of Code-Specific Anchors. In 9% of regression cases, the feedback lacks code-specific identifiers, such as variable names, function signatures, or references to relevant lines or blocks. Without such anchors, suggestions are harder to map to the code and easier to misapply. Strengthening anchoring requirements in the alignment metric is a promising direction for improving grounding.

Therefore, DPO-f+ is preferred overall, but its remaining weakness is that it can still produce feedback that is fluent and helpful-sounding without being diagnostic, grounded, and concise enough for real code understanding.

7.3 Future Applications and Integration↩︎

Our failure analysis suggests that effective deployment of DPO-f+ requires application-specific constraints to reduce regression modes. In particular, four needs emerge: (i) verbosity control to prevent signal dilution; (ii) explicit diagnostic framing to state the cause before prescribing edits; (iii) structured separation between diagnosis and fix to preserve pedagogical value; and (iv) code-specific anchoring to ground suggestions in the source. These findings motivate future work that combines refined alignment metrics with interface affordances, such as sectioned outputs and length budgets, to improve robustness across settings.

(1) Open-Source Development. DPO-f+ could be adapted as a review assistant that improves review throughput without increasing triage burden. By mining preference pairs from repository traces, such as pull-request discussions and CI outcomes, the framework could be optimized for Actionability and Contextual Relevance. To reduce the failure modes identified in Section 7.2, such an assistant should enforce lightweight constraints, including identifier mentions and bounded-length fix plans, so suggestions remain concrete and transferable across unfamiliar codebases.

(2) Computer Science Education. In educational settings, DPO-f+ could serve as a scalable TA-style assistant. Unlike standard LLMs that may encourage “answer-dumping,” the DPO-f+ objective can be weighted toward Understandability and Explainability. A diagnosis-first structure that states what is wrong and why it fails may better target student misconceptions and reduce the cognitive load of interpreting raw code patches. Evaluation in this setting should extend beyond execution correctness to include learning outcomes, such as time-to-first-pass and performance on follow-up tasks.

(3) Collaborative Software Teams. For professional teams, DPO-f+ could be aligned with internal architectural standards and historical review practices. To reduce misapplication in complex systems, future versions should prioritize Completeness and Contextual Relevance, requiring references to affected modules or APIs and concrete code anchors when possible. Incorporating such grounding requirements into the alignment metric may help organizations use AI feedback to reduce defects while maintaining coding standards.

Across these domains, moving from surface-level fluency to structured, diagnostic feedback is an important step toward making AI-driven code review more reliable and pedagogically sound.

8 Conclusion↩︎

Large language models are increasingly used for software engineering tasks such as code repair. However, developers still struggle to interpret model outputs, which limits effective human–AI teaming. Prior work has largely focused on optimizing generated code while giving less attention to the natural-language feedback that can support code comprehension and iterative improvement. To address this gap, we present DPO-f+, a framework that aligns code-repair feedback with developer needs through profile-specific criteria to better support code comprehension. The framework (1) formalizes profile- and domain-specific metrics for feedback alignment, (2) automatically constructs preference pairs from code-repair tasks, (3) fine-tunes models using Direct Preference Optimization (DPO) with a reward, and (4) evaluates performance at scale through an automated protocol. Empirically, DPO-f+ outperforms both the baseline and standard DPO in generated-code accuracy and feedback alignment. On novice programming tasks, it improves Pass@1 by 5.71 percentage points over the baseline and 3.30 points over standard DPO. On more advanced tasks (SWE-bench Lite), it improves the issue-resolution rate by 4.67 points over the baseline and 1.67 points over DPO. Across both settings, it also achieves the highest overall feedback-alignment scores. We further conducted a human study (\(n=200\)), in which developers reported greater satisfaction with feedback generated by DPO-f+. By aligning feedback to developer needs, DPO-f+ reframes LLM-assisted repair from one-shot output delivery into a collaborative sense-making process, offering a practical path toward improved code comprehension and human–AI teaming in software engineering.

9 Data Availability↩︎

Our source data and scripts are available at https://doi.org/10.5281/zenodo.19337617.

References↩︎

---

1. https://anonymous.4open.science/r/dpo_f-D1D0↩︎

2. https://www.prolific.com/↩︎