Regression Accumulation in Multi-Turn LLM Programming Conversations
July 02, 2026
Abstract
In LLM-assisted software development, coding is often iterative. In this paper, single-turn refers to settings where the LLM model responds in one request with one code suggestion, whereas multi-turn refers the model responds across multiple sequential requests within the same conversation. The regression accumulation refers to the later code suggestion break the requirements introduced in earlier turns. Reliability depends not only on satisfying the current turn, but also on preserving requirements in earlier turns. Yet, there are limited evaluations of the accumulation of regression in multi-turn coding conversations.
To address this gap, we conduct an empirical study of multi-turn regression in LLM programming. We construct 542 tasks from HumanEval+ and MBPP+ benchamrks, construct each task with an 8-turn requirement-evolution chain. We evaluate six LLM models on 26,016 turn instances (\(542 \times 6 \times 8\)). At each turn, we check if the current code suggestion still passes earlier benchmark tests, and we analyze 384 failure cases drawn from the failure population in our 26,016 turn instances to derive and validate a taxonomy of multi-turn regression bugs through independent four-annotator labeling. Our results show that regression accumulation appears across all six models, with 40% to 73% of tasks losing previously correct behavior over the full conversation. Final-turn quality is lower than initial-turn quality across models, and especially in turns adding input validation or broader input types. Manual analysis shows that Cross-Turn Conflict, where later code conflicts with earlier requirements, is the main failure class. We further find that Verification Gate, which checks new code against prior tests and triggers rollback and retry, is the only strategy that consistently improves all models, raising final-turn quality from 75.8% to 87.9% on DeepSeek-V3 and from 31.6% to 47.3% on Llama-3.1-8B.
These findings suggest that strong performance on single-turn tasks can overestimate reliability in multi-turn coding conversations. Future evaluation and tool design should test whether later code suggestions preserve requirements from earlier turns and should include Verification Gate mechanisms.
1 Introduction↩︎
Software programs are rarely developed in a single step. They evolve through repeated modifications that add features, broaden input handling, restructure interfaces, or improve performance. A central engineering risk in this process is regression, in which a change breaks behavior that previously worked correctly. Regression testing is the typical response to this risk [1]Both early and later software evolution research suggest that continual modification can increase system complexity and maintenance risk over time [2]–[4].
AI code-generation systems further highlight this maintenance issue. Developers increasingly use these systems in interactive coding conversations rather than relying only on one-shot prompts [5]. In a single-turn setting, the model answers one prompt with one code suggestion. In a multi-turn setting, the model continues the same conversation across several turns and produces a new code suggestion after each follow-up instruction. In such sessions, each turn may address a local requirement, but the current code suggestion must remain consistent with the code suggestions and requirements established in earlier turns [1], [2].
Current evaluation practice does not capture this risk well because it is misaligned with how these systems are actually used. In practice, developers increasingly rely on multi-turn coding conversations, where each new code suggestion is built on what earlier turns already established. By contrast, single-turn benchmarks such as HumanEval [6] and MBPP [7] ask whether a model can produce a correct code suggestion for one prompt. Recent multi-turn benchmarks [8]–[11] show that models are less likely to produce a code suggestion that satisfies the latest instruction once requirements are distributed across several turns in the same conversation. However, they still focus mainly on whether the code suggestion at the current turn satisfies the latest instruction. They do not directly test whether requirements established in earlier turns are still preserved after later changes. This is not a problem with coding tasks themselves. It is a limitation of what current evaluation practice chooses to measure. As a result, a model can appear successful at turn \(t\) while silently breaking requirements introduced at turns \(1\) to \(t-1\). This mismatch can lead us to overestimate reliability in realistic workflows and leave users to recover broken functionality through extra debugging and repair. We therefore need to understand how much regression accumulates across turns, what kinds of failures dominate it, and which interaction-level controls can reduce it.
We still lack an evaluation of whether later-turn code suggestions preserve requirements established in earlier turns. This paper addresses that gap by studying whether AI-generated code suggestions preserve requirements established in earlier turns. We construct 8-turn evolution chains and evaluate each code suggestion against the benchmark tests. We conduct this study on 542 tasks from HumanEval+ and MBPP+ using GPT-4o, DeepSeek-V3, Qwen2.5-Coder-32B, Qwen3-32B, DS-R1-Distill-32B, and Llama-3.1-8B, for a total of 26,016 turn instances (\(542 \times 6 \times 8\)). Our results show regression accumulation across all model families, with the steepest losses after error-handling and input-generalization requests. Our manual analysis shows that these failures more often reflect incompatibilities between new code suggestions and earlier-turn requirements than simple context loss. We also find that mitigation strategies such as Verification Gate, which checks each new code suggestion against earlier-turn tests and allows one retry from the last passing one, are effective when multi-turn coding conversations explicitly verify whether new code suggestions preserve earlier-turn requirements.
The main contributions from this work are:
We present a benchmark construction method and executable evaluation protocol for studying regression accumulation in multi-turn LLM programming. The protocol combines 542 HumanEval+ and MBPP+ tasks with 8-turn evolution chains and a fixed regression oracle derived from benchmark-provided tests.
We conduct a systematic empirical study across six LLMs and 26,016 turn instances to analyze how regression accumulation evolves across turns, model families, and requirement change types.
We develop a multi-turn bug taxonomy with eight multi-turn bug types in three classes, plus Baseline Failure for Turn 1, from manual analysis of 384 failure instances and four-annotator labeling.
We formulate and evaluate interaction-level mitigation strategies for multi-turn coding, including a verification-based rollback-and-retry design for code suggestions that introduce regression.
2 Related Work↩︎
2.1 Software Evolution and Regression Under Change↩︎
Regression under change has long been a central concern in software engineering. As systems evolve, new functionality must be integrated without breaking behavior that already works. Classic software evolution research explains why this risk persists. Lehman and Belady [2] argue that software subject to continual modification tends to accumulate complexity and quality risk. Maintenance research commonly distinguishes corrective, adaptive, perfective, and preventive changes [12]. Other work classifies the concrete edits developers make during software evolution [3]. Regression testing operationalizes this concern by checking whether existing behavior survives new modifications [1]. We adopt this lens for conversational code generation. In our setting, each turn acts as a maintenance step, and the central question is whether the generated update preserves prior behavior.
2.2 Interactive and Multi-Turn LLM Coding↩︎
Recent surveys and perspective papers position LLM-based development as a growing software engineering research area [13], [14]. Empirical work also shows that developers use LLM assistants through shared, multi-turn interactions in collaborative settings such as pull requests and issues [5]. These studies motivate evaluating coding assistants in iterative workflows rather than only through one-shot generation.
Recent benchmarks have begun to study multi-turn code generation directly. MT-Sec [8], CodeIF-Bench [9], CodeFlowBench [10], and CodeAssistBench [15] all evaluate code generation across sequences of instructions. Outside software tasks, Laban et al. [11] show that multi-turn performance declines across a broad range of domains. These studies establish that turn history matters, but most code benchmarks still score each turn against tests written only for the current request. That setup captures turn-level instruction following. It does not reveal whether later edits preserve prior behavior.
Our protocol instead keeps the benchmark’s original tests as a fixed regression oracle at every turn, making regression directly observable. A model can satisfy the latest instruction while still breaking behavior introduced earlier in the conversation. We build on HumanEval [6], MBPP [7], and their enhanced variants HumanEval+ and MBPP+ [16], because their validated tests support controlled longitudinal evaluation under repeated change.
2.3 Failure Modes and Repair in LLM-Generated Code↩︎
Studies of LLM code suggestions have cataloged common single-turn failure modes such as incorrect logic, missing edge cases, and API misuse [17]. A separate line of work investigates iterative repair through self-debugging, execution feedback, and learned self-correction [18]–[22]. These systems aim to improve one candidate solution through repeated repair attempts.
We study a different failure setting. The model is not revising the same task until it passes. It is responding to a sequence of distinct change requests whose effects must compose over time. This difference exposes bug patterns that do not appear in single-turn evaluation, including omission of earlier-turn requirements and failures caused by accumulated code changes. In our paper, prior work on long-conversation drift [23] motivates recap-based prompting that restates prior requirements, while one-step recoverability [24] motivates a rollback-and-retry strategy that recovers from code suggestions that introduce regression using the last known-good code.
3 Study Design↩︎
Our study was carried out in five stages. We begin with 542 Python seed tasks drawn from two established benchmarks: 164 Python function-level tasks from HumanEval+ and 378 Python function-level tasks from MBPP+. We include all tasks from both benchmarks and combine them into a single seed-task pool by applying the same fixed 8-turn prompt structure and benchmark-provided tests to every task. We then convert each task into a fixed 8-turn programming conversation. At Turn 1 (T1), the model is asked to produce an initial code suggestion from the benchmark’s original task prompt. For Turns 2 to 8 (T2 to T8), we define controlled requirement changes using six software-maintenance change types and instantiate them with fixed prompt templates, each designed to modify code suggestions from earlier turns while preserving previously established requirements. Next, we execute this conversation once for each task on each model. After each turn, we re-run the benchmark tests to measure whether requirements established in earlier turns are preserved. This process yields per-turn code suggestions, test outcomes, turn-by-turn regression results, and the failing code suggestions, failed tests, and error messages returned by Python execution or assertion failures that are used in the downstream analyses for RQ1 to RQ3. Figure 1 summarizes this workflow and data flow.
3.1 Task Construction↩︎
We source 542 Python seed tasks from two established benchmarks, HumanEval+ [16] (164 Python function-level tasks) and MBPP+ [16] (378 Python function-level tasks). Both benchmarks are widely used in LLM code generation research and provide rigorously validated test suites. We include all tasks regardless of complexity, using the complexity of the benchmark reference solution as a control variable rather than a selection criterion.
To account for differences in seed-task size, we use the lines of code (LOC) in the benchmark reference solution as a coarse structural proxy [25], [26]. We do not treat LOC alone as a complete measure of software complexity. Rather, we use it only to stratify these short function-level benchmark tasks by reference-solution size. Based on this measure, we group tasks into three strata: easy (1–3 lines, \(n\)=322), medium (4–8 lines, \(n\)=162), and hard (\(\geq\)9 lines, \(n\)=58). We also consider a complementary notion of task difficulty for the models themselves. Specifically, we use Turn 1 (T1) pass rate to capture how solvable each task is at the initial code-suggestion turn, so that we can analyze how initial solvability interacts with subsequent regression accumulation.
Each seed task is instantiated as the same fixed 8-turn conversation by filling a small set of benchmark-derived fields into the fixed turn templates introduced later in this section. These fields include the target function name
({entry_point}) and, for the restructuring turn, the derived class name ({class_name}). This deterministic instantiation keeps prompt construction consistent across tasks and avoids introducing additional variation from
task-specific prompt writing.
Regression Test Strategy. A key challenge arises at Turn 6 (T6) because, unlike earlier turns, it changes how the benchmark tests call the code, i.e., from standalone functions to class methods. To preserve the same
benchmark-provided tests across turns, we continue to use the benchmark’s original test suite throughout the conversation. For Turns 1 to 5 (T1 to T5), the tests are executed unchanged. For Turns 6 to 8 (T6 to T8), we deterministically rewrite only the
invocation sites by prepending object instantiation and replacing calls of the form f(args) with _obj.f(args). The assertions themselves are left unchanged. This design keeps the evaluation tied to the benchmark’s validated tests
while preventing T6 failures from being counted merely because the required refactoring changes the call interface.
3.2 Cross-Turn Dependency Design↩︎
Existing multi-turn benchmarks often decompose tasks into independent subtasks [8] or separately verifiable instructions [9]. Such designs are useful for measuring whether a model can satisfy the requirement stated at the current turn. However, they are less suitable for studying regression accumulation because a later turn can be solved without modifying or preserving code suggestions from earlier turns. In that setting, a model may succeed at the current turn even if it would fail to maintain earlier-turn requirements under continued code evolution. Our goal is different. We want later turns to modify code suggestions produced earlier in the same conversation and to preserve established requirements. This design choice is consistent with software maintenance settings in which later modifications depend on and interact with earlier ones [2]–[4], [27]. We therefore construct each subsequent turn so that it must be applied on top of the current code suggestion rather than as an independent new solution. For example, Turn 4 (T4) adds caching to the code suggestion that already includes Turn 2 (T2) validation and Turn 3 (T3) type generalization, and Turn 6 (T6) refactors the accumulated functionality from Turns 1 to 5 (T1 to T5) into a class while preserving the same requirements. Concretely, if T2 requires the code suggestion to reject invalid inputs and T3 requires it to accept string representations of numeric inputs, then a later turn such as T4 must add caching without removing either the validation introduced in T2 or the input conversion introduced in T3.
3.3 Requirement Change Types↩︎
Turn 1 is treated separately as the initial code-suggestion baseline. For Turns 2 to 8, we derive a pool of maintenance-like follow-up changes from prior software-maintenance, software-evolution, and source-code-change literature [3], [12], [28]. We therefore select six change types for benchmark construction based on three benchmark-design requirements. Each change type should represent a common follow-up modification, create explicit preservation obligations, and be instantiable consistently across many benchmark tasks. We do not claim that these six types exhaust all realistic maintenance turns. Rather, they provide a controlled subset for reproducible comparison across tasks and models.
Functional Extension: add new functionality while preserving requirements established in earlier turns [12], [28], [29].
Error Handling Enhancement: add validation and exception handling to existing code [30].
Input Generalization: extend the code suggestion to support additional input forms [12], [28].
Performance Optimization: improve efficiency while keeping outputs unchanged [31].
Interface Restructuring: refactor the program structure or interface while preserving requirements established in earlier turns [32].
Non-functional Augmentation: add cross-cutting support such as logging or call statistics such as counts of calls, cache hits, and errors [33].
All 542 tasks follow a fixed 8-turn sequence with one baseline turn and seven controlled requirement changes (Table 1). The ordering is designed so that later turns add new requirements while preserving those established earlier in the conversation, and the same sequence is applied to all tasks for comparability. Early turns establish the initial code suggestion and then add constraints through validation and broader input support. Turn 4 (T4) adds caching, Turn 5 (T5) adds new functionality, and Turn 6 (T6) refactors the accumulated code suggestion into a class. Turn 7 (T7) adds cross-cutting support such as logging or call statistics, and Turn 8 (T8) optimizes the class method’s core algorithm while preserving caching, validation, type conversion, logging, and call statistics. The sequence also intentionally repeats some broad change families at different stages. Turn 5 (T5) revisits functional extension after validation, type generalization, and caching have already been added, and Turn 8 (T8) revisits algorithm optimization after the codebase has been restructured and augmented. This design lets us compare how similar modification families behave under different levels of accumulated conversational state. We do not claim that this is the only realistic ordering of maintenance turns.
Fixed Prompt Templates. We do not ask any LLM to invent a different follow-up turn for each task. Instead, for each turn we use the same fixed prompt wording across tasks and fill in only a small number of task-specific fields copied
from the benchmark. These fields include the target function name (entry_point), the benchmark’s original task prompt at Turn 1 (T1), and the class name used for the refactoring turn at Turn 6 (T6), which is deterministically derived from
entry_point. Figure 2 summarizes the eight turn templates. For example, the Turn 2 (T2) template always asks the model to add input validation while preserving requirements established in earlier
turns. The only task-specific difference is the name of the function being modified. This design removes prompt variability as a confound while keeping the benchmark prompts recognizable and executable.
The templates provide a controlled and reproducible representation of common follow-up programming turns. We standardize the prompt structure and keep the wording stable, while allowing only benchmark-derived fields to vary. Some templates include
concrete code hooks such as _cache, clear_cache(), and get_stats(self). These hooks make each turn-specific change operationally precise, comparable across tasks, and directly checkable by the same downstream
regression tests.
As shown in Figure 2, Depends on identifies which previously introduced functionality must be understood and modified to complete the current turn. Preserve identifies which requirements established in earlier turns must remain intact after the change. Together, these two fields make the cross-turn dependency explicit. The resulting 8-turn sequence covers all six change types with restructuring at the midpoint, remains consistent with empirical evidence that developer-facing LLM interactions are typically short multi-turn conversations [5], and is long enough to expose regression accumulation across turns in the same conversation [11]. In our setting, this length is also sufficient to reveal the full regression trajectory. The full wording of all eight templates is provided in our replication package (https://anonymous.4open.science/r/multi-turn-llm-regression-E73E).
All 542 tasks receive the same template at each turn. This choice ensures a controlled comparison across tasks, even though some turns may be less natural for certain problems and some prompt details may be more explicit than in unconstrained developer conversations. We accept this trade-off because the benchmark is designed to study regression under a fixed and reproducible sequence of evolving requirements. Our goal is not to replay the full diversity of natural maintenance conversations. It is intended to expose models to a stable set of follow-up changes that systematically require modifications of existing code while preserving requirements established in earlier turns. Our regression tests therefore evaluate whether later code suggestions preserve requirements established in earlier turns, not whether every turn is equally natural for every task.
| Turn | Turn Type | Coupling Scope |
|---|---|---|
| T1 | Initial Code Suggestion | None (baseline) |
| T2 | Error Handling Enh. | T1 control flow |
| T3 | Input Generalization | T1 algorithm, T2 validation |
| T4 | Add Caching | T1–T3 consistency |
| T5 | Functional Extension | T1–T4 collectively |
| T6 | Interface Restructuring | T1–T5 all requirements |
| T7 | Non-functional Aug. | T2,T4,T6 cross-cutting |
| T8 | Algorithm Opt. | T1–T7 full codebase |
3.4 Taxonomy Development↩︎
To characterize failures that emerge only after requirements accumulate across turns, we develop a new taxonomy of multi-turn bugs and use it as the coding scheme for the annotation study. This taxonomy is not directly adopted from an existing bug taxonomy. Instead, it is derived bottom-up from our failure cases while being informed by qualitative coding and thematic-synthesis practice in software engineering [34]–[36]. We began with an initial calibration subset of 100 failure cases, selected to cover turn positions, change types, models, and failure severities, and asked the three coders (all co-authors) to independently read and open-code the observed failure patterns without pre-fixing the final labels. This bottom-up process produced a provisional pool of fine-grained failure codes rather than predefined high-level classes. The three coders then compared their codes, merged synonymous or overlapping concepts, split labels whose boundary covered more than one mechanism, and removed candidate codes that were too rare, too ambiguous, or not distinguishable enough to support stable annotation rules. We refined the remaining labels by writing explicit definitions, inclusion criteria, exclusion criteria, and illustrative examples for each candidate label. A label was retained in the final scheme only if it captured a recurring failure pattern, had a boundary that annotators could distinguish from neighboring labels, and could be supported by a concrete decision rule.
Across subsequent calibration rounds, the same three authors applied the evolving codebook to shared cases and used disagreements to refine the label set rather than to optimize agreement prematurely. These disagreements repeatedly traced back to three issues: label overlap, overly abstract definitions, and underspecified decision rules. We therefore added distinguishing rules and made the single-label decision rule explicit for boundary cases.
Once the concrete labels and their boundaries stabilized, we grouped them into higher-level classes based on their primary root cause. In other words, this higher-level organization was introduced after the label set had stabilized, not imposed before coding. We aligned this organization with established software engineering concepts such as regression bugs [1], code decay [2], and feature interaction [37]. We also compared the resulting scheme against existing bug taxonomies for AI-generated code suggestions [17] to identify multi-turn failure patterns that are not well captured by prior single-turn classifications. After the label set and class structure stabilized, we froze the codebook for the full annotation study.
The finalized scheme operationalizes annotation as a bug-classification form rather than a free-response questionnaire. After freezing the codebook, four of the study’s authors independently applied it to the 384 sampled failures. For each failed turn instance, annotators inspect the original task prompt, the turn number and change type, the current-turn prompt for the failing turn (for example, Turn \(n\) or T\(n\)), the last correct code suggestion from the previous successful turn (for example, Turn \(n\!-\!1\)), the failing code suggestion, the failed tests and error messages, and the regression rate. They then diagnose which previously established requirement or constraint has been broken and encode that diagnosis by assigning one bug label from the frozen codebook. We present these options to annotators as closed-choice labels grouped under the higher-level classes, with Baseline Failure reserved for Turn 1 cases. The high-level failure mechanism is inferred from the selected label’s class membership rather than being recorded through a separate question. After full annotation begins, the taxonomy is no longer substantially revised, so that agreement statistics remain interpretable. We apply the frozen codebook in the 384-sample annotation study described in Section 4.4. We present the taxonomy itself, its agreement statistics, and its empirical distribution in RQ2.
3.5 Mitigation Strategy Design↩︎
We design mitigation strategies for two risks that arise in multi-turn coding conversations. One is that the model may lose track of constraints established earlier in the conversation, which is consistent with prior observations of multi-turn quality loss and context drift [11], [23]. The other is that a new code suggestion may break requirements established in earlier turns when a fresh requirement is applied to already-evolving code, a recurring concern in change-impact and ripple-effect analysis [4], [27]. We therefore formulate two lightweight interaction-level strategies and evaluate them individually and in combination. Both strategies operate within the ongoing conversation rather than as post-hoc repair procedures, allowing us to assess whether they reduce regression as requirements accumulate.
3.5.1 S1: Snowball Recap.↩︎
Snowball Recap is designed for cases in which the model may lose track of requirements established earlier in the conversation. In our protocol, each new turn is appended to the existing dialogue, so later turns are issued under an increasingly long interaction history. The strategy therefore makes earlier constraints explicit again before each later turn instead of relying on the model to recover them from the growing conversation context alone. Following prior evidence that restating relevant context can reduce quality loss in long LLM interactions [11] and prior observations that long conversations can cause models to lose track of earlier requirements [23], this strategy prepends a recap prefix before each turn \(t \geq 2\). The prefix has three fixed parts: a header stating that all requirements from earlier turns must continue to hold, one summary line for each earlier turn in the format “T\(i\). [change type]: summary.”, and a fixed transition line before the prompt for the current turn. These summary lines are not generated by another model. Instead, they are produced deterministically from two metadata fields already stored in the benchmark instance, namely the turn index and the turn’s change type. For example, Turn 2 contributes the recap line “T2. [Error Handling Enhancement]: add input validation (TypeError/ValueError) to all public methods without rejecting previously valid inputs,” and Turn 3 contributes “T3. [Input Generalization]: accept string representations of numeric inputs and convert them, while preserving existing validation and logic.” For Turn 4, the final prompt is the fixed header, followed by the recap lines for Turns 2 and 3, and then the original Turn 4 prompt.
3.5.2 S2: Verification Gate.↩︎
Verification Gate is designed for cases in which a newly generated code suggestion immediately breaks requirements that were preserved at the previous turn, a familiar regression risk under successive change [1], [4], [27]. Unlike post-hoc repair methods that ask the model to fix a code suggestion only after it has already failed [18], [19], this strategy checks each turn immediately after generation and applies the one-step recoverability principle of Jain et al. [24]. After each turn \(t \geq 2\), we execute the same benchmark-provided tests used throughout the study and compare the current regression pass rate with that of the preceding turn. The gate is triggered only when \(RPR_t < RPR_{t-1}\) and a last passing code suggestion is available. In that case, the current code suggestion is rejected and the model receives a rollback prompt with three fixed elements: the number of failed regression tests, the full code of the last passing code suggestion, and the original requirement for the current turn. Concretely, the prompt tells the model that its previous attempt caused \(n_{\text{fail}}\) of \(n_{\text{total}}\) regression tests to fail, provides the last correct code block, and asks it to apply the current requirement to that version instead. The model is then given a single retry. We retain the retry only when it passes at least as many benchmark tests as the rejected code suggestion; otherwise, the current code suggestion is kept. For example, suppose Turn 3 has already added support for string representations of numeric inputs, and Turn 4 asks the model to add caching. If the first Turn 4 code suggestion adds caching but breaks the earlier input-conversion requirement, the gate rejects that code suggestion, restores the last passing Turn 3 version, and asks the model to generate Turn 4 again from that point. This design bounds intervention cost to at most one additional generation per turn while making the rollback criterion fully observable and reproducible.
3.5.3 S3: Combined (S1 + S2).↩︎
The combined strategy applies both recap and verification gating in the same conversation. It tests whether the two interventions address complementary failure mechanisms or largely overlap in practice.
All three strategies are model-agnostic. They require neither fine-tuning nor access to the internal model. Their overhead is also bounded. Snowball Recap adds prompt tokens only, and Verification Gate introduces at most one retry per turn.
4 Experimental Setup↩︎
4.1 Models Under Study↩︎
We evaluate six LLMs, including two commercial models (GPT-4o [38] and DeepSeek-V3 [39]) and four open-source models (Qwen2.5-Coder-32B [40], Qwen3-32B [41], DS-R1-Distill-32B [42], and Llama-3.1-8B [43]). This set allows us to compare commercial models and to compare a code-specialized model (i.e., Qwen2.5-Coder-32B) with general-purpose models (i.e., GPT-4o, DeepSeek-V3, Qwen3-32B, and Llama-3.1-8B). It also includes a reasoning-oriented model (i.e., DS-R1-Distill-32B) alongside standard-generation models. The commercial models are accessed through APIs, whereas the open-source models are evaluated under a shared local inference setup. Within the local group, Qwen2.5-Coder-32B and Qwen3-32B isolate code specialization at the same nominal size, DS-R1-Distill-32B adds a reasoning-oriented variant, and Llama-3.1-8B provides a smaller scale contrast.
All four open-source models are served locally on a single 32 GB NVIDIA RTX 5090 through vLLM [44] with AWQ 4-bit quantization and
max-model-len set to 32,768. GPT-4o and DeepSeek-V3 are accessed through their official OpenAI-compatible APIs.
| Model | Params | Type | Deploy | Comparison |
|---|---|---|---|---|
| GPT-4o | — | General | API | Commercial baseline |
| DeepSeek-V3 | 671B | General | API | MoE architecture |
| Qwen2.5-Coder-32B | 32B | Code | Local | Code-specialized |
| Qwen3-32B | 32B | General | Local | General (same family) |
| DS-R1-Distill-32B | 32B | Reason | Local | Reasoning model |
| Llama-3.1-8B | 8B | General | Local | Scale comparison |
4.2 Evaluation Protocol↩︎
For each task on each model, we run one 8-turn conversation consisting of an initial code-suggestion turn followed by seven controlled requirement changes. All six models are evaluated on the same 542 tasks, comprising 164 HumanEval+ tasks and 378 MBPP+ tasks, totaling 3,252 conversations and 26,016 turn instances.
Each conversation starts with a fixed system prompt that asks the model to return the full code suggestion in a single Python code block. At Turn 1, the user message is the benchmark prompt for the initial code suggestion. At each subsequent turn, the user message is the fixed benchmark request for that turn. At Turn \(t\), the input contains the system prompt, the user requests from Turns 1 to \(t\), and the model responses from all earlier turns.
After each response, we extract the code suggestion by taking the last fenced Python block. If no fenced block is present, we use the raw response text as the candidate program. We then evaluate that program against the regression oracle for the current turn. For function-level tasks, Turns 1 to 5 use the benchmark’s original tests, and Turns 6 to 8 use deterministically adapted versions of those same tests after the interface is restructured into a class. The adaptation prepends object instantiation and rewrites direct function calls into method calls, but leaves the original assertions unchanged. For each test, we build a temporary Python script containing the code suggestion and the test, run it in an isolated subprocess, and enforce a 15-second timeout. Assert-style tests are executed independently and in parallel, whereas unittest-style suites are executed in a single subprocess.
4.3 Metrics↩︎
We use one turn-level metric, Regression Pass Rate, which captures how much previously correct behavior is still preserved at each turn. The remaining three metrics summarize regression accumulation over the full conversation.
Regression Pass Rate (\(RPR_t\)). For a given task at turn \(t\), \(RPR_t = p_t / n_t\), where \(p_t\) is the number of benchmark tests that pass at turn \(t\) and \(n_t\) is the total number of benchmark tests for that task. Because the same benchmark tests are preserved across turns, a lower \(RPR_t\) directly indicates that more prior behavior has been broken.
Degradation (\(\Delta\)). \(\Delta = RPR_8 - RPR_1\). This metric captures the net change in quality from the first to the last turn.
Degradation Rate. This is the fraction of tasks for which \(\Delta < 0\). It measures how often quality declines over the conversation.
Survival Rate. This metric is computed only for tasks with \(RPR_1 = 100\%\). It reports the fraction of those initially solved tasks that still satisfy \(RPR_t \geq 50\%\) at turn \(t\).
4.4 Bug Annotation Protocol↩︎
To answer RQ2, we annotate a stratified sample of failure instances. We set the sample size to 384 using Cochran’s standard formula for proportion estimates with a 95% confidence level, a 5% margin of error, and the conservative choice \(p=0.5\) [45]. From all turn instances with \(RPR_t < 1.0\), we then stratify this sample across turn number and model so that the annotation set covers the main parts of the experimental space.
The four annotators are the first four authors. Each of the four independently labels every sampled case using the frozen codebook developed in Section 3.4. For each sample, annotators inspect the most recent earlier code suggestion that passed the regression oracle, the failing code suggestion at the current turn, the failed tests and error messages, and the current-turn request. They then assign one bug label from the predefined codebook as the root-cause label.
To reduce within-annotator inconsistency, each annotator labels every sample in three passes and keeps the within-annotator majority label as the final individual judgment. We then aggregate the four final judgments across annotators. Of the 384 samples, 277 are unanimous, and 107 are non-unanimous. For non-unanimous cases with a majority label, we keep the majority label. For the remaining 42 cases without a majority label, the senior researcher resolves the tie. We report inter-rater agreement in RQ2 using Fleiss’ \(\kappa\), because each of the 384 samples receives one nominal bug label from each of four annotators, and Fleiss’ \(\kappa\) provides a chance-corrected agreement measure for more than two raters, as in prior software engineering studies with multi-rater labeling [46], [47]. We compute it on the final annotator labels before majority aggregation so that it reflects how consistently the bug taxonomy can be applied independently.
5 Results↩︎
5.1 RQ1: How much previously correct behavior is lost as requirements accumulate across turns in LLM coding conversations?↩︎
We organize the results in three parts. Figure 3 shows the end-to-end trajectories from Turn 1 to Turn 8 for all six models. We report per-turn \(RPR\) values and identify where the largest losses occur in Table 3. Table 4 shows the same pattern after restricting each model to tasks it solved at Turn 1.
Finding 1. Regression accumulates for all six models, but its magnitude differs substantially by model family. As shown in Figure 3, every trajectory trends downward from Turn 1 to Turn 8. No model finishes at or above its initial regression pass rate. Across models, 40–73% of tasks lose previously correct behavior over the full conversation. DeepSeek-V3 and GPT-4o remain strongest at Turn 8, reaching 75.8% and 72.4% respectively. The 32B open models form a middle tier, with final-turn \(RPR\) values ranging from 51.7% to 57.9%. Llama-3.1-8B shows the largest loss and falls to 31.6%.
Finding 2. Strong Turn 1 performance does not necessarily translate into stronger multi-turn resilience. DeepSeek-V3 begins below GPT-4o at Turn 1, yet retains more quality by Turn 8. Llama-3.1-8B declines much more sharply than the larger commercial models and finishes far below the 32B open models at Turn 8. Reasoning specialization does not appear to help in this setting, since DS-R1-Distill-32B reaches 51.7% at Turn 8, whereas Qwen3-32B reaches 57.9%.
Finding 3. The per-turn results show that regression losses are front-loaded and are concentrated in a small number of change types. For five of the six models, the largest losses occur at Turn 2 or Turn 3. These error-handling and input-generalization turns are disruptive because the new validation or conversion logic must remain compatible with both the task’s original behavior and earlier-turn requirements.
| GPT-4o | DeepSeek | Qwen-Coder | Qwen3 | DS-R1 | Llama-8B | ||||||||
| T | Turn Type | \(RPR\) | LOC | \(RPR\) | LOC | \(RPR\) | LOC | \(RPR\) | LOC | \(RPR\) | LOC | \(RPR\) | LOC |
| 1 | Initial Impl. | 90.2 | 9 | 87.6 | 15 | 84.5 | 18 | 72.7 | 50 | 77.5 | 43 | 76.1 | 8 |
| 2 | Error Handling | 79.9 | 18 | 81.3 | 31 | 75.4 | 30 | 62.8 | 76 | 71.3 | 47 | 63.0 | 16 |
| 3 | Input General. | 75.9 | 28 | 80.4 | 49 | 63.5 | 42 | 49.5 | 99 | 24.7 | 175 | 49.5 | 25 |
| 4 | Add Caching | 75.1 | 42 | 78.4 | 74 | 62.4 | 59 | 59.3 | 88 | 47.6 | 124 | 47.8 | 37 |
| 5 | Func.Ext. | 75.1 | 53 | 78.5 | 103 | 62.3 | 79 | 63.8 | 94 | 51.4 | 124 | 47.8 | 49 |
| 6 | Interface Restr. | 74.9 | 55 | 78.5 | 107 | 61.8 | 83 | 66.5 | 89 | 56.8 | 122 | 47.4 | 50 |
| 7 | Non-func.Aug. | 74.8 | 79 | 78.3 | 147 | 58.8 | 94 | 59.7 | 135 | 49.9 | 163 | 47.7 | 67 |
| 8 | Algorithm Opt. | 72.4 | 85 | 75.8 | 163 | 55.0 | 87 | 57.9 | 139 | 51.7 | 166 | 31.6 | 71 |
3pt
6pt
No model shows sustained recovery once quality drops. Some local improvements appear, such as Qwen3 between T5 and T6, but they do not reverse the overall trend. In other words, once regression appears in the conversation, later turns rarely restore the lost behavior on their own.
Finding 4. Regression accumulation appears for both easier and harder seed tasks. For DeepSeek-V3, the hard group (\(\geq\)9 LOC in the benchmark reference solution) declines from 82.5% at Turn 1 to 62.7% at Turn 8, while the easy group (1 to 3 LOC) declines from 88.3% to 77.4%.
Finding 5. Regression accumulation remains substantial even among tasks solved perfectly at Turn 1 (i.e., tasks with \(RPR_{T1}=100\%\)). To isolate the regression signal from baseline inability, we perform this analysis separately for each model and retain only the tasks on which that model passes all benchmark tests at Turn 1. We then report the median task-level \(RPR_{T8}\), the percentage of tasks with \(RPR_{T8}<100\%\), and the percentage of tasks with \(RPR_{T8}=0\%\) in Table 4.
| Model | \(N\) | Med.\(RPR_{T8}\) | % With Regr. | % Collapsed |
|---|---|---|---|---|
| GPT-4o | 427 | 100.0% | 47.8% | 3.0% |
| DeepSeek-V3 | 414 | 100.0% | 45.9% | 1.9% |
| Qwen2.5-Coder-32B | 440 | 98.1% | 54.5% | 14.3% |
| Qwen3-32B | 397 | 98.2% | 50.9% | 10.6% |
| DS-R1-Distill-32B | 428 | 92.8% | 61.2% | 29.2% |
| Llama-3.1-8B | 333 | 16.1% | 81.7% | 34.2% |
The ordering of the models by median task-level \(RPR_{T8}\) is unchanged after restricting each model to tasks it solved at T1. On this restricted subset, the median task-level \(RPR_{T8}\) remains 100% for GPT-4o and DeepSeek-V3, while the 32B open models also retain high median \(RPR_{T8}\) values but still exhibit collapse rates ranging from 10.6% to 29.2%. The two weakest models, i.e., DS-R1-Distill-32B and Llama-3.1-8B, lose a substantial fraction of the tasks they solve at Turn 1 (T1) to total failure by Turn 8.
Regression accumulates for all six LLM models in multi-turn conversations and are concentrated in specific change types (i.e., error-handling and input-generalization requests). Subsequent changes to requirements in a multi-turn conversation can introduce regression even when the first-turn code suggestion is correct.
5.2 RQ2: What multi-turn-specific bugs emerge, and how are they distributed across taxonomy categories and requirement change types?↩︎
To answer RQ2, we analyze 384 manually labeled failure cases (Section 4.4). Table ¿tbl:tab:bug-taxonomy-dist? reports the taxonomy and its distribution. The reported counts and percentages use the finalized label per case, so each annotated case contributes to exactly one row in the distribution. Baseline Failure (BF) is reserved for Turn 1 failure and is therefore interpreted separately from multi-turn regression failures.
2.5pt
@llp3.35cmrr@ ID & Name & Definition & Count & %
BF & Baseline Failure & T1 implementation error, not a multi-turn bug & 48 & 12.5
CL-1 & Context Loss & Forgets or overwrites a constraint from a prior turn & 63 & 16.4
CL-3 & Partial Recall & Remembers function structure but misses internal details & 18 & 4.7
& 81 & 21.1
CC-1 & Semantic Collision & New logic contradicts prior logic, producing wrong results & 102 & 26.6
CC-2 & Interface Mismatch & New code breaks prior interfaces or signatures & 1 & 0.3
CC-3 & Integration Inconsistency & Individually correct components fail to integrate & 1 & 0.3
CC-4 & Over-Guarding & New validation rejects inputs the specification treats as valid & 110 & 28.6
& 214 & 55.7
AC-1\(^\star\) & Regression via Omission & Silently drops necessary prior code during regeneration & 3 & 0.8
AC-2\(^\star\) & Complexity Collapse & Code exceeds capacity, causing syntax errors or truncation & 38 & 9.9
& 41 & 10.7
6pt
Finding 6. Cross-Turn Conflict is the major source of manually observed regression failures. As shown in Table ¿tbl:tab:bug-taxonomy-dist?, Cross-Turn Conflict accounts for 55.7% of labeled cases, compared with 21.1% for Context-Loss and 10.7% for Accumulation. This indicates that later revisions more often break previously correct behavior through incompatibility with existing code than through simple forgetting.
Finding 7. The distribution is concentrated in two conflict types, and their locations across change requests are interpretable. Over-Guarding (CC-4) contributes 28.6% and Semantic Collision (CC-1) contributes 26.6% of all labeled failures. Together they account for 55.2% of all annotated cases. Within CC-4, more than half of the cases arise from error-handling and input-generalization requests (59/110, 53.6%), in which stricter checks must preserve previously valid behavior. Within CC-1, half of the cases involve functional extension and performance optimization (51/102, 50.0%), in which new logic is added on top of existing behavior.
Finding 8. Accumulation-specific failures are present, but less frequent. Complexity Collapse (AC-2) appears in 9.9% of annotated failures and Regression via Omission (AC-1) appears in 0.8%. These patterns are explicitly separated in our taxonomy because they are not well represented in prior single-turn LLM bug taxonomies [17]. Four annotators independently labeled all 384 samples, with unanimous agreement on 277 samples and Fleiss’ \(\kappa=0.799\) (substantial agreement [48]), which supports the reliability of the reported distribution.
Manual labels indicate that multi-turn regression is primarily an integration problem rather than solely a Context-Loss problem. Over-Guarding and Semantic Collision account for most cases, while accumulation-specific failures remain observable as code grows.
5.3 RQ3: To what extent can interaction-level mitigation strategies preserve prior behavior in multi-turn coding conversations?↩︎
To answer RQ3, we evaluate whether interaction-level controls can mitigate regression accumulation during the conversation itself. We compare Snowball Recap (S1), Verification Gate (S2), and their combination (S3). S1 prepends a deterministic recap of prior turns to each new request. In S2, after each turn, we run regression tests and trigger the gate only when \(RPR_t < RPR_{t-1}\) and a last passing code suggestion exists. We then reject the current code suggestion, send the model a rollback prompt containing the failed-test count, the full last passing code suggestion, and the current-turn requirement, and allow exactly one retry. We retain the retry only when it passes at least as many benchmark tests as the rejected code suggestion; otherwise, the current code suggestion is kept. S3 applies the recap (S1) and the verification gate (S2) together. Each strategy is run on all 542 tasks with the full 8-turn chain on DeepSeek-V3 and Llama-3.1-8B. Table 5 reports \(RPR_{T8}\), the fraction of tasks with regression, and average gate triggers per task. A gate trigger is counted whenever the current code suggestion lowers \(RPR\) relative to the previous turn.
| Model | Strategy | \(RPR_{T1}\) | \(RPR_{T8}\) | % With Regr. | Gates |
|---|---|---|---|---|---|
| DeepSeek | Baseline | 87.6 | 75.8 | 43.2% | — |
| S1 (Recap) | 91.2 | 80.1 | 36.7% | — | |
| S2 (Gate) | 91.1 | 87.9 | 12.5% | 1.08 | |
| S3 (Both) | 90.5 | 87.0 | 13.8% | 1.21 | |
| Llama-8B | Baseline | 76.1 | 31.6 | 72.9% | — |
| S1 (Recap) | 76.1 | 35.8 | 68.3% | — | |
| S2 (Gate) | 76.1 | 47.3 | 54.8% | 2.29 | |
| S3 (Both) | 76.1 | 42.3 | 59.4% | 2.33 |
Finding 9. Verification Gate is the only strategy that produces large and stable mitigation gains across both models. Under S2, DeepSeek-V3 improves from \(RPR_{T8}=75.8\%\) to 87.9% and reduces tasks with regression from 43.2% to 12.5%. Llama-3.1-8B improves from 31.6% to 47.3% and reduces tasks with regression from 72.9% to 54.8%. In contrast, Snowball Recap alone yields smaller improvements (DeepSeek-V3 80.1%, Llama-3.1-8B 35.8%), and combining recap with gating does not surpass S2 (S3 reaches 87.0% and 42.3%). This result indicates that online behavioral verification and rollback, rather than prompt restatement alone, is the main driver of mitigation in multi-turn coding.
Finding 10. Mitigation effectiveness depends on model scale and correction frequency. Under S2, the average number of triggered gates is 1.08 per task on DeepSeek-V3 and 2.29 on Llama-3.1-8B. The smaller model requires more than twice as many corrections and still produces lower-quality results. This suggests that Verification Gate removes a substantial share of regressions, but recovery after a failed turn remains harder for lower-capacity models.
Our results show that Verification Gate is the only strategy that consistently and substantially mitigates multi-turn regression accumulation across both models. Snowball Recap yields smaller gains, and combining the recap with the gate does not outperform the gate alone.
6 Discussion↩︎
6.1 Interpretation of the results↩︎
LLM-assisted programming is increasingly used in iterative settings where requirements evolve over multiple turns. In this setting, reliability is not only about satisfying the current request but also requires preserving requirements established in earlier turns. Our results show that this preservation requirement is where current models fail systematically. Across all six models, regression accumulates as conversations proceed. For five of the six models, the largest early losses appear at Turn 2 or Turn 3, where new validation or input-generalization logic must be integrated without breaking earlier-turn requirements. Stronger models degrade more gradually and finish at higher \(RPR\) levels, whereas weaker models lose quality more sharply as requirements accumulate across later turns. The dominant failures are cross-turn incompatibilities rather than isolated single-turn mistakes.
We combine fixed multi-turn evolution chains, regression-oracle tracking at every turn, and a manually validated taxonomy of multi-turn bugs. This design separates first-turn code-suggestion errors from failures introduced by later requirement changes. It also makes model-level and turn-level comparisons reproducible.
The mitigation results extend this contribution from diagnosis to intervention. Snowball Recap helps when the model loses track of requirements established in earlier turns by restating them before each new turn. Verification Gate addresses a different problem: a later code suggestion may still break earlier-turn requirements even when those requirements are present in the conversation context. In that case, prompt restatement alone is not enough. Verification Gate is therefore the only strategy that yields consistent gains in both stronger and weaker models. Snowball Recap yields smaller gains, and adding it to Verification Gate does not improve on Verification Gate alone. This pattern suggests that the main problem in our setting is not simply loss of earlier-turn context, but failure to preserve earlier-turn requirements.
These results also suggest several directions for future LLMs for code generation. This points to a need for models that treat later turns less as isolated generation steps and more as constrained code updates over an evolving code state. More specifically, future systems may need stronger mechanisms for preserving earlier-turn requirements, checking compatibility between new and existing logic, and deciding when to revise from a last known-good code suggestion instead of continuing from a faulty one. Our mitigation results further suggest that these capabilities should not be left entirely to prompting. They are likely to benefit from tighter integration between code generation, regression testing, and turn-level acceptance checks.
6.2 Implications↩︎
(1) Implication for evaluation design. Single-turn scores are insufficient to characterize reliability in conversational programming. Evaluation protocols should track preservation across turns and include regression-aware metrics, because turn-local correctness can coexist with failure to preserve requirements established in earlier turns. In practice, this means that benchmark design should pair evolving requirements with stable regression oracles, so that each new turn is evaluated against both new functionality and earlier-turn requirements. Without this structure, evaluations can overestimate model robustness in maintenance-like workflows.
(2) Implication for failure analysis. Cross-Turn Conflict is the major failure class, with Over-Guarding and Semantic Collision accounting for most manually labeled failures. This suggests that future analyses should focus on how new constraints and new logic interact with requirements established in earlier turns, rather than attributing multi-turn failure primarily to context loss. A direct consequence is that failure taxonomies and debugging workflows for multi-turn coding should represent interaction faults explicitly, for example, incompatible validation changes, conflicting assumptions between turns, or integration breakage after local edits.
(3) Implication for tool construction. Tooling should prioritize verification-aware interaction loops, including turn-level regression checks and rollback-based retry from a known-good state. Our results indicate that this pattern is more effective than recap-first prompting, especially when requirements continue to evolve over turns. For deployment, this implies that practical assistants should treat each new code suggestion as a candidate change that requires acceptance checks, rather than assuming that additional prompt context alone will preserve earlier-turn requirements.
6.3 Threats to Validity↩︎
Internal validity. Our benchmark uses a fixed 8-turn sequence and controlled prompt templates. This improves comparability but cannot represent all natural conversation patterns. We mitigate this by applying the same protocol to all models, adding stratified analyses, and reporting initially solved-task results. We also set the temperature to 0.0 for reproducibility, while acknowledging that stochastic decoding in deployment may behave differently.
Construct validity. We measure preservation with turn-level regression pass rate (\(RPR\)) against benchmark tests. This captures tested behavioral preservation but not all quality dimensions, such as readability or untested edge cases. To reduce measurement artifacts, we reuse benchmark-validated tests and apply deterministic adaptation for interface restructuring turns. For taxonomy labeling, we use a frozen codebook, independent four-annotator coding, adjudication, and substantial agreement (\(\kappa=0.799\)).
External validity. Our tasks are Python function-level problems from HumanEval+ and MBPP+, so effect sizes may not directly transfer to other languages, repository-scale maintenance, or human-in-the-loop settings. We partially broaden coverage by evaluating six models from both the commercial and open-source families. Further replications on larger repositories and additional languages are needed.
7 Conclusion↩︎
LLM-assisted programming is increasingly used in iterative software development workflows where requirements evolve over multiple turns. In this setting, reliability depends on whether newly generated code preserves behavior that was correct in earlier turns. This paper investigates the preservation problem through regression accumulation in multi-turn coding conversations.
We conduct a controlled empirical study on 542 tasks, six models, and 26,016 turn instances. We provide a reproducible benchmark protocol for turn-level regression evaluation, a taxonomy of multi-turn regression bugs from a four-annotator analysis of 384 stratified failures. We also evaluate interaction-level mitigation strategies based on recap and verification gate mechanisms.
Our findings show that regression accumulation appears across all evaluated models. We find that Cross-Turn Conflict is the major failure class, and that Verification Gate is the only mitigation strategy that yields consistent gains across both stronger and weaker models. Taken together, these results suggest that conversational code generation should be treated as an evolving maintenance process rather than a sequence of isolated prompt outcomes.
Data Availability↩︎
Our benchmark (542 tasks \(\times\) 8 turns), all experimental results across 6 models, bug annotations, and analysis scripts are available at https://anonymous.4open.science/r/multiturn-code-bugs.