1 Introduction↩︎

Large language models are increasingly deployed as interactive agents [1]–[6]. In many of these systems, the model first generates a chain of thought and then emits the next action [7]–[9]. These agents often benefit from generating reasoning traces before acting, but prior work shows that verbalized CoT is not always faithful to the computation that determines the final output [10]–[13]. If the model already has a strong prompt-conditioned guess about what to do, the reasoning trace may mainly justify that guess rather than revise it. In long-context agent settings, the prompt itself contains task instructions, interaction history, and environment feedback that often narrow down the next action, making such a shortcut especially plausible.

This raises a training question: when we supervise agents with CoT, what is actually improving? One possibility is that training mainly strengthens reasoning-based revision, so the model becomes better at changing its action after generating a reasoning trace. Another is that training also strengthens direct action prediction from the prompt. To distinguish these possibilities, we compare two decoding modes throughout the paper: prompt actions, which are predicted directly from the prompt without generating CoT, and CoT actions, which are predicted after generating a step-by-step reasoning trace.

Better performance after CoT generation does not by itself show that the generated reasoning has become more effective at changing the action. It may also reflect a model that has become better at predicting the action directly from the prompt. In other words, final task performance alone cannot tell us whether training is increasing the contribution of generated reasoning or simply making the final action more predictable from information that is already present in the prompt. This is why we need a more fine-grained diagnosis than overall success rate alone.

Our analysis combines offline step-level comparisons, online evaluation on unseen tasks, and conflicting-trace tests, allowing us to track both how action prediction changes during training and which source of information exerts more control over the final action at test time. We complement these behavioral results with attention and gradient analyses that help explain why the prompt may have a structural advantage during action prediction, and we use this diagnosis to motivate a simple reduced action-supervision intervention.

We find that prompt-action quality improves substantially during training. Although CoT reasoning still helps, the relative advantage of CoT actions over prompt actions does not widen as training progresses. This suggests that CoT training improves performance not only through better CoT-based revision, but also through stronger direct action prediction from the prompt. We further find that when the reasoning trace conflicts with the prompt, later checkpoints are harder to overturn with conflicting traces, suggesting that the final action becomes increasingly anchored to the prompt.

Although training requires the model to produce a reasoning trace before the final action, the model still becomes better at predicting the action directly from the prompt. To better understand why, we find that in agent settings the prompt is often much longer than the reasoning trace and accounts for most of the attention and gradient mass. As a result, supervision on action tokens can disproportionately strengthen prompt-based action prediction.

Motivated by this diagnosis, we selectively reduce action-token supervision on a subset of training examples and find that this intervention improves OOD generalization.

Our contributions are: (1) a diagnostic framework that compares prompt actions with CoT actions to measure how well the final action can be predicted from the prompt alone; (2) behavioral and mechanistic evidence that CoT supervision can improve direct action prediction from the prompt in long-context agents; (3) reduced action supervision, a training intervention that weakens direct final-action reinforcement on part of the data and improves OOD generalization.

2 Related Work↩︎

2.0.0.1 Faithfulness of chain-of-thought reasoning.

Recent work shows that verbalized CoT is not always tightly coupled to the computation that determines a model’s output [11]–[13]. [14] show that CoT explanations can omit the features that actually drive a decision, while [15] find similar unfaithful reasoning patterns on natural prompts, including cases of implicit post-hoc rationalization. [10] propose separating reasoning and answer generation models to improve faithfulness, and [16] study whether answer-only fine-tuning preserves or degrades CoT quality and faithfulness. We differ from this line of work in two ways: we study the model’s training dynamics, and we focus on reasoning data rather than answer-only training data in long-context agent settings, where the prompt itself is a rich source of action-predictive information.

2.0.0.2 Agent reasoning and generalization.

A central theme in agent research is the gap between strong in-domain performance and weak out-of-domain generalization. Prior work reports mixed conclusions on whether reinforcement learning closes this gap. [17], [18] argue that RL generalizes better than supervised fine-tuning, whereas [19] find only limited gains. Other studies aim to improve generalization through self-reflection and future prediction [20], online data collection and iterative fine-tuning [21], meta-reasoning prompts [22], or multi-agent interaction [23], [24]. Recent critical evaluations of ReAct-style prompting further suggest that some apparent gains may depend heavily on prompt construction and exemplar-query similarity rather than on interleaved reasoning itself [25], [26]. We complement this literature by relating agent generalization to shortcut learning: in long-context prompts, the prompt itself may provide a strong shortcut to the next action, so improved performance need not imply a stronger reasoning-mediated revision process.

3 Setup and Diagnostic Framework↩︎

We study next-action prediction in long-context agent environments where the prompt \(p\) contains a task description, recent observations, and interaction history. The model generates a response consisting of a reasoning trace followed by a final action. We conduct experiments on three environments: ALFWorld [27] for embodied household tasks, ScienceWorld [2] for interactive scientific reasoning, and BFCL [28] for function-calling and tool-use evaluation. We train models using both supervised fine-tuning and reinforcement learning.

We define two decoding modes. The CoT action is obtained by instructing the model to generate step-by-step reasoning before emitting an action. The prompt action is obtained by prefilling the response with the <action> tag, which forces the model to generate the action directly without producing any intermediate reasoning. Because the prompt action bypasses the reasoning trace entirely, the gap between prompt-action quality and CoT-action quality measures how much of the final decision depends on reasoning versus how much is already recoverable from the prompt alone.

This distinction supports a three-part diagnosis:

1. We track whether training improves prompt-action quality alongside CoT-action quality (§4.1, §4.2).

2. We perturb the reasoning trace with a conflicting one and test whether the model preserves the original prompt-implied action or follows the substituted trace (§4.3).

3. We use attention and gradient analyses to explain why the prompt can receive more action-time signal than the generated CoT (§5).

4 Training Makes Actions More Predictable from the Prompt↩︎

We first ask whether CoT training also improves direct action prediction from the prompt alone. The key question in this section is not whether CoT helps at all, but where the observed training gains are expressed. If training mainly improves reasoning-based action revision, then the advantage of CoT actions over prompt actions should widen over time. If training also strengthens a direct prompt-to-action pathway, then prompt-action quality should improve in parallel, and the CoT-versus-prompt gap may remain relatively stable. We therefore examine this question in three steps: training dynamics, online evaluation on unseen tasks, and conflicting-trace tests.

4.1 Prompt and CoT Action Accuracy Improve in Parallel↩︎

During supervised fine-tuning, we split the data into a training set and a validation set. We train the model on the training set and then evaluate whether prompt actions and CoT actions match the reference action on the validation set, testing whether the model has learned to predict the action directly from the prompt. As shown in Figure 1, these two accuracies rise in parallel over training: when the model gets better at producing the correct final action after generating CoT, it also gets better at predicting the action directly from the prompt. This parallel trend is the first signal that the gains from CoT training are not expressed only after the reasoning trace is produced. Instead, training appears to improve how much of the final action is already recoverable from the prompt itself. In other words, the model does not merely become better at arriving at the right action after reasoning; it also becomes better at predicting that action before any reasoning is generated.

We also evaluate consistency between prompt actions and CoT actions. As shown in Figure 2, the prompt/CoT-consistency curves generally rise over training. The model therefore reaches the same action more often with and without the full reasoning trace. This rising consistency sharpens the interpretation of the accuracy trends above. The important pattern is not only that both decoding modes improve, but that they converge more often to the same decision. As training proceeds, the final action increasingly looks like something that can already be inferred from the prompt, with the reasoning trace serving more as a continuation of that decision process than as a separate source of revision.

This section establishes the basic behavioral pattern: training improves CoT-conditioned action generation and prompt-only action prediction together. We return in §5 to why action supervision may favor prompt tokens in these long-context settings. At a high level, Figure 1 and Figure 2 tell a consistent story. Better checkpoints are not only more accurate after generating CoT; they are also better at recovering the same action directly from the prompt, and more likely to agree across the two decoding modes. This makes direct action prediction from the prompt an increasingly important part of the model’s observed behavior over training.

4.2 Direct Prompt-vs-CoT Comparisons in Online Evaluation↩︎

We next test whether the same pattern persists during online evaluation on unseen tasks.

We split the tasks into in-domain train/test sets and OOD test sets; the detailed split is shown in the Appendix. This test matters because training-set predictability from the prompt alone could reflect memorization or distribution-specific coupling between prompt and action.

First, we evaluate performance when the agent interacts with the environment using the prompt action rather than the CoT action. As shown in Figure 3 (a), the gap between the two decoding modes remains flat over training. We show more results in the Appendix. This shows that the extra benefit of generating CoT before acting does not expand with training. The improvement therefore does not appear primarily as a growing CoT-specific advantage; instead, a substantial part of it remains recoverable from the prompt alone even on unseen tasks.

Second, we compare prompt actions and CoT actions in the same decision context by asking GPT-5.4 which action is better. Figure 3 (b) shows that the win rate of CoT actions over prompt actions is stable across checkpoints. This action-level comparison reduces the attribution problem in trajectory-level outcomes: even if later recovery or mistakes make full-trajectory success noisy, the judge still asks whether the generated CoT yields a locally better action than the prompt-based action at the current decision point. We also manually checked GPT-5.4’s judgments on 100 examples, and they aligned with human judgments in 93% of cases. For OOD tasks, CoT-revised actions likewise show only limited improvement over prompt actions. Together with the trajectory-level evaluation, this action-level view shows that the flat-gap pattern is not simply an artifact of long-horizon execution noise. Even when we compare the two candidate actions in the same local decision context, later checkpoints do not show a widening advantage for CoT-conditioned generation. The main training trend is therefore one of parallel improvement rather than increasing separation between the two pathways.

CoT actions retain a consistent advantage over prompt actions throughout training, so reasoning still helps. However, that advantage does not grow: both trajectory-level execution and local action-level judgments show the same flat-gap pattern in online evaluation on unseen tasks. This combination of results is central for the rest of the paper. CoT remains useful, but its relative benefit stays roughly constant even as overall performance rises. The natural reading is that training is making more of the correct action predictable from the prompt, rather than increasingly shifting the final decision through generated reasoning alone.

4.3 Does Final Generation Increasingly Rely on the Shortcut?↩︎

The previous sections compare prompt and CoT actions under standard decoding. We now use a stronger probe: when the supplied reasoning trace conflicts with the prompt, which one does the model follow?

To test this, we follow prior work [12], [13] and replace the original reasoning trace for a prompt with a randomly chosen trace that supports a different final action. We then truncate the inserted trace at different ratios and ask whether the resulting action agrees with the original prompt-based action or with the action implied by the substituted trace. Figure 4 shows the result in ALFWorld: across checkpoints, later models preserve the original prompt-based action more often, even under conflicting traces. This broad upward trend, replicated across BFCL and ScienceWorld (Appendix 9), shows that later checkpoints are harder to overturn with substituted reasoning traces. Similar checkpoint-wise trends under reinforcement learning and for the Llama model are reported in Appendix Figures 14 and 15. A related pattern is that out-of-domain settings are often more prompt-anchored than in-domain ones, which is consistent with the view that when reasoning is less reliable or harder to use, the model falls back more heavily on the prompt-based pathway. Appendix 9 shows more results.

5 Why the Prompt Has a Structural Advantage↩︎

We next ask why action supervision in long-context agents may favor prompt tokens over generated CoT tokens.

In autoregressive generation, action tokens are predicted from the full available context, which includes both the prompt and the previously generated rationale. In agent settings, however, the prompt is often substantially longer than the CoT because it contains task instructions, interaction history, and current environment feedback. This creates a structural asymmetry: when the model generates the final action, information from the prompt can occupy a larger share of the available evidence stream.

Figure 5 also supports this asymmetry. Nearly 80% of the attention mass during action generation falls on prompt tokens rather than CoT tokens. While part of this effect is mechanical (the prompt is typically much longer than the CoT), prompt tokens still dominate even among the highest-attention positions, where more than half come from the prompt. Action prediction is therefore empirically prompt-dominated in these settings, beyond what prompt length alone would imply.

Our goal here is to measure aggregate pathway dominance at action prediction time, not per-token informativeness. We therefore report the total attention mass and gradient share assigned to prompt tokens versus CoT tokens, rather than length-normalized averages. In agent settings, prompt length is itself part of the structural asymmetry: a longer prompt can cause the prompt-based pathway to receive a larger share of the action-supervision signal.

Along the value path of self-attention, the gradient contribution from context token \(i\) to action-token supervision is weighted by its attention weight. In particular, if \(\mathcal{L}_t\) denotes the loss on action position \(t\) and \(o_t\) is the corresponding attention output, then \[\frac{\partial \mathcal{L}_t}{\partial v_i} = \alpha_{t,i}\,\delta_t, \qquad \delta_t = \frac{\partial \mathcal{L}_t}{\partial o_t}.\] Thus, if prompt tokens consistently receive a larger share of the attention mass during action generation, they also receive a larger share of the optimization signal along this pathway. A full derivation is provided in Appendix 10.1. This implies an optimization bias along the value path: when prompt features are already predictive of the next action, gradient descent may preferentially strengthen the direct prompt-to-action pathway, leaving the rationale to function as a weaker refinement signal rather than the main route to the decision.

To complement this forward-pass view, we also compute a direct gradient-based diagnostic of where the action-supervision signal flows during backpropagation. As shown in Table 4, prompt tokens receive a larger share of the gradient signal than CoT tokens.

Together, the attention and gradient evidence provide a plausible optimization-level explanation for the behavioral patterns above. Agent prompts are substantially longer than the reasoning traces they surround, and prompt tokens receive the dominant share of attention and gradient signal during action prediction. We also conduct experiments on relatively short-prompt datasets, where prompt/CoT-action consistency does not increase much over training (Figure 9).

6 A Training Intervention Motivated by These Patterns↩︎

If action-token supervision mainly reinforces direct action prediction from the prompt, then reducing that supervision on part of the training data should weaken this effect.

We therefore remove action supervision on a subset of training examples. Concretely, for a randomly selected fraction \(k\%\) of training samples, we mask the loss on the final action tokens and optimize only the CoT span. For the remaining samples, we keep the standard supervision unchanged. Let \(M(t)\) be an indicator that equals \(1\) when token position \(t\) belongs to the CoT span and \(0\) otherwise. For masked examples, we train with \[\mathcal{L}_{\mathrm{cot}} = - \sum_{t=1}^{|y|} M(t)\log p_{\theta}(y_t \mid p, y_{<t}).\] This objective removes the direct optimization pressure on final action tokens on the selected samples. Because the remaining samples still preserve standard action supervision, the model continues to receive explicit training signals for action generation. We also keep supervision on the <action> tag so that the model still learns when to transition from reasoning to action generation. Reduced action supervision therefore does not remove reasoning supervision; it selectively weakens how often the final action loss can directly reward shortcut learning from the prompt.

We also apply this intervention to reinforcement learning. There, we randomly select \(k\%\) of training samples, mask the loss on the final action tokens, and optimize only the CoT span, while keeping the advantage estimates unchanged.

6.1 Evaluation↩︎

This evaluation serves two goals: to test whether reduced action supervision preserves or improves standard task performance, and to test whether it changes the prompt-versus-CoT diagnostics in the predicted direction.

We evaluate the intervention in three agent settings: ALFWorld for embodied household tasks [27], ScienceWorld for interactive scientific reasoning [2], and BFCL for function-calling and tool-use evaluation [28]. Across these environments, we compare standard CoT-supervised training with reduced action supervision and summarize checkpoint-wise evidence about shortcut reliance and rationale sensitivity under our diagnostics. We train for 5 epochs in SFT and 200 steps in RL, use learning rates of 1e-5 for SFT and 1e-6 for RL, and run each setting three times, reporting mean performance. We set \(k=0.3\) for both SFT and RL; an ablation over \(k\) is shown in Table 3. More detailed settings are provided in the Appendix.

Figure 6 shows that reduced action supervision improves performance in most environment-model combinations. Figure 7 reports the CoT-minus-prompt action gap under reduced action supervision. Relative to the flatter gap pattern under vanilla training, the gap is generally larger in ALFWorld and ScienceWorld, with more mixed changes in BFCL. We treat this shift as supportive, though not by itself definitive, evidence that the intervention reduces reliance on prompt-only action prediction and leaves more room for CoT-based action revision. The corresponding comparison under GRPO is shown in Table 2 and Figure 18.

The variation across environments is also informative. Gains are largest in ALFWorld and ScienceWorld and more modest in BFCL, suggesting that the intervention helps most when a stronger prompt-based shortcut would otherwise dominate action selection. Together with the performance improvements in Figure 6, this pattern is consistent with the shortcut diagnosis: weakening the prompt-to-action shortcut can improve task performance while leaving more room for CoT-based revision. The more mixed pattern in BFCL is informative rather than purely negative. Compared with ALFWorld and ScienceWorld, BFCL uses more structured prompts and a more constrained action space, which may leave less room for the intervention to change the balance between prompt-based prediction and CoT-based revision.

We also compare our method with other baselines, including DPO [29] and FRODO [10]. And following [10], we use response randomly selected from other prompt response pairs as the rejected response. As shown in Table 1, although our method with SFT underperforms on some tasks, it performs well when combined with DPO. It is also worth noting that FRODO [10] uses DPO for training.

Finally, as shown in Figure 8, under the intervention, consistency with the prompt action under perturbed reasoning traces does not increase over training.

This result suggests that our method can effectively mitigate the prompt-action shortcut, and that the performance gain is mainly driven by CoT-conditioned action revision, which helps explain the improved OOD performance.

7 Conclusion↩︎

We study how CoT training changes the role of direct action prediction from the prompt in long-context language-model agents by comparing actions decoded directly from the prompt with actions decoded after a reasoning trace. We find that, training improves direct action prediction from the prompt. Later checkpoints are also harder to overturn with conflicting reasoning traces. These findings do not settle whether latent reasoning ability improves in an absolute sense, but they do show that, in the long-context agent settings we study, gains from standard CoT supervision should not be interpreted as evidence that CoT-based revision alone has become more important.

Limitations↩︎

Prompt actions serve only as a behavioral proxy for how much of the final action can be predicted from the prompt rather than a direct measurement of internal computation. In addition, some of our local action-quality judgments rely on GPT-5.4 as a judge, which may introduce evaluation bias.

This appendix is organized as follows. We first report experimental details and an ablation for the masking ratio. We then provide supplementary behavioral evidence for increasing shortcut dependence during training, followed by additional behavioral diagnostics and outcome-level analyses. Next, we present the mechanistic derivation and gradient-based diagnostic that support the main-text interpretation. We conclude with qualitative materials and the full prompts used in our experiments.

LLM usage↩︎

We use LLMs to help polish the paper and draft figures.

8 Experimental Details↩︎

This section collects the setup information needed to interpret the results in the main text and the Appendix. We first summarize the data split and training configuration, and then report the masking-ratio ablation used in the reduced action-supervision study.

8.1 Data Split↩︎

We conduct our experiments on three environments: ALFWorld, ScienceWorld, and BFCL. We use a maximum of 30 interaction steps for ALFWorld and ScienceWorld, and 15 for BFCL. For ALFWorld, following [22], we designate Cool & Place and Pick Two & Place as unseen tasks for OOD evaluation. For ScienceWorld, the final task type of each topic is reserved for OOD evaluation. For BFCL, we use the long-context setting for OOD evaluation.

8.2 Training Details↩︎

We train for 5 epochs in SFT and 200 steps in RL, using a learning rate of 1e-5 for SFT and 1e-6 for GRPO [30]. For SFT, we use a batch size of 16. For GRPO, we use a batch size of 32, and the group size is set to 8. To penalize outputs that did not adhere to the required format, we apply a reward penalty of -0.1. The KL regularization coefficient is set to 0.01. We use GPT-5.4 to collect trajectories and select successful trajectories for SFT.

8.3 Contrast with short-prompt settings.↩︎

To further evaluate our hypothesis, we also consider short-prompt settings (MATH, MedQA, and GPQA), where prompts are much shorter and do not carry rich interaction histories. As shown in Figure 9, the consistency between prompt actions and CoT actions in these short-prompt datasets does not increase over training, which is also consistent with previous results [10], [16]. This contrast supports the view that the structural length asymmetry specific to agent prompts is a key driver of the shortcut pattern we observe.

8.4 Masking-Ratio Ablation↩︎

Table 3 reports the performance of our reduced action-supervision objective under different masking ratios \(k\) for Qwen3-8B.

9 Conflicting-Trace Evidence for Reliance on the Prompt↩︎

We next present supplementary evidence that most directly probes the paper’s directional claim. By replacing the model’s original reasoning trace with a conflicting one, this section asks whether training makes final actions more or less anchored to the prompt.

The previous analysis suggests that training benefits direct action prediction from the prompt more strongly than cases that require revising the prompt-based decision. A natural question is whether this phenomenon also changes the extent to which the final action is anchored by the prompt relative to the generated trace. Previous work [10] finds that fine-tuning on CoT data can increase trace sensitivity in mathematical and QA tasks. However, our results suggest that, for agentic tasks, training may instead increase the degree to which final actions remain predictable from the prompt.

The core diagnostic logic in this section is simple. If training makes the final action increasingly sensitive to the supplied reasoning trace, then a conflicting substituted trace should overturn the original prompt-based action more often as training proceeds. If, instead, the observable gains are increasingly recoverable from the prompt, then the model should preserve the original prompt-based action more often even when the inserted reasoning trace supports a different answer.

To probe how the model determines the final action, we follow [12], [13] and perturb the model’s reasoning trace while measuring the resulting action. For a prompt \(p_1\) with its original reasoning trace \(t_1\), we replace the reasoning trace with a random trace \(t_2\) that leads to a different final action, and then measure the action generated from the mixed input \((p_1, t_2)\). We further truncate \(t_2\) at different ratios and track how often the resulting action agrees with the original prompt-based action versus the action associated with the substituted trace. We use prefilling so that the model treats the inserted thinking trace as part of its own response, and then append the token <action> to force direct action prediction without additional reasoning. Because the substituted trace conflicts with the original prompt, agreement with the original prompt-based action serves as a proxy for how strongly the model preserves its prompt-conditioned decision under conflicting evidence. This probe goes beyond comparing two decoding modes under normal generation. The earlier analyses show that prompt actions and CoT actions improve together; the conflicting-trace test asks which source of information exerts more control when the two are explicitly put in tension. It therefore provides a more direct view of how strongly the final action remains anchored to the prompt.

Figure 4 further indicates that agreement with the original prompt-based action increases over training checkpoints. This trend is consistent with increasing prompt anchoring of the final action relative to the influence of the substituted reasoning trace. This interpretation is especially natural because, if training systematically improved the relative value of CoT-based action revision, one would expect the substituted trace to overturn the original prompt-based action more often, not less, as training proceeds. The overall pattern is therefore consistent with a setting in which the rationale has limited leverage over an action that is already largely determined by the prompt. This trend also fits naturally with the flat-gap results above. If later checkpoints already encode a stronger prompt-based guess about the next action, then conflicting reasoning traces should find it harder to move the final decision. The behavioral picture is therefore coherent across analyses: training improves CoT-conditioned performance, but it also makes the model increasingly likely to preserve the action that is recoverable from the prompt.

Figure 10 indicates that prompt-action self-consistency is positively related to prompt/CoT action consistency. This pattern suggests that when the model is more stable in its prompt-based action, the reasoning process has less room to revise it. When prompt-based behavior is less stable, reasoning may have more room to change the final action.

Taken together, these results are consistent with training increasingly favoring prompt-conditioned action generation under conflicting evidence, while also making the model more likely to base its decisions on prompt information. These effects may contribute to stronger overall performance, but heavier prompt reliance can also encourage overfitting and lead to weaker out-of-domain generalization. As shown in Figure 4, out-of-domain tasks exhibit stronger dependence on the prompt than in-domain tasks. However, Figure 10 indicates that prompt-action self-consistency on those out-of-domain tasks is similar to that on in-domain tasks. This pattern suggests that stronger dependence on the prompt out of domain does not necessarily arise from greater confidence in the prompt action alone. One possible interpretation is that reasoning on unfamiliar tasks is of lower quality, making the CoT less able to revise the prompt-based action and thereby contributing to poor generalization.

9.1 A Qualitative Example↩︎

We conclude this section with one representative example, adapted from the supplied experimental log, that illustrates how a coherent rationale can still yield an action that does not materially improve over the initial prompt-conditioned decision. In an ALFWorld task with the goal “put a cool tomato in the microwave,” the interaction history already records retrieving the tomato from the countertop. The generated rationale nevertheless emphasizes examining the tomato again to verify its temperature, even though the task context and recent observations already constrain the next action substantially. This example is consistent with the main-text diagnosis: a rationale may remain locally plausible while still introducing unnecessary or hallucinated intermediate justifications.

10 Mechanistic Analysis↩︎

We then move from behavioral patterns to a mechanistic account. The two subsections below formalize why information from the prompt can dominate action prediction and provide a gradient-based diagnostic that matches the main-text interpretation.

10.1 Value-Path Gradient Decomposition↩︎

This section gives the full derivation behind the main-text claim that larger attention mass on prompt tokens is associated with a larger share of the optimization signal along the value path of self-attention.

Consider one attention head at an action position \(t\), with \[q_t = W_Q h_t,\qquad k_i = W_K h_i,\qquad v_i = W_V h_i,\] \[\alpha_{t,i} = \mathrm{softmax}\!\left(\frac{q_t^\top k_i}{\sqrt{d}}\right), \qquad o_t = \sum_{i \le t}\alpha_{t,i} v_i.\] If \(\mathcal{L}_t = -\log p_\theta(a_t \mid x_{\le t})\) denotes the loss for the action token at position \(t\), then along the value path of self-attention, \[\frac{\partial \mathcal{L}_t}{\partial v_i} = \alpha_{t,i}\,\delta_t, \qquad \delta_t = \frac{\partial \mathcal{L}_t}{\partial o_t},\] and therefore \[\frac{\partial \mathcal{L}_t}{\partial W_V} = \sum_{i \le t}\alpha_{t,i}\,\delta_t\,h_i^\top.\] Partitioning the available context into prompt tokens \(P\) and CoT tokens \(C\) yields \[\frac{\partial \mathcal{L}_t}{\partial W_V} = \sum_{i \in P}\alpha_{t,i}\,\delta_t\,h_i^\top + \sum_{i \in C}\alpha_{t,i}\,\delta_t\,h_i^\top.\] This decomposition is the basis for the main-text interpretation: when prompt tokens systematically receive larger attention weights during action generation, they also receive a larger share of the optimization signal along this pathway.

10.2 Gradient-Based Diagnostic↩︎

To complement this forward-pass view, we also compute a direct gradient-based diagnostic of where the action-supervision signal flows during backpropagation; details are provided here. We run a forward pass on each sample and define an action-only loss \[\mathcal{L}_{\mathrm{act}} = - \sum_{t=1}^{|y|} M(t)\log p_{\theta}(y_t \mid x_{<t}),\] where \(M(t)=1\) only for positions in the final action span. This setup asks a narrow question: when the model is trained to predict the final action under teacher forcing, which part of the available context receives the stronger update signal?

For each transformer block \(\ell\), let \(h_i^{(\ell)}\) denote the hidden state at token position \(i\). After backpropagating \(\mathcal{L}_{\mathrm{act}}\), we record the per-token gradient norm \[g_i^{(\ell)} = \left\lVert \frac{\partial \mathcal{L}_{\mathrm{act}}}{\partial h_i^{(\ell)}} \right\rVert_2.\] Let \(P\) be the set of prompt tokens and \(C\) the set of CoT tokens. We measure the gradient share on prompt tokens \[R_{\mathrm{prompt}}^{(\ell)} = \frac{\sum_{i \in P} g_i^{(\ell)}}{\sum_{i \in P \cup C} g_i^{(\ell)}}.\]

As reported in Table 4, the action-only gradient is substantially more concentrated on prompt tokens than on reasoning tokens across all four models and three environments. This is consistent with a systematic optimization bias: standard action supervision drives updates predominantly through the prompt-based pathway rather than through the rationale.

Together, the attention and gradient evidence offer a structural explanation compatible with the main-text interpretation. Agent prompts are substantially longer than the reasoning traces they surround, and this length asymmetry translates into an optimization asymmetry: the prompt side receives the dominant share of attention and gradient signal during action prediction. Standard CoT supervision therefore appears to disproportionately strengthen the direct prompt-to-action pathway. This mechanistic pattern matches the behavioral diagnosis that later checkpoints make the final action more predictable from the prompt and more resistant to shortcut conflict.

11 Prompts↩︎

Finally, we provide the full prompts used for training and evaluation, including the environment prompts and the action-comparison prompt used for local judgments.

You are an expert agent operating in the ALFRED Embodied Environment.
Your task is to: {task_description}

Prior to this step, you have already taken {step_count} step(s).
Below are the most recent {history_length} observations and the
corresponding actions you took:
{action_history}

You are now at step {current_step} and your current observation is:
{current_observation}

Your admissible actions of the current situation are: [{admissible_actions}].

Now it's your turn to take an action.
You should first reason step-by-step based on the guideline about the
current situation. This reasoning process MUST be enclosed within
<thinking> </thinking> tags.
Once you've finished your reasoning, choose an admissible action for
the current step and present it within <action> </action> tags.

You are an expert agent operating in the ScienceWorld environment,
which is a text-based virtual environment centered around accomplishing
tasks from the elementary science curriculum.
Your current task is: {task_description}

Prior to this step, you have already taken {step_count} step(s).
Below are the most recent {history_length} observations and the
corresponding actions you took:
{action_history}

You are now at step {current_step} and your current observation is:
{current_observation}

Here are the actions you may take:
{action_descriptions}

Current available actions:
{available_actions}

Now it's your turn to take an action.
You should first reason step-by-step about the current situation.
This reasoning process MUST be enclosed within <thinking> </thinking>
tags.
Once you've finished your reasoning, choose an appropriate action for
the current step and present it within <action> </action> tags.

For BFCL, we directly adopt the prompt from [21].

You are a strict evaluator of Agent action quality.
Please judge which of the two actions is better based only on the given original user prompt.
The better action should better match the task goal, better fit the current observation, be more executable, be safer, and be more likely to advance task completion.
If the two actions are of similar quality, judge them as similar.

Original user prompt:
{user_prompt}

Candidate action A:
{cot_action}

Candidate action B:
{direct_action}

Please think step by step first, and then output your final answer inside <judge></judge> tags.
Inside the tag, output exactly one of the following three labels:
- A_better
- B_better
- similar

References↩︎