1 Introduction↩︎

In recent years, Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across various domains, achieving breakthrough progress particularly in complex tasks such as mathematical problem solving, code generation, and logical reasoning [1]–[4]. To further elicit the deep reasoning capabilities of these models, Reinforcement Learning (RL) has been widely employed to fine-tune LLMs, optimizing the model policy by maximizing the expected reward of the generated content [5], [6]. Within this mainstream paradigm, researchers typically model reasoning tasks as a single-turn interaction process: the model receives a specific question or instruction as input and generates a Chain-of-Thought (CoT) or final answer in a single pass, after which the environment provides reward feedback based on the quality of the output [7], [8].

However, the aforementioned tasks are predominantly single-turn. In contrast, LLM agents frequently encounter multi-turn tasks characterized by outcome-only rewards [9]. When addressing such tasks, directly employing outcome rewards for reinforcement learning often leads to slow convergence due to the sparsity of reward signals and the absence of fine-grained feedback. Moreover, for algorithms like GRPO [10], the model may fail to learn successful trajectories that are not sampled during training, thereby limiting its performance.

To mitigate these challenges, several approaches, such as [11], [12], utilize manually crafted dense process rewards to alleviate the reward sparsity problem. By assigning an intermediate reward to each step of the trajectory, these methods provide richer reward signals, resulting in faster model convergence. However, methods relying on customized dense process rewards face a potential problem: if the dense rewards are not perfectly accurate, they may be inconsistent with the task’s outcome rewards. This inconsistency can bias the training direction and ultimately degrade the model’s final performance.

In this work, we introduce Reward-Swap Policy Optimization (RSPO), a general reinforcement learning framework that leverages dense process rewards to enhance training with sparse outcome rewards. As illustrated in Figure 1, in our cyclic framework, agent \(A\) trains on dense rewards for \(k\) steps to become Agent \(B\). Agent \(B\) explores and collects diverse trajectories, which are stored with outcome rewards to update Agent \(A\) off-policy. The key insight behind RSPO is that trajectories sampled by models trained solely with outcome rewards are often confined within a behavioral space dictated by those rewards. In contrast, training with dense process rewards enables the model to explore beyond this restricted space, thereby generating more diverse trajectories to enhance training. The overall pipeline of RSPO is depicted in Figure 2.

Extensive experiments conducted on ALFWorld, an embodied agent environment simulating household tasks, and WebShop, a web search interaction task, demonstrate that applying our RSPO method to various reinforcement learning algorithms—including GRPO [10], GiGPO [13], and PPO [14]—yields significant performance gains. In experiments utilizing Qwen2.5-1.5B-Instruct [15], RSPO achieves a 5.7% improvement over the GRPO baseline and an 8.6% improvement over the PPO baseline in the ALFWorld environment. Similarly, in the WebShop environment, RSPO yields a 5.5% gain over GRPO and a 12% gain over PPO. Averaged across all tasks, RSPO delivers performance improvements of 4.87% on GRPO, 3.6% on GiGPO, and 6.6% on PPO. These results demonstrate the effectiveness of RSPO in addressing such multi-turn tasks characterized by outcome rewards.

2 Related Work↩︎

2.1 Reinforcement Learning for LLMs↩︎

Reinforcement Learning (RL) is critical for optimizing LLMs[5], particularly for reasoning tasks[16], [17]. However, standard Actor-Critic methods like PPO[14] incur high computational costs due to the value network. To address this, recent "critic-free" methods estimate advantages using statistical baselines instead. Examples include GRPO[10], RLOO[18], and REINFORCE++[19], which utilize group, leave-one-out, or global batch averages, respectively. Other methods, such as  [7], [8], [20], [21], follow a similar paradigm. These methods significantly lower training overhead. Furthermore, to improve data usage efficiency, several works have proposed incorporating off-policy data[22]–[24], demonstrating that reusing historical trajectories can effectively facilitate model training.

2.2 Reinforcement Learning for LLM-based Agents↩︎

Recent work applies RL to enhance tool utilization in LLMs[25]–[27]. Several methods focus on training stability and credit assignment: RAGEN[28] filters low-variance trajectories, while GiGPO[13] and MT-GRPO[29] utilize turn-level advantages. Furthermore, ARPO [30] employs additional branching at high-entropy tokens, thereby concentrating computational resources on the optimization of high-entropy steps. AEPO [31] further refines the branching strategy and advantage estimation mechanisms of ARPO. SPEAR [32] accelerates training via self-imitation[33] and curriculum learning from replay data, while [34] integrate implicit world modeling and self-reflection to guide action selection. IGPO[35] introduces an intrinsic reward mechanism by calculating the information gain based on the conditional probability of the ground truth.

In this work, we introduce a reward-swap reinforcement learning mechanism, which leverages diverse trajectories induced by dense process rewards to facilitate training with sparse outcome rewards. Our proposed RSPO is compatible with existing reinforcement learning methods and can work synergistically to achieve superior performance.

2.3 Utilizing Process Rewards in LLMs↩︎

When an LLM agent engages in multi-turn interactions with an environment and receives an outcome reward only at the end of a trajectory, direct training with outcome rewards often leads to slow convergence due to reward sparsity. To address this, several approaches accelerate learning by constructing dense process rewards. StepAgent [36] generates step-wise rewards by imitating and reflecting upon expert trajectories. PRIME [37] employs implicit process reward modeling to continuously optimize the reward model during training, thereby yielding more accurate process rewards. Similarly, ReasonRAG [38] utilizes Shortest Path Reward Estimation, approximating process rewards via Monte Carlo-style estimation with step-based penalties. SPA-RL [11] decomposes the final reward into fine-grained intermediate signals through a step-wise progress attribution mechanism. Building upon the ReAct [39] framework, RLVMR [12] introduces meta-reasoning labels to derive more sophisticated composite rewards. However, while these dense process rewards can accelerate model training, such methods remain susceptible to potential inconsistencies between process rewards and outcome rewards.

3 Preliminaries↩︎

3.1 Task setup↩︎

We formulate the multi-turn interaction between the LLM agent and the environment as a Markov Decision Process (MDP) [40]. For a task \(x \sim p(X)\), the agent observes a state \(\boldsymbol{s}_t \in \mathcal{S}\) at step \(t\) and samples an action \(\boldsymbol{a}_t \sim \pi_\theta(\cdot | \boldsymbol{s}_t)\) based on parameters \(\theta\). The environment then returns the next state \(\boldsymbol{s}_{t+1}\) and a reward \(\boldsymbol{r}_t\). This process yields a trajectory \(\boldsymbol{\tau} = \{(\boldsymbol{s}_t, \boldsymbol{a}_t, \boldsymbol{r}_t)\}_{t=1}^T\). Reinforcement learning aims to optimize \(\theta\) by maximizing the expected discounted return \(\mathbb{E}_{\boldsymbol{\tau}\sim\pi_\theta}\left[\sum_{t=1}^{\infty}\gamma^{t-1}\boldsymbol{r}_t\right]\), where \(\gamma \in (0,1)\) is the discount factor.

3.2 Group Relative Policy Optimization (GRPO)↩︎

Recently, critic-free methods like GRPO have proven highly effective for LLM optimization. By eliminating the value function required by PPO, GRPO reduces memory and computational resource consumption. For each task \(x\), GRPO samples \(N\) trajectories \(\{\boldsymbol{\tau}_1, \dots, \boldsymbol{\tau}_N\}\) and normalizes the rewards to compute the advantage:

\[A(\boldsymbol{\tau}_i)=\frac{R(\boldsymbol{\tau}_i) - \text{mean}\bigl(\{R(\boldsymbol{\tau}_j)\}_{j=1}^N\bigr)}{{\text{std}}\bigl(\{R(\boldsymbol{\tau}_j)\}_{j=1}^N\bigr)}.\]

The objective function is formulated as: \[\begin{align} \label{eq:grpo95objective1} \mathcal{J}_{\mathrm{\text{GRPO}}} = &\mathbb{E}_{\substack{x \sim p(X), \{\boldsymbol{\tau}_i\}_{i=1}^N \sim \pi_{\theta_{\text{old}}}}} \biggl[ \frac{1}{NT} \sum_{i=1}^{N} \sum_{t=1}^{T} \min\Bigl( r_{i,t}(\theta){A}_{i,t},\, \text{clip}\bigl(r_{i,t}(\theta), 1 \pm \epsilon\bigr) {A}_{i,t}) \Bigr) \biggr] \nonumber \\ &-\beta \mathbb{D}_{\mathrm{KL}}\!\bigl(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\mathrm{ref}}(\cdot \mid x)\bigr), \end{align}\tag{1}\] where \(r_{i,t}\) is the importance sampling ratio \(\frac{\pi_{\theta}(a_t^{i}\mid x,\,s_t^{i})}{\pi_{\theta_{\mathrm{old}}}(a_t^{i}\mid x,\,s_t^{i})}\). To ensure training stability, the KL divergence term penalizes excessive deviation of the policy \(\pi_{\theta}\) from the reference model \(\pi_{\mathrm{ref}}\), scaled by the factor \(\beta\).

4 Reward-Swap Policy Optimization↩︎

4.1 Motivation↩︎

The motivation for our method in addressing multi-turn tasks with sparse outcome rewards stems from the human learning process regarding delayed rewards. When humans are confronted with a task characterized by delayed rewards and only outcome feedback, identifying an effective direction initially can be challenging. To overcome this, we often establish artificial immediate rewards—intermediate signals that we hypothesize will contribute to the final objective, even if they are not perfectly aligned with the task’s outcome rewards. We then optimize our choices based on these immediate rewards until certain attempts successfully complete the task and elicit the final outcome feedback. Subsequently, we retrospectively analyze these successful trajectories to reflect and learn, discerning which specific steps genuinely contributed to the long-term goal. By reinforcing these effective steps, we proficiently master the task.

4.2 RSPO↩︎

Based on this motivation, we derive the overall training pipeline of RSPO. In this framework, the tasks are formulated similarly: they require the agent to engage in multi-turn interactions with the environment, which returns an outcome reward only upon task completion. Here, the immediate rewards correspond to dense process rewards; while beneficial for achieving the final objective, they may exhibit potential inconsistencies with the task’s outcome rewards.

Following this rationale, we first train the initial agent \(A\) using dense rewards for \(k\) steps to obtain an intermediate agent \(B\). Agent \(B\) then interacts with the environment to collect a batch of trajectories, which are stored in a replay buffer \(\mathcal{D}\). Subsequently, we sample successful trajectories from \(\mathcal{D}\) to facilitate the training of the initial agent \(A\) for another \(k\) steps, resulting in an updated agent \(A'\). Finally, by repeating these steps, the performance of both Agent \(A\) and Agent \(B\) is continuously improved through this iterative refinement process. The overall framework is illustrated in Figure 2, and the pseudocode for RSPO is provided in the Appendix 8.

When Agent \(A\) is updated using the task’s outcome rewards, its optimization objective is formulated as follows: \[\label{eq:all} \mathcal{J}_{\text{Total}}(\pi_\theta)=(1-\alpha)\mathcal{J}_{\text{On}}(\pi_\theta) + \alpha\mathcal{J}_{\text{Off}}(\pi_\theta),\tag{2}\] where \(\mathcal{J}_{\text{On}}\) and \(\mathcal{J}_{\text{Off}}\) denote updates using self-generated on-policy data and off-policy data sampled from the replay buffer, respectively, and \(\alpha\) represents the proportion of off-policy data.

Moreover, since proximal policy gradient methods such as PPO and GRPO typically necessitate on-policy training, an excessive amount of off-policy data may induce a significant deviation in the state-action distribution relative to the current policy, thereby compromising training stability. Consequently, we prioritize the use of self-generated on-policy trajectories during training. In our practical experiments, the value of \(\alpha\) is set to \(1/8\).

4.2.1 Sampling Strategy↩︎

When sampling from the replay buffer, the choice of sampling strategy—such as random sampling, variance-based sampling, or reward-based sampling—significantly impacts the quality of the data utilized for updates. Guided by the conclusions from [23] and the original insight underlying our motivation (i.e., learn from successful trajectories), we adopt reward-based sampling as our definitive strategy. Specifically, this method prioritizes the sampling of trajectories that yield the highest rewards.

In the meanwhile, to ensure a fair comparison, we guarantee that the amount of data used for training RSPO is consistent with that of the baselines. For example, if the train data size for GRPO training is \(16\), RSPO also uses a train data size of \(16\), with the data size for self-generated on-policy trajectories and sampled off-policy trajectories being \(14\) and \(2\), respectively.

4.2.2 Clipping center shift↩︎

Proximal policy gradient methods, such as PPO and GRPO, typically employ a clipping operation to prevent excessive policy updates. However, when utilizing off-policy data for updates, a discrepancy exists between the distribution of the current policy and that of the policy used to sample the off-policy data. Directly applying the standard clipping range \((1-\epsilon, 1+\epsilon)\) may result in the erroneous clipping of valid data, rendering their contribution to the gradient update zero. Consequently, for off-policy data, we adopt the Generalized Clipping Mechanism proposed in  [41]:

\[\label{eq:geppo} \text{clip} \left( \frac{\pi_{\theta}(a \mid s)}{\pi_{B}(a \mid s)}, \frac{\pi_{\theta_{\text{old}}}(a \mid s)}{\pi_{B}(a \mid s)} \pm \epsilon\right),\tag{3}\] where \(\pi_{\theta}\) and \(\pi_{\theta_{\text{old}}}\) represent the current policy being updated and the old policy (i.e., the policy before the update), respectively, and \(\pi_B\) represents the policy that generates off-policy data (i.e., the policy corresponding to Agent \(B\) in Figure 2). The Generalized Clipping Mechanism introduces an offset to the clipping center, rendering the constraint more reasonable. The components of Equation 2 are provided below:

\[\begin{align} \label{eq:gigpo95objective1} \mathcal{J}_{\mathrm{\text{On}}} = &\mathbb{E}_{\substack{x \sim p(X), \{\boldsymbol{\tau}_i\}_{i=1}^{N} \sim \pi_{\theta_{\text{old}}}}} \biggl[ \frac{1}{NT} \sum_{i=1}^{N} \sum_{t=1}^{T} \min\Bigl( r^{\mathrm{on}}_{i,t}(\theta){A}^{\mathrm{on}}_{i,t},\, \text{clip}\bigl(r^{\mathrm{on}}_{i,t}(\theta), 1 \pm \epsilon\bigr) {A}^{\mathrm{on}}_{i,t}) \Bigr) \biggr] \nonumber \\ &-\beta \mathbb{D}_{\mathrm{KL}}\!\bigl(\pi_\theta( \cdot\mid x) \,\|\, \pi_{\mathrm{ref}}( \cdot\mid x)\bigr), \end{align}\tag{4}\] \[\begin{align} \label{eq:gigpo95objective} \mathcal{J}_{\mathrm{\text{Off}}} = &\mathbb{E}_{\substack{{x, \tau} \sim \mathcal{D}}} \biggl[ \frac{1}{NT} \sum_{i=1}^{N} \sum_{t=1}^{T} \min\Bigl( r^{\mathrm{off}}_{i,t}(\theta) {A}^{\mathrm{off}}_{i,t},\, \text{clip}\bigl(r^{\mathrm{off}}_{i,t}(\theta), \frac{\pi_{\theta_{\text{old}}}(a \mid s)}{\pi_{B}(a \mid s)} \pm \epsilon\bigr) {A}^{\mathrm{off}}_{i,t}) \Bigr) \biggr] \nonumber \\ &-\beta \mathbb{D}_{\mathrm{KL}}\!\bigl(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\mathrm{ref}}(\cdot \mid x)\bigr), \end{align}\tag{5}\] where \[r_{i,t}^{\mathrm{on}} = \frac{\pi_{\theta}\bigl(a_t^{i}\mid x,\,s_t^{i}\bigr)}{\pi_{\theta_{\mathrm{old}}}\bigl(a_t^{i}\mid x,\,s_t^{i}\bigr)}, r_{i,t}^{\mathrm{off}} = \frac{\pi_{\theta}\bigl(a_t^{i}\mid x,\,s_t^{i}\bigr)}{\pi_{B}\bigl(a_t^{i}\mid x,\,s_t^{i}\bigr)}.\]

For GRPO and PPO, advantage estimation employs group-based estimation and the Generalized Advantage Estimator (GAE) [42], respectively.

4.2.3 Dense process reward model↩︎

Following SPA-RL [11], we train a dense process reward model using a collected dataset \(\mathcal{D}_{dense}=\bigl\{(\hat{\tau}_{i},\,R_{i})\bigr\}_{i=1}^{M}\), where \(R_{i}\) is the final reward for trajectory \(\hat{\tau}_{i}\). We incorporate a Multi-Layer Perceptron (MLP) after the final hidden layer of the pre-trained LLM \(\pi_{pre}\), utilizing a tanh activation to constrain the intermediate reward \(r\) within \((-1, 1)\). The formulation is given by: \[\hat{r}_{t} = \tanh(\mathrm{MLP}\bigl(h_t\bigr)), \quad h_t = f_{\pi_{pre}}(s_t, a_t),\] where \(s_t\) is the context at step \(t\), \(f_{\pi_{pre}}\) denotes the encoding operation of \(\pi_{pre}\), and \(\hat{r}_t\) is the step-level reward. The total predicted reward for a trajectory is aggregated as: \[\hat{R} = \sum_{t=1}^{T} \hat{r}_{t},\] where \(T\) denotes the total number of steps in trajectory \(\tau\). The model is optimized by minimizing the MSE loss: \[\mathcal{L}_{\text{P}} = \frac{1}{N} \sum_{i=1}^{N} \bigl(\hat{R}_{i} - R_{i}\bigr)^2,\] where \(\hat{R_i}\) and \(R_i \in [0, 1]\) are the predicted and ground-truth rewards, respectively. It is important to note that the dense process rewards need not be derived exclusively through the aforementioned method; we merely adopted one of many available approaches to obtain them.

5 Experiments↩︎

In this section, we evaluate the performance of RSPO when applied to various reinforcement learning algorithms. We aim to analyze the effectiveness of RSPO from the following perspectives: (1) the performance improvements achieved by RSPO across various benchmarks when integrated with different reinforcement learning methods; (2) ablation study on sampling strategies; (3) the contribution of trajectories induced by dense rewards to performance improvement; (4) the potential inconsistency of dense rewards.

5.1 Experimental Setup↩︎

5.1.0.1 Environments.

We evaluated our method on two agent benchmarks:

(1) ALFWorld [9]: An embodied control benchmark simulating a household environment. By leveraging an interactive TextWorld interface, it aligns textual abstract environments with embodied physical environments, enabling agents to complete specified tasks via text instructions. It primarily encompasses six household task categories: Pick & Place (Pick), Examine in Light (Look), Clean & Place (Clean), Heat & Place (Heat), Cool & Place (Cool), and Pick Two & Place (Pick2).

(2) WebShop [43]: A web search and interaction benchmark. In this environment, the agent interacts with a simulated shopping website to search for and select products, ultimately executing a "buy now" action. The environment subsequently provides a final reward based on the extent to which the selected product matches the specified requirements.

5.1.0.2 Agent models.

We use Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct [15] as our base models. For the dense reward model, we use Llama-3.2-3B-Instruct as the base model.

5.1.0.3 Baselines.

We evaluate RSPO across three common and representative reinforcement learning algorithms: PPO, GRPO, and GiGPO. For GRPO and GiGPO, the dense reward is calculated as the sum of the predicted values over the entire trajectory. Specifically, GiGPO is categorized into two variants based on whether the standard deviation (std) is excluded from step-level advantage calculation: GiGPO_w/ std and GiGPO_w/o std. We adopt the best-performing GiGPO variants for each environment as reported in [13]: GiGPO_w/ std for ALFWorld and GiGPO_w/o std for WebShop.

Furthermore, we compared our approach against SPEAR combined with GRPO and GiGPO (SPEAR+GRPO and SPEAR+GiGPO; note that SPEAR was not applied to PPO). It is worth noting that the original SPEAR experiments utilized larger training data size and total epochs. To ensure a fair comparison, we reproduced the SPEAR results by aligning these hyperparameters with the settings used in our experiments.

5.1.0.4 Training Details.

Following the GiGPO codebase [13], we set the train data size to \(16\) and rollout group size \(N=8\). For RSPO-based methods, the train data size comprises \(14\) on-policy and \(2\) off-policy data size. Consequently, the total batch size for PPO is \(128 (16 \times 8)\), and for RSPO-PPO is \(112 (14 \times 8)\). In each RSPO loop, both Agent \(A\) and Agent \(B\) are updated for \(3\) steps respectively. All experiments run for a total of \(150\) steps (counted based on the update steps of the final obtained model). Full training settings and hyperparameter details are provided in Appendix 7. In the meanwhile, Appendix 9, 10, and 11 present additional experiments, including hyperparameter sweeps, robustness tests against dense reward noise, and a qualitative analysis of the impact of dense rewards on exploration.

5.2 Performance↩︎

Table 1 demonstrates the superior performance of RSPO. As observed, RSPO demonstrates consistent performance gains across both 1.5B and 7B model scales, with improvements reaching up to 12%. Specifically, using the 1.5B base model, RSPO surpasses GRPO by 5.7% on ALFWorld and 5.5% on WebShop. Compared to GiGPO, RSPO achieves margins of 1.6% on ALFWorld and 7.9% on WebShop. The improvements over PPO are even more pronounced, exhibiting gains of 8.6% and 12% on ALFWorld and WebShop, respectively.

With the 7B base model, the overall performance gains are less substantial compared to the 1.5B model; however, it still achieves average improvements of 2.37% and 3.96% on ALFWorld and WebShop, respectively. We attribute this to the stronger foundational capabilities of the 7B model compared to its 1.5B counterpart, which enable it to spontaneously plan and select a greater diversity of trajectories. Consequently, its reliance on the diverse trajectories induced by the dense reward model is diminished, as the model itself is capable of generating a sufficient variety of successful trajectories. This intrinsic capability results in performance improvements that are less pronounced than those observed with the 1.5B model.

Additionally, we reproduce the performance of SPEAR+GRPO and SPEAR+GiGPO under the same settings for training data size and total epochs; however, the results were suboptimal. We attribute this to the fact that the original SPEAR paper uses a training data size of 32 and 350 total epochs, while our experiments use a data size of 16 and 150 epochs. Given SPEAR’s inherently slower learning speed, it yields suboptimal performance.

5.3 Ablation Study↩︎

In WebShop, we conduct ablation studies on the replay buffer’s sampling strategies and data sources to validate the efficacy of our reward-based sampling mechanism and the trajectories induced by dense rewards. For all experiments, we employ RSPO+GRPO as the baseline method with Qwen2.5-1.5B-Instruct as the base model. The comprehensive results are presented in Table 2.

5.3.1 Sampling Strategy↩︎

We evaluated three distinct strategies for sampling from the replay buffer:

(1) Random Sampling: Samples are selected uniformly at random from the replay buffer.

(2) Variance-based Sampling: This method prioritizes samples exhibiting the highest intra-group reward variance, under the hypothesis that samples with higher variance provide richer information.

(3) Reward-based Sampling: This approach prioritizes samples with the highest rewards, aiming to focus learning on the diverse successful trajectories induced by dense rewards. This is the final method adopted in our work.

As shown in the upper section of Table 2, reward-based sampling significantly outperforms alternative sampling methods. This indicates that the agent effectively acquires superior performance from the diverse successful trajectories induced by dense rewards, which aligns with our motivation to learn by reviewing successful trajectories.

In addition, we observed that the model performs poorly with random sampling, exhibiting severe fluctuations in the success rate curve during training. We attribute this to the fact that, at the early stage of training, the data collected by the agent is mainly negative samples. Therefore, random sampling results in off-policy data that is also largely negative. These negative samples compromise training stability—particularly in off-policy scenarios—thereby degrading final performance [44].

5.3.2 Data Sources for the Replay Buffer↩︎

Prior works such as  [23], [24], [32] employed the model’s own historical trajectories for off-policy training. We evaluate a similar setup by populating the replay buffer with Agent \(A\)’s self-generated historical trajectories (\(\text{RSPO}_{H}\)). Additionally, we replaced the dense rewards directly with outcome rewards (\(\text{RSPO}_{O}\), while maintaining other procedures unchanged) to validate the performance gains attributed to dense rewards.

As shown in the bottom section of Table 2, the final performance when training with self-generated historical trajectories is significantly inferior to that of our method. Similarly, replacing dense rewards with outcome rewards also resulted in a performance decline. This demonstrates that trajectories induced by the dense rewards are more beneficial, primarily because they expand the model’s exploration space. Furthermore, we observed that when the amount of training data is held constant, using the model’s own historical trajectories does not outperform the original baseline. Its primary advantage lies in improving data usage efficiency rather than final performance.

5.4 Inconsistency of Dense Rewards↩︎

To demonstrate that the improvements achieved by RSPO are not merely a consequence of the high quality of the dense reward model itself, we also evaluate the performance of training GRPO exclusively with dense rewards in WebShop (using Qwen2.5-1.5B-Instruct as the base model).

The results are illustrated in the Figure 3. As observed, when training relies solely on dense rewards, the model’s success rate (blue curve) initially increases but subsequently declines, whereas the average dense reward of the sampled trajectories during training (green curve) exhibits a steady upward trend. This indicates that the model focuses on optimizing the dense reward metric; however, since the dense rewards are not perfectly aligned with the task’s outcome rewards, this leads to the phenomenon of "Reward Hacking" [45]. This finding underscores the necessity of the RSPO, which consistently utilizes outcome rewards for the final model training.

6 Conclusions↩︎

In this paper, we propose RSPO, a reinforcement learning framework that leverages the rich information from dense process rewards to facilitate training with sparse outcome rewards. This approach addresses both the potential inconsistency of dense process rewards and the sparsity of outcome rewards. Through an iterative process, RSPO enables a model trained with dense process rewards to interact with the environment, generating a wider variety of trajectories. These trajectories are stored in a replay buffer and utilized to assist in training with sparse outcome rewards.

By ensuring that the final model is consistently trained using the task’s outcome rewards, RSPO avoids the issues arising from the potential inconsistency of dense process rewards. Simultaneously, utilizing data from the replay buffer expands the model’s exploration space, ultimately elevating its performance ceiling. Experimental results in complex agent environments demonstrate that RSPO yields consistent performance improvements across various reinforcement learning methods, providing a plug-and-play training paradigm for multi-turn LLM agents.

7 Implementation Details↩︎

All our experiments are implemented based on VeRL-Agent. [13]. Throughout our experiments, we strictly adhered to the licensing terms for academic use associated with LLaMA-3.2-3B-Instruct. The licensing terms can be found at the following link: https://huggingface.co/meta-llama/Llama-3.2-1B/blob/main/LICENSE.txt.

7.1 Hyperparameter Settings↩︎

To ensure a fair comparison, we follow the settings of GiGPO and retained most of the primary parameters from VeRL-Agent. For the additional hyperparameters introduced, the off-policy data ratio \(\alpha\) is set to \(1/8\). For GRPO and GiGPO, the train data size is set to \(16\). For RSPO-based methods, the train data size comprises \(14\) on-policy and \(2\) off-policy data size. Consequently, the total batch size for PPO is \(128 (16 \times 8)\), and for RSPO-PPO is \(112 (14 \times 8)\).

In each RSPO loop, we trained the agent for \(k\) steps using dense rewards, followed by \(k\) steps using outcome rewards, where \(k=3\) (the total step count is calculated based on the update steps of the final obtained model). When populating the replay buffer, the training data size was set to \(14 \times 2 = 28\), and the rollout group size \(N=8\). All experiments run for a total of \(150\) steps (counted based on the update steps of the final obtained model).

7.2 Computing resources↩︎

All experiments are conducted on workstations equipped with 380 CPU cores, 2.2TB memory, and 8 GPUs with 96GB memory. In WebShop, the 1.5B and 7B models are trained on 2 and 8 GPUs respectively for GRPO and GiGPO, whereas PPO required 4 and 8 GPUs respectively. In ALFWorld, the 1.5B model is trained on 8 GPUs, while the 7B model uses 2 workstations with 16 GPUs.

8 Pseudo Code↩︎

Algorithm [alg:code] summarizes the full training procedure of RSPO. Each RSPO loop consists of two distinct training phases. In the first phase, the model \(\pi_{dense}\) is trained for \(k\) steps using dense rewards. The trained \(\pi_{dense}\) then interacts with the environment to generate a batch of trajectories, which are labeled with outcome rewards and stored in the replay buffer \(\mathcal{D}\). Subsequently, the process enters the second phase, where the model \(\pi_{\theta}\) is trained for \(k\) steps using outcome rewards. The training dataset employed here is a mixture of on-policy data generated by \(\pi_{\theta}\) and off-policy data sampled from the replay buffer. This cycle is repeated to continuously refine \(\pi_{\theta}\) and \(\pi_{dense}\).

This training paradigm ensures that the final model \(\pi_{\theta}\) is consistently trained with task outcome rewards, thereby avoiding "Reward Hacking" issues stemming from potential inconsistencies in dense process rewards. Simultaneously, it expands the model’s exploration space by leveraging data from the replay buffer, ultimately elevating the model’s performance ceiling.

9 Hyperparameter Sensitivity Analysis↩︎

We conduct additional ablation studies on the WebShop using Qwen2.5-1.5B-Instruct model. To ensure a strictly fair comparison across all hyperparameter sweeps, we re-run the baseline and all variants under a unified experimental setting (e.g., consistent random seeds and compute budget). While absolute scores vary slightly due to seed variance, the relative performance gains and trends remain consistent with the main paper.

9.1 Sensitivity Analysis of Off-policy Data Ratio \(\alpha\)↩︎

We evaluated \(\alpha \in \{1/16, 1/8, 1/4, 1/2\}\) while fixing \(k=3\). The results are compared against the GRPO baseline under the same settings.

When \(\alpha\) is kept small (e.g., \(1/16\) to \(1/8\)), the method effectively leverages diverse trajectories to improve performance without disrupting training stability. As \(\alpha\) increases to \(1/4\) or \(1/2\), performance drops significantly. This aligns with our discussion in Section 4.2: excessive off-policy data introduces severe distribution shifts, which destabilizes the policy update.

9.2 Sensitivity Analysis of Loop Steps \(k\)↩︎

We evaluated \(k \in \{1, 3, 5, 10\}\) while fixing \(\alpha=1/8\).

We observe an "inverted-U" trend, indicating that \(k\) must be balanced to avoid under-exploration or overfitting to dense rewards. When \(k=3\), this setting achieves the best balance, yielding the highest Score and Success Rate (SR). When \(k=1\), the performance is slightly lower compared to the optimal setting. And when \(k \ge 5\), the performance reverts to the baseline level. Optimizing against the dense reward model for too long may lead to signs of reward hacking, resulting in no improvement in final performance.

These experiments confirm that RSPO is robust and effective as long as hyperparameters are selected within a reasonable range (i.e., a small \(\alpha\) to maintain stability and a moderate \(k\) to balance exploration).

10 Sensitivity to Dense Reward Quality↩︎

To investigate the sensitivity of RSPO to the quality of dense rewards, we conducted an ablation study on WebShop using Qwen2.5-1.5B-Instruct. Specifically, we introduced Gaussian noise \(\mathcal{N}(0, \sigma)\) to the dense rewards used by the agent during exploration.

As shown in the table, when the noise is relatively small (\(\sigma=0.1\)), the performance remains robust (SR: 67.2 vs 64.1), demonstrating that RSPO does not require high-precision dense rewards to be effective. Even with significant noise (\(\sigma=0.3\)), the performance drop is moderate, and the model does not collapse. This validates our core insight: the dense reward in RSPO serves primarily as a "guide" for exploration rather than a ground truth. As noted in Figure 3 of our paper, training solely with the dense reward (even without added noise) leads to reward hacking. RSPO succeeds because the "Reward-Swap" mechanism uses the true outcome reward to filter out trajectories, ensuring that even if the dense reward is imperfect or noisy, the final policy remains aligned with the task objective.

11 Impact of Dense Rewards on State Space Exploration↩︎

To investigate how dense rewards contribute to performance, we analyzed the diversity of visited states during training. We counted the number of identical states visited by the agent within a fixed window. A higher count indicates the agent is stuck in local optima or repeating behaviors, while a lower count implies better exploration of the state space.

As the training progresses, GRPO tends to revisit the same states (count increases to 1167), suggesting it is stuck in a local behavioral space. In contrast, RSPO significantly reduces the number of identical states (dropping to 623), demonstrating that Agent B successfully guides the model to explore a much broader and more diverse state space.

12 Limitations↩︎

Due to computational resource constraints, we did not conduct experiments on models exceeding 7B parameters; consequently, the performance gains of RSPO on larger-scale models remain to be fully validated. Additionally, the off-policy data ratio is kept constant throughout our experiments. Employing an adaptive ratio (e.g., a decaying schedule) could potentially enhance the final results. We identify these as promising directions for future work.

References↩︎