1 Introduction↩︎

On-policy reinforcement learning (RL) has become a cornerstone of modern artificial intelligence, underpinning recent advances ranging from the fine-tuning of large language models (LLMs) for alignment and reasoning [1]–[3] to complex locomotion and manipulation in embodied agents [4], [5]. By directly optimizing policies against reward signals—whether derived from human feedback, verifiable outcomes, or physical interactions—policy gradient methods provide a flexible and scalable framework for transforming raw predictive models into goal-directed behaviors.

Despite their empirical success, a fundamental challenge persists: ensuring stable and monotonic policy improvement without sacrificing learning efficiency or data utilization. This challenge is traditionally addressed through trust-region methods, most notably Trust Region Policy Optimization (TRPO) [6]. While recent works have sought to improve TRPO’s scalability via Quasi-Newton methods [7], Kronecker-factored approximations [8], or low-rank updates [9], the computational burden of second-order optimization remains a barrier. Consequently, first-order approximations such as Proximal Policy Optimization (PPO) [10] and its variants (e.g., GRPO [3]) have become widespread. These methods rely on the policy ratio \(\rho_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\text{old}}(a_t|s_t)\) and enforce stability through a heuristic clipping mechanism that truncates \(\rho_t\) within a fixed interval.

While clipping has proven effective in practice [11], it introduces a fundamental limitation: stability is enforced through a pointwise, binary decision. When the policy ratio exceeds a predefined threshold, gradients are abruptly truncated, regardless of the associated return signal. As a result, highly informative samples—such as an LLM discovering a novel reasoning trajectory or a robot executing a previously unseen maneuver—may be suppressed precisely because they induce large policy updates [12]. Moreover, this rigid boundary renders historical data unusable once it becomes slightly stale, forcing on-policy algorithms to discard valid experiences and rely exclusively on freshly collected samples. Although prior work has proposed various heuristics to relax clipping, including asymmetric bounds or gradient-based modifications [13]–[15], these approaches remain fundamentally constrained by the pointwise nature of clipping.

In this work, we argue that this limitation is not inherent to trust-region optimization itself, but rather a consequence of its particular approximation. Revisiting the original trust-region objective, we show that, in the local regime, constraining policy divergence is approximately equivalent to controlling the second-order statistics of the policy ratio—specifically, its variance. This observation reveals variance regularization as a principled, distributional alternative to hard clipping. Unlike pointwise truncation, variance-based constraints softly scale updates according to their divergence, preserving gradient information from high-value outliers while naturally down-weighting stale or off-policy samples. Building on this insight, we propose Ratio-Variance Regularized Policy Optimization (R\(^2\)VPO). By formulating variance control as a Lagrangian dual constraint, R\(^2\)VPO employs a primal–dual optimization scheme that jointly maximizes expected returns and minimizes the variance of importance weights, unifying stable on-policy optimization with efficient off-policy data reuse.

We empirically evaluate R\(^2\)VPO across diverse domains to verify its generality. As summarized in Figure 1, R\(^2\)VPO demonstrates consistent superiority over strong clipping-based baselines across a wide spectrum of tasks. In the realm of LLM fine-tuning, we conduct extensive experiments on state-of-the-art models across diverse thinking paradigms. R\(^2\)VPO achieves a remarkable macro-average relative gain of +35% (up to +138% on smaller models) and outperforms strong baselines like GRPO in 6 out of 7 settings. Crucially, the off-policy variant (R\(^2\)VPO-OFF) effectively leverages stale data to improve sample efficiency. Extending our evaluation to continuous control, R\(^2\)VPO significantly outperforms PPO on DeepMind Control Suite benchmarks [16], exhibiting superior exploration in sparse-reward settings and preventing performance collapse in dynamic tasks. These findings establish ratio-variance control as a highly promising direction for advancing general reinforcement learning ¹.

2 Related Work↩︎

2.0.0.1 Approximations of the Trust Region.

Trust region methods ensure monotonic policy improvement by constraining the divergence between consecutive policies [6], [17], grounded in Natural Gradient descent [18]. To address the cost of Fisher Information Matrix computation, various optimizations have been proposed, ranging from K-FAC approximations (ACKTR) [8] and Quasi-Newton methods [7] to efficient low-rank updates [9]. Others have focused on theoretical convergence via metric-aware [19] or stochastic frameworks [20], and extended these principles to multiagent domains [21]. Despite these advances, PPO [10] remains dominant due to its simplicity, utilizing heuristic clipping. Recently, methods like SPO [12] and FixPO [22] have revisited KL-regularization, replacing clipping with direct penalties. Similarly, SB-TRPO [23] incorporates safety bias via trust regions. However, these approaches often rely on fixed coefficients or heuristics. R\(^2\)VPO advances this direction by deriving the variance of policy ratio as a local second-order approximation. Unlike heuristic penalties, our variance-based formulation naturally captures optimization curvature and, by solving the Lagrangian dual, adaptively adjusts constraint strength to avoid hyperparameter sensitivity.

Beyond TRPO/PPO-style approximations, several non-clipping trust-region methods have enforced information-theoretic constraints more directly. Relative Entropy Policy Search [24] optimizes policy updates under an explicit KL constraint and solves the resulting dual problem, while subsequent non-parametric policy search methods further study limited information loss under trust-region constraints. V-MPO [25] formulates policy optimization from a maximum-a-posteriori perspective with KL-constrained E/M-style updates. More recently, differentiable trust-region layers [26] impose constraints by projecting neural policy outputs through differentiable layers. These methods demonstrate that clipping is not the only way to realize trust-region control. Our work is complementary: rather than solving an exact KL-constrained update, introducing projection layers, or estimating Fisher-preconditioned directions, R\(^2\)VPO extracts the shared local second-order geometry of a broad class of f-divergences and implements it as a simple quadratic regularizer directly in importance-ratio space. This makes it particularly easy to integrate into PPO/GRPO-style LLM and replay-based pipelines.

2.0.0.2 On-Policy Optimization in LLMs and Robotics.

PPO has become the de facto standard across domains, dominating robotics tasks [5], [27]–[30] and LLM alignment [1], [31]. Recognizing that standard clipping indiscriminately truncates high-value signals, recent heuristics have emerged: DAPO [13] and GRPO-CH relax clipping bounds; TOPR [15] employs asymmetric thresholds; and GPPO [14] retains gradient direction. These methods, however, remain “patches” on the clipping paradigm. R\(^2\)VPO fundamentally differs by enforcing a distributional constraint via variance regulation, providing a soft, adaptive regularization for both the discrete and continuous action spaces.

2.0.0.3 Off-Policy Efficiency and Scalability.

In general RL, off-policy algorithms such as SAC [32] and TD3 [33] achieve superior sample efficiency but require separate Critic networks, which poses scalability hurdles for 7B+ models. While works like Off-Policy TRPO [34], [35] have introduced monotonic improvement guarantees for off-policy reuse, adapting them to critic-free LLM fine-tuning remains challenging. Alternative approaches using Importance Sampling (IS) corrections (e.g., V-trace [36], Retrace [37]) often suffer from high variance with stale data. R\(^2\)VPO circumvents these bottlenecks by leveraging ratio variance as a natural stabilizer. By penalizing the variance of importance weights, our framework enables robust off-policy reuse without complex correction terms or extra Value networks, combining the efficiency of methods like Off-Policy TRPO with the scalability of critic-free architectures.

3 Preliminaries↩︎

We consider a standard RL framework modeled as a Markov Decision Process (MDP), defined by the tuple \(\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{P}, r, \gamma \rangle\). Here, \(\mathcal{S}\) denotes the state space, \(\mathcal{A}\) is the action space, and \(\mathcal{P}: \mathcal{S} \times \mathcal{A} \rightarrow \Delta(\mathcal{S})\) represents the transition dynamics. At each time step \(t\), the agent observes a state \(s_t \in \mathcal{S}\), selects an action \(a_t \in \mathcal{A}\) according to a stochastic policy \(\pi_\theta(a_t|s_t)\) parameterized by \(\theta\), receives a scalar reward \(r(s_t, a_t)\), and transitions to the next state \(s_{t+1} \sim \mathcal{P}(\cdot|s_t, a_t)\). The discount factor \(\gamma \in [0, 1)\) balances immediate and future rewards. The standard goal in RL is to find an optimal policy \(\pi^*\) that maximizes the expected discounted cumulative return \[\mathcal{J}(\pi_\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t=0}^\infty \gamma^t r(s_t, a_t) \right],\] where \(\tau = (s_0, a_0, s_1, a_1, \dots)\) denotes a trajectory sampled under the policy \(\pi_\theta\).

Optimizing \(\mathcal{J}(\pi_\theta)\) directly via vanilla policy gradients often suffers from instability due to large step sizes. To ensure monotonic improvement, Trust Region Policy Optimization (TRPO) [6] maximizes a surrogate objective subject to a strict KL-divergence constraint on the policy update, which guarantees policy improvement provided the new policy stays within a “trust region” of the old policy \[\label{eq:trpo95constraint} \begin{align} \max_{\theta}\quad & \mathbb{E}_{t} \left[ \rho_t(\theta)\,\hat{A}_t \right] \\ \text{s.t.}\quad & \mathbb{E}_{t} \left[ D_{\text{KL}}\left( \pi_{\text{old}}(\cdot|s_t)\,\|\,\pi_\theta(\cdot|s_t) \right) \right] \le \delta \end{align}\tag{1}\] where \(\rho_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\text{old}}(a_t|s_t)}\) is the probability ratio, \(\hat{A}_t\) is the advantage function estimator, and \(\delta\) is a divergence limit. Although expressed in terms of KL divergence, the trust-region constraint implicitly enforces that the policy ratio remains close to one over the data distribution.

Due to the high computational cost of enforcing Eq. 1 via second-order optimization, first-order methods such as Proximal Policy Optimization (PPO) [10] and Group Relative Policy Optimization (GRPO) [3] have become standard. These algorithms replace the explicit trust-region constraint with a heuristic clipping objective: \[\label{eq:clip95obj} \begin{align} \mathcal{L}^{\text{CLIP}}(\theta) = \mathbb{E}_{t} \Big[ & \min \Big( \rho_t(\theta)\,\hat{A}_t, \\ & \text{clip}\!\left(\rho_t(\theta), 1-\epsilon, 1+\epsilon\right)\,\hat{A}_t \Big) \Big]. \end{align}\tag{2}\] Clipping enforces stability by imposing a fixed, pointwise bound on the policy ratio. While effective at preventing destructive updates, this binary mechanism tightly couples stability to the hard truncation of \(\rho_t(\theta)\). As a result, it indiscriminately discards gradient information from samples with high divergence or staleness, limiting both learning efficiency and the reuse of off-policy data. This limitation motivates the need for a more principled regularization that controls divergence broadly without erasing valuable high-leverage signals.

4 Method↩︎

In this section, we revisit trust-region policy optimization from the perspective of the policy ratio. We show that, across a broad class of trust-region constraints, the local geometry of policy divergence is universally governed by the variance of the policy ratio. This insight leads to a simple yet principled alternative to heuristic clipping: a variance-regularized objective that enables stable updates and naturally supports off-policy data reuse.

4.1 From \(f\)-Divergence to Policy Ratio Variance↩︎

To establish a generalized framework, we revisit the trust region constraint. While TRPO specifically employs the Kullback-Leibler (KL) divergence, we consider the broader family of \(f\)-divergences [38], denoted as \(D_f(\cdot\|\cdot)\). Let \(\pi_{\text{off}}\) be the behavior policy (or the policy at the previous iteration) and \(\pi_\theta\) be the current policy. The trust region constraint can be formulated as \[\label{eq:f95constraint} \mathbb{E}_{s \sim d^{\pi_{\text{off}}}} \left[ D_f(\pi_\theta(\cdot|s) \| \pi_{\text{off}}(\cdot|s)) \right] \leq \delta,\tag{3}\] where the divergence is defined as \(D_f(P\|Q) = \mathbb{E}_{x\sim Q}[f(P(x)/Q(x))]\) and \(f: (0, \infty) \to \mathbb{R}\) is a convex function with \(f(1)=0\). While valid for any convex \(f\), directly optimizing this constraint is often intractable. Here, we demonstrate that for any valid \(f\)-divergence, the local geometry of the trust region is dominated by the variance of the policy ratio.

The proof is provided in Appendix 5. This high-order term indicates that the approximation is accurate in a small neighborhood around \(\rho_\theta \approx 1\). Thus, ratio variance constitutes the fundamental second-order surrogate underlying a wide class of trust-region constraints.

As summarized in Table 1, this property holds uniformly across the \(f\)-divergence family. In the local trust-region regime, common divergences—including Forward/Reverse KL, Jensen–Shannon, Hellinger, and \(\chi^2\)—all collapse to a scaled ratio-variance term \(\frac{f''(1)}{2}\mathrm{Var}[\rho_\theta]\). Figure 2 empirically validates this approximation, showing tight alignment between exact divergences and their quadratic surrogates across representative metrics. Additional visualizations are provided in Appendix 8.

4.2 Primal-Dual Formulation↩︎

Leveraging the approximation established in Prop. 1, we recast the intractable \(f\)-divergence constrained problem into a tractable variance-constrained optimization. By substituting the variance term \(\mathbb{E}[(\rho_t(\theta)-1)^2]\) for the divergence constraint, the optimization objective becomes \[\label{eq:var95problem} \begin{align} \max_{\theta} \quad & \mathbb{E}_{(s_t,a_t)\sim\pi_{\text{off}}} \!\left[\rho_t(\theta)\,\hat{A}_t\right] \\ \text{s.t.} \quad & \mathbb{E}_{(s_t,a_t)\sim\pi_{\text{off}}} \!\left[(\rho_t(\theta)-1)^2\right] \leq \delta . \end{align}\tag{4}\] Solving this constrained problem directly is challenging due to the hard boundary on the variance. To address this, we employ the method of Lagrange multipliers to relax the constraint into a penalty term, transforming the problem into an unconstrained saddle-point optimization.

To understand the mechanism of this objective, we analyze the gradient of the Lagrangian \(\mathcal{L}\) with respect to the policy \[\begin{align} \label{eq:gradient} \nabla_\theta \mathcal{L}(\theta,\lambda) &= \mathbb{E}_{(s_t,a_t)\sim\pi_{\mathrm{off}}} \Big[ \underbrace{ \big( \hat{A}_t - 2\lambda(\rho_t(\theta)-1) \big) }_{\text{Regularized Advantage}} \,\rho_t(\theta) \nonumber \\ &\qquad\qquad\qquad \times \nabla_\theta \log \pi_\theta(a_t \mid s_t) \Big]. \end{align}\tag{6}\]

The gradient in Eq. 6 reveals the core mechanism of R\(^2\)VPO. The term \(2\lambda(\rho_t(\theta)-1)\) acts as a dynamic, instance-dependent regularizer that modulates the original advantage \(\hat{A}_t\). Unlike hard clipping, which zeroes out the gradient when the ratio exceeds a threshold \(\nabla \mathcal{L}=0\), this formulation preserves the gradient direction but scales its magnitude:

• Near-Identity Updates: When the policy ratio is close to \(1\) (i.e., \(\pi_\theta \approx \pi_{\text{off}}\)), the penalty term vanishes, and the update is driven purely by the task advantage \(\hat{A}_t\).

• High-Divergence Updates: When the policy deviates significantly, the penalty term grows linearly with the deviation. This induces a continuous attenuation of high-divergence updates, in contrast to the discontinuous truncation imposed by clipping.

To rigorously justify substituting the established hard clipping mechanism with our variance-based control, we derive a bound relating the two approaches. We show that the “clipping error”—the divergence between the true off-policy objective and the heuristically clipped surrogate—is strictly upper-bounded by the policy ratio variance.

The proof is provided in Appendix 6. Theorem 4 offers a formal justification for R\(^2\)VPO. By explicitly minimizing the ratio variance (via the penalty \(\lambda\)), R\(^2\)VPO naturally minimizes the potential overestimation error that clipping was heuristically designed to prevent. Unlike hard clipping, which completely discards signals outside the \(\epsilon\)-bound, variance control softly penalizes them proportional to their squared deviation, maintaining gradient flow while reducing the clipping-induced error controlled by the ratio variance.

R\(^2\)VPO should be interpreted as a local trust-region surrogate rather than an exact trust-region method with global monotonic improvement guarantees. Unlike TRPO, which enforces an explicit KL constraint and derives a monotonic improvement bound under idealized assumptions, our method replaces the exact divergence by its local second-order ratio-variance approximation. Therefore, the guarantee provided by Proposition 1 is local: when the policy ratio remains close to one, the variance penalty captures the leading-order geometry of the trust region. In exchange for giving up exact global monotonicity, R\(^2\)VPO obtains a simple first-order objective that operates directly in the ratio space used by PPO/GRPO-style training and naturally supports stale-data reuse.

4.3 Implementation Instances↩︎

The theoretical framework developed in Theorem 3 directly gives rise to R\(^2\)VPO, a practical algorithm that replaces heuristic clipping with a principled variance-based regularization. Concretely, the clip operator and its associated thresholds are substituted by a simple quadratic penalty \((\rho_t(\theta)-1)^2\), yielding a clean objective that is fully aligned with the primal–dual formulation in Eq. 5 . This design not only simplifies implementation but also provides a unified interface for both on-policy and off-policy training, as summarized in Algorithm [alg:r2vpo95combined].

4.3.0.1 On-Policy Training (R\(^2\)VPO-ON).

In the standard on-policy setting, R\(^2\)VPO mirrors the workflow of PPO/GRPO but employs the Lagrangian objective. A flexibility of our framework lies in the option of the dual variable \(\lambda\), which governs the regularization strength. It can be configured in two modes depending on stability requirements:

1. Fixed Regularization: \(\lambda\) is treated as a static hyperparameter (e.g., \(\lambda=0.04\)) to simplify implementation.

2. Adaptive Lagrangian Tuning: \(\lambda\) is dynamically optimized via dual gradient descent to strictly enforce the trust region constraint \(\delta\).

In the adaptive setting, \(\lambda\) is updated at each step via \[\label{eq:lambda95update} \lambda \leftarrow \max(0, \lambda - \eta_\lambda (\delta - \mathbb{E}[(\rho_t-1)^2])).\tag{7}\] This automatic tuning ensures the policy remains within the trust region without manual hyperparameter searching.

4.3.0.2 Off-Policy Training (R\(^2\)VPO-OFF).

Crucially, the variance-based formulation extends naturally to off-policy learning by robustly tolerating data staleness. Unlike standard on-policy methods that must discard data immediately after optimization, R\(^2\)VPO-OFF leverages a Replay Buffer \(\mathcal{D}\) to reuse historical reasoning trails. During the inference phase, we collect experience tuples \(\tau = \langle q, o, \log \pi_{\text{off}}(o \mid q), r, \hat{A} \rangle\) and store them in \(\mathcal{D}\), explicitly recording the behavior policy \(\pi_{\text{off}}\) at generation time. And in the training phase, we sample mini-batches uniformly from \(\mathcal{D}\). The variance penalty \((\rho_t-1)^2\) automatically scales the gradient contribution: fresh data contributes fully, while stale data is softly penalized via the “Regularized Advantage” mechanism, preventing instability. Similar to the on-policy variant, the regularization coefficient \(\lambda\) can be either fixed or adaptively tuned. Beyond sample efficiency, this off-policy capability facilitates a decoupled, asynchronous architecture ideal for LLM post-training. Since generation (inference) and optimization (training) often have mismatched throughputs, R\(^2\)VPO-OFF allows Actor nodes to focus exclusively on high-throughput generation while Learner nodes asynchronously update the model using buffered data. The updated weights are periodically synchronized to Actors, maximizing overall system efficiency.

5pt

5 Experiment↩︎

In this section, we conduct a comprehensive evaluation to verify the efficacy of R\(^2\)VPO. Our experiments are designed to answer two primary research questions (RQ):

• RQ1 (Asymptotic Performance): Does ratio variance regularization provide a more stable and effective optimization landscape than heuristic hard clipping, leading to superior reasoning capabilities?

• RQ2 (Sample Efficiency): Can R\(^2\)VPO effectively leverage off-policy data via replay buffers to significantly reduce the computational cost (rollouts) required for convergence?

To strictly address these questions, our experimental design spans the full spectrum of action modalities. We first evaluate R\(^2\)VPO in the high-dimensional discrete action space of LLMs, specifically on complex mathematical reasoning benchmarks, to rigorously test its exploration capabilities and data reuse efficiency. Complementing this, we extend our assessment to the continuous action space of robotic locomotion and manipulation to verify the algorithm’s robustness and physical stability.

For LLM reasoning, models are fine-tuned with a global batch size of \(128\) and learning rate of \(1\times10^{-6}\), utilizing an adaptive dual-update strategy for \(\lambda\). We evaluate both on-policy (R\(^2\)VPO-ON) and off-policy (R\(^2\)VPO-OFF) regimes; the latter employs a FIFO replay buffer (capacity \(4\)) with an Update-to-Data (UTD) ratio of \(2\) to efficiently leverage historical data. For continuous control, agents utilize MLP networks as backend. We leverage massive parallelism (\(2,048\) concurrent environments) for robust gradient estimation. Unlike the LLM setting, we employ a fixed dual factor \(\lambda\) given the dense reward signals in robotic tasks for training stability. Full hyperparameter configurations are detailed in Appendix 9.

5.1 Complex Reasoning with LLMs↩︎

We utilize the DAPO-Math-17K dataset [13] for training. To assess generalization, we evaluate performance on five challenging mathematical reasoning benchmarks: AIME 2024, AIME 2025 [39], AMC 2023 [40], HMMT Feb 2025 [41], and OlymMath [42]. Our experiments cover a comprehensive set of seven base models spanning both Fast Thinking (e.g., Qwen3-1.7B/8B-Fast [43], openPangu-1B [44]) and Slow Thinking paradigms (e.g., DeepSeek-Distill-Qwen-1.5B [3], openPangu-7B [44], Qwen3-4/8B-Thinking [43])². For comparative analysis, we benchmark R\(^2\)VPO against four state-of-the-art policy gradient methods: GRPO [3], which employs standard symmetric clipping; GRPO-CH [13], a variant with relaxed upper clipping thresholds; GPPO [14], which prioritizes gradient direction preservation; and TOPR [15], which utilizes asymmetric estimators for off-policy corrections.

Table 2 demonstrates the consistent superiority of R\(^2\)VPO. R\(^2\)VPO-OFF achieves the highest macro-average accuracy of 46.33% (+35% relative gain over base models), significantly outperforming strong baselines like GRPO (+24%) and GPPO (+30%). The gains are most dramatic in Fast Thinking models, where Qwen3-1.7B-Fast and Qwen3-8B-Fast improve by 138% and 110%, respectively. As analyzed in Appendix 10.1, these performance jumps coincide with a substantial increase in response length, suggesting that variance-aware optimization successfully unlocks extended reasoning capabilities in smaller models. Furthermore, R\(^2\)VPO-OFF outperforms its on-policy counterpart in 5 out of 7 settings, confirming that variance regularization effectively mitigates the instability of stale data. And, R\(^2\)VPO-ON remains highly competitive, notably surpassing the off-policy variant on openPangu-7B (57.55%), which validates the stability of the core variance-regularized objective even without data replay.

5.2 Continuous Robotic Control↩︎

To demonstrate generality beyond discrete reasoning, we evaluate R\(^2\)VPO in on-policy manner on continuous control tasks from DM control [16], covering locomotion (e.g., Walker, Cheetah) and manipulation (e.g., Ball_in_Cup). As illustrated in Figure 5, R\(^2\)VPO exhibits significantly improved robustness compared to the PPO baseline. In sparse-reward tasks like Ball_in_Cup, PPO fails to learn (flatlining near zero), whereas R\(^2\)VPO successfully solves the task, evidencing superior exploration. In dense-reward tasks like Cheetah_Run, PPO suffers from catastrophic performance collapse during late-stage training. In contrast, R\(^2\)VPO exhibits steady improvement without the late-stage collapse observed in PPO, suggesting that explicitly controlling the ratio variance yields a more stable optimization landscape than heuristic clipping. Full results on 10 environments are provided in Appendix 10.2.

5.3 Analysis of R\(^2\)VPO under Ratio↩︎

5.3.0.1 The Necessity of Variance-Aware Constraints.

Figure 6 (a) visualizes a critical conflict in standard methods: high-value exploration tokens (red points) naturally exhibit high variance and often fall into the clipping region. Standard mechanisms indiscriminately zero out gradients for these informative samples. R\(^2\)VPO addresses this by replacing binary clipping with a distributional “soft brake”. This design is validated by the dynamics of the adaptive dual variable \(\lambda\) shown in Figure 6 (c). During early training, \(\lambda\) remains naturally inactive (\(\approx 0\)), effectively simplifying the objective to an unconstrained form. This implies that the strict clipping enforced by GRPO during early phases is unnecessary and potentially detrimental. As the policy diverges later in training, \(\lambda\) gradually increases (red dashed line), applying a stricter penalty exactly when needed to ensure stability. Consequently, the adaptive \(\lambda\) strategy yields superior asymptotic performance compared to both fixed-\(\lambda\) and GRPO-CH baselines.

5.3.0.2 Robustness to Data Staleness.

We further investigate the impact of off-policy data staleness by varying the replay buffer capacity (\(rb \in \{2, 4, 8\}\)) on the Qwen3-1.7B-Fast model. As shown in the performance curves in Figure 6 (b), GRPO is highly sensitive to staleness, exhibiting significant degradation as the buffer size increases. In sharp contrast, R\(^2\)VPO demonstrates remarkable robustness. To understand the mechanism behind this, Figure 6 (c) visualizes the distribution of policy ratios \(\rho_t\) under varying staleness. We observe that in the context of LLM fine-tuning, increasing staleness does not lead to excessively extreme ratio distributions; this bounded behavior provides empirical support for the reliability of the second-order Taylor expansion underlying our variance-based constraint, even in off-policy regimes. R\(^2\)VPO effectively tolerates the resulting distribution shifts (as shown by the evolving average ratio in the inset plot) throughout training. This confirms that the variance-based penalty allows for the safe handling of stale data without the destructive truncation inherent in hard clipping. Additional visualizations of ratio dynamics across training steps are provided in Appendix 10.1.

5.3.0.3 Ablations on the dual factor \(\lambda\).

To validate the effectiveness of our adaptive constraint mechanism, we compare two distinct strategies on the Qwen3-8B-Fast model: a static assignment (\(\lambda=0.04\)) and our proposed dynamic dual-update mechanism (\(\lambda_{\text{adaptive}}\)). As shown in Figure 6 (c), both variance-regularized variants consistently outperform the GRPO-CH baseline on all benchmarks, confirming the fundamental efficacy of the ratio variance penalty. Crucially, the adaptive variant yields superior asymptotic performance over the static one. The visualization of \(\lambda_{\text{adaptive}}\) (red dashed line) reveals a distinct trajectory: it remains near zero during the early “warm-up” phase and gradually increases as training progresses. This confirms our hypothesis that while early exploration benefits from relaxed constraints, a stricter penalty is increasingly necessary to stabilize updates as the policy diverges from the behavior distribution.

6 Conclusion↩︎

In this work, we proposed R\(^2\)VPO, a principled framework that transcends the limitations of heuristic hard clipping in general RL. By theoretically establishing ratio variance as a superior, distributional proxy for the trust region, we shifted the paradigm from indiscriminate “pointwise truncation” to adaptive “soft regularization”, which preserves gradient signals from high-divergence yet informative exploration, and naturally accommodates the distribution shifts inherent in off-policy learning. Empirical evaluations across two distinct domains confirm that R\(^2\)VPO significantly outperforms standard clipping-based baselines in both asymptotic performance and sample efficiency. While variance regularization provides a robust approximation of the trust region, the penalty term can disproportionately dominate the optimization objective in regimes of extreme stochasticity or large policy divergence. This may result in overly conservative updates that slow down convergence in early training phases. Future work will investigate adaptive scaling mechanisms to mitigate the impact of excessive variance and extend this framework to complex, multi-turn Agentic RL workflows, where stability in long-horizon planning is critical.

Acknowledgements↩︎

We sincerely thank all co-authors for their valuable contributions, insightful discussions, and continuous support throughout this work. We are also grateful to the reviewers for their constructive comments and suggestions, which helped us improve the clarity, positioning, and overall quality of the paper.

Impact Statement↩︎

This paper introduces R\(^2\)VPO, a framework aimed at enhancing the stability and sample efficiency of reinforcement learning in both Large Language Models (LLMs) and robotic control. The primary societal impact of this work lies in its potential to significantly reduce the computational resources required for training capable AI systems. By achieving convergence with approximately fewer rollouts and enabling efficient off-policy data reuse, our method contributes to the goals of “Green AI”, lowering the carbon footprint associated with large-scale model post-training.

Furthermore, by improving the training stability of continuous control policies, this work supports the development of safer and more reliable embodied agents, reducing the risks of physical accidents during deployment. However, as with all advancements that enhance the reasoning and control capabilities of general-purpose AI, there remains a risk of misuse if applied to optimize harmful objectives. We advocate for the responsible deployment of these algorithms, accompanied by robust safety alignment protocols.

7 Theoretical Analyses↩︎

8 Variance Approximation of divergence↩︎

To theoretically justify replacing standard KL-divergence constraints with our variance-based regularization, we empirically analyze the relationship between the variance of the importance ratio \(\rho_t(\theta) = \pi_\theta(a_t|s_t) / \pi_{\text{off}}(a_t|s_t)\) and various statistical divergences. As illustrated in Figure 7, we compare \(\text{Var}(\rho_\theta)\) against six standard divergence metrics: Forward/Reverse KL-Divergence, Jensen-Shannon Divergence, Hellinger Distance, \(\chi^2\)-Divergence, and \(\alpha\)-Divergence (\(\alpha=0.5\)). The plots reveal a strong linear correlation in the near-policy region, aligning with the second-order Taylor expansion where \(D_{f}(\pi_\theta || \pi_{\text{off}}) \approx \frac{f''(1)}{2} \text{Var}(\rho_\theta)\). This confirms that penalizing the variance of \(\rho_t\) effectively acts as a stable, computationally efficient trust-region constraint.

9 Hyperparameters↩︎

We list the detailed hyperparameters used for both LLM mathematical reasoning and continuous robotic control experiments in Table 3.

                             |

                             |

                             |

10 More Experimental Results↩︎

10.1 Results on LLMs Training↩︎

We present the performance comparison of all LLMs across different benchmarks, as shown in Table 4.

3.8pt

To further investigate the optimization efficiency and behavioral changes induced by our method, we visualize the training dynamics across all seven base models.

10.1.0.1 Comparison with an Exact KL Penalty.

We additionally compare R\(^2\)VPO with an exact KL-penalty variant, which replaces the quadratic ratio-variance term with an empirical KL penalty. As shown in Figure 8, the two methods exhibit similar training dynamics in the studied regime, consistent with the observation that the empirical ratio distributions remain concentrated near one. This supports our interpretation that, in practical PPO/GRPO-style updates, the main trade-off is not improved global exactness, but simplicity and deployability: exact KL penalties preserve the full divergence geometry, whereas R\(^2\)VPO keeps the shared local second-order structure and implements it as a lightweight ratio-space regularizer.

10.1.0.2 Reward Convergence.

As illustrated in Figure 9, R\(^2\)VPO exhibits superior sample efficiency. In both Fast and Slow thinking paradigms, R\(^2\)VPO-OFF consistently achieves faster reward convergence and higher final returns. This advantage is particularly pronounced in the Qwen3-1.7B-Fast and DeepSeek-Distill-Qwen-1.5B models, where baselines like GRPO struggle to improve or plateau early. This supports our claim that variance-aware regularization provides a robust signal for policy improvement, preventing the vanishing gradients often caused by aggressive clipping.

10.1.0.3 Reasoning Length Evolution.

Figure 10 tracks the average number of tokens per response. We observe two distinct behavioral shifts correlated with performance gains:

• Expansion in Fast Thinking: For models like Qwen3-1.7B-Fast and Qwen3-8B-Fast, R\(^2\)VPO triggers a rapid expansion in reasoning length. This aligns with their dramatic performance jumps, suggesting that the method successfully unlocks the models’ capacity for extended reasoning chains.

• Retention in Slow Thinking: For models initialized with long-context capabilities (e.g., Qwen3-4B-Thinking), RL training naturally condenses the output. However, unlike GRPO, which often suffers from excessive length reduction (length collapse), R\(^2\)VPO preserves significantly more reasoning tokens. This indicates that our variance constraint protects valuable, complex reasoning paths from being prematurely pruned, maintaining the depth of thought required for hard benchmarks.

10.1.0.4 Detailed Analysis of Ratio Dynamics.

To further validate the stability of R\(^2\)VPO under off-policy conditions, we visualize the evolution of policy ratio distributions across varying degrees of data staleness (\(rb \in \{0, 2, 4, 8\}\)) and training phases in Figure 11. Crucially, the distributions remain tightly concentrated around \(\rho_t=1\) (indicated by the red vertical line) even at maximum staleness, providing strong empirical evidence that the optimization trajectory stays within the valid regime of our second-order variance approximation throughout the training process.

10.2 Results on Robotics Tasks↩︎

To provide a qualitative overview of our experimental settings, we visualize the scenes of the selected DeepMind Control Suite tasks in Figure 12. These environments cover a diverse range of challenges, including high-dimensional locomotion and sparse-reward manipulation.

The full performance comparisons on these robotic tasks are reported in Figure 13. These results demonstrate that R\(^2\)VPO significantly outperforms PPO across the majority of tasks. In particular, R\(^2\)VPO exhibits superior sample efficiency and asymptotic performance in complex environments such as WalkerRun and CheetahRun, while maintaining robust learning in sparse-reward settings like CartpoleSwingupSparse.

10.2.0.1 High-Dimensional Control with Stabilized Actor-Critic Training.

We further evaluate Humanoid-Run and Dog-Run in Figure 14, two more challenging high-dimensional continuous-control tasks. In our initial implementation, both vanilla PPO and vanilla R\(^2\)VPO fail to train reliably, suggesting that these tasks require additional actor-critic stabilization beyond the lightweight setup used in the main experiments. We therefore adopt a WPO [45]-inspired stabilization setup. Under this stronger configuration, both methods become trainable, and R\(^2\)VPO continues to outperform the corresponding PPO baseline. We note that Dog-Run remains sensitive to task-specific hyperparameters: the later-stage performance drop is likely caused by a conservative configuration that limits exploration after early improvement. A full environment-specific sweep may further improve both methods.

11 Computational Infrastructure↩︎

All LLM fine-tuning experiments were conducted on a high-performance computing cluster consisting of Huawei Ascend Atlas A3 nodes.

To assess the computational overhead of our method, we report the average wall-clock time per training iteration across different model scales in Table 5. For the off-policy variant (R\(^2\)VPO-OFF), the inclusion of a replay buffer and a higher Update-to-Data (UTD) ratio of \(2\) results in slightly longer iteration times: approximately 300–400 seconds for 1B models, \(\approx\)​500 seconds for 4B models, and 600–700 seconds for 7B/8B models.

In contrast, the on-policy variant (R\(^2\)VPO-ON) requires approximately 50–100 seconds less per iteration than its off-policy counterpart. Crucially, the runtime of R\(^2\)VPO-ON is strictly comparable to other baseline methods (e.g., GRPO), confirming that the calculation of the variance penalty introduces negligible computational overhead.

References↩︎