1 Introduction↩︎

With the emergence of reasoning-oriented models such as DeepSeek-R1 [1], Reinforcement Learning (RL) has become an increasingly important paradigm for post-training large language models (LLMs), particularly for improving instruction following, alignment, and reasoning capabilities [2]–[4]. Due to the massive scale of LLM, modern LLM RL pipelines often separate rollout generation from gradient computation: responses are sampled by an inference engine (e.g. vLLM [5] or SGLang [6]), whereas log-probabilities and updates are computed by a training engine (e.g. FSDP [7] or Megatron [8]). Even under synchronized parameters, implementation differences in precision, decoding, or serving backends can make the training policy \(\textcolor{blue}{\pi}\) and inference policy \(\textcolor{red}{\mu}\) assign different probabilities to the same trajectories. This inevitably creates the notorious training-inference mismatch issue [9], [10].

Existing algorithmic and infrastructure remedies reduce this mismatch by correcting sampler-side ratios [9], filtering unstable samples [10], decaying learning rates [11], or narrowing system-level discrepancies [12], [13]. These works aim to address the mismatch on the training side by improving the stability of the policy optimization process. However, due to the ever-existing training-inference mismatch, there is no guarantee that effective improvement of the training policy necessarily take the same effect for the inference policy.

This suggests an objective-level view of training-inference mismatch. Since the policy used for rollout and deployment is induced by the inference engine, a model update should not be judged solely by whether it improves the training policy, i.e., the canonical optimization objective of LLM RL algorithms. Instead, the more natural question is:

In face of the objective misalignment revealed above, in this work we aim to break the canonicity of improving the training policy, and propose a new policy optimization principle for LLM RL, named Monotonic Inference Policy Improvement (MIPI): the policy optimization objective should be established to improve the performance of the inference policy monotonically. Following this principle, we propose a two-step LLM RL framework, called Monotonic Inference Policy Update (MIPU). As illustrated in 1, the recipe of MIPU is to decompose the MIPI objective (i.e., \(J(\textcolor{blue}{\mu_{k+1}})-J(\textcolor{blue}{\mu_k})\)) into three terms, and optimize it via a sampler-referenced policy update step (i.e., the direct ‘Step 1’) and a inference-gap-aware acceptance step (i.e., the indirect ‘Step 2’). For Step 1, the training engine performs a sampler-referenced policy update to generate a candidate model that theoretically ensures the halfway monotonicity \(J(\textcolor{red}{\pi_{k+1}})-J(\textcolor{blue}{\mu_k}) \ge 0\). For Step 2, the synchronized candidate is evaluated through an inference-gap-aware acceptance criterion, where the accepted update further ensures the remaining half monotonicity \(J(\textcolor{blue}{\mu_{k+1}}) -J(\textcolor{red}{\pi_{k+1}}) \ge 0\).

In the experiments, we evaluate MIPU under FP8-quantized rollout, a high-mismatch setting where inference-side quantization amplifies the training-inference discrepancy. Across Qwen3-4B and Qwen3-1.7B, MIPU achieves the best average performance and more stable training dynamics. Further analysis shows that Step 1 improves candidate updates, while Step 2 filters unreliable synchronized candidates using an inference-side criterion.

The contributions of this paper are summarized as follows:

• We propose a new learning objective for LLM RL, i.e., MIPI, which aligns with the purpose de facto and circumvents the training-inference mismatch issue.

• We propose a new LLM RL framework, i.e., MIPU, that realizes the MIPI principle in a two-step manner. MIPU allows flexible implementation of both sampler-referenced policy update step and inference-gap-aware update acceptance step.

• Experiments under FP8-quantized rollout show that MIPU improves both performance and training stability. And additional analysis verifies the complementary roles of Step 1 and Step 2 in candidate construction and inference-gap-aware acceptance.

2 Preliminaries↩︎

2.0.0.1 Reinforcement Learning for LLMs

Reinforcement Learning (RL) [14] is general paradigm for sequential decision-making problems. RL has demonstrated its great ability of optimizing pre-trained LLMs [15], [16]. In this paradigm, sequential token generation is modeled as a Markov Decision Process (MDP) \(M = (\mathcal{S}, \mathcal{A}, P, R, \gamma)\), where a state \(s_t = (q, y_{1:t}) \in \mathcal{S}\) is the prompt with the output generated so far, and the action \(a_t \in \mathcal{A}\) is the next token selected from the vocabulary \(\mathcal{V}\). Hence, the transition \(P(s_{t+1}|s_t, a_t)\) is deterministic in this context. An episode starts from a prompt \(s_0\) (out of a predefined set \(\mathcal{D}_0\)) and terminates at an end-of-sequence token or by the maximum sequence length \(H\). The reward \(R(s_t, a_t)\) signal is issued by either a rule-based reward function or a learned reward model. In the scope of this paper, we consider the verifiable reward function. For any non-terminal timestep \(t < T-1\), \(R(s_t, a_t)\) is \(0\); on completion, the terminal reward, denoted by \(R(\tau)\) for the whole sequence, equals \(1\) if \(\tau\) produces a correct and well-formatted answer and \(0\) otherwise.

The policy \(\pi_\theta(a_t \mid s_t)\) in the MDP is the LLM itself, parameterized by \(\theta\), and it defines a probability distribution of next-token generation. We use \(d^{\pi_{\theta}}_{\tau}\) to denote the distribution of the output sequence \(\tau\) generated by \(\pi_{\theta}\) and use \(d^{\pi_{\theta}}_{s,a}, d^{\pi_{\theta}}_{s}\) for the state-action pairs \((s,a)\) and the state respectively. The learning objective of an RL policy is to maximize the reward function, i.e., \(\pi^* = \arg \max_{\pi_{\theta}} J(\pi_{\theta})\), where \(J(\pi) = \mathbb{E}_{\tau \sim \pi} \left[ R(\tau) \right]\).

2.0.0.2 Monotonic Policy Improvement and Proximal Policy Gradient

Optimizing RL policy according to the vanilla policy gradient \(\nabla_{\pi}J(\pi)\) often turns out to be ineffective as the performance is sensitive to the step size. To this end, [17] revisits the policy difference [18] for arbitrary two policies \(\pi\) and \(\pi^{\prime}\) as defined below: \[\Delta(\pi^{\prime},\pi) := J(\pi^{\prime}) - J(\pi) = \mathbb{E}_{s,a \sim d^{\pi^{\prime}}} \left[ A^{\pi}(s,a) \right], \label{eq:exact95policy95difference}\tag{1}\] where \(d^{\pi^{\prime}}\) denotes the discounted state visitation distribution induced by policy \(\pi^{\prime}\), and let \(A^{\pi}(s,a)\) denote the advantage of taking action \(a\) at state \(s\) under policy \(\pi\). Then, a natural idea is to ensure monotonic policy improvement during iterative policy optimization. Formally, given an old policy \(\pi_k\), the goal of monotonic policy improvement is to find a new policy \(\pi_{k+1}\) such that \(J(\pi_{k+1})-J(\pi_k)\geq 0\), i.e., \(\Delta(\pi_{k+1}, \pi_k) \ge 0\).

However, directly optimizing 1 is infeasible because it depends on the visitation distribution of the updated policy, i.e., \(\pi_{k+1}\). [17] propose Trust-Region Policy Optimization (TRPO) to approximate the exact objective difference by replacing the updated visitation distribution with the old-policy distribution and estimating with the importance sampling ratio: \[\widetilde{\Delta}(\pi_{k+1},\pi_k) := \mathbb{E}_{s\sim d^{\pi_k},\,a\sim\pi_k(\cdot|s)} \left[ \frac{\pi_{k+1}(a\mid s)}{\pi_k(a\mid s)} A^{\pi_k}(s,a) \right]. \label{eq:ratio95surrogate}\tag{2}\] This local surrogate, denoted by \(\tilde{\Delta}\), forms the basis of practical proximal policy optimization algorithms. TRPO maximizes such a surrogate subject to a KL-based trust-region constraint between the old and updated policies, while Proximal Policy Optimization (PPO) [19] replaces the constrained optimization with a clipped probability-ratio objective that discourages excessively large policy updates. Based on PPO, Group Relative Policy Optimization (GRPO) [20] adopts a group-based advantage estimation which is free of learning the value function.

2.0.0.3 Training-Inference Mismatch in LLM RL

As discussed above, modern LLM RL pipelines may induce different action distributions in the training and inference engines, even when they share synchronized model parameters. We model this training-inference mismatch by distinguishing the training policy \(\textcolor{blue}{\pi}\) from the inference policy \(\textcolor{red}{\mu}\): at update step \(k\), with the same parameters, \(\textcolor{blue}{\pi_k}\) is the policy represented inside the training engine, whereas \(\textcolor{red}{\mu_k}\) is the policy actually used for rollout generation and deployment in the inference engine. The training-inference mismatch manifests as \(\textcolor{blue}{\pi}\neq\textcolor{red}{\mu}\), even though they share the same model parameters. This mismatch thus results in new learning issues regarding non-stationarity and suboptimality different from the notorious ones in traditional RL [21]–[23]. In the following of this paper, we will use the notations to explicitly highlight the distinction between training policies and inference policies. This distinction provides the basis for our method, where we revisit policy improvement from the perspective of the inference policy rather than the training-side surrogate.

3 Related Works↩︎

3.0.0.1 Reinforcement learning for LLM post-training

RL has become a prominent paradigm for LLM post-training, especially for improving reasoning, coding, and long-horizon tasking capabilities. A major recent direction is reinforcement learning with verifiable rewards (RLVR), where outcome rewards can be obtained from automatically checkable signals such as mathematical answers, code execution, or task success. Representative works include GRPO [20], DAPO [2], and GSPO [24], which develop scalable RL recipes for mathematical and reasoning tasks. In parallel, recent studies have also explored off-policy or data-reuse variants of reinforced fine-tuning [25], [26], highlighting the importance of correcting distributional mismatch during LLM RL. More recently, RL has also been adopted in agentic model training, where models learn from environment feedback in tool-use, coding, and multi-turn interaction settings. Examples include GLM-4.5 [27], Kimi K2.5 [28], and ROME [29]. Together, these works show the broad potential of RL for improving LLM capabilities, while also highlighting the fragility of RL training under long-horizon rollouts, complex optimization dynamics, and system-level implementation details.

3.0.0.2 Training-inference mismatch in LLM RL

Recent work has identified training-inference mismatch as an important source of instability in modern LLM RL systems. A common line of algorithmic approaches incorporates sampler-side information into the training update. For example, TIS [9] uses the training-to-sampler probability ratio as a clipped correction weight, while MIS [10] filters tokens or sequences with extreme mismatch signals. Learning rate decay [11] provides another optimization-side remedy by reducing the update magnitude when mismatch-related instability emerges. In parallel, infrastructure-level approaches study how system design choices affect training-inference mismatch. For example, [13] show that numerical precision can be a direct source of mismatch and propose using FP16 rollout to reduce the discrepancy between training and inference engines. [12] further investigate low-precision RL systems and analyze how quantized rollout affects reasoning-oriented RL training. These works reduce mismatch from the system side, while sampler-aware methods such as TIS and MIS correct or filter the training update from the algorithm side. Our work is complementary: instead of only reducing mismatch, we study its objective-level consequence and ask whether a synchronized update should be accepted as the next inference policy.

4 The New Objective:       Monotonic Policy Improvement on the Inference Side↩︎

As introduced in 2, training-inference mismatch separates the training policy \(\textcolor{blue}{\pi}\) from the inference policy \(\textcolor{red}{\mu}\). As illustrated in 1, a training-side improvement can still lead to a harmful inference policy after synchronization. Thus, when \(\textcolor{blue}{\pi}\neq\textcolor{red}{\mu}\), improving the training policy does not necessarily imply improving the inference policy: \[J(\textcolor{blue}{\pi_{k+1}}) - J(\textcolor{blue}{\pi_k}) \geq 0 \;\nRightarrow\; J(\textcolor{red}{\mu_{k+1}}) - J(\textcolor{red}{\mu_k}) \geq 0. \label{eq:mismatch95implication}\tag{3}\]

We now instantiate this mismatch in the GRPO objective. In GRPO training, the training-side improvement is implemented by the clipped surrogate \[\begin{align} \mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E}_{x\sim\mathcal{D},\, \{y_i\}_{i=1}^{G} \sim \textcolor{red}{\mu_k}(\cdot\mid x)} \left[ \frac{1}{G} \sum_{i=1}^{G} \min \left( r_i(\theta)\hat{A}^{\textcolor{red}{\mu_k}}_i, \operatorname{clip} \left( r_i(\theta), 1-\epsilon, 1+\epsilon \right) \hat{A}^{\textcolor{red}{\mu_k}}_i \right) \right], \label{eq:grpo95objective} \end{align}\tag{4}\] where \(r_i(\theta)=\frac{\textcolor{blue}{\pi_\theta}(y_i\mid x)}{\textcolor{blue}{\pi_k}(y_i\mid x)}\) is the training-side probability ratio, and optimizing 4 yields the next training policy \(\textcolor{blue}{\pi_{k+1}}:=\textcolor{blue}{\pi_{\theta^\star}}\). The superscript in \(\hat{A}^{\textcolor{red}{\mu_k}}_i\) emphasizes that the GRPO advantage is not an explicit estimate of the old training-policy advantage \(A^{\textcolor{blue}{\pi_k}}\). Instead, it is computed from the response group sampled by the inference policy \(\textcolor{red}{\mu_k}\): GRPO normalizes the reward of \(y_i\) using the mean and standard deviation of rewards within \(\{y_j\}_{j=1}^G\sim\textcolor{red}{\mu_k}(\cdot\mid x)\).

Therefore, beyond the policy-reference mismatch discussed above, GRPO introduces an additional mismatch-induced bias through its advantage estimation. Unlike the standard policy surrogate in 2 , whose weighting term is the old-policy advantage \(A^{\textcolor{blue}{\pi_k}}\), GRPO weights the update by the group-relative statistic \(\hat{A}^{\textcolor{red}{\mu_k}}_i\) induced by inference rollouts. When \(\textcolor{red}{\mu_k}\neq\textcolor{blue}{\pi_k}\), this statistic is no longer aligned with the advantage of the old training policy. Thus, under training-inference mismatch, the standard GRPO update is affected by both a ratio-level mismatch and an advantage-level bias.

More importantly, these mismatches reveal a deeper objective-level misalignment. The standard GRPO surrogate optimizes a training-policy update, but the resulting synchronized inference policy is the one ultimately used for deployment. Under training-inference mismatch, the training-policy transition \(\textcolor{blue}{\pi_k}\rightarrow\textcolor{blue}{\pi_{k+1}}\) may differ from the inference-policy transition \(\textcolor{red}{\mu_k}\rightarrow\textcolor{red}{\mu_{k+1}}\). This motivates us to move beyond correcting individual terms in the training objective and instead ask what policy improvement should mean when these two transitions differ.

4.1 MIPI: Monotonic Inference Policy Improvement↩︎

To address this problem, we propose Monotonic Inference Policy Improvement (MIPI), a policy improvement principle for LLM RL under training-inference mismatch.MIPI aims to align the update with the inference-policy transition rather than only with the training-side surrogate. Formally, MIPI takes the inference-policy improvement \(J(\textcolor{red}{\mu_{k+1}})-J(\textcolor{red}{\mu_k})\) as the target quantity.

To make this principle explicit, we decompose the inference-policy improvement as follows: \[\begin{align} J(\textcolor{red}{\mu_{k+1}}) - J(\textcolor{red}{\mu_k}) = \underbrace{ J(\textcolor{red}{\mu_{k+1}}) - J(\textcolor{blue}{\pi_{k+1}}) }_{\text{\textcircled{1} post-update inference gap}} + \underbrace{ J(\textcolor{blue}{\pi_{k+1}}) - J(\textcolor{blue}{\pi_k}) }_{\text{\textcircled{2} training-side update}} + \underbrace{ J(\textcolor{blue}{\pi_k}) - J(\textcolor{red}{\mu_k}) }_{\text{\textcircled{3} pre-update inference gap}} . \end{align} \label{eq:deployed95improvement95decomposition}\tag{5}\]

This decomposition provides the formal basis of MIPI. It separates the inference-policy improvement into a candidate construction part and a post-synchronization verification part. Accordingly, as illustrated in 1, MIPU realizes MIPI through the following two steps:

4.2 Step 1: Sampler-Referenced Policy Update↩︎

Step 1 targets Terms 2⃝+3⃝ in 5 , which telescope to \(J(\textcolor{blue}{\pi_{k+1}})-J(\textcolor{red}{\mu_k})\). Equivalently, this grouped target can be written as \(\Delta(\textcolor{blue}{\pi_{k+1}},\textcolor{red}{\mu_k})\). Following 2 , viewing this target relative to the sampler \(\textcolor{red}{\mu_k}\) leads to the sampler-referenced weighting term \((\textcolor{blue}{\pi_\theta}/\textcolor{red}{\mu_k})A^{\textcolor{red}{\mu_k}}\). As discussed after 4 , GRPO uses the rollout-induced group-relative statistic \(\hat{A}^{\textcolor{red}{\mu_k}}_i\), so this formulation keeps both the probability ratio and the advantage statistic aligned with the sampler.

By contrast, the standard GRPO surrogate in 4 clips the training-side ratio \(\textcolor{blue}{\pi_\theta}/\textcolor{blue}{\pi_k}\), although the samples are generated by the inference policy \(\textcolor{red}{\mu_k}\).

Thus, the proximal constraint is centered at the old training policy \(\textcolor{blue}{\pi_k}\) rather than at the sampler \(\textcolor{red}{\mu_k}\). Following the terminology of [9], we refer to this direct sampler-referenced variant as PPO-IS, instead clips the full trainer-to-sampler ratio \(\rho_i(\theta)=\textcolor{blue}{\pi_\theta}(y_i\mid x)/\textcolor{red}{\mu_k}(y_i\mid x)\). However, this full ratio conflates two different effects: the pre-update training-inference mismatch and the current training update: \[\rho_i(\theta) = \underbrace{ \frac{ \textcolor{blue}{\pi_k}(y_i\mid x) }{ \textcolor{red}{\mu_k}(y_i\mid x) } }_{w_i^k:\;\text{pre-update mismatch}} \cdot \underbrace{ \frac{ \textcolor{blue}{\pi_\theta}(y_i\mid x) }{ \textcolor{blue}{\pi_k}(y_i\mid x) } }_{r_i(\theta):\;\text{current update}} . \label{eq:ratio95factorization}\tag{6}\] When \(w_i^k\) is already outside the clipping range, PPO-IS can over-constrain the update even when the current update ratio \(r_i(\theta)\) remains close to one.

This factorization suggests separating the pre-update mismatch correction from the clipping of the current update. Following the same terminology, Vanilla-IS follows this idea by using the full correction weight \(w_i^k\) while clipping only \(r_i(\theta)\), but the unbounded correction can introduce large variance. We therefore adopt the truncated variant TIS, which replaces \(w_i^k\) with \(\bar{w}*i^k=\min(w_i^k,w*{\max})\) and still applies PPO clipping only to \(r_i(\theta)\). Detailed comparisons among PPO-IS, Vanilla-IS, and TIS are provided in 8.

Accordingly, Step 1 keeps a truncated sampler-referenced correction \(\bar{w}_i^k\) while applying PPO-style clipping only to the current-update ratio \(r_i(\theta)\). The Step-1 surrogate is \[\begin{align} \mathcal{J}_{\mathrm{S1}}(\theta) = \mathbb{E}_{x\sim\mathcal{D},\, \{y_i\}_{i=1}^{G}\sim \textcolor{red}{\mu_k}(\cdot\mid x)} \left[ \frac{1}{G} \sum_{i=1}^{G} \bar{w}_i^k \min \left( r_i(\theta)\hat{A}^{\textcolor{red}{\mu_k}}_i, \operatorname{clip} \left( r_i(\theta), 1-\epsilon, 1+\epsilon \right) \hat{A}^{\textcolor{red}{\mu_k}}_i \right) \right]. \end{align} \label{eq:step195surrogate}\tag{7}\]

The truncated weight \(\bar{w}_i^k\) retains the sampler-referenced correction while preventing large pre-update discrepancies between \(\textcolor{blue}{\pi_k}\) and \(\textcolor{red}{\mu_k}\) from dominating the gradient. The proximal constraint is still applied only to \(r_i(\theta)\), the part of the ratio corresponding to the current training-side update \(\textcolor{blue}{\pi_k}\rightarrow\textcolor{blue}{\pi_\theta}\). Optimizing 7 yields a candidate training policy \(\textcolor{blue}{\pi_{k+1}}:=\textcolor{blue}{\pi_{\theta^\star}}\).

Step 1 provides a sampler-referenced proposal for the combined term \(J(\textcolor{blue}{\pi_{k+1}})-J(\textcolor{red}{\mu_k})\) in 5 . It does not, however, determine whether the synchronized inference policy \(\textcolor{red}{\mu_{k+1}}\) realizes the gain proposed by the trainer. This post-update consistency is checked in Step 2.

4.3 Step 2: Inference-Gap-Aware Update Acceptance↩︎

After Step 1, the candidate training policy \(\textcolor{blue}{\pi_{k+1}}\) is synchronized to the inference engine, producing \(\textcolor{red}{\mu_{k+1}}\). The remaining term in 5 is the post-update inference gap \(T_{\mathrm{post}} = J(\textcolor{red}{\mu_{k+1}}) - J(\textcolor{blue}{\pi_{k+1}})\).

A negative value of \(T_{\mathrm{post}}\) means that the inference policy underperforms its training-side counterpart after synchronization. In our proxy, this corresponds to an unfavorable mismatch pattern: the inference policy assigns insufficient probability to positive-advantage responses or excessive probability to negative-advantage responses relative to the training policy. Such updates are therefore unreliable, even if they appear beneficial on the training side.

By the performance difference identity, the direct form of the post-update gap \(T_{\mathrm{post}}=\Delta(\textcolor{red}{\mu_{k+1}},\textcolor{blue}{\pi_{k+1}})\) requires the advantage under the candidate training policy \(\textcolor{blue}{\pi_{k+1}}\), which is not directly available in GRPO-style training. We therefore use the reverse identity, which anchors the advantage term at \(\textcolor{red}{\mu_{k+1}}\)and makes it compatible with validation rollouts from the synchronized inference policy. This yields a validation-based proxy with an importance correction from \(\textcolor{red}{\mu_{k+1}}\) to \(\textcolor{blue}{\pi_{k+1}}\): \[\begin{align} T_{\mathrm{post}} &= - \Delta(\textcolor{blue}{\pi_{k+1}}, \textcolor{red}{\mu_{k+1}}) = - \mathbb{E}_{s,a\sim d^{\textcolor{blue}{\pi_{k+1}}}} \left[ A^{\textcolor{red}{\mu_{k+1}}}(s,a) \right] \\ &\leadsto \widehat{T}_{\mathrm{post}} = - \mathbb{E}_{x\sim\mathcal{D}_{\mathrm{val}},\,y_i\sim\textcolor{red}{\mu_{k+1}}} \left[ \rho_i \hat{A}^{\textcolor{red}{\mu_{k+1}}}_i \right]. \end{align} \label{eq:post95gap95proxy}\tag{8}\] Here, \(\hat{A}^{\textcolor{red}{\mu_{k+1}}}_i\) is the group-relative advantage computed from validation responses sampled by \(\textcolor{red}{\mu_{k+1}}\). The symbol \(\leadsto\) emphasizes that \(\widehat{T}_{\mathrm{post}}\) is a stable proxy motivated by the reverse identity, rather than an exact estimator of \(T_{\mathrm{post}}\). Since the reverse form takes expectation under \(\textcolor{blue}{\pi_{k+1}}\) while validation responses are sampled from \(\textcolor{red}{\mu_{k+1}}\), we introduce an importance weight. For each completion \(y_i\), we use the length-normalized sequence ratio \[\rho_i = \exp \left( \frac{1}{T_i} \sum_{t=1}^{T_i} \log \frac{ \textcolor{blue}{\pi_{k+1}}(y_{i,t}\mid x,y_{i,<t}) }{ \textcolor{red}{\mu_{k+1}}(y_{i,t}\mid x,y_{i,<t}) } \right). \label{eq:geometric95sequence95ratio}\tag{9}\] This length-normalized ratio serves as a more stabilized importance weight for approximating the reverse-form expectation, avoiding the high variance of full sequence-level importance sampling ratio.

The proxy in 8 serves as a risk signal. Since \(\widehat T_{\mathrm{post}}=-\mathbb{E}_{\textcolor{red}{\mu_{k+1}}}[\rho_i\hat{A}^{\textcolor{red}{\mu_{k+1}}}_i]\), a more negative value indicates that \(\rho_i\) is positively associated with the validation advantage statistic. In other words, compared with the synchronized inference policy, the candidate training policy assigns relatively higher probability to positive-advantage responses or lower probability to negative-advantage responses. This suggests that the training-side proposal may not be realized by the inference engine. It does not prove that the inference policy is worse than the previous one, but marks the update as unreliable under training-inference mismatch.

Therefore, we use the post-update gap proxy as an acceptance test: \(\widehat{T}_{\mathrm{post}} \geq -c\), where \(c\geq 0\) is a tolerance parameter for noise in the proxy. This rule tolerates mild negative proxy values but rejects candidates with a clear negative post-update gap signal. If passing the test, we accept the candidate and set \(\textcolor{blue}{\pi_k}\leftarrow\textcolor{blue}{\pi_{k+1}}\) and \(\textcolor{red}{\mu_k}\leftarrow\textcolor{red}{\mu_{k+1}}\). Otherwise, we reject the candidate and roll back both the trainer state and the inference engine to the previous checkpoint. The implementation details are provided in 7.2.

Overall, Step 1 turns the trainer update into a sampler-referenced candidate, and Step 2 filters candidates with a large post-update inference gap. Algorithm 2 summarizes the full MIPU procedure, including checkpointing, synchronization, validation, and rollback. This mechanism does not provide a formal monotonic-improvement guarantee, but it reduces the risk of accumulating updates whose gains are not realized by the inference policy.

5 Experiments↩︎

We evaluate MIPU under FP8-quantized rollout, where quantized inference amplifies training-inference mismatch. Our experiments are organized around three research questions:

• RQ1: Does MIPU improve performance and training stability under high training-inference mismatch? (5.2)

• RQ2: Do Step 1 and Step 2 play complementary roles in MIPU? (5.3)

• RQ3: Is Step 2 effective because of its inference-gap signal, or simply because it rejects more updates? (5.4)

5.1 Experiments Setup↩︎

5.1.0.1 Training.

We train Qwen3-1.7B and Qwen3-4B under the FP8-quantized rollout, where rollout generation uses FP8-quantized inference and therefore induces a high training-inference mismatch. All analyses beyond the main results are conducted on Qwen3-4B. The detailed hyperparameter choices are provided in 7.1. For training data, we use DAPO-Math-17 [2] and DeepMath-103K [30] as our training corpora. Qwen3-1.7B is trained on 5759 examples filtered from DAPO, while Qwen3-4B is trained on 1491 examples filtered from DeepMath. The filtering retains problems that the base model can solve with non-trivial but non-saturated success rates, ensuring that the training set provides meaningful reward variation for RL.

5.1.0.2 Evaluation.

We evaluate all methods on five mathematical reasoning benchmarks: MATH-500 [31], AIME24 [32], AMC23 [33], Minerva [34], and OlympiadBench [35]. We report pass@1 accuracy for all benchmarks. For the small benchmarks AIME24 and AMC23, we use avg@16 to reduce evaluation variance and improve evaluation reliability. All reported scores are computed using the same evaluation protocol across methods. For diagnostics, we report the inference-training K3-KL (denoted as Mismatch-K3 in the following figures), computed on rollout tokens sampled from \(\mu\). Specifically, with \(d_t=\log \textcolor{blue}{\pi}(y_t\mid s_t)-\log \textcolor{red}{\mu}(y_t\mid s_t)\), we average \(\exp(d_t)-d_t-1\), which estimates \(D_{\mathrm{KL}}(\textcolor{red}{\mu}\|\textcolor{blue}{\pi})\).

5.1.0.3 Baselines.

Our work focuses on stabilizing the RL objective under training-inference mismatch, which is orthogonal to system-level infrastructure optimization. We therefore do not compare with infrastructure methods. For the main comparison, we compare against the standard GRPO baseline and two optimization-side stabilization baselines, MIS and LR-decay. Since MIPU instantiates Step 1 with a TIS-style correction, we analyze TIS separately as the Step 1-only variant in Sec. 5.3. The implementation details are in 7.

5.2 Main Results↩︎

1 reports the main results under FP8-quantized rollout, the high-mismatch setting where quantized inference amplifies the gap between the training policy \(\pi\) and the inference policy \(\mu\). On both model scales, MIPU achieves the best average performance, reaching \(66.71\%\) on Qwen3-4B and \(53.97\%\) on Qwen3-1.7B. For Qwen3-4B, the gains are most visible on AMC23 (\(85.00\%\)) and Minerva (\(45.96\%\)); for Qwen3-1.7B, the gains are most visible on MATH-500 (\(86.52\%\)) and OlympiadBench (\(59.52\%\)). Beyond peak accuracy, the Stable column in 1 shows that MIPU remains stable until the end of training, while several baselines suffer from collapse or sharp degradation.

It is worth noting that some baselines can achieve competitive intermediate performance, but their gains are not sustained under continued training. This distinction is important in high-mismatch RL, where a transient peak may be followed by rapid degradation once mismatch-driven errors accumulate. Therefore, 3 further visualizes the training dynamics under the Qwen3-4B FP8-quantized rollout. Baseline, MIS, and LR-decay can reach reasonable intermediate performance but later degrade during continued training. In contrast, MIPU maintains a stable trajectory after reaching high performance. Together, the table and training curve show that MIPU improves both peak performance and long-horizon training stability in the high-mismatch FP8-quantized rollout.

5.3 Ablation Studies↩︎

We ablate MIPU under the representative Qwen3-4B FP8-quantized rollout. We compare the baseline GRPO update, Step 1 only, Step 2 only, and the full Step 1+Step 2 framework. This ablation is designed to disentangle two complementary roles: Step 1 constructs sampler-referenced candidate updates, while Step 2 decides whether the synchronized inference policy should accept the candidate according to \(\widehat{T}_{\mathrm{post}}\).

2 and 4 show that Step 1 and Step 2 address different failure modes. The baseline update is centered at the training policy \(\pi_k\) even though rollouts are sampled from \(\mu_k\), which makes the synchronized inference policy vulnerable to FP8-induced mismatch and can eventually lead to collapse. Step 2 alone can prevent collapse by rejecting candidates with large negative \(\widehat{T}_{\mathrm{post}}\), but it cannot improve the quality of the underlying baseline update. Thus, when the baseline rarely proposes useful candidates, Step 2 mostly keeps the previous inference policy rather than creating further improvement. Its acceptance rule improves safety, but it cannot compensate for poor candidate quality when the proposal update itself remains biased by the training-inference mismatch.

Step 1 improves learning by constructing sampler-referenced candidate updates before synchronization. However, Step 1 alone accepts every synchronized candidate as the next inference policy, so mismatch fluctuations can still accumulate. The full method combines the two roles: Step 1 produces better candidate updates, and Step 2 filters candidates whose post-update inference gap suggests that the gain may not be realized by \(\mu_{k+1}\).

In summary, Step 1 and Step 2 address different bottlenecks: Step 1 improves the candidate update direction, while Step 2 controls whether the synchronized candidate should be accepted as the next inference policy. Strong performance requires corrected updates from Step 1 together with inference-gap-aware acceptance from Step 2. This division of roles explains why the full method improves both the final score and the stability of the training trajectory, rather than merely shifting the best checkpoint along an unstable run.

5.4 Analysis of Step 2: Inference-Gap-Aware Acceptance↩︎

5.4.0.1 Post-update gap as an inference-side signal.

We first inspect whether the post-update gap proxy \(\widehat{T}_{\mathrm{post}}\) carries a meaningful mismatch signal. As shown in 5 (a), under FP8-quantized rollout, the smaller Qwen3-1.7B model exhibits a larger training-inference mismatch than Qwen3-4B. Consistently, its \(\widehat{T}_{\mathrm{post}}\) has a larger magnitude and oscillates more strongly, indicating a more volatile inference-side gap under quantized rollout. Moreover, both models show a training-dependent pattern: the mismatch increases during training, and \(\widehat{T}_{\mathrm{post}}\) also shifts from initially low values toward higher values. These observations suggest the use of Step 2 in high-mismatch settings, where \(\widehat{T}_{\mathrm{post}}\) carries a structured inference-side signal rather than behaving as uninformative boundary-level noise. This also motivates using \(\widehat{T}_{\mathrm{post}}\)as an acceptance signal: it reflects whether the synchronized inference policy remains consistent with the candidate update in a mismatch-relevant direction, rather than merely measuring the magnitude of the mismatch.

5.4.0.2 Signal-conditioned rollback.

We next test whether the benefit of Step 2 can be explained merely by reducing the number of accepted updates. We compare it with a random rollback control that rejects candidate updates without using \(\widehat{T}_{\mathrm{post}}\). Since the Step 2-only run, without Step 1 correction, has an overall rollback rate of around \(70\%\) within the first \(500\) steps, we set the random rollback probability to \(70\%\). In the comparison window shown in 5 (b), random rollback is even more conservative than Step 2, rejecting \(67.0\%\) of candidate updates compared with \(53.5\%\) for Step 2.

Despite rejecting more updates, random rollback still collapses after reaching a transient peak, whereas Step 2 maintains a stable score trajectory. This shows that Step 2 is not a generic update-sparsification mechanism. Its stability comes from conditioning the acceptance decision on the post-update gap proxy \(\widehat{T}_{\mathrm{post}}\), which helps filter unreliable synchronized candidates rather than randomly reducing the number of accepted updates. Random rollback is more conservative in terms of accepted-update count, but it cannot distinguish harmful synchronized candidates from benign ones. By contrast, Step 2 uses \(\widehat{T}_{\mathrm{post}}\) to identify candidates whose training-side gains are unlikely to be realized by the inference policy. Overly conservative rejection may preserve a stale inference policy, whereas inference-gap-aware acceptance allows useful candidates to pass while filtering candidates with mismatch-relevant risk.

We further analyze the sensitivity of the acceptance tolerance in 9, showing that Step 2 requires calibrated acceptance rather than simply stricter rejection.

6 Conclusion↩︎

We revisit training-inference mismatch in LLM reinforcement learning from an objective-level perspective. Rather than treating mismatch only as a system discrepancy to be reduced, we show that it changes the object of policy improvement: standard RL updates optimize the training policy, while deployment depends on the inference policy. Based on this view, we formulate Monotonic Inference Policy Improvement (MIPI) and propose Monotonic Inference Policy Update (MIPU), which realizes MIPI through sampler-referenced candidate construction and inference-gap-aware candidate acceptance. Experiments under FP8-quantized rollout show that MIPU improves average reasoning performance and training stability, and further analysis confirms the rationality of Step 2. These results suggest that training-inference mismatch should not be treated only as a low-level system discrepancy, but also as a change in the objective of policy improvement.

6.0.0.1 Limitations.

Due to significant computational constraints, our experiments are currently limited to moderate-scale models. Future work should examine whether the same mismatch patterns also appear in larger models and more diverse RL training systems. In addition, Step 2 should be viewed as a flexible design space rather than a fixed implementation. Our validation-based post-update gap proxy provides one effective instantiation for evaluating synchronized candidates from the deployment side. Future work may further improve this component with more efficient estimators, stronger acceptance criteria, or optimization schemes that directly leverage the post-update inference gap.

7 Implement Details↩︎

7.1 Hyperparmeters↩︎

The major hyperparameter choices are shown in 3.

7.2 Implementation Details↩︎

7.2.0.1 Baseline.

We use vanilla GRPO with the same rollout, reward, and optimization settings as other methods. In implementation, we additionally use the dual-clipped policy loss. For a token-level ratio \(r_{i,t}(\theta)=\pi_\theta(y_{i,t}\mid x,y_{i,<t})/\pi_k(y_{i,t}\mid x,y_{i,<t})\) and group-relative advantage \(\hat{A}_i\), the standard clipped objective is \(\ell^{\mathrm{clip}}_{i,t}=\min\!\left(r_{i,t}(\theta)\hat{A}_i,\operatorname{clip}(r_{i,t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_i\right)\). The dual-clipped objective is then \[\ell^{\mathrm{dc}}_{i,t} = \begin{cases} \ell^{\mathrm{clip}}_{i,t}, & \hat{A}_i \ge 0,\\ \max\!\left(\ell^{\mathrm{clip}}_{i,t}, (1+2\epsilon)\hat{A}_i\right), & \hat{A}_i < 0, \end{cases}\] where we set \(\epsilon=0.2\). The baseline maximizes the average dual-clipped GRPO objective over sampled responses.

7.2.0.2 Ours.

For Step 1, we instantiate the sampler-referenced update with token-level truncated importance weights. Specifically, for each generated token, we compute the training-to-inference mismatch weight \(w^k_{i,t}\) and set \(w_{\max}=2\). The weight is applied after the dual-clipped GRPO objective is computed, yielding the Step 1 token-level objective \[\mathcal{J}_{\mathrm{S1}} = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{T_i} \sum_{t=1}^{T_i} \bar{w}^k_{i,t}\, \ell^{\mathrm{dc}}_{i,t} \right].\] Vanilla-IS follows the same implementation except that it uses the untruncated weight \(w^k_{i,t}\) instead of the truncted \(\bar{w}^k_{i,t}\).

For Step 2, after the candidate training policy \(\pi_{k+1}\) is synchronized to the inference engine as \(\mu_{k+1}\), we estimate the post-update inference gap proxy \(\widehat{T}_{\mathrm{post}}\) on a validation batch. The candidate is accepted if \(\widehat{T}_{\mathrm{post}}\ge -c_t\), where \(c_t\ge 0\) is a dynamic acceptance tolerance parameter. Motivated by the observation in 5 (a) that \(\widehat{T}_{\mathrm{post}}\) is initially low and then gradually increases under FP8 rollout, we use a larger tolerance in the early stage to avoid over-rejecting useful candidates. The threshold is linearly annealed during the first \(100\) steps: \[c_t = c_{\mathrm{start}} + \frac{\min(t,100)}{100} \left( c_{\mathrm{end}}-c_{\mathrm{start}} \right).\] For Qwen3-1.7B, we set \(c_{\mathrm{start}}=4\times 10^{-3}\) and \(c_{\mathrm{end}}=1\times 10^{-3}\). For Qwen3-4B, we set \(c_{\mathrm{start}}=1\times 10^{-3}\) and \(c_{\mathrm{end}}=0\). After the first \(100\) steps, the tolerance is fixed at \(\tau_{\mathrm{end}}\). We further analyze the effect of different tolerance choices in 9.

7.2.0.3 MIS.

We implement MIS as a token-level mismatch-filtering baseline. For each token, we compute the same training-to-inference ratio \(w^k_{i,t}\). Tokens whose mismatch weight exceeds the upper bound are masked out: \[m^k_{i,t} = \mathbb{1} \left[ w^k_{i,t}\le \bar{w} \right], \qquad \bar{w}=2.\] The MIS objective applies this mask to the dual-clipped GRPO objective: \[\mathcal{J}_{\mathrm{MIS}} = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^{G} \frac{1}{T_i} \sum_{t=1}^{T_i} m^k_{i,t}\, \ell^{\mathrm{dc}}_{i,t} \right].\]

7.2.0.4 LR-decay.

We implement LR-decay following the original decay rule. Let \(\eta_t\) denote the learning rate at training step \(t\). At every decay interval \(T_{\mathrm{decay}}\), the learning rate is halved until it reaches a lower bound \(\eta_\infty\): \[\eta_t = \begin{cases} \max\!\left(\eta_{t-1}/2,\eta_\infty\right), & t \equiv 0 \pmod{T_{\mathrm{decay}}},\\ \eta_{t-1}, & \text{otherwise}. \end{cases}\] The model parameters are then updated using \(\eta_t\). Following the authors’ guidance, we set \(T_{\mathrm{decay}}=249\) for Qwen3-1.7B and \(T_{\mathrm{decay}}=234\) for Qwen3-4B under FP8 rollout.

8 Analysis of Step 1: Sampler-Referenced Update↩︎

Step 1 implements the sampler-referenced update derived in 6 . For readability, we omit token or trajectory arguments in this section, and all ratios are evaluated on sampled rollouts. A direct implementation is PPO-IS, which clips the total trainer-to-sampler ratio: \[\boldsymbol{PPO-IS:} \qquad \omega_{\mathrm{PPO\text{-}IS}} = \operatorname{clip}\!\left( \frac{\textcolor{blue}{\pi_{\theta}}}{\textcolor{red}{\mu_k}}, 1-\epsilon, 1+\epsilon \right).\] Although this ratio directly references the sampler, it mixes the pre-update mismatch between \(\textcolor{blue}{\pi_k}\) and \(\textcolor{red}{\mu_k}\) with the current update from \(\textcolor{blue}{\pi_k}\) to \(\textcolor{blue}{\pi_{\theta}}\). As a result, the ratio can already deviate substantially from \(1\) even before the current update is applied. This makes the PPO clipping overly active and therefore less informative, because the trust-region constraint is no longer centered on the current training-side update itself. This phenomenon is also consistent with the analysis in [9]. Empirically, 6 (b) shows that PPO-IS has a much larger clipped ratio than the other variants, and 6 (a) shows that, although it reduces the measured inference-training KL, it fails to produce a sustained upward learning trajectory.

We therefore use the factorization \[\frac{\textcolor{blue}{\pi_{\theta}}}{\textcolor{red}{\mu_k}} = \underbrace{ \frac{\textcolor{blue}{\pi_k}}{\textcolor{red}{\mu_k}} }_{\text{pre-update mismatch correction}} \cdot \underbrace{ \frac{\textcolor{blue}{\pi_{\theta}}}{\textcolor{blue}{\pi_k}} }_{\text{current update ratio}},\] which separates the pre-update mismatch correction from the current training-side update. This leads to two decomposed variants: \[\begin{gather} \boldsymbol{Vanilla-IS:} \qquad \omega_{\mathrm{Vanilla\text{-}IS}} = \frac{\textcolor{blue}{\pi_k}}{\textcolor{red}{\mu_k}} \cdot \operatorname{clip}\!\left( \frac{\textcolor{blue}{\pi_{\theta}}}{\textcolor{blue}{\pi_k}}, 1-\epsilon, 1+\epsilon \right), \\[0.6em] \boldsymbol{TIS:} \qquad \omega_{\mathrm{TIS}} = \min\!\left( \frac{\textcolor{blue}{\pi_k}}{\textcolor{red}{\mu_k}}, C \right) \cdot \operatorname{clip}\!\left( \frac{\textcolor{blue}{\pi_{\theta}}}{\textcolor{blue}{\pi_k}}, 1-\epsilon, 1+\epsilon \right). \end{gather}\]

Vanilla-IS keeps the trust-region constraint on the current update ratio \(\textcolor{blue}{\pi_{\theta}}/\textcolor{blue}{\pi_k}\), which is more suitable than directly clipping \(\textcolor{blue}{\pi_{\theta}}/\textcolor{red}{\mu_k}\). However, its mismatch-correction weight \(\textcolor{blue}{\pi_k}/\textcolor{red}{\mu_k}\) is unbounded. When a rollout token is sampled with low probability under \(\textcolor{red}{\mu_k}\), this weight can become large and significantly amplify gradient variance, as also discussed in [9]. This explains why Vanilla-IS recovers a clearer upward learning trend than PPO-IS, but still exhibits larger gradient-norm spikes in 6 (a).

TIS further truncates the mismatch-correction weight while preserving the decomposed form. As shown in 6, this yields the best stability-performance trade-off among the Step 1 variants: compared with PPO-IS, it avoids overly aggressive clipping, and compared with Vanilla-IS, it controls variance amplification from large mismatch weights. We therefore use TIS to construct the Step 1 candidate update in the full method.

In summary, directly clipping \(\textcolor{blue}{\pi_{\theta}}/\textcolor{red}{\mu_k}\) makes the trust-region constraint depend on the pre-existing mismatch, which can render the update overly conservative. The decomposed form separates mismatch correction from the current update, and the truncated version used by TIS further controls variance, making it the most effective Step 1 implementation in our setting.

9 Analysis of the Tolerance Paremeter \(c\)↩︎

We further study how the acceptance tolerance affects Step 2. In the main method, the synchronized candidate is accepted if \(\widehat{T}_{\mathrm{post}}\ge -c\), where \(c\ge 0\) is a tolerance parameter. For sensitivity analysis, we keep the same boundary form \(\widehat{T}_{\mathrm{post}}\ge -c\) and also test \(c<0\), which corresponds to a stricter-than-zero acceptance rule rather than a tolerance. Thus, \(c>0\) allows mild negative proxy values, \(c=0\) gives strict non-regression under the proxy, and \(c<0\) requires a positive margin.

7 shows that stricter acceptance is not always beneficial. When the acceptance rule is too strict, Step 2 rejects many early candidates. This is especially harmful because \(\widehat{T}_{\mathrm{post}}\) is initially low under FP8-quantized rollout and gradually increases during training, as observed in 5 (a). As a result, overly strict acceptance slows down learning and leads to a lower performance plateau. In particular, the stricter setting enters persistent rollback after around \(280\) steps, suggesting that conservative rejection alone cannot provide sufficient policy improvement.

These results show that rollback rate itself is not the objective. Although rejecting more updates may appear safer, excessive rollback can prevent the inference policy from moving to a better region. The dynamic tolerance used in MIPU provides a calibrated acceptance rule: it starts with a larger tolerance to allow useful early candidates, and gradually tightens the criterion as the post-update gap becomes more stable. This balances early learning efficiency with later-stage protection against mismatch-driven unreliable updates.

10 Licenses↩︎

The licenses of used assets in this paper are listed as follows:

• MATH500: MIT License

• AIME24: Apache-2.0 License

• AMC23: No License

• Minerva Math: No License

• OlympiadBench: Apache-2.0 License

• DAPO-Math-17k: Apache-2.0 License

• DeepMath-103k: MIT License

• Qwen3 Models: Apache-2.0 License

• ROLL: Apache-2.0 License

• vLLM: Apache-2.0 License

• Megatron: Apache-2.0 License

References↩︎