1 Introduction↩︎

Reinforcement fine-tuning (RFT) has become a cornerstone for enhancing the reasoning capabilities of large language models (LLM) [1]–[6]. Among existing approaches, Group Relative Policy Optimization (GRPO) has gained wide adoption due to its efficiency and scalability [7]–[9]. However, GRPO suffers from a well-documented drawback: entropy collapse: As training progresses, policy entropy declines monotonically, sampled outputs converge to nearly identical solutions, and the model prematurely adopts a deterministic policy with limited exploration [10]–[12]. This severely restricts the ability of RFT to discover diverse reasoning strategies. Existing remedies typically rely on entropy regularization and clip refine, which inevitably introduce bias and instability into the optimization objective, creating a trade-off between exploration and stability [13]–[16]. EFRame is the first to achieve large-scale entropy control, but it remains a qualitative analysis without conducting a quantitative study of entropy [17].

Although the issue of entropy collapse in GRPO has been repeatedly noted, it remains unresolved: there is still no principled algorithm capable of precisely regulating entropy throughout training. Existing approaches often treat entropy only as a side indicator—for instance, using it to split steps or guide auxiliary heuristics—while the policy entropy itself continues to collapse as training progresses [15], [18]–[20]. As a result, the relationship among entropy, exploration, and performance remains unclear: it is uncertain whether entropy is a sufficient proxy for exploration or whether exploration itself consistently improves training outcomes. If significant performance improvement can be observed by adjusting entropy to a better range, it would indicate that exploration plays a crucial role in this process. If arbitrary entropy control can be achieved during RFT, it would make it possible to realize exploration at any desired degree and, in turn, establish a principled connection among entropy, exploration, and performance.

Motivated by these challenges, we propose Arbitrary Entropy Policy Optimization (AEPO), a novel policy gradient that directly addresses entropy collapse. AEPO replaces conventional entropy bonuses with a REINFORCE policy gradient [21], [22] applied to samples drawn from temperature-adjusted distributions. Entropy is further stabilized through temperature regulation, ensuring that batch-level entropy remains oscillating around an arbitrary constant \(\mathcal{H}\) throughout training, as shown in Fig. 1. AEPO achieves entropy control through three key design components:

• Policy gradient as regularization: Instead of relying on a conventional entropy bonus, we employ a full policy gradient term applied to samples that naturally exhibit high- or low-entropy properties. This design ensures that entropy never dominates the optimization objective, allowing the model to monotonically explore toward higher accuracy throughout training.

• Distribution as regularization: Entropy is regulated via temperature-adjusted distributions. Given a predefined entropy threshold \(\mathcal{H}\), when the previous step’s entropy \(\mathcal{H}(\pi_{\theta_{\text{old}}}) < \mathcal{H}\), we regard the distribution under \(T_{\text{high}}>1\) as a better candidate and mix in a small proportion of its samples. Conversely, when \(\mathcal{H}(\pi_{\theta_{\text{old}}}) \geq \mathcal{H}\), we instead sample from the distribution under \(T_{\text{low}}<1\).

• REINFORCE as regularization: In Reinforcement Learning with Verifiable Reward (RLVR), the REINFORCE algorithm can filter out negative samples in an unbiased manner, allowing positive samples to form a unidirectional gradient toward a better distribution. This guides the model to optimize from positive samples that align with the target distribution. If negative samples were included at this stage, the entropy control mechanism would fail.

In summary, our contributions are threefold:

• Controllable entropy: We propose Arbitrary Entropy Policy Optimization that can stabilize entropy at arbitrary target levels, effectively eliminating entropy collapse in GRPO.

• Entropy–performance relation: We find that merely adjusting entropy can directly influence training performance, providing explicit evidence for the correlation between entropy, exploration, and performance. Moreover, we observe a non-monotonic trend in which performance first increases and then decreases as entropy grows, highlighting the existence of an optimal entropy regime.

• Generalizability: Beyond entropy control, AEPO provides a broader paradigm for RFT. When a target distribution \(\pi^{*}\) is identified as superior to the current policy \(\pi_{\theta}\), samples from \(\pi^{*}\) can be used to construct a REINFORCE policy gradient as a regularization term, which allows \(\pi\) to progressively approximate \(\pi^{*}\) during long-horizon training.

2 Related Work↩︎

RFT has become a central paradigm for post-training large language models (LLMs), aligning them with human feedback and task-specific objectives. Early methods such as RLHF leveraged policy optimization (e.g., PPO) to encode human preferences [23]–[26], while Direct Preference Optimization (DPO) [27] later improved efficiency by optimizing policies directly from preference data. Recent models like DeepSeek-R1 [9] and Kimi-1.5 [28] extend RFT through hybrid reward formulations and scalable optimization. Among them, Group Relative Policy Optimization (GRPO) [7], [8] has become the de facto baseline for reasoning-focused RFT, yet suffers from entropy collapse that limits exploration of diverse reasoning strategies.

Entropy has long been regarded as a proxy for exploration in reinforcement optimization. Classical methods employ entropy regularization to stabilize training and encourage diversity [21], [22]. More recent studies extend this idea to GRPO by introducing entropy bonuses into rewards or advantages [14]–[16]. However, such approaches only yield coarse-grained effects: entropy still collapses as training proceeds, or the added bias destabilizes optimization. EFRame [17] represents the first framework to achieve large-scale entropy control through conditional sampling and replay, yet its treatment remains qualitative and lacks quantitative characterization of the entropy–exploration–performance relationship.

In summary, existing methods lack a principled mechanism to precisely regulate entropy throughout training. Moreover, the role of entropy in driving exploration and its connection to downstream performance has not been quantitatively established. Our work addresses this gap by proposing Arbitrary Entropy Policy Optimization (AEPO), which enables controllable entropy regulation and provides explicit evidence of a non-monotonic relationship between entropy, exploration, and reasoning performance.

3 Preliminary↩︎

Our work focuses on fine-tuning LLM using Reinforcement Learning (RL) for tasks with verifiable solutions, such as mathematical reasoning and code generation.

Suppose the LLM is a softmax policy, that is \[\begin{align} \pi_{\theta}(o_{t}|q_{t})=\frac{\text{exp}(l(q_{t},o_{t}))}{\sum_{o'_{t}}\text{exp}(l(q_{t},o'_{t}))}, \end{align}\] where \(q_{t}\) is the concatenation of query q followed by \(o_{<t}\), and \(l(q_{t},o_{t})\) is the logit of token \(o_{t}\) given input \(q_{t}\). Furthermore, given a temperature T, we define: \[\begin{align} \pi_{\theta}^{T}(o_{t}|q_{t})=\frac{\text{exp}(l(q_{t},o_{t})/T)}{\sum_{o'_{t}}\text{exp}(l(q_{t},o'_{t})T)}. \end{align}\]

3.1 Policy-gradient based RL algorithms↩︎

Given a query \(q\), let \(o\) denote a response sampled from policy \(\pi_{\theta}\) for query \(q\). Given a reward function: \[R(q,o) =\boldsymbol{1}[o=o^{*}],\] where \(o^{*}\) is the reference response for query \(q\), the policy objective is: \[\begin{align} \mathcal{J}(\theta) = \mathbb{E}_{q \sim P(Q), o \sim \pi_{\theta}(O|q)}\sum_{t=1}^{|o|}[R(q,o)]. \end{align}\] To optimize the objective function, it is a common practice to use the Policy Gradient algorithm for gradient estimation: \[\begin{align} &\nabla_{\theta}\mathcal{J}_{REINFORCE}(\theta)= \mathbb{E}_{q \sim P(Q), o \sim \pi_{\theta}(O|q)}\\ &\quad \quad \quad \quad \quad \quad \sum_{t=1}^{|o|}[\nabla_{\theta} \text{log}\pi_{\theta}(o_{t}|q,o_{<t}) \cdot R(q,o)],\\ &\nabla_{\theta}\mathcal{J}_{GRPO}(\theta)=\mathbb{E}_{q \sim P(Q), \{o_i\}_{i=1}^{G} \sim \pi_{\theta}(O|q)}\\ &\quad \quad \quad \quad \quad \quad \frac{1}{G} \sum_{i=1}^G \frac{1}{|o_i|} \sum_{t=1}^{|o_i|}[\nabla_{\theta} \text{log}\pi_{\theta}(o_{t}|q,o_{<t}) \cdot \hat{A}_{i,t}], \end{align}\]

where \(R(q,o)\) is the reward for a query–response pair \((q,o)\), and \(\hat{A}_{i,t}\) denotes the estimated advantage. To reduce gradient variance, GRPO extends REINFORCE by introducing group-wise normalization: for each query \(q\), it samples \(G\) responses \(\{o_i\}_{i=1}^{G}\) and normalizes their rewards to compute the relative advantage for stable optimization.

\[\begin{align} \hat{A}_{i,t} = \frac{R(q,o_{i})-\text{mean}(\{R(q,o_{j})\}_{j=1}^{G})}{\text{std}(\{R(q,o_{j})\}_{j=1}^{G})}. \end{align}\]

3.2 Entropy-regularization variants↩︎

In traditional RL, it is common to add an entropy term to the objective to prevent the policy from becoming overly deterministic. Prior work has also explored various approaches in this direction for LLM training.
Entropy-Reg Given a query \(q\), let \(o\) denote a response sampled from policy model \(\pi_{\theta}\) for query \(q\). For each token \(o_{t}\) in response \(o\), we denote the token-level entropy as: \[\begin{align} \mathcal{H}_{t}(\pi_{\theta}):=-\mathbb{E}_{o_{t} \sim \pi_{\theta}(\cdot|q,o_{<t})}[\text{log}\pi_{\theta}(o_{t}|q,o_{<t})], \end{align}\] and then we can further denote that: \[\begin{align} \mathcal{H}(\pi_{\theta}):=\mathbb{E}_{q \sim P(Q), o \sim \pi_{\theta}(O|q)}\frac{1}{|o|}\sum_{t=1}^{|o|}\mathcal{H}_{t}(\pi_{\theta}). \end{align}\] In maximum entropy RL, we optimize for the entropy-regularized objective as follows: \[\begin{align} \mathcal{J}&_{MaxEnt}(\theta)=\mathcal{J}(\theta)+\lambda \cdot \mathcal{H}(\pi_{\theta}) = \nonumber \\ &\mathbb{E}_{q \sim P(Q), o \sim \pi_{\theta}(O|q)}\sum_{t=1}^{|o|}[R(q,o)-\lambda\cdot\text{log}\pi_{\theta}(o_{t}|q,o_{<t})]. \end{align}\]

Entropy-Adv [15] proposed an entropy-guided advantage shaping method. The key idea is to inject an entropy-based term into the advantage function during policy optimization. They define an entropy-based advantage term \(\psi(\mathcal{H}_{t})\) and use it to shape advantage: \[\begin{align} &\psi(\mathcal{H}_{t})=\text{min}(\beta \cdot \mathcal{H}_{t}^{detach},\frac{|\hat{A}_{i,t}|}{\kappa}),\\ &A_{i,t}^{\text{shaped}}=\hat{A}_{i,t}+\psi(\mathcal{H}_{t}), \end{align}\] where\(\beta>0\; and\; \kappa >1\). The entropy term \(\mathcal{H}_{t}^{detach}\) is detached from the computational graph during backpropagation, acting as a fixed offset to the original advantage. The policy gradient of the algorithm retains a format similar to that in GRPO, where only the advantage \(\hat{A}_{i,t}\) is replaced by the shaped one: \[\begin{align} \nabla_{\theta}\mathcal{J}^{\text{shaped}}(\theta)=&\mathbb{E}_{q \sim P(Q), o \sim \pi_{\theta}(O|q)},\\ &\sum_{t=1}^{|o|}[\nabla_{\theta} \text{log}\pi_{\theta}(o_{t}|q,o_{<t}) \cdot A_{i,t}^{\text{shaped}}]. \end{align}\]

4 Method↩︎

Arbitrary Entropy Policy Optimization (AEPO) is designed to achieve precise and stable control of policy entropy during RFT. Unlike conventional entropy-regularized methods, AEPO does not introduce explicit entropy bonuses; instead, it adjusts the training distribution and policy gradient to regulate entropy implicitly yet controllably. This section first introduces the key premises underlying AEPO’s design, which reveal the relationship between temperature, entropy, and policy updates, and then details the algorithmic formulation of AEPO.

4.1 Premise↩︎

The design of AEPO is built upon two empirical premises that connect temperature-based sampling to entropy dynamics. These premises establish the foundation for controllable entropy modulation without introducing bias into the optimization objective.

Previous studies, such as [29], show that increasing the sampling temperature generally broadens the model’s output distribution and raises its entropy. Although the correspondence between higher temperature and higher entropy is not a universally proven principle, it holds—or approximately holds—under most intuitive and commonly used model settings. In the context of RFT, temperature thus serves as a practical external variable for indirectly regulating entropy, enabling controllable entropy adjustment.

The proof can be seen in [31] and [14]. \(\mathcal{H}\) indicates the policy entropy of policy model, and \(\text{Cov}\) denotes covariance, \(\pi^k_\theta\) is the policy at step \(k\), and \(A^{\pi^k}(s,a)\) is the advantage function of action \(a\) under state \(s\). This result indicates that when positive actions have a low probability. If the probability of sampling low-probability actions can be increased, the model will be optimized toward higher entropy.

When positive samples are drawn from a higher-temperature version of the current policy distribution, training with these samples increases the model’s entropy, promoting exploration. Conversely, sampling from a lower-temperature distribution produces entropy reduction, guiding the model toward more deterministic behavior. Figure 3 illustrates this phenomenon: the entropy increases under high-temperature sampling and decreases under low-temperature sampling. Meanwhile, entropy increase is typically slower than entropy decrease, which aligns with intuition. This dynamic forms the empirical basis for AEPO’s temperature-controlled entropy feedback loop, enabling bidirectional regulation of entropy around a target value.

4.2 AEPO↩︎

Building upon the premises introduced above, AEPO is designed to achieve stable and controllable entropy regulation throughout RFT. Its core idea is to replace explicit entropy bonuses with a REINFORCE regularization term that adjusts the sampling distribution according to the current entropy state. In this way, AEPO modulates exploration implicitly—through data distribution—rather than directly through the loss function.

Formally, AEPO augments the standard GRPO objective with an additional policy gradient term applied to samples drawn from temperature-adjusted distributions. The overall policy gradient can be expressed as:

\[\begin{align} & \nabla_{\theta}\mathcal{J}_{AEPO}(\theta) \nonumber \\ =& \underbrace{ \mathbb{E}_{q \sim P(Q), \{o_{i}\}_{i=1}^{G} \sim \pi_{\theta}(O|q)}\sum_{t=1}^{|o|}[\nabla_{\theta} \text{log}\pi_{\theta}(o_{t}|q,o_{<t}) \cdot \hat{A}_{i,t}]}_{\text{GRPO form policy gradient}} \nonumber \\ +& \underbrace{\alpha\cdot \mathbb{E}_{q \sim P(Q), o \sim \textcolor{red}{\pi_{\theta}^{T}(O|q)}}\sum_{t=1}^{|o|}[\nabla_{\theta} \text{log}\pi_{\theta}(o_{t}|q,o_{<t}) \cdot \textcolor{red}{R(q,o)}]}_{\text{REINFORCE form policy gradient}}. \end{align}\]

The implementation of AEPO’s loss function is shown in Eq. 2, which consists of three key design components:

Policy gradient as regularization. AEPO replaces conventional entropy regularization with a full policy gradient term that simultaneously enables entropy control and prevents entropy from dominating the optimization process. As illustrated in Fig. 4, entropy regularization alone tends to drive the optimization toward two extremes—either entropy collapse or entropy explosion. In the former case, the regularization term is too weak to reverse the monotonic entropy decay; in the latter, entropy becomes an irreversible dominant factor in optimization. By contrast, AEPO constrains entropy fluctuation within a narrow and stable range through the policy gradient mechanism, making it remarkably robust to hyperparameters.

Distribution as regularization. AEPO regulates the optimization direction by adjusting the expected sampling distribution \(\pi^{*}\) derived from the current policy. When the observed entropy \(\mathcal{H}(\pi_{\theta_{\text{old}}})\) is below the target threshold \(\mathcal{H}\), AEPO samples from the higher-temperature distribution \(\pi^{T_{\text{high}}}_{\text{old}}\) to encourage exploration. Conversely, when \(\mathcal{H}(\pi_{\theta_{\text{old}}})\) exceeds \(\mathcal{H}\), AEPO samples from the lower-temperature distribution \(\pi^{T_{\text{low}}}_{\text{old}}\) to promote stability. This bidirectional regulation mechanism achieves fine-grained entropy control, allowing the policy to maintain equilibrium between exploration and convergence.

REINFORCE as regularization. In RLVR, the reward space is binary. This property allows REINFORCE to filter out negative samples in an unbiased manner, ensuring that the gradient is formed from positive samples that align with the desired distribution. Consequently, AEPO’s regularization term produces a one-sided optimization signal that guides the policy toward higher-quality behavior distributions.

5 Experiments↩︎

To validate the effectiveness of our methods, we present the experimental setup and results in the following sections. Our work is based on the EasyR1 and VeRL frameworks [32], [33], and we compare with the RL baselines GRPO [7] and its entropy-regularization variants [13], [15].

5.1 Experimental setup↩︎

Model and Dataset: We conduct experiments to evaluate the effectiveness of AEPO in RFT for mathematical reasoning tasks. The base model is Qwen2.5-Math-7B [34], an LLM specialized for mathematical problem solving. For training, we use the DAPO-17K dataset [10], which contains diverse problem instances curated for RFT.

Benchmark: Evaluation is performed on a broad suite of mathematical reasoning benchmarks, including AIME24 [35], AMC [36], College Math [37], GSM8K [38], MATH [36], Minerva Math [39], and Olympiad [36]. These benchmarks collectively span a wide range of difficulty levels, from grade school arithmetic to advanced competition-level mathematics, and together they cover nearly all mainstream benchmarks for mathematical reasoning, enabling a comprehensive assessment of AEPO’s impact on reasoning performance across diverse tasks.

Implementation: We initialize all policy models and conduct experiments on 8 A800 GPUs (40GB). Unless otherwise specified, we follow the default EasyR1 settings: maximum response length of 2048, global batch size 128, rollout batch size 512, rollout group size \(G\)=5, temperature 1.0, learning rate \(10^{-6}\), \(\epsilon = 0.2\), and a binary reward. For AEPO, \(T_{\text{high}} = 1.2\), \(T_{\text{low}} = 0.8\), we replace 60 temperature-adjusted positive samples in each batch. For entropy-regularized baselines, we set \(\lambda=0.03\) for Entropy-Reg, and \(\beta=0.4\), \(\kappa=2\) for Entropy-Adv [15]. To exclude potential confounding factors, we did not apply KL divergence in any of our experiments.

5.2 Main results↩︎

Table 2 summarizes the performance of AEPO compared with GRPO and entropy-based baselines across seven mathematical reasoning benchmarks. We observe that AEPO consistently outperforms all competing methods, achieving the best results on every benchmark. In particular, AEPO yields an average score of 61.36, representing a relative improvement of +3.40 points over GRPO. Notably, the gains are especially pronounced on high-difficulty datasets such as AIME24 and AMC, where exploration plays a critical role in finding correct reasoning paths.

Compared to entropy-regularized variants, AEPO also exhibits clear advantages. Both Entropy-Reg and Entropy-Adv deliver only marginal improvements over GRPO. As shown in Fig. 5, the entropy regularization term in Entropy-Reg introduces significant bias: during mid-stage training, entropy begins to surge uncontrollably, indicating that entropy gradually replaces accuracy as the dominant optimization signal, with accumulated bias distorting the learning process. Entropy-Adv, on the other hand, fails to fundamentally reverse the trend of entropy collapse, leaving the model with insufficient exploration. In contrast, AEPO achieves consistent improvements across all benchmarks, underscoring the benefits of principled entropy control over heuristic entropy bonuses.

5.3 Entropy and exploration↩︎

To further examine the relationship between entropy and exploration, we evaluate AEPO under different entropy thresholds \(\mathcal{H}\). As shown in Table 2, AEPO consistently surpasses GRPO across all benchmarks regardless of the entropy target, with strong improvements on challenging datasets such as AIME24 and MATH. At different thresholds, AEPO maintains clear advantages, indicating that principled entropy regulation yields benefits across a wide range of entropy levels.

Figure 1 further highlights this effect. While GRPO exhibits monotonic entropy collapse, AEPO stably maintains entropy around arbitrary preset levels simply by adjusting the hyperparameter \(\mathcal{H}\), since AEPO is not sensitive to the parameter \(\alpha\), as long as it is chosen within a reasonable range. This controllability allows us to probe the direct impact of entropy on exploration and downstream performance. Notably, we observe a non-monotonic trend: performance first improves as entropy rises from low to moderate levels, but begins to decline when entropy becomes excessively high. This provides explicit evidence that moderate entropy fosters effective exploration, while overly high entropy disperses optimization and harms accuracy.

Together, these results confirm that AEPO not only eliminates entropy collapse but also clarifies the link between entropy, exploration, and performance, offering a controllable pathway to balance exploration with convergence.

6 Ablation Study↩︎

To assess the contribution of each component in AEPO, we design two ablation studies to verify the necessity of distribution as regularization and REINFORCE as regularization for achieving effective entropy control.

6.1 Distribution as regularization↩︎

One critical component of AEPO is the use of temperature-adjusted distributions for entropy control. In our design, the REINFORCE regularization term samples from a modified distribution, where the temperature \(T\) is adaptively adjusted based on the previous step’s entropy. This adjustment ensures that positive samples carry either higher or lower entropy as required, thereby stabilizing the overall entropy around the target threshold \(\mathcal{H}\).

To validate the necessity of this design, we replace the temperature-adjusted distribution with the original distribution, as shown in Table [tab:loss95ablation]. The results demonstrate that when REINFORCE samples are drawn directly from the original policy distribution (i.e., without temperature adjustment), entropy control collapses: the policy entropy monotonically decreases during training, similar to GRPO. More importantly, the average benchmark score drops to 57.39, which is even worse than standard GRPO and far below AEPO (61.36). This shows that the variance of REINFORCE gradients, when unregularized by distribution adjustment, further degrades optimization stability and downstream reasoning accuracy.

These findings confirm that distribution adjustment is indispensable for AEPO: it directly enables controllable entropy regulation, reduces variance, and consistently improves reasoning performance across benchmarks. Moreover, it provides strong evidence for the role of policy gradient as regularization: the REINFORCE term in AEPO influences the exploration ability of the GRPO component through entropy control, thereby shaping the overall optimization dynamics during training.

6.2 REINFORCE as regularization↩︎

Another essential component of AEPO is the use of REINFORCE gradients as a replacement for conventional entropy bonuses. In principle, REINFORCE allows unbiased estimation of gradients from positive samples while discarding negative ones, thereby forming a unidirectional optimization signal toward better distributions. This mechanism is critical for maintaining stable entropy control: when negative samples are included, the entropy-regularizing effect contributed by positive samples is counteracted, and the policy entropy eventually collapses.

To examine the role of REINFORCE, we conduct an ablation where the regularization term is still sampled from temperature-adjusted distributions, but the advantage function \(\hat{A}_t\) is used directly without filtering negative samples. As shown in Table [tab:loss95ablation], this variant fails to prevent entropy collapse, leading to degraded performance across benchmarks. The average score drops to 58.14. This confirms that filtering negative samples via REINFORCE is indispensable.

7 Conclusion and discussion↩︎

In this paper, we propose Arbitrary Entropy Policy Optimization (AEPO), a principled RFT framework that addresses one of the most persistent challenges in RFT—precise and stable entropy control. Unlike traditional entropy regularization methods that trade off exploration against stability, AEPO achieves controllable entropy regulation through a unified design that integrates policy gradient, distribution, and REINFORCE as regularization components. This formulation eliminates the entropy collapse phenomenon in GRPO and maintains policy entropy within an arbitrarily specified range, enabling balanced and consistent exploration throughout training.

Extensive experiments across seven mathematical reasoning benchmarks demonstrate that AEPO consistently outperforms entropy-based baselines, exhibiting greater stability, generalization, and robustness to hyperparameters. More importantly, AEPO reveals a non-monotonic relationship between entropy and reasoning performance, showing that moderate entropy fosters exploration while excessive entropy impairs optimization—offering the first quantitative evidence linking entropy dynamics to reasoning capability in large language models.

Beyond solving entropy collapse, AEPO establishes a generalizable paradigm for learning under target distributions, suggesting its applicability to broader domains such as multi-modal alignment and long-horizon reasoning. In summary, AEPO transforms entropy from a passive statistical indicator into an active, tunable variable that fundamentally governs exploration, stability, and performance in RFT.

Declaration of AI↩︎

AI is only used for translation and language polishing in this paper.

References↩︎