1 Introduction↩︎

Alongside the scaling of model parameters, chain-of-thought (CoT) reasoning [1]–[3] has emerged as a critical scaling dimension for test-time compute, establishing itself as a foundational capability for large language models (LLMs) [4]–[6] to solve complex tasks. Recent work [7] has demonstrated that reinforcement learning with verifiable rewards (RLVR) can effectively enhance the reasoning abilities. In particular, the paradigm of “zero RL” [8] diverges from traditional imitation learning by initiating RL directly from a pretrained base model, entirely bypassing the need for supervised CoT data, which has proven capable of incentivizing emergent problem-solving strategies. This breakthrough indicates that powerful reasoning capabilities can be cultivated purely through trial-and-error reinforcement learning, circumventing the dependency on costly and curated reasoning demonstrations.

Over the past year, the research community has shown considerable interest in eliciting reasoning behaviors directly from base models. Extensive efforts have focused on high-quality data curation, generating diverse chain-of-thought datasets to bootstrap reasoning capabilities [9]–[11]. Building upon these foundations, a variety of policy optimization algorithms have been proposed to stabilize [12]–[14] and enhance optimization efficacy [15]–[17]. To accommodate these intensive algorithmic workloads, scalable training infrastructures have also undergone a parallel evolution [18]–[21]. However, constrained by computational demands, the vast majority of existing studies rely on relatively small models. While these studies provide valuable insights, a fundamental question remains unanswered: how do the training dynamics and emergent capabilities of zero RL evolve when applied to a trillion-parameter model? Larger models possess substantially richer pretrained knowledge and latent capabilities that are amenable to optimization [8]. Exploring zero RL at this massive scale is therefore essential to charting the true boundaries of self-evolved reasoning.

To meaningfully explore this frontier, our primary goal is to elicit high-quality reasoning behaviors from the model, which naturally requires us to first define what makes a high-quality reasoning process. While reaching the correct final answer is essential, the quality of the reasoning path itself is equally important. We believe an ideal CoT trace should not only be correct but also be readable to humans for easy verification, and computationally concise to save inference resources. However, we find that a naive application of zero RL severely deviates from these ideals, exposing several critical limitations. First, existing reasoning traces often suffer from poor readability. They typically lack logical formatting and clear structure, which makes it hard for humans to follow and interpret the thought process. Second, standard algorithms like GRPO often introduce an implicit length bias by assigning disproportionate credit to longer sequences. While token-level loss encourages exploration in early training, it eventually causes uncontrolled length growth and token redundancy, rendering the inference process highly inefficient. Furthermore, different problems inherently require varying levels of reasoning depth, yet standard training pipelines typically produce a single-mode model constrained by a fixed response budget.

In this work, we present a trillion-parameter self-iteration pipeline that trains models for emergent reasoning from scratch. A key insight from our exploration is that stabilizing training at the 1T scale only requires a few simple yet critical modifications. Guided by this principle, our pipeline avoids heavy engineering. It simply combines clipped importance ratio policy gradient with training-inference ratio correction for incentivizing reasoning, self-distillation for compression and stabilization, sample-level loss normalization for stable training, and tier-based adaptive training for flexible reasoning depth. We also implement infrastructure optimizations, including mixed-precision control and context parallelism for stable and efficient training.

Directly answering our central question, our trillion-parameter experiments offer a profound validation of the “bitter lesson” in artificial intelligence: massive computation and scale ultimately overtake human-engineered heuristics. We summarize our insights into three key findings: First, scaling dictates the capability ceiling. The 1T model exhibits vastly superior sample efficiency and reaches a significantly higher performance bound compared to its 104B counterpart. Second, our analysis sheds light on the “discovery vs. sharpening” debate, suggesting that both phenomena coexist as sequential stages. An initial “discovery” phase unlocks dormant pathways to expand the reasoning boundary, followed by a prolonged “sharpening” phase where the model refines its policy within this newly established boundary. Third, and most strikingly, we observe the spontaneous emergence of advanced cognitive strategies. While researchers historically engineered complex pipelines to incentivize improved thinking mechanisms, Ring-2.5-1T-Zero renders these hand-crafted heuristics redundant. Driven purely by this self-iterative training process without any human-annotated data, the model autonomously converges on several key emergent behaviors as optimal mathematical solutions: anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety.

Finally, we evaluate Ring-2.5-1T-Zero on seven challenging mathematical benchmarks and find that our model achieves competitive performance comparable to frontier models. Furthermore, to assess CoT quality beyond mere final-answer accuracy, we propose a structured evaluation framework spanning three dimensions: comprehensibility, reproducibility, and efficiency. Under these metrics, our model demonstrates clear advantages in generating structured, human-readable, and concise reasoning traces.

Our main contributions are summarized as follows:

\(\bullet\) We demonstrate that with only minor algorithmic and system improvements (e.g., clipped importance sampling, training-inference ratio correction, and mixed-precision control), our self-iterative training process remains stable and efficient, successfully training Ring-2.5-1T-Zero to achieve competitive performance.

\(\bullet\) We introduce a CoT quality evaluation framework that assesses reasoning traces along three dimensions: comprehensibility, reproducibility, and efficiency. This framework goes beyond final-answer accuracy and provides diagnostic signals for understanding the quality of the reasoning process itself.

\(\bullet\) We provide an empirical validation of the bitter lesson at scale, demonstrating that scaling to a trillion-parameter model unlocks higher sample efficiency and performance ceilings while enabling the autonomous emergence of advanced cognitive strategies, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, which renders hand-crafted designs redundant.

2 Evaluation Metrics for Chain-of-Thought Quality↩︎

Evaluating the quality of Chain-of-Thought (CoT) reasoning remains an open challenge. Currently, most evaluations [22], [23] rely on final-answer accuracy as the sole metric. However, accuracy only measures the final outcome, treating the underlying reasoning process as a black box. As LLMs are increasingly deployed in complex tasks and agentic workflows, CoT traces serve a dual purpose: they provide human-readable explanations and act as critical intermediate steps for downstream agent systems. Therefore, a comprehensive evaluation must go beyond mere correctness.

Prior to our experiments, we pre-defined several essential characteristics that an ideal CoT trace should possess. Guided by these expectations, we construct a structured evaluation framework that assesses CoT quality along three complementary dimensions: Comprehensibility, Reproducibility, and Efficiency. Comprehensibility ensures the logic is transparent and easy for humans to read; Reproducibility measures whether the reasoning strategy can be effectively and efficiently learned by weaker models; and Efficiency evaluates whether the generation is concise and computationally cost-effective. These three dimensions are independent of, yet highly complementary to, the correctness of the final answer.

2.1 Comprehensibility↩︎

Comprehensibility measures the degree to which a human reader can follow the reasoning steps without requiring external context or experiencing high cognitive load. A transparent CoT trace should feature a coherent logical flow, explicit causal dependencies between steps, and no hallucinated claims. When any of these properties are violated, the trace fails its primary explanatory purpose, undermining user trust.

To evaluate comprehensibility quantitatively, we employ an LLM-as-a-Judge framework via pairwise comparison. Given the same problem, a judge LLM evaluates CoT traces from different models and selects the one with stronger logical coherence, clearer articulation, and fewer reasoning flaws. The complete evaluation prompt is detailed in Appendix 10.

2.2 Reproducibility↩︎

Reproducibility assesses whether an agent lacking prior domain knowledge (e.g., a human with novice or a weaker student model) can effectively and efficiently learn the underlying reasoning process and acquire comparable task-solving abilities. An ideal trace encodes generalizable problem-solving strategies rather than intuitive leaps, enabling weaker student models to learn the underlying logic.

We measure reproducibility through knowledge distillation. By fine-tuning a weaker model on the generated CoT traces, we use the downstream performance gain of the student as a proxy. A larger improvement indicates that the teacher’s traces carry rich and transferable reasoning skills.

2.3 Efficiency↩︎

Efficiency evaluates the model’s ability to solve a problem concisely, avoiding unnecessary verbosity, redundant derivations, or circular logic. Furthermore, as CoT traces are heavily integrated into agent workflows, generation efficiency directly translates to faster inference speeds and lower token costs. An efficient CoT isolates the essential path to the solution without sacrificing clarity or correctness.

In practice, we measure efficiency by calculating the average token count of correct CoT traces. Among valid solutions, fewer tokens indicate higher efficiency, demonstrating that the model has identified a direct and optimal path to the answer rather than meandering through unrelated explorations.

3 Methodology↩︎

In this section, we present a training framework that elicits and scales reasoning capabilities directly from a base model without relying on human-annotated data. A core design principle of our framework is minimalism. Guided by our finding that overly complex algorithmic designs are often unnecessary, we only introduce optimization elements when they are strictly required to guide the model’s iterative learning. We first outline the multi-stage training pipeline (Section 3.1), which progressively builds reasoning capabilities through three RL stages and an intermediate self-distillation phase. Subsequently, we detail the infrastructure optimizations (Section 3.2) that make long-context RL stable and efficient at scale.

3.1 Zero Reinforcement Learning Training Pipeline↩︎

Our training pipeline consists of four distinct phases. In the first RL stage (Section 3.1.1), we use a clipped importance ratio policy gradient, combined with KL regularization and a token-level loss, to stimulate reasoning from the base pretrained model. Next, we apply self-distillation (Section 3.1.2) to compress the learned reasoning traces and bridge the gap between training and inference. The second RL stage (Section 3.1.3) resumes RL training using sample-level loss normalization for stable optimization. Finally, the third RL stage (Section 3.1.4) introduces tier-based training, equipping the model with adaptive response modes for different difficulty levels.

3.1.1 First Stage RL: Reasoning Elicitation↩︎

In Zero-RL, we train directly from a pretrained base model that lacks initial reasoning capabilities. Such a model rarely generates step-by-step derivations because reasoning tokens have very low probabilities under the base policy. To elicit these latent capabilities, the initial RL phase must aggressively amplify low-probability reasoning tokens. Therefore, we adopt a clipped importance-sampling policy gradient [16]. Unlike standard PPO-clip, which entirely zeroes out gradients for tokens outside the clipping range, this approach applies a stop-gradient solely to the clipped ratio while allowing gradients to flow through for all tokens. This ensures every generated token contributes to the learning process, which is crucial for building reasoning abilities from scratch. Specifically, the objective is formulated as: \[\mathcal{J}(\theta) = \mathbb{E}_{\substack{q \sim \mathcal{D} \\ \{o_i\}_{i=1}^G \sim \pi_{\textcolor{red}{\text{S}}}}} \left[ \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} \operatorname{sg}\!\left(\hat{\rho}_{i,t}\right) \cdot \hat{A}_{i,t} \cdot \log \pi_{\textcolor{blue}{\text{M}}}^{\theta}(o_{i,t} \mid q, o_{i,<t}) \right], \label{eq:pg-obj}\tag{1}\] where \(q\) is the input question sampled from dataset \(\mathcal{D}\), \(\{o_i\}_{i=1}^G\) are \(G\) rollout responses from the inference engine \(\pi_{\textcolor{red}{\text{S}}}\), \(o_{i,t}\) is the \(t\)-th token of the \(i\)-th response, \(\hat{A}_{i,t}\) is the advantage estimate computed using GRPO-style group-normalized rewards [8], \(\operatorname{sg}(\cdot)\) denotes the stop-gradient operator, and \(\hat{\rho}_{i,t}\) is the clipped importance ratio defined below.

However, amplifying low-probability tokens amplifies a critical system-level vulnerability: the numerical discrepancy between the training engine (Megatron) and the inference engine (SGLang). Differences in floating-point precisions and kernel implementations yield slightly divergent logits. To prevent these micro-discrepancies from compounding into macroscopic training collapse, we replace the numerator of the importance sampling ratio with the actual training-engine logits: \[\begin{align} \rho_{i,t} &= \frac{\pi_{\textcolor{blue}{\text{M}}}^{\theta}(o_{i,t} \mid q, o_{i,<t})}{\pi_{\textcolor{red}{\text{S}}}^{\theta_{\text{old}}}(o_{i,t} \mid q, o_{i,<t})}, \tag{2} \\ \hat{\rho}_{i,t} &= \operatorname{clip}\!\left(\rho_{i,t},\; \epsilon_{\text{low}},\; \epsilon_{\text{high}}\right), \tag{3} \end{align}\] where \(\pi_{\textcolor{blue}{\text{M}}}^{\theta}\) is the current policy parameterized by \(\theta\) in the training engine, \(\pi_{\textcolor{red}{\text{S}}}^{\theta_{\text{old}}}\) is the rollout policy from the inference engine using previous parameters \(\theta_{\text{old}}\), and \(\hat{\rho}_{i,t}\) is the clipped ratio. We do not apply a lower bound on the IS weight. Instead, we only set an upper bound \(\epsilon_{\text{high}}\) to prevent excessively large updates. This computed ratio accurately reflects the true divergence between training and inference engines. To further stabilize training, we apply a KL divergence penalty against the frozen reference model, which prevents the policy from drifting too far from the reference model: \[\mathcal{L}_{\text{KL}}(\theta) = \sum_{i=1}^{G} \sum_{t=1}^{|o_i|} D_{\text{KL}}\!\left(\pi_{\textcolor{blue}{\text{M}}}^{\theta}(\cdot \mid q, o_{i,<t}) \;\|\; \pi_{\text{ref}}(\cdot \mid q, o_{i,<t})\right), \label{eq:kl}\tag{4}\] where \(\pi_{\text{ref}}\) is the frozen reference policy model.

Finally, because base models tend to produce short responses, we explicitly encourage longer generation by adopting a token-level loss that sums over all tokens without normalizing by the response length \(|o_i|\), as shown in Equation 1 . This ensures that longer correct responses accumulate stronger gradient signals. To progressively improve reasoning capabilities, we gradually expand the response window length in a curriculum manner throughout training. The combined first-stage RL objective Ring-2.5-1T-Zero-I is: \[\mathcal{L}_{\text{Ring-2.5-1T-Zero-I}}(\theta) = -\mathcal{J}(\theta) + \beta \cdot \mathcal{L}_{\text{KL}}(\theta) \label{eq:total-loss}\tag{5}\] where \(\beta\) is the KL penalty coefficient that balances exploration and stability.

3.1.2 Self Distillation: Compression and Stabilization↩︎

While the first-stage RL successfully elicits reasoning capabilities, the unconstrained token-level loss introduces two critical drawbacks. First, the generated Chain-of-Thought (CoT) traces become excessively long, often burdened with redundant steps, circular logic, or unnecessary verbosity. Second, this extreme length severely destabilizes the RL process. Because the numerical discrepancies between the training and inference engines accumulate through the training process, it drastically inflates the training-inference gap, leading to optimization collapse.

To resolve these dual issues of verbosity and instability before proceeding to the next RL stage, we introduce a self-distillation phase to reset the model. Specifically, we use the first-stage expert policy \(\pi_{\text{expert}}\) to sample multiple rollouts for each query. We then apply a two-step length refinement process: first, we select the shortest correct reasoning trace among these rollouts; second, we prompt the model to self-evaluate this selected trace and filter out any remaining segments it identifies as redundant. This mechanism effectively trims circular logic and compresses verbose derivations without sacrificing correctness. Finally, this refined, high-quality corpus is used to fine-tune the original base model via standard supervised learning: \[\mathcal{L}_{\text{self-distillation}}(\theta) = -\mathbb{E}_{q \sim \mathcal{D},\, o \sim \pi_{\text{expert}}} \left[ \sum_{t=1}^{|o|} \log \pi_{\theta}(o_t \mid q, o_{<t}) \right], \label{eq:distill}\tag{6}\] where \(o\) represents the length-refined response, and \(\pi_{\theta}\) is the base pretrained model. By distilling with shortened responses, the model acquires the expert’s reasoning abilities while producing much more concise traces and effectively wipes out the accumulated numerical errors between the engines, which provides a highly capable, concise, and stable foundation for the subsequent RL stages.

3.1.3 Second Stage RL: Sustained Optimization↩︎

After self-distillation, the model exhibits a much smaller training-inference gap. This provides a stable starting point for the next phase of RL. To maintain stability, we keep the same clipped importance-sampling policy gradient and ratio correction used in the first stage.

The key change in this stage is our loss calculation strategy. In the first stage, a token-level loss is necessary to encourage longer responses. However, the model now naturally produces long chains of thought. Continuing with a token-level loss would cause the output length to grow without bound, destroying the conciseness gained from self-distillation. To control response length, we pivot to a sample-level loss, which keeps the gradient magnitude independent of the output length: \[\mathcal{L}_{\text{Ring-2.5-1T-Zero-II}}(\theta) = - \mathbb{E}_{\substack{q \sim \mathcal{D} \\ \{o_i\}_{i=1}^G \sim \pi}} \left[ \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \operatorname{sg}\!\left(\hat{\rho}_{i,t}\right) \cdot \hat{A}_{i,t} \cdot \log \pi_{\theta}(o_{i,t} \mid q, o_{i,<t}) \right], \label{eq:total-loss-stage2}\tag{7}\] where the only change from Equation 1 is the \(\frac{1}{|o_i|}\) normalization factor. Additionally, we remove the KL penalty in this stage. Since the distilled model already provides a strong starting point, a KL constraint would unnecessarily restrict further exploration. This sample-level formulation, along with the removal of KL regularization, ensures stable optimization over extensive RL steps.

3.1.4 Third Stage RL: Adaptive Reasoning Depth↩︎

After two stages of RL, the model demonstrates strong reasoning abilities but operates in a single mode with a fixed-length budget. In real-world scenarios, different questions require different levels of reasoning depth. Simple questions need short answers, while complex problems benefit from extended reasoning paths. To cultivate cognitive routing, we equip the model with multiple response modes by introducing a tier-based training strategy in the third stage.

We partition the training questions into three difficulty tiers, each governed by a specific maximum token length and a corresponding system prompt \(p_k\): \[\mathcal{T} = \left\{ \mathcal{T}_{\text{l}},\; \mathcal{T}_{\text{m}},\; \mathcal{T}_{\text{h}} \right\}, \label{eq:tiers}\tag{8}\] where \(\mathcal{T}_{\text{l}}\), \(\mathcal{T}_{\text{m}}\), and \(\mathcal{T}_{\text{h}}\) denote the low, medium, and high difficulty tiers. The third stage RL objective Ring-2.5-1T-Zero-III incorporates these prompts: \[\mathcal{L}_{\text{Ring-2.5-1T-Zero-III}}(\theta) = -\sum_{k \in \{\text{l}, \text{m}, \text{h}\}} \mathbb{E}_{\substack{q \sim \mathcal{D}_k \\ \{o_i\}_{i=1}^G \sim \pi(\cdot \mid p_k, q)}} \left[ \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \operatorname{sg}\!\left(\hat{\rho}_{i,t}\right) \cdot \hat{A}_{i,t} \cdot \log \pi_{\theta}(o_{i,t} \mid p_k, q, o_{i,<t}) \right], \label{eq:pg-obj-stage3}\tag{9}\] where \(\mathcal{D}_k\) is the subset of questions assigned to tier \(k\), and \(p_k\) is the corresponding system prompt. The rollout policy \(\pi\) generates responses conditioned on both the system prompt and the question, truncating the output at the given window size. This mechanism teaches the model adaptive compute allocation: it learns to respond concisely to trivial queries while reserving exhaustive CoT generation for complex problems. During deployment, users can explicitly control this behavior via the system prompt.

3.2 Infrastructure Optimization↩︎

Scaling RL to long context windows exposes critical bottlenecks in both numerical stability and communication latency. We address these challenges through two targeted optimizations. First, we use mixed-precision control to ensure training stability (Section 3.2.1). Second, we optimize context parallelism to maximize throughput (Section 3.2.2).

3.2.1 Mixed-precision Control↩︎

During RL training, the importance ratio \(\rho_{i,t}\) computes the ratio of two token probabilities. These probabilities are derived from logits using a softmax function, which involves exponentiation. The exponential function naturally amplifies small numerical errors. A minor error in a logit can blow up into a massive gap in the probability ratio, completely destabilizing the training process. Consequently, any forward-pass component involving exponentiation is highly sensitive to precision. Specifically, this points to two key areas: the attention softmax and the LM head.

To solve this problem, we implement a simple yet effective mixed-precision strategy. We keep the main model body in BF16 but compute both the attention softmax and the LM head in FP32. By using high precision strictly in these two sensitive locations, we prevent small rounding errors from growing into large RL losses. In practice, this targeted adjustment eliminates sudden spikes in the loss and successfully closes the numerical gap between the training engine (Megatron) and the inference engine (SGLang).

3.2.2 Context parallelism Optimization↩︎

Training with long context windows requires context parallelism (CP) to partition a single long sequence across multiple devices. Standard ring attention [24] solves this by passing key-value (KV) chunks sequentially in a circle. However, this design forces each device to wait for a complete round of communication with all other devices to finish its attention calculation. As the scale of parallelism grows, this sequential dependency creates a severe latency bottleneck.

To break this bottleneck, we tailor our CP strategy to our model’s hybrid architecture, which consists of MLA and Lightning Attention layers. For the MLA layers [25], we adopt an all-to-all CP strategy. This approach reshuffles data along the head dimension using a single collective operation. As a result, each device can independently compute full attention for a subset of heads across the entire sequence. Because MLA compresses keys and values into a low-rank latent space, the all-to-all communication volume is drastically reduced compared to sending full KV tensors. For the Lightning Attention layers [26], the calculation depends on a small, fixed-size KV state matrix. Instead of passing this state sequentially around the ring, we use a single AllGather operation. This broadcasts all local states to every device at once, allowing them to finish their computations immediately. These optimizations are mathematically equivalent to standard ring attention and produce identical gradients.

4 Experiments↩︎

This section presents our experimental evaluation. We first describe the experimental setup, then report the main results to compare our models against current frontier models, and finally provide a detailed analysis to validate our key design choices.

4.1 Experimental Setup↩︎

We summarize the key components of our experimental setup below. This covers model configurations, training details, reward design, and the evaluation metrics.

Base models. We conduct zero RL training starting from two pretrained Ling-2.5 base models [27] without any supervised fine-tuning: The models are: (1) Ling-2.5-1T-Base, a 1-Trillion parameter Mixture-of-Experts (MoE) model with 63B activated parameters. (2) Ling-2.5-flash-Base, a 104B parameter MoE model with 7.4B activated parameters. Both models are trained from scratch using our four-stage pipeline. This progresses from First Stage RL to Self-Distillation, Second Stage RL, and finally Third Stage RL.

Training infrastructure. We run all experiments on 320 \(\times\) H200 GPUs. We adopt a hybrid architecture where Megatron serves as the training engine and SGLang serves as the rollout engine. The RL training process is orchestrated by the Areal [21] framework.

Hyperparameters. Our training first follows an off-policy RL setup with a batch size of 512 and a minibatch size of 32, then uses a batch size of 256 and a minibatch size of 32. Across all RL stages, we use the Adam optimizer with \(\beta_1 = 0.9\), \(\beta_2 = 0.999\), a constant learning rate of \(2 \times 10^{-6}\), and a weight decay of \(0.01\). For each question, we generate \(G = 16\) rollout responses at a temperature of \(1.0\). To encourage exploration, we do not impose a lower bound on the importance sampling ratio. We only clip the upper bound with \(\epsilon_{\text{high}} = 5.0\) to prevent excessively large updates. In the first stage RL, we use a token-level loss combined with a KL penalty (\(\beta = 10^{-4}\), K3 divergence). We update the reference model every 400 steps with the latest checkpoint. This ensures the KL anchor remains relevant as the policy improves. We progressively expand the response window from 4k to 64k tokens in a curriculum method, doubling the context window every 800 training steps. After the first stage RL, we curate and filter high-quality responses from the first stage RL expert model. We then fine-tune the base model for 3 epochs with a sequence length of 64k and a learning rate of \(7 \times 10^{-5}\). In the second stage RL, we switch to a sample-level loss and remove the KL penalty. This adjustment prevents uncontrolled length growth and enables highly stable training. In third stage RL, we partition questions into Low (4k), Medium (16k), and High (64k) difficulty tiers. Each tier receives a specific system prompt. This teaches the model to adapt its reasoning depth based on the available inference budget.

Reward design. Throughout training, the reward consists of an accuracy component and a format component: \[r_i = r_{\text{acc},i} + r_{\text{format},i},\] where \(r_{\text{format},i} \in \{0, 1\}\) checks whether the response follows the required structure with <think>... </think> and <answer>...</answer> tags, and \(r_{\text{acc},i} \in \{0, 1\}\) measures the final answer’s correctness. In early training stages, we use easily verifiable problems. Following previous work [8], their correctness can be judged by deterministic rule-based matching. In later stages, problems become significantly harder. Answers may take multiple valid forms or require semantic understanding. Therefore, rule-based verification becomes unreliable. We replace \(r_{\text{acc},i}\) with an LLM-as-a-Judge evaluator (i.e., Qwen3-Next-80B-A3B-Instruct). This judge receives both the model’s response and the reference answer to output a binary correctness judgment.

Evaluation metrics. We evaluate our models on seven challenging mathematical reasoning benchmarks: AIME 2024, AIME 2025, AIME 2026, HMMT February 2025, HMMT November 2025, HMMT February 2026, and IMOAnswerBench. Following  [8], we set the temperature to 0.6 and top-k to 0.95 during inference. For all evaluations, we report pass@1 accuracy averaged over 64 runs to minimize variance. For our multi-tier model (Third Stage RL), we report results under three distinct inference modes: High (64k budget), Medium (16k budget), and Low (4k budget).

4.2 Main Results↩︎

Table 1 presents the main results comparing our Ring-2.5-1T-Zero models against state-of-the-art proprietary models across various mathematical reasoning benchmarks. Furthermore, Figure 2 illustrates the comprehensive training dynamics of our First Stage RL. By dividing the training into two distinct phases with progressively difficult data (i.e., starting with relatively simpler data followed by harder data to push the upper bounds of reasoning), we ensure a stable and continuous optimization process.

Zero RL achieves competitive reasoning from scratch. Without relying on any human-annotated data, Ring-2.5-1T-Zero(First Stage RL) already achieves an impressive 84.2% accuracy on AIME 2026. This compelling result underscores that our First Stage RL can effectively bootstrap advanced mathematical reasoning directly from a pretrained base model. The progressive curriculum, which gradually scales the response window from 4k to 64k tokens, combined with importance ratio correction, stably and effectively incentivizes the model to develop emergent chain-of-thought (CoT) behaviors. This suggests that as long as the exploration space is properly constrained via careful regularization, large language models inherently possess the capacity to self-discover complex logical deductions without human-annotated trajectories.

Multi-stage training yields consistent improvement. Each subsequent stage in our pipeline yields measurable performance gains, validating the overall efficacy of our multi-stage methodology. Specifically, the self-distillation phase stabilizes the model and prepares it for extended RL optimization. Subsequently, the Second Stage RL incorporates a sample-level loss to facilitate sustained performance improvements, while the Third Stage RL applies tier-based training to embed adaptive reasoning depth into the model’s capabilities. By transitioning the granularity of supervision from token-level (First Stage RL) to sample-level (Second Stage RL), and using self-distillation as an anchor, we create a much smoother optimization landscape for sustained reasoning improvement.

Adaptive inference modes offer flexible performance-efficiency trade-offs. The three inference tiers of Ring-2.5-1T-Zero(i.e., Low, Medium, and High) provide users with fine-grained control over reasoning depth during inference. While the Medium (TT=16k) and Low (TT=4k) modes effectively reduce inference latency and compute costs while maintaining competitive performance, we observe a slight performance drop in the Third Stage RL compared to the Second Stage peak. This decline is mainly due to two factors: first, a lack of high-quality, ultra-long reasoning data limits the High mode’s upper bound. Second, jointly training across three different lengths introduces negative transfer, where the training signals from the Low and Medium modes slightly pull down the High mode’s peak capability. Despite this, our adaptive strategy successfully shifts compute scaling from a rigid, one-size-fits-all approach to a dynamic, query-dependent allocation, proving that many queries simply do not require exhaustive reasoning.

Scaling model size amplifies Zero RL benefits. By comparing the 104-billion-parameter Ring-2.5-flash-Zero with the 1-trillion-parameter Ring-2.5-1T-Zero, we observe that larger model capacity disproportionately benefits from the Zero RL paradigm. The performance gap widens significantly on harder benchmarks. This trend provides a clear demonstration that aggressive parameter scaling is of paramount importance, particularly in the Zero RL setting. Under the zero RL regime, a model’s ability to explore is strictly bounded by its internal world model. More parameters provide the necessary representational capacity and broader internal knowledge base required to autonomously navigate the massive search space.

4.3 Other Evaluations of CoT Quality↩︎

Beyond evaluating final-answer accuracy, we assess the quality of our model’s CoT reasoning trajectories across three complementary dimensions: comprehensibility, reproducibility, and efficiency. As demonstrated below, our zero-RL model not only achieves high accuracy but also produces reasoning traces that are highly readable, easily transferable, and remarkably token-efficient.

4.3.1 Comprehensibility↩︎

Beyond quantitative scaling, we observe an intriguing qualitative phenomenon: the 1T model spontaneously develops structurally segmented reasoning. Even without explicit formatting constraints, it naturally demarcates reasoning phases using markers like Step 1: and Step 2: (See Section 6.2). Unlike the highly sanitized summaries typical of human reference solutions, these emergent traces capture an authentic, search-like trajectory characterized by trial-and-error, self-reflection, and parallel exploration. The spontaneous emergence of these step markers offers a profound insight. It suggests that the model does not merely engage in a continuous, greedy generation stream; rather, it exhibits a form of localized forward planning, setting cognitive intents for the upcoming chunk of tokens before actually generating them.

To evaluate comprehensibility quantitatively, we conduct pairwise comparisons using an LLM-as-a-Judge protocol. Across all 90 AIME problems (2024 to 2026), we compare our model’s reasoning traces against four strong baselines (GLM-5.1, Kimi-k2.6, MiniMax-M2.7, and Qwen3.5-397B), focusing on logical coherence, causal explicitness, and the absence of hallucinations. As shown in Figure 3 (a), our model achieves dominant win rates across all comparisons. This confirms that the emergent structured reasoning reflects a genuinely superior logical organization that human readers find easier to understand and follow.

4.3.2 Reproducibility↩︎

A key indicator of CoT quality is whether the reasoning traces can serve as an effective training signal for weaker models. We evaluate this reproducibility by distilling our model’s reasoning traces into Qwen2.5-32B and Llama3.3-70B-Instruct models, and compare them against the same config distilled from DeepSeek-R1.

Notably, we achieve our distillation results using only 100K data samples, a mere fraction compared to the 800K samples utilized by DeepSeek-R1. As shown in Figure 3 (b), despite using significantly less data, models distilled from our traces consistently outperform their DeepSeek-R1 counterparts. We observe a 5.8-point gain on Qwen-32B (78.4 vs.) and a 4.5-point gain on Llama-70B (74.5 vs.). This demonstrates that our step-segmented reasoning traces provide a richer, more transferable learning signal, resulting in vastly superior sample efficiency during knowledge distillation.

4.3.3 Efficiency↩︎

Finally, we assess whether our model achieves high accuracy using fewer reasoning tokens. We identify a subset of problems where all five models (ours and the four baselines) answer correctly. We then compare the average token count of their CoT traces on this shared subset.

As shown in Figure 3 (c), on the mutually solved AIME problems, our model uses an average of only 6,368 tokens. This is less than half of the other models. This substantial efficiency advantage suggests that our CoT helps the model eliminate redundant exploratory steps, allowing it to arrive at correct solutions via much more direct logical paths.

4.4 Detailed Analysis↩︎

In this subsection, we present detailed ablation studies to validate our key design choices. Due to computational resource constraints, we conduct all ablation experiments on the Ling-2.5-flash-Base model.

4.4.1 The Impact of Different RL Algorithms↩︎

We compare four RL algorithms under identical settings: GRPO [8], DAPO [15], CISPO [16], and GSPO [12]. All methods share the same base model, learning rate, and training data. They differ exclusively in how the policy gradient is computed. As shown in Figure 4, both CISPO and DAPO build upon GRPO by amplifying the gradient signal for low-probability tokens. This effectively reduces the all-failed group ratio much faster than vanilla GRPO. Both algorithms also achieve faster reward growth and a more rapid increase in sequence length. However, this acceleration comes at the cost of stability. CISPO is the most prone to entropy collapse and training instability, followed by DAPO and then GRPO. The speed of reward improvement and the degree of instability follow the exact same ordering: CISPO \(>\) DAPO \(>\) GRPO. This confirms our hypothesis: amplifying low-probability tokens is highly effective for stimulating reasoning from scratch, but it requires careful stabilization. In contrast, GSPO uses a sample-level loss to explicitly maintain high entropy. While this prevents entropy collapse, it provides very little incentive for the model to explore longer sequence lengths. Consequently, it is ill-suited for bootstrapping early reasoning abilities.

4.4.2 The Impact of RL Stabilization Strategies↩︎

The analysis above reveals a fundamental trade-off. Algorithms that aggressively amplify low-probability tokens learn faster but are inherently less stable. Therefore, we investigate two complementary stabilization strategies that preserve fast learning while maintaining training robustness.

KL penalty.

We compare training runs with and without a KL divergence penalty against the reference model (Eq. 4 ). As shown in Figure 5, removing the KL penalty causes catastrophic failure. The log-probability difference between the training and rollout engines grows unboundedly. Entropy collapses, and the reward eventually crashes. With the KL penalty, all training metrics remain stable. The numerical gap stays controlled, entropy remains at a healthy level to preserve exploration, and the reward improves steadily. The KL penalty acts as a vital regularizer. It constrains the overall policy drift, prevents premature entropy collapse, and sustains the model’s exploratory capacity.

Mismatch mitigation strategies. In our off-policy RL setup, the inference engine (SGLang) generates rollouts, while the training engine (Megatron) computes gradients. Standard policy optimization computes the importance ratio using logits generated purely by the inference engine. However, because of floating-point differences, the two engines produce subtly different logits for the exact same weights. This numerical mismatch is severely magnified when we amplify low-probability tokens. We propose a straightforward fix. We replace the numerator of the importance ratio with the actual training engine logits \(\pi^{\theta}_{\textcolor{blue}{\text{M}}}\) (Eq. 2 ). This entirely eliminates the need for the inference engine to recompute logits. We compare three configurations: (1) Baseline: standard ratio using only SGLang logits. (2) + IcePop: the baseline augmented with clipped importance ratio thresholding. (3) + Ours: using Megatron as the numerator and SGLang as the denominator. As shown in Figure 6, the baseline collapses within roughly 800 steps. The log-probability gap explodes, entropy drops to zero, and the reward crashes. IcePop delays this collapse but ultimately fails around step 2700. Our approach successfully maintains stable RL training. The log-probability difference stays near zero, entropy is preserved, and the reward improves consistently. Importantly, our method avoids threshold tuning and saves computation time.

4.4.3 The Importance of Format Reward↩︎

The format reward \(r_{\text{format},i}\) forces the model to comply with structural rules. We compare two format specifications: Format A requires only a single opening tag: <think>...</think>.... Format B requires double-closed tags with an implicit termination rule: <think>...</think><answer>... </answer>. Under Format B, the response must end with the EOS token after the final closing tag.

As shown in Figure 7, Format A causes the sequence length to increase rapidly while the reward remains completely stagnant. Upon inspecting the raw text, we found that the model exploits the loose format. It appends endless garbled text after the answer without ever properly stopping. In contrast, Format B strictly requires the EOS token. Only properly terminated rollouts receive any credit. This strict rule prevents degenerate length growth driven by trailing garbage. We adopt Format B across all training stages.

4.4.4 Length Inertia in zero RL Training↩︎

One might expect a reasoning model to naturally adjust its output length based on the actual difficulty of a problem. However, our experiments reveal a fundamental flaw in standard RL training. We call this phenomenon “length inertia”. During the early stages, policy optimization relies on a token-level loss, which implicitly assigns higher credit to longer sequences. As training progresses, the model discovers a lazy shortcut, which learns that merely generating more tokens is a mathematically safer way to secure high cumulative rewards. Consequently, without explicit control mechanisms, the response length grows continuously over time.

To explicitly demonstrate this inertia, we track the sequence length of simple questions that the model answers perfectly on its first rollout. As shown in Figure 9 (c), the model does not maintain a concise answer for these already solved problems. Instead, the sequence length expands unconditionally as training continues. This proves that the standard RL training paradigm makes the model lazy. It inflates its token usage due to a strong forward inertia, completely ignoring the actual cognitive requirements of the task.

We further validate this blind expansion by comparing training runs with 16k and 32k response windows (Figure [fig:window-size]). The model does not save the extra 32k capacity for truly difficult questions. Instead, it scales up its verbosity uniformly across all problems. The 32k setup produces average responses nearly twice as long as the 16k setup, yet it yields marginal rewards. This confirms that the model blindly fills the available context window rather than adapting to cognitive complexity.

4.4.5 Hyperparameter Analysis↩︎

Learning rate. We evaluate learning rates of \(1\times10^{-6}\), \(2\times10^{-6}\), and \(3\times10^{-6}\). As shown in Figure 8 (a) and Figure 8 (d), all three settings converge to highly similar reward levels and sequence lengths. This proves that training is robust to learning rate variations within this scale.

Rollout number. We vary the number of rollout samples per question across \(G \in \{8, 16, 32\}\). As shown in Figure 8 (b) and Figure 8 (e), \(G=32\) converges fastest per training step. However, \(G=8\) converges fastest in terms of actual wall-clock time due to lower computational overhead. In implementation, we provide a better trade-off between gradient variance reduction and computational efficiency by adopting \(G=16\) as a balanced default.

Loss reduction. We compare token-level and sample-level loss reduction. Token-level reduction assigns weight to every single token equally. This naturally favors and encourages longer responses. In contrast, sample-level reduction normalizes the loss by the sequence length. As shown in Figure 8 (c) and Figure 8 (f), token-level reduction significantly promotes CoT length growth. Meanwhile, sample-level reduction keeps the response length perfectly flat. This confirms our design choice: token-level loss is essential for initially incentivizing long reasoning chains from scratch.

5 Discussion↩︎

5.1 How Much Does Model Size Affect Training?↩︎

We investigate how model scale influences zero RL training by comparing Ling-2.5-flash-Base (104B parameters) and Ling-2.5-1T-Base (1T parameters) under identical training configurations. As shown in Figure 9 (a), both models improve steadily over time. However, the larger model demonstrates two distinct advantages. First, it reaches a higher performance ceiling. After 3,600 steps, Ring-2.5-1T-Zero-I achieves 89.06% on AIME 2024 and 83.28% on AIME 2025. In contrast, Ring-2.5-flash-Zero only reaches 71.72% and 63.54% even after 5,200 steps. Second, it exhibits superior sample efficiency. Ring-2.5-1T-Zero maintains a consistently faster rate of reasoning improvement throughout the entire training process. These results confirm that massive parameter capacity is a fundamental prerequisite for effective zero RL.

5.2 Does Reinforcement Learning Really Improve the Reasoning Boundary of LLMs?↩︎

A long-debated question in the RL community is whether RLVR genuinely expands the model’s reasoning boundary [28] or merely sharpens its existing distribution [29]. To resolve this binary debate, we follow [29] and track pass@1024 during early training. This metric measures the probability that at least one out of 1024 samples solves the problem correctly.

As shown in Figure 9 (b), our results reveal that zero RL is actually a distinct two-phase phenomenon. As we can see, pass@1024 increases during the initial phase of training but eventually plateaus. In contrast, pass@1 accuracy continues to climb steadily throughout the entire training process. This phenomenon provides a clear insight. In the early stage, RL actively expands the model’s reasoning boundary by discovering new reasoning patterns that were dormant during pretraining. The training data effectively unlocks latent capabilities that the model had not previously exhibited. In the later stage, the focus shifts. RL sharpens the output distribution and refines the policy. It teaches the model to consistently produce correct solutions within its established reasoning boundary, thereby driving the continuous improvement in pass@1. This shift proves that once the dormant capabilities are fully unlocked, the “discovery” phase ends. The RL process then smoothly transitions into the “sharpening” phase, where it ceases to push the boundary further and instead focuses on refining the policy to perfectly exploit its established limits.

5.3 How to Improve Reasoning Capabilities While Keeping Efficiency?↩︎

GRPO with a token-level loss inherently encourages longer responses [7], [16]. This happens because the gradient magnitude scales directly with the sequence length. While this property is highly effective for bootstrapping chain-of-thought reasoning early on, it creates a severe tension with inference efficiency. The model quickly learns to generate longer outputs even when the extra text provides no improvement in accuracy. To mitigate this length bias, existing approaches typically incorporate explicit length penalties into the reward function [30] or adjust normalization strategies [31]. However, these techniques fall short of actively regulating response length. Relying on the model to self-prune is inherently passive, which often fails to reliably curb verbosity, leaving the model prone to generating unconstrained and redundant outputs.

In contrast, we address this tension through a two-phase approach. First, during the self-distillation phase, we actively trim the reasoning trajectories. We remove repetitive verification steps, compress verbose sub-derivations, and eliminate circular reasoning before distilling the data back into the base model. This process yields a student model that retains the expert’s logical rigor but produces much shorter outputs. Second, in Second Stage RL, we switch to a sample-level loss normalization. This eliminates the mathematical bias toward longer responses. It allows the model to optimize for accuracy without uncontrolled length growth. While this pipeline achieves a practical compromise, it remains a heuristic workaround. An ideal approach must jointly optimize reasoning quality and token efficiency within a single, unified RL objective.

5.4 How to Balance Training Efficiency and Stability?↩︎

In zero RL for mathematical reasoning, low-probability tokens hold immense value [16]. These tokens often represent novel problem-solving strategies, creative proof techniques, or unconventional approaches that the base model initially ignores. Amplifying these specific tokens dramatically accelerates learning. Our experiments show that explicitly boosting low-probability correct tokens leads to significantly faster convergence than vanilla GRPO. However, this speed comes at a severe cost to stability. When the importance ratio \(\rho_{i,t}\) becomes too large, it magnifies the learning signal. Unfortunately, it simultaneously magnifies any underlying numerical inconsistencies between the training and inference engines.

Prior studies have explored various strategies to stabilize RL training, including importance ratio clipping [27], sampling ratio correction [13], and strict KL divergence constraints [7]. Guided by the principle of minimal modifications, our design demonstrates that the joint application of ratio correction and a KL divergence penalty is the most critical mechanism for regulating training stability. Furthermore, we find that the preceding self-distillation phase also inherently stabilizes the base model, laying a more robust foundation for subsequent policy updates. Specifically, our solution maximizes the benefit of low-probability tokens while deploying strict stabilization mechanisms. First, a KL divergence penalty constrains the overall distribution drift. This prevents the policy from collapsing into a narrow, degenerate set of token sequences. Second, our training-inference ratio correction fundamentally eliminates the floating-point discrepancy. This prevents the amplification process from blowing up numerical errors. Together, these dual mechanisms allow us to enjoy extremely fast convergence while maintaining perfectly stable training over thousands of steps.

5.5 How Does Native RL-based CoT Compare to Distillation-based Approaches at Scale?↩︎

Native zero RL possesses a fundamental advantage. It requires no external teacher and can independently push the reasoning boundaries of frontier models. Conversely, distillation remains highly sample-efficient. In our experiments, simply distilling trajectories from the trained first-stage RL expert into Ling-2.5-flash-Base produces performance that matches or exceeds prolonged zero RL training on the flash model itself. Nevertheless, distillation merely transfers existing knowledge. Large-scale pretrained models harbor massive, untapped potential. Only continued exploration can unlock this deep reasoning capability, something that static distillation alone can never achieve.

5.6 What Are the Fundamental Limits of RL-based Scalable CoT?↩︎

\(\bullet\) Training data distribution. We identify a fundamental mismatch between natural real-world data and optimal RL training data. As illustrated in Figure 9 (d), while the real world inherently presents a long-tailed difficulty distribution heavily skewed toward simple problems, effective model training does not need to preserve this long tail. Mimicking this natural frequency offers no benefits to an RL model; rather, feeding it an excess of trivial problems wastes computational resources and stalls the learning process. Instead, the model strictly requires a dynamic curriculum where the problem difficulty scales continuously alongside its growing capabilities. Interestingly, we observe a similar long-tail inefficiency regarding general world knowledge. Because most natural text contains basic and repetitive facts, we must artificially maximize the density of complex agent data during the mid-training phase. By actively filtering out easy data and forcing the model to engage with highly difficult trajectories, we push it out of its comfort zone and successfully expand its reasoning boundaries.

\(\bullet\) Model capacity. Model scale dictates both the performance ceiling and the learning speed. Larger models possess richer internal representations. Therefore, they are far more likely to produce surprising emergent behaviors and achieve stronger final results.

\(\bullet\) Context window. Longer reasoning chains unlock complex, multi-step derivations. Due to hardware resource constraints, we restrict our training to a 64k context window. We firmly believe that expanding this window further will directly unlock new levels of mathematical capability.

\(\bullet\) Pretrained priors. Zero RL can only bootstrap reasoning from the knowledge already embedded during pretraining. If specific mathematical concepts or proof techniques are absent from the pretraining data, RL cannot magically invent them. The pretrained model’s world knowledge and pattern library form a rigid upper bound. RL can sharpen and optimize within this boundary, but it cannot fundamentally transcend it.

5.7 Differences from Prior Work↩︎

While DeepSeek-R1 [8] pioneers the application of pure reinforcement learning for reasoning, our research at the 1-trillion parameter scale introduces several key advancements. We summarize our main differences from prior work into four distinct areas.

\(\bullet\) First, we shift the focus to the intrinsic quality of the reasoning process. While previous works mainly evaluate final answer accuracy, we critically assess the intermediate chain-of-thought trajectories. To this end, we propose a structured evaluation framework that quantifies reasoning quality across three specific dimensions: comprehensibility, reproducibility, and efficiency.

\(\bullet\) Second, we expand the zero RL paradigm into a multi-stage self-iterative process. DeepSeek-R1 shows that starting zero RL directly from a base model is highly effective. We build upon this success by designing a progressive pipeline that continuously improves the reasoning capabilities of the model. Notably, in our final training stage, we introduce multiple reasoning modes, enabling a single model to dynamically adapt its reasoning depth to the problem’s complexity. Furthermore, guided by minimalist design principles, we implement only the strictly essential algorithmic and infrastructural optimizations, ensuring immense scalability and training stability.

\(\bullet\) Third, our findings provide a strong empirical validation of the bitter lesson in artificial intelligence. We demonstrate that massive computation and model scale surpass human-engineered heuristics. By scaling our training to the 1-trillion parameter regime, we unlock significantly higher performance ceilings and achieve superior training efficiency.

\(\bullet\) Finally, and most strikingly, we observe the spontaneous emergence of highly complex cognitive strategies. While previous RL models exhibited basic self-reflection, Ring-2.5-1T-Zero progresses to much more sophisticated thinking mechanisms, obviating the need for complex, hand-crafted reasoning pipelines. Driven entirely by our self-iterative training process without human-annotated data, the model autonomously converges on five advanced emergent behaviors: anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety. It naturally discovers these strategies as optimal paradigms for solving complex mathematical problems.

6 A Bitter Lesson↩︎

The history of AI research repeatedly demonstrates a “bitter lesson”: general methods that leverage computation and scale ultimately overtake human-engineered heuristics. In the context of LLM reasoning, researchers have invested heavily in crafting specialized prompts, curated SFT datasets, and complex pipelines to force models to present in a structured format, verify their own answers, or explore parallel paths [3], [32], [33]. Indeed, in our own experiments on the 104B model, we found it necessary to introduce a hand-designed reward to successfully guide the model toward structured formatting and self-verification. However, when scaling to the 1-Trillion parameter model, such hand-crafted heuristics become completely redundant. Under pure zero RL, the 1T model spontaneously discovers these advanced cognitive strategies from scratch, without any explicit supervision or auxiliary rewards. In this section, we showcase five striking emergent behaviors that validate this bitter lesson.

6.1 Anthropomorphic Reasoning Traces↩︎

Interestingly, our case analysis reveals another fascinating phenomenon: during complex problem-solving, the model frequently exhibits anthropomorphic reasoning traces. These traces manifest as simulated emotional states and informal meta-commentary, which we categorize into three distinct psychological behaviors: (1) Simulated Frustration and Venting (e.g., using colloquialisms like “brain fart” or mild profanities such as “shit” to signal error detection); (2) Simulated Slacking and Guesswork (e.g., “fudge”, “wing it”, reflecting a simulated tendency to bypass rigorous computation); and (3) Self-Praise and Playful Banter (e.g., “genius”, “nailed it”, or humor like “jk”). We hypothesize that these corpus-induced artifacts stem from internet forum discussions and raw human thought processes in the pre-training data. This suggests the model has learned to mimic not only the mathematical logic of human experts, but also their underlying cognitive friction and emotional dynamics.

6.2 Structured Format↩︎

Without any formatting instructions beyond a simple “reason step by step”, the model naturally organizes its thoughts into highly structured, pedagogically clear traces. It spontaneously adopts explicit step numbering, clear phase transitions, and distinct intermediate summaries. This demonstrates that structural clarity is not merely a stylistic preference that needs to be taught via human annotations; rather, it emerges as a natural, optimal strategy for the model to organize its own long-context attention. More importantly, from an algorithmic perspective, this natural structure suggests that LLM reasoning can form a higher-level action space, extending beyond standard token-level actions or agent-style task steps, which opens up new pathways for future reinforcement learning optimization.

6.3 Parallel Reasoning↩︎

We observe that the model spontaneously branches into alternative strategies instead of getting trapped in a single, narrow chain of thought. Effectively executing a self-contained “Tree-of-Thought” search within a single linear rollout, the model evaluates competing approaches, compares their outcomes, and only commits to a final answer when multiple independent lines of evidence converge. This proves that parallel exploration is a naturally optimal algorithm discovered by RL, not just an engineering trick requiring special optimization.

6.4 Context Anxiety↩︎

Finally, we identify a fascinating, albeit flawed, emergent behavior that we term context anxiety. As the model approaches what it perceives as its maximum token limit, it experiences a strategic panic. It actively aborts its complex reasoning chain to force a heuristic guess. It consciously prioritizes structural completeness over mathematical rigor, demonstrating a deep, learned awareness of the game’s rules: failing to format correctly yields zero reward, but guessing at least offers a non-zero probability of success.

6.5 Self-Verification↩︎

Instead of blindly plowing forward, the model spontaneously learns to verify its intermediate derivations. Once it reaches a candidate conclusion, it actively re-examines its initial assumptions, cross-checks against known formulas, and substitutes the result back into the original constraints to ensure logical consistency. Remarkably, it executes these rigorous sanity checks purely out of a learned necessity to secure the final correctness reward, entirely bypassing the need for human-designed verification templates.

7 Related Work↩︎

7.1 Reinforcement Learning with Verifiable Reward↩︎

Reinforcement learning with verifiable rewards (RLVR) has become a powerful paradigm for unlocking the reasoning capabilities of large language models. DeepSeekMath [7] first introduced GRPO, which eliminates the need for a separate critic model by relying on group-level relative rewards. Building upon this, DeepSeek-R1 [8] proved that pure RL applied directly to base models can spontaneously incentivize emergent chain-of-thought reasoning. This “zero RL” approach requires absolutely no supervised fine-tuning. Following this breakthrough, numerous open-source initiatives have explored bootstrapping reasoning from scratch. SimpleRL-Zoo [34] and Open-Reasoner-Zero [35] systematically identified key hyperparameter choices necessary for maintaining training stability. Furthermore, AceReason-Nemotron [36] demonstrated that native RL can surpass static distillation when the underlying model has sufficient capacity. On the algorithmic front, researchers continue to refine policy optimization for mathematical reasoning. DAPO [15] introduced asymmetric clipping and dynamic sampling to enhance large-scale stability. Meanwhile, VAPO [37] proposed a value-based augmented PPO to explicitly support long-chain reasoning. Unlike these prior works, our research focuses on the numerical and structural bottlenecks of scaling zero RL. We adopt the clipped importance ratio policy optimization [16] and introduce a training-inference ratio correction to eliminate logit discrepancies. We then build a multi-stage pipeline that progressively shifts from a token-level to a sample-level loss and employs tier-based training to give the model adaptive reasoning depth.

7.2 Large-scale Model Training↩︎

Training trillion-parameter models requires a highly sophisticated combination of parallelism strategies. Megatron-LM [38], [39] pioneered tensor and pipeline parallelism for massive model architectures. Concurrently, ZeRO [40] dramatically reduced memory redundancy by partitioning optimizer states across devices. More recently, MegaScale [41] detailed robust production-level solutions for training across tens of thousands of GPUs. For RL-specific workloads, frameworks such as veRL [19] and OpenRLHF [18] provide highly optimized distributed orchestration and serve as vital foundations for the open-source community. Building upon these systems, our work successfully scale zero RL to a 1-Trillion parameter model while utilizing optimized context parallelism to efficiently support stable long context training.

8 Conclusion↩︎

In this work, we addressed a fundamental question: How did zero reinforcement learning behave and scale at the trillion-parameter level? Using a simple design, we built a stable and efficient pipeline to apply zero RL directly to a pretrained 1T-parameter base model. We showed that complex engineering and human-designed rules were not necessary for scaling. Instead, a few simple changes, such as clipped importance sampling, training-inference ratio correction, and mix-precision control, were enough to solve system bottlenecks and develop high-quality reasoning from scratch.

Our experiments gave strong evidence for the “bitter lesson” in artificial intelligence, showing that simple, scalable computation worked better than complex human rules. We summarized our key findings into three main points. First, scale greatly improved both model capability and training efficiency, as the 1T model learned much faster and performed much better than smaller models. Second, we revealed how RL works for reasoning by identifying a clear two-phase learning process. This process started with a discovery phase that unlocked reasoning skills, followed by a sharpening phase that made the model more precise. Third, the 1T model developed advanced reasoning behaviors on its own, such as anthropomorphism, structured formatting, self-verification, parallel exploration, and context anxiety, making hand-crafted human rules unnecessary.

While our model achieved strong results on seven math benchmarks, we also prioritized the actual quality of its reasoning process. Specifically, we evaluated Ring-2.5-1T-Zero on readability, reproducibility, and efficiency, finding that it consistently produced clear, concise, and highly readable solutions. By sharing this setup and our results, we provided a clear, reproducible path for future research on trillion-parameter models, which ultimately brought valuable benefits to the wider AI community. Indeed, by demonstrating that a simple design was sufficient to scale RL to 1T parameters, we lowered the engineering barriers for large-scale training, thereby providing an open playbook to help other researchers avoid expensive trial-and-error and build more efficient, self-evolving systems.

9 Comparison of Our CoT with Other Models↩︎

In this section, we compare the Chain-of-Thought (CoT) reasoning traces generated by our model against those produced by other frontier models. As presented below, our model produces reasoning traces that exhibit high token efficiency and a much clearer, highly structured format.

10 LLM-as-a-Judge Evaluation Prompts↩︎

Assessing the intrinsic quality of the underlying reasoning process requires a deep logical review. To achieve this at scale, we employ an LLM-as-a-Judge protocol for evaluations. This automated judge strictly examines the reasoning traces based on three core criteria. These criteria are logical coherence, causal explicitness, and the absence of hallucinations. Specifically, we conduct pairwise comparisons between the Chain-of-Thought (CoT) traces of Ring-2.5-1T-Zero and four state-of-the-art baselines (i.e., GLM-5.1, Kimi-k2.6, MiniMax-2.7, and Qwen3.5-397B). We perform the evaluations across the AIME 2024, AIME 2025, and AIME 2026 benchmarks. For each match, the judge receives a single math problem alongside two competing reasoning traces. We randomly shuffle the presentation order of Model A and Model B to strictly eliminate position bias. The judge then delivers a structured verdict of A, Tie, or B.

{
  "analysis": "<2-4 sentence comparative analysis>",
  "verdict": "<A | Tie | B>"
}

11 Showcase of Our Reasoning Traces↩︎

In this section, we present a showcase of Chain-of-Thought (CoT) reasoning traces generated by Ring-2.5-1T-Zero. As demonstrated below, our model autonomously produces highly structured and readable derivations without the need for external formatting supervision.

References↩︎