1 Introduction↩︎

Post-training, especially reinforcement learning, has become central to advancing the capabilities of large language models [1]–[4]. Its effectiveness depends heavily on data engineering: which samples are used, how to sort the samples, and batching strategies. These choices shape the training signal at every step, making data engineering an important factor for improving both training efficiency and final performance.

Existing post-training data engineering pipelines typically rely on external feedback signals, including human preferences [5], [6], verifier outcomes  [7]–[9], rollout pass rates [10]–[12], and difficulty signals [13]–[16]. These signals have proven useful for data selection and curriculum learning.

However, external signals are often costly to obtain and to apply throughout training [17], leaving the rich data-feedback signals embedded in model internals largely underexplored. Recent work has shown that internal representations can guide data selection in pre-training [18], [19] and supervised fine-tuning [20]–[23], suggesting that model internals encode structure actionable for training. Whether they can play a similar role in post-training data engineering for reinforcement learning remains an open question.

Mechanistic interpretability research [24]–[26] continuously explores how to obtain and understand model internals. As a recent advance, Sparse Autoencoders (SAEs) decompose LLM hidden representations into sparse, fine-grained feature activations [27]–[29], providing fine-grained and disentangled perspectives of LLM internals. While recent pioneering work [30] adopts LLM hidden representations in RL data selection, exploring the fine-grained feature space offered by SAE may lead to more holistic and precise modeling of data properties with model internals.

Therefore, we study the method using SAE activations to capture three intrinsic properties of post-training data: (1) Diversity: distances and clusters in the internal space can measure how broadly a batch covers distinct feature regions and reasoning patterns. (2) Difficulty: sparse activation patterns can reflect the actual demands that a problem imposes on the model, going beyond shallow features such as length or topic. (3) Quality: internal activations can help distinguish samples from the target distribution from noisy or off-distribution raw data. These three properties correspond to concrete data engineering operations: batching strategy, curriculum ordering, and data filtering.

Based on these findings, we propose SaeRL, an intrinsic framework for RL post-training data engineering based on SAE activations. SaeRL uses SAE to model three data properties: quality, difficulty, and diversity. SaeRL then proceeds in three steps: (1) an SAE-based quality probe filters the data pool toward target-distribution samples; (2) samples are clustered in SAE space and sorted by calibrated difficulty within each cluster, forming local easy-to-hard trajectories; (3) batches are interleaved across clusters and moderately mixed by swapping a small tail portion between nearby batches, improving coverage while preserving within-batch coherence.

Experiments on mathematical reasoning show that SaeRL improves performance and efficiency across model scales and RL algorithms. Ablation studies show that batching strategy, curriculum ordering, and data filtering each contribute to the final results. These results suggest that SaeRL improves post-training data engineering by jointly modeling data diversity, sample difficulty, and data quality with SAEs.

Our contributions are twofold: (1) We frame model internals as actionable signals for post-training data engineering. (2) We propose SaeRL, which grounds SAE-based quality, difficulty, and diversity signals in concrete data engineering operations for efficient LLM post-training. We hope that this work can facilitate future research on intrinsic data engineering and actionable mechanistic interpretability [31].

2 Motivating Finding↩︎

We conduct a preliminary study to examine whether SAE activations encode actionable signals for post-training data engineering. We find that they capture three intrinsic data properties—diversity, difficulty, and quality—motivating the design of SaeRL.

2.1 SAE Can Predict Data Diversity↩︎

SAE representations encode diversity-relevant semantic information. Since data diversity corresponds to coverage over distinct topics and skills, we examine whether SAE activations capture such semantic variation by testing their ability to predict external topic labels.

We use DeepMath [32], a large-scale mathematical reasoning dataset with annotated topic labels, in our pilot study. Given an SAE representation \(z_i\) for a data sample, we train a linear probe to predict topic labels at three levels of granularity: \[\hat{t}_i = f_T(z_i).\] As shown in Table 1, SAE features substantially outperform the majority-class baseline across all granularities, including \(82\) leaf topics. This indicates that SAE activations encode topic-level semantic structure, making SAE space a reliable basis for measuring data coverage and diversity in post-training data engineering.

2.2 SAE Can Predict Data Difficulty↩︎

SAE representations encode difficulty-relevant information. Data difficulty is reflected in internal activation patterns—problem meanings, symbolic structure, and required skills—making SAE activations a natural interface for extracting difficulty signals. Given the SAE representation \(z_i\), we train an ElasticNet [33] regressor to predict a continuous difficulty score: \[\hat{d}_i = f_D(z_i).\] As shown in Table 2, SAE features strongly predict in-distribution difficulty and retain a positive signal under distribution shift, indicating that SAE activations capture difficulty-relevant structure beyond shallow cues such as length or topic. This makes them a reliable basis for difficulty-aware curriculum construction.

2.3 SAE Can Predict Data Quality↩︎

SAE representations encode quality-relevant information. Data quality reflects whether a training example is reliable, well-formed, and aligned with the target reasoning distribution. These properties are only partially captured by surface statistics such as length, step count, or TeX ratio.

We use PRM800K [34] as the validation setting, as its step-level process labels provide a reliable proxy for solution quality. We convert these labels into numeric scores (\(+1\!\to\!1\), \(0\!\to\!0.5\), \(-1\!\to\!0\)) and average them within each example to obtain a continuous sample-level quality score. Given the SAE representation \(z_i\), we train a ridge regressor to predict this score: \[\hat{q}_i = f_Q(z_i).\] As shown in Table 3, SAE features outperform both the mean baseline and a metadata-only baseline, improving test Pearson correlation from \(0.2100\) to \(0.3715\) over metadata features. This suggests that SAE activations capture quality-relevant structure beyond shallow cues, supporting their use for quality-aware data filtering.

3 Methodology↩︎

Based on the motivating findings above, we propose SaeRL, an offline data engineering framework for reinforcement learning post-training that uses SAE to model three intrinsic data properties—diversity, difficulty, and quality—and maps them to concrete operations: batching strategy, curriculum ordering, and data filtering.

3.1 SAE Representation↩︎

SAEs decompose dense model activations into sparse, interpretable feature activations  [28], providing a structured interface for extracting content-level signals from model internals. Given a sample \(x_i\), we extract token-level SAE activations separately from its prompt and solution spans, aggregating each via mean and max pooling to capture both sustained and localized activation patterns. The unified representation is \[\phi(x_i) = \bigl[z_i,\, m_i\bigr],\] where \(z_i\) concatenates the pooled SAE activations over both spans, and \(m_i\) is a small set of shallow metadata features (e.g., length statistics, TeX ratio, digit ratio); the SAE part contains \(960\) features and \(m_i\) contains \(26\).

3.2 Diversity-driven Batching Strategy↩︎

We model batch diversity by clustering samples in SAE space and applying moderate batch mixing. Empirically, we find that batch diversity in SAE space has a concave relationship with downstream performance: moderate cross-cluster mixing improves over pure-cluster batches, while excessive mixing hurts optimization (Section 5.2). Appendix 8 provides a bias–variance perspective analysis on this finding.

3.2.0.1 Clustering.

We cluster samples using SAE features and metadata via MiniBatchKMeans  [35] at \(K{=}10\), capturing model-internal structure such as mathematical semantics, problem format, and skill patterns.

3.2.0.2 Moderate batch mixing.

Each batch is paired with a partner batch drawn from a nearby curriculum stage, matched by similar average difficulty and sequence length but required to have a different dominant cluster, with a small tail portion exchanged between the two batches.

3.3 Difficulty-driven Curriculum Ordering↩︎

We model sample difficulty from SAE representations and use it to construct a cluster-first easy-to-hard curriculum.

3.3.0.1 Difficulty proxy and calibration.

As described in Section 2.2, we train a lightweight ElasticNet regressor on a small difficulty-labeled subset (\(|L|{=}3\text{k}\)) to estimate sample difficulty, producing a raw score \(\hat{d}_i = f_D(\phi(x_i))\) for each sample.

Since scores may vary in scale across clusters, we apply cluster-wise calibration using a global mapping with shrinkage-based cluster corrections: \[r_i = \text{Calibrate}\bigl(\hat{d}_i,\, c_i\bigr),\] where \(c_i\) is the cluster assignment of \(x_i\) and \(r_i\) is the final ranking score used for curriculum ordering.

3.3.0.2 Cluster-first curriculum.

Within each cluster, samples are sorted by \(r_i\) into fixed-size batches, forming local easy-to-hard trajectories. The global curriculum then interleaves batches across clusters stage by stage, with moderate batch mixing applied within each stage.

3.4 Quality-driven Data Filtering↩︎

We model sample quality from SAE representations to filter noisy data before curriculum ordering. The probe formalizes this as binary classification: given a sample \(x_i\), it outputs the probability of belonging to the target distribution, \[s_i = p_\psi\bigl(y_i = 1 \mid \phi(x_i)\bigr),\] implemented as a SGD-trained linear classifier  [36] over SAE activations, trained on a subset of source-labeled samples. High-scoring samples are then selected by a fixed threshold \(\mathcal{D}_\tau = \{x_i : s_i \geq \tau\}\) or top-\(k\) ranking \(\mathcal{D}_k = \operatorname{TopK}_{x_i}(s_i)\), filtering the noisy data pool toward the target distribution and providing a higher-quality data source for post-training.

4 Main Experiment↩︎

We evaluate SaeRL in the mathematical reasoning domain, focusing on downstream performance, training efficiency, and noisy-data selection.

4.1 Experiment Setup↩︎

4.1.0.1 Models and training.

We train two model scales, Qwen2.5-Math-1.5B and Qwen2.5-Math- 7B [37], on DeepMath-103K [32] with a batch size of \(128\) to test the generality of SaeRL. We denote SaeRL trained with GRPO  [8] and DAPO  [9] as SaeRL^G and SaeRL^D, respectively. We train an SAE on layer-27 activations of Qwen3-1.7B [38] as the shared encoder for all data engineering operations, demonstrating that a single SAE trained on one model can effectively guide post-training data engineering for other model families and larger scales. Additional details are provided in Appendix 10.2.

4.1.0.2 Evaluation.

We instantiate SaeRL in the mathematical reasoning domain and evaluate on six benchmarks spanning a wide difficulty range: GSM8K  [39] and AMC23 (lower), MATH500  [34] and MinervaMath  [40] (mid), and OlympiadBench  [41] and AIME24 (competition-level), which are referred to as GSM8K, AMC, MATH, MNV, OLPD, and AIME, respectively. We report Pass@8 for AIME24 and Avg@8 for the remaining five benchmarks.

4.1.0.3 Baselines.

We compare SaeRL against five baselines. Vanilla GRPO  [8] and DAPO  [9] serve as RL algorithm baselines without curriculum, and we pair SaeRL with both to test whether its benefits are consistent across RL algorithms. Difficulty Curriculum Learning  [13] uses externally provided difficulty labels, testing whether SAE-based signals add value beyond human annotations. ADARFT  [14] estimates difficulty from rollout accuracy, representing rollout-based curriculum methods. GAINRL  [30] selects data via compressed hidden-state representations, serving as the closest internal-signal baseline to directly test whether sparse SAE features outperform dense alternatives.

4.2 Training Performance↩︎

[tab:main95result] shows that SaeRL improves average accuracy across RL algorithms, baselines, and model scales. At the \(1.5\mathrm{B}\) scale, SaeRL improves both GRPO and DAPO, showing that the SAE-based curriculum is not specific to a particular RL algorithm. Compared with Difficulty Curriculum Learning, ADARFT, and GAINRL, SaeRL obtains stronger overall performance, indicating that sparse SAE activations provide a more useful signal than external difficulty labels, rollout accuracy, or compressed hidden states. At the \(7\mathrm{B}\) scale, SaeRL^G again achieves the best average result among the compared methods, suggesting that a shared SAE trained on a smaller model can still guide data engineering for larger models.

4.3 Training Efficiency↩︎

SaeRL improves training efficiency by reducing both training steps and preparation cost.

[tab:speedup] evaluates convergence speed by measuring how many training steps each method needs to reach a shared target accuracy. At the \(1.5\mathrm{B}\) scale, SaeRL accelerates both GRPO and DAPO. SaeRL^D gives the fastest average convergence, and SaeRL^G requires fewer average steps than GRPO, ADARFT, and GAINRL. At the \(7\mathrm{B}\) scale, SaeRL^G reaches the target in the fewest average steps. These results show that SAE-guided data engineering improves convergence across different model scales and RL algorithms.

SaeRL also demonstrates efficiency gains. The Difficulty baseline and ADARFT achieve comparable convergence speed but require LLM-generated labels or multiple rollouts per problem at substantial cost—ADARFT takes approximately \(17.33\) H100 GPU hours with a reduced rollout budget (Appendix 10.3). In contrast, SaeRL trains the difficulty proxy from a small labeled subset of \(3{,}000\) samples, and SAE encoding for the full dataset of \(103{,}022\) samples takes about \(0.5\) H100 GPU hours. Thus, SaeRL obtains its convergence gains with substantially lower preprocessing overhead.

4.4 Noisy Data Selection↩︎

We further evaluate whether SAE activations support the selection of high-quality samples from a target distribution within a larger mixed noisy pool. We use DeepMath as the target distribution: it is constructed from NuminaMath [42] and other open mathematical sources through decontamination, difficulty filtering, and answer-verifiability filtering [32], making recovery from its source family a meaningful test of quality discrimination. We formulate the task as follows. The raw pool \(\mathcal{D}_{\mathrm{raw}}\) consists of \(103{,}022\) DeepMath samples mixed with \(107{,}021\) samples from the source corpus NuminaMath-1.5, giving \(|\mathcal{D}_{\mathrm{raw}}| = 210{,}043\). The probe is trained to recover the DeepMath subset using only \(d = 960\) SAE features obtained by mean/max pooling over prompt and solution tokens.

4pt

The SAE-only source/style probe achieves \(0.9911\) ROC-AUC and \(0.9910\) AP on the holdout split, indicating that DeepMath-like high-quality samples are highly separable in the SAE activation space. As shown in 4, after applying the fixed probe to \(\mathcal{D}_{\mathrm{raw}}\), the \(p_{95}\) threshold retains \(103{,}121\) samples, with \(95.37\%\) DeepMath purity and \(95.46\%\) recall. Direct top-\(50\mathrm{k}\) selection by the probe score further improves the DeepMath purity to \(99.92\%\). These results suggest that the SAE-based probe captures fine-grained DeepMath-like activation signatures, enabling stable high-quality data selection from noisy data.

5 Analysis↩︎

We analyze the sources of SaeRL’s gains across four dimensions: component contribution, batch diversity control, robustness, and interpretability.

5.1 Ablation Study↩︎

SaeRL relies on the joint effect of batching strategy, curriculum ordering, data filtering. Difficulty sorting defines the easy-to-hard trajectory, cluster-first grouping preserves local coherence in SAE activation space, and moderate batch mixing adds limited cross-cluster coverage without disrupting the trajectory.

5 shows that removing difficulty sorting causes the largest degradation, confirming that the easy-to-hard trajectory is central to SaeRL. The w/o Clus & Mix variant removes cluster assignments and therefore cannot perform moderate batch mixing, leaving a difficulty-only curriculum. Its drop indicates that difficulty sorting alone is insufficient, and SAE-space grouping provides useful local coherence.

Comparing w/o Diff with w/o Diff & Mix shows that mixing without difficulty sorting does not improve the curriculum and can even weaken it. In contrast, the full SaeRL outperforms the variants that remove either difficulty sorting or cluster-based batch construction, indicating that moderate batch mixing is most effective when it is applied on top of an already structured cluster-first, easy-to-hard curriculum.

5.2 Batch Diversity Analysis↩︎

The cluster-first curriculum introduces moderate cross-cluster batch mixing to balance within-batch gradient coherence and cross-cluster coverage. The mixing strength, controlled by the number of tail samples swapped between batches, directly governs this trade-off. To verify that moderate mixing is indeed optimal and to characterize how sensitivity to mixing strength affects downstream performance, we compare five curriculum variants that differ only in this parameter: \[\mathcal{M}=\{\texttt{mix0},\texttt{mix4},\texttt{mix8},\texttt{mix16},\texttt{mix32}\},\] where mix0 is the cluster-first curriculum with no mixing, and larger indices correspond to stronger cross-cluster mixing. All other components of Saerl are held fixed.

We quantify batch diversity by the mean in-batch \(k\)-NN distance (\(k{=}5\)) computed in the two-dimensional SAE projection space, and measure downstream performance by the average mean@8 across the six evaluation benchmarks used in the main experiments. Figure 3 reports both peak performance at step 800 and the number of steps required to reach a fixed threshold \(\tau = 43.0\%\).

The results reveal a clear non-monotonic relationship. Performance improves steadily from mix0 to mix8, and mix8 reaches \(\tau\) in the fewest training steps. Beyond this point, further increasing the mixing strength to mix16 and mix32 degrades both final accuracy and convergence speed—despite mix32 achieving the highest measured diversity. This suggests that beyond a moderate level, cross-cluster mixing disrupts within-batch gradient coherence more than it reduces cluster-local bias.

This pattern is consistent with the bias–variance decomposition in Appendix 8, which shows that the mixing utility is a concave function of mixing strength with a unique interior optimum. The practical takeaway is that effective batch construction requires balancing two competing objectives: preserving local SAE-space coherence to stabilize optimization, while introducing limited cross-cluster coverage to reduce directional bias.

5.3 Batch Size Analysis↩︎

[tab:diff-bs] shows that SaeRL remains effective across batch sizes. Under Avg@8, SaeRL outperforms GRPO at both \(B=128\) and \(B=512\), indicating that the curriculum remains effective beyond the default training batch size. Under Pass@8, increasing the batch size narrows the gap between the two methods. This suggests that larger batches may dilute the structural benefit of an ordered learning trajectory.

5.4 Interpretability Analysis↩︎

Beyond downstream performance, we examine whether SaeRL exposes interpretable structure at the cluster and feature levels during curriculum construction. Additional details are provided in Appendix 11.

5.4.0.1 Cluster-level structure.

Comparing \(\mathcal{C}\) with DeepMath topic metadata \(\mathcal{T}\) yields low alignment (purity \(= 0.1095\), \(\mathrm{NMI} = 0.0881\)), indicating that SAE clusters do not reproduce the human-defined topic taxonomy. Rather, inspection of cluster statistics and representative examples reveals that clusters capture curriculum-relevant properties including problem format, reasoning structure, solution profile, and difficulty. Several clusters also show enrichment for recognizable mathematical areas such as limits, combinatorics, group theory, and integration. SAE clusters thus characterize the data along axes more relevant to curriculum construction than external topic labels.

5.4.0.2 Feature-level signals.

The difficulty proxy relies primarily on SAE activations rather than shallow metadata. Among the top-20 features ranked by LightGBM [43] gain, 19 are SAE-derived and only 1 is a metadata feature; among the top-100, only 3 are metadata features. Within the SAE features, solution-side mean activations dominate, consistent with sustained solution-side patterns providing the strongest correlational signal for difficulty. Prompt-side max activations also contribute, reflecting localized cues in the problem statement such as symbolic structure or diagrammatic format. A high-activation audit in Appendix 11 further shows that individual high-gain features exhibit recurring semantic tendencies spanning abstract algebra, advanced analysis, geometry, combinatorics, and number-theoretic reasoning.

Taken together, these analyses show that SaeRL provides not only an effective curriculum ordering, but also an auditable decision pathway. Each sample can be inspected through its activation group, difficulty-related feature signals, and position within the curriculum.

6 Related Work↩︎

We discuss two trends behind our work: post-training data engineering is becoming more adaptive, and model internals are increasingly used as training signals.

6.1 Post-training Data Engineering↩︎

Post-training data engineering controls which examples are used, when they appear, and how training budget is allocated, which strongly shapes final model behavior  [5]–[7], [44]. Existing methods have evolved from (1) task-, goal-, or replay-based curricula  [13], [45], [46], to (2) quality- and capability-aware selection of compact high-value data or boundary-level prompts  [10], [14]–[16], [47]–[49], and then to (3) optimization- and resource-aware selection using gradients, influence, rollout utility, or distribution schedules  [11], [12], [30], [50]–[54].

However, they still rely mainly on external or scalar signals; SaeRL instead grounds data engineering in model-internal structure.

6.2 Model Internals for LLM Training↩︎

Model internals have moved from post-hoc analysis toward training-time feedback. Existing work uses (1) logit- or loss-based signals for data filtering and token or instruction selection  [55], [56], (2) gradients or influence estimates for data selection and weighting  [50]–[52], [57], and (3) hidden states or activations for efficient example selection or representation-level intervention  [30], [58].

These approaches show that internals can guide training, but they often reduce internal structure to coarse signals. SaeRL instead uses sparse autoencoder features [27]–[29], which provide sparse and fine-grained activation signals for data engineering, extending beyond prior SAE-based uses for tuning-data diversity [23] and preference modeling [59] to the RLVR post-training setting.

7 Conclusion↩︎

We propose SaeRL, a post-training data engineering framework grounded in model-internal sparse representations. SaeRL uses SAE activations as a shared representation space to model three intrinsic data properties—diversity, difficulty, and quality—and grounds each in a concrete data engineering operation: batching strategy, curriculum ordering, and data filtering. Experiments on mathematical reasoning demonstrate consistent gains in accuracy and convergence efficiency across model scales and RL algorithms, with a single SAE transferring effectively across model families. These results suggest that model internals are a powerful and practical source of signals for post-training data engineering, opening a direction complementary to external feedback-based approaches.

Limitations↩︎

7.0.0.1 Domain scope.

Our empirical validation focuses on mathematical reasoning with verifiable rewards. This setting provides a controlled testbed for studying curriculum construction, since both training feedback and evaluation outcomes can be measured reliably. However, the extent to which the same SAE-space structure transfers to other post-training settings remains to be established, including code-centric RL, agentic RL, tool-use and multi-step decision-making, and general instruction-following.

7.0.0.2 Limited supervision.

Although SAERL reduces the need for large-scale labeling or rollout-based scoring, it is not fully unsupervised. The difficulty proxy uses a small difficulty-labeled subset, and the quality probe relies on source or distribution labels as supervision. Future work may explore weaker forms of supervision, self-calibrated scoring, or fully unsupervised criteria for constructing SAE-guided curricula.

7.0.0.3 Theoretical scope.

Our analysis treats proximity in SAE space as a proxy for semantic similarity and gradient coherence. This yields an optimization-level interpretation of the observed coherence–coverage trade-off, but it does not prove a causal relationship between SAE distance and training dynamics. Establishing such a guarantee would require direct gradient-level measurements on the representation space.

Ethical Considerations↩︎

This section discusses the ethical considerations and broader impact of this work.

7.0.0.4 Potential Risks.

SaeRL uses model-internal SAE representations for post-training data engineering. Although intended to improve efficiency and inspectability, such signals could be misused to optimize data for unsafe behaviors. We restrict our experiments to mathematical reasoning with verifiable rewards and recommend safety filtering and human oversight for broader applications.

7.0.0.5 Intellectual Property.

The models, datasets, benchmarks, and software frameworks used in this work are publicly available research artifacts and are used in accordance with their respective licenses and terms of use. Any released code or processed artifacts will follow the corresponding license requirements.

7.0.0.6 Intended Use.

SaeRL is intended as a research framework for post-training data engineering, including curriculum construction, batch organization, data filtering, and interpretability-oriented analysis. It is not intended for developing harmful models, evading safety mechanisms, or optimizing data for malicious capabilities.

7.0.0.7 Documentation of Artifacts.

We will document the released artifacts with sufficient detail to support reproducibility, including the data-processing pipeline, SAE feature extraction, curriculum construction, data filtering, and evaluation setup.

7.0.0.8 AI Assistants in Research or Writing.

We use AI assistants for code development assistance and language polishing. All AI-assisted content is reviewed and edited by the authors, who remain responsible for the final scientific claims, experiments, and writing.

8 A Bias–Variance View of Moderate Batch Mixing↩︎

We provide a bias–variance perspective on why moderate cross-cluster batch mixing can improve optimization.

8.0.0.1 Setup.

For each sample \(x_i\), let \(g_i = \nabla_\theta \ell(x_i;\,\theta)\) denote its per-sample gradient. Since SAE activations \(z_i\) approximate the model’s internal representation of \(x_i\), we assume \(g_i = G(z_i) + \varepsilon_i\) for a locally Lipschitz \(G\colon \mathbb{R}^d \to \mathbb{R}^p\), where \(\varepsilon_i\) captures SAE approximation error and residual nonlinear effects. Under this assumption, samples nearby in SAE space tend to produce similar gradients.

8.0.0.2 Pure-cluster bias.

Consider a batch of size \(b\) drawn from cluster \(c\), with gradient mean \(\mu_c\) and covariance \(\Sigma_c\), and let \(G_t\) denote the target gradient. The MSE of \(\hat{G}_c = \frac{1}{b}\sum_{i=1}^{b} g_i\) decomposes as \[\operatorname{MSE}_0 = \|G_t - \mu_c\|^2 + \frac{1}{b}\operatorname{tr}(\Sigma_c), \label{eq:mse0}\tag{1}\] where the two terms are cluster-local bias and estimation variance, respectively. Pure-cluster batches have low variance but may be biased when \(\mu_c\) deviates from \(G_t\).

8.0.0.3 Effect of mixing.

Suppose the mixed batch contains \(\lfloor \rho b \rfloor\) samples from cluster \(d\) and \((1-\rho)b\) from cluster \(c\), with the two clusters uncorrelated. Let \(r_c = G_t - \mu_c\) and \(v = \mu_d - \mu_c\). The net MSE change relative to equation 1 is \[\Delta(\rho) = -2\rho\langle r_c, v\rangle + \rho^2 \|v\|^2 + \frac{\rho}{b}\operatorname{tr}(\Sigma_d - \Sigma_c), \label{eq:delta}\tag{2}\] When \(v \neq 0\), \(\Delta(\rho)\) is a convex quadratic in \(\rho\) with minimizer \(\rho^\dagger = \operatorname{clip}(\rho^*, 0, 1)\), where \(\rho^* = A/2C\), \(A = 2\langle r_c, v\rangle - \frac{1}{b} \operatorname{tr}(\Sigma_d - \Sigma_c)\), and \(C = \|v\|^2\); when \(v = 0\), the optimum is attained at an endpoint. If \(\rho^* \in (0,1)\), the mixing utility \(U(\rho) = -\Delta(\rho)\) admits an interior maximum: too little mixing leaves cluster-local bias uncorrected, while too much weakens within-batch gradient coherence.

9 Technical Details of the SaeRL Pipeline↩︎

This appendix provides additional technical details for the offline data-processing steps used in Section 3.

9.1 SAE Sample Representation↩︎

Each sample \(x_i\) is divided into a prompt span \(S_i^p\) and a solution span\(S_i^s\). Let \(\mathcal{F}\) denote the retained SAE feature set, and let \(a_t\in\mathbb{R}^{|\mathcal{F}|}\) be the SAE activation vector at token \(t\). For each span, we summarize token-level SAE activations using mean pooling and max pooling. The SAE representation \(z_i\) is obtained by concatenating the mean-pooled and max-pooled activations from both the prompt and solution spans: \[z_i= \big[ \bar a_{S_i^p}; a_{S_i^p}^{\max}; \bar a_{S_i^s}; a_{S_i^s}^{\max} \big].\] For clustering and difficulty estimation, we further append shallow metadata features \(m_i\), including length statistics, TeX ratio, and digit ratio, and use \(\phi_i=[z_i;m_i]\) as the full feature vector. In our experiments, \(z_i\) is 960-dimensional and \(m_i\) is 26-dimensional.

9.2 Clustering and Difficulty Calibration↩︎

We cluster samples in the feature space using MiniBatchKMeans with \(K=10\). Each sample is assigned to the nearest cluster centroid based on its full feature vector \(\phi_i\).

To estimate sample difficulty, we use a small difficulty-labeled subset \(\mathcal{L}\) with \(|\mathcal{L}|=3000\). For each labeled sample, the difficulty label is denoted by \(d_i^\star\). We train an ElasticNet difficulty proxy \(f_D\) on the labeled subset and use it to produce the raw difficulty prediction \(\hat{d}_i=f_D(\phi_i)\).

Because only a limited number of difficulty labels are available, the raw proxy score is calibrated using a global calibration map fitted on the small labeled subset, together with a shrinkage-based cluster residual. Let \(g\) denote the global calibration map fitted on \(\mathcal{L}\).

For each cluster, we compute the average residual between the labeled difficulty \(d_i^\star\) and the globally calibrated prediction \(g(\hat{d}_i)\) over labeled samples in that cluster. If a cluster has no labeled samples, its residual is set to zero. The residual is then scaled by a shrinkage weight \(\lambda_c=n_c/(n_c+\tau_{\mathrm{sh}})\), where \(n_c\) is the number of labeled samples in cluster \(c\) and \(\tau_{\mathrm{sh}}>0\) controls the strength of shrinkage toward the global calibration.

The final difficulty score is \[r_i = g(\hat{d}_i) + \lambda_{c_i}\Delta_{c_i}.\] A larger \(r_i\) corresponds to a higher estimated difficulty.

9.3 Curriculum Ordering and Moderate Batch Mixing↩︎

Within each cluster, samples are sorted by the calibrated difficulty score \(r_i\) in ascending order, resulting in a local easy-to-hard curriculum. The ordered samples in each cluster are then partitioned into fixed-size batches. The global curriculum interleaves these batches across clusters stage by stage, which preserves local easy-to-hard trajectories while maintaining coverage across different clusters.

After the cluster-first curriculum has been constructed, we apply moderate batch mixing. For an ordered batch of size \(b\), we keep the first \(b-h\) samples fixed and exchange only the last \(h\) tail samples with another batch. The partner batch is selected from a local curriculum window and must satisfy three conditions: it should have similar average calibrated difficulty, similar average sequence length, and a different dominant cluster. The dominant cluster of a batch is defined as the most frequent cluster label among its samples.

Since only the tail block is exchanged, this operation introduces limited cross-cluster mixing while largely preserving the local curriculum structure within each batch.

9.4 SAE-based Data Selection↩︎

Raw data selection is formulated as a binary classification problem over SAE representations. Let \(\mathcal{D}_{\mathrm{raw}}\) denote the raw candidate pool. For each candidate sample, \(y_i=1\) indicates membership in the target DeepMath-like distribution, while \(y_i=0\) indicates otherwise.

The quality probe uses only the SAE representation \(z_i\), without metadata. We train a linear classifier with SGD and use its predicted probability as the selection score: \[s_i = p_\psi(y_i=1\mid z_i) = \sigma(w^\top z_i+b),\] where \(\psi=(w,b)\) and \(\sigma\) is the logistic sigmoid.

Given the score \(s_i\), we use either threshold-based selection or fixed-size top-\(k\) selection. The threshold rule selects samples with scores above a quality threshold \(\gamma\), which controls selection precision. The top-\(k\) rule selects the \(k\) highest-scoring samples, which fixes the number of selected samples.

10 Implementation Detail↩︎

This appendix provides the training hyperparameters and baseline implementation details used in the experiments.

10.1 Sparse Autoencoder Training Details↩︎

We train the SAE using the OpenSAE framework on layer-\(27\) activations of Qwen3-1.7B, with an expansion factor of \(64\) to support fine-grained feature extraction. The training corpus consists of FineWeb-Edu [60] and Wikipedia, totaling \(80\)GB. Training was conducted on \(4\) A100 GPUs and completed in approximately \(29\) hours, giving a total cost of \(116\) A100 GPU hours.

10.2 Hyperparameters↩︎

We maintain consistent core hyperparameters across both model scales (1.5B and 7B) to support a controlled comparison. Table 6 summarizes the detailed training configuration on verl.

10.3 Baseline Implementations↩︎

For the Difficulty Curriculum Learning baseline [13], we sort the DeepMath-103K dataset [32] by its provided difficulty labels in ascending order and sample progressively throughout training.

To ensure a fair comparison, we train the ADARFT baseline [14] using GRPO [8]. ADARFT estimates problem difficulty from rollout accuracy, originally using Avg@128. Given the scale of the 103K problem dataset, we adapt this standard and use Avg@16 as the difficulty proxy for our ADARFT implementation, allowing us to simulate the ADARFT curriculum strategy within computational viability constraints.

11 Interpretability Details of SAE-Guided Curriculum Construction↩︎

This appendix supplements the interpretability analysis in Section 5.4. We examine three types of evidence: the relation between SAE clusters and external topic annotations, the feature groups that provide predictive signal for difficulty estimation, and the intermediate variables exposed by the curriculum construction procedure.

4pt

6pt

4pt

11.1 Cluster-Level Structure↩︎

We compare the SAE cluster assignments on DeepMath with human-annotated topic metadata. The overall cluster–topic alignment is low: cluster-topic purity is \(0.1095\), NMI is \(0.0881\), and ARI is \(0.0394\). These results indicate that SAE clusters do not simply reproduce the human-defined mathematical topic taxonomy.

We therefore further inspect each cluster using top leaf topics, topic entropy, difficulty distribution, solution length, and representative samples. Table 7 provides a summary of the resulting cluster structure.

Some clusters exhibit clear topical enrichment, such as limits, combinatorics, group theory, and integration. At the same time, clusters also differ in problem format, solution length, symbolic density, proof orientation, and mean difficulty. These summaries should therefore be understood as coarse descriptions of activation-space structure, rather than as one-to-one topic labels.

This distinction is necessary because the external topic taxonomy and the SAE representation characterize different aspects of a sample. Topic labels describe the subject category assigned by the dataset, whereas SAE clusters are formed according to model-internal activation patterns. A cluster may therefore contain multiple mathematical topics while still preserving structure relevant to curriculum construction, such as similar solution profiles, reasoning formats, or difficulty ranges.

11.2 Feature-Level Signals↩︎

We next examine which input feature groups provide predictive signal for difficulty estimation. The curriculum pipeline uses prompt-side and solution-side SAE features, together with a small set of metadata features. For feature-group analysis, we train an auxiliary LightGBM model on the same difficulty-labeled subset and rank input columns by gain. This model is used only for interpretability analysis; the actual curriculum ranking scores are still produced by the calibrated difficulty proxy described in Section 3.

Table 8 reports the source distribution of the top-20 and top-100 gain-ranked features.

The top-ranked features come primarily from SAE activations rather than metadata. Among the top-20 features, 19 are SAE-derived and 1 is metadata; among the top-100 features, 97 are SAE-derived and 3 are metadata. Within these features, solution-side mean features form the largest group, which is consistent with sustained solution-side activations providing correlational signals for difficulty estimation. Prompt-side max features also appear frequently, indicating that localized strong activations in the problem statement provide additional predictive cues.

To complement the feature-group statistics, we further inspect high-activation samples for several high-gain SAE features. Table 9 summarizes the semantic tendencies observed in these samples.

These high-activation samples exhibit several recurring patterns, including abstract algebra, measure and integration, discrete combinatorics, geometric formats, elementary algebra, and number-theoretic expressions. These descriptions indicate semantic tendencies only. SAE features are not strictly monosemantic: the same SAE feature may be activated in different contexts, and the same mathematical pattern may be distributed across multiple SAE features.

11.3 Procedure-Level Inspectability↩︎

The final curriculum order is constructed from a sequence of explicit intermediate variables. For each sample, the pipeline records its SAE representation, metadata, cluster assignment, raw difficulty prediction, calibrated ranking score, batch assignment, and, when moderate mixing is applied, its tail-swap partner. These variables allow researchers to inspect how a sample moves from the representation space to its final position in the curriculum.

References↩︎

---

1. Equal contribution.↩︎