1 Introduction↩︎

Large Language Models (LLMs) are widely used across various tasks, but their inference cost remains a major bottleneck. A significant portion of this cost comes from the multilayer perceptron (MLP) blocks inside each transformer layer. Therefore, optimizing MLP blocks can speed up on-device LLM inference. Recent works [1], [2] have shown that these activations contain significant sparsity that can be exploited by thresholding MLP gate activations, which can speed up LLM inference without modifying the overall transformer architecture. However, deciding which activations to remove and when to apply the modified computation remains a challenge.

A common strategy to address this challenge is to calibrate one activation threshold per layer from activation statistics from a calibration dataset. CATS [1] uses a percentile-based approach to choose these layerwise thresholds. But percentile-based calibration focuses primarily on how many activations are removed, instead of examining how much damage a given threshold causes to the resultant MLP output. Two adjacent thresholds in the activation layer can have drastically different amounts of information lost in the final output. Even more broadly, activation modification is applied uniformly across all tokens for a generation prompt, even though the impact of the modification is not necessarily uniform across tokens.

In this work, we study sparsification of the MLP layer in LLMs at two levels. First, we propose Sensitivity-aware Thresholding for Sparsity (SATS), a threshold calibration method that replaces percentile-based threshold selection with a sensitivity-aware method based on layerwise MLP output distortion. SATS chooses thresholds to satisfy a local layer sensitivity budget, and the layerwise final operating points are chosen to satisfy a global target sparsity. Secondly, we introduce a lightweight token routing method that decides whether to use the dense MLP path or the sparse path (that uses the activation thresholding) for each token. The routing method only requires the token identity, which allows it to be simplified into a look-up table with negligible effect on the inference speed.

We evaluate both of the methods on llama 3.1 8B [3] and Qwen 3 8B [4]. Across both model families, SATS provides improved quality compared to percentile-based thresholding at matched realized sparsity. We then show that token routing further improves the quality-throughput trade-off relative to static sparse or dense model execution. Our main contributions are as follows:

• We propose SATS, a sensitivity-aware threshold calibration method for activation sparsification that preserves the existing runtime activation threshold masking method

• We show that SATS improves over percentile-based threshold calibration at matched realized sparsity on two recent open-weight LLM families.

• We introduce a token-identity-based routing method that selects between the dense and sparse computation paths at inference with minimal routing overhead

• We show that token routing improves the quality-throughput trade-off relative to the static sparse execution while remaining faster than dense inference.

2 Background↩︎

We focus on the transformer-based LLMs that use Gated MLP blocks, such as llama [5] and Qwen [6] families. Given an input vector \(x\), the gated MLP block can be represented using the following equation: \[\mathrm{Gated\text{-}MLP}(x) := (\mathrm{Activation}(xW_{\mathrm{gate}})*(xW_{\mathrm{up}}))W_{\mathrm{down}}\]

where \(W_{gate}\), \(W_{up}\) and \(W_{down}\) represent the weights in the MLP layer. While the activation in the MLP layer can vary, SiLU [7] is a widely adopted activation, which can be represented using the following equation:

\[\mathrm{SiLU} := x*\mathrm{sigmoid}(x) = \frac{x}{1+e^{-x}}\]

CATS [1] seeks to utilize activation sparsity in the MLP layers by applying specific thresholds to sparsify the values in the activation. Given a sparsity level \(k\) (e.g. 50%) as input, the activations over a random subset of the training data, limited to 500 data points, are calculated. This is used to compute the cutoff threshold as the kth percentile of the resulting values. \[t := \min\{t^{\prime} : F(t^{\prime}) \ge k\}\]

After calculating the threshold, CATS applies the following activation function on top of the existing activation function in the MLP layer. \[\mathrm{CATS}_{t}(x_{j}) := \begin{cases} x_{j}, & \mathrm {if}\;|x_{j}| \ge t \\ 0, & \mathrm{if }\;|x_{j}| < t\end{cases}\]

For the input \(SiLU(xW_{gate})\), this becomes: \[\mathrm{CATS}_{t}(\mathrm{SiLU}(xW_{gate})) := \begin{cases} \mathrm{SiLU}(xW_{gate}), & \mathrm {if }\;|\mathrm{SiLU}(xW_{gate})| \ge t \\ 0, & \mathrm{if }\;|\mathrm{SiLU}(xW_{gate})| < t\end{cases}\]

We extend the CATS framework for more damage-aware thresholding. CATS controls the global sparsity level, but does not explicitly account for the damage caused by removing specific neurons. Our proposed Sensitivity-Aware Thresholding for Sparsity (SATS) framework replaces percentile-based sparsity calibration with a sensitivity-aware threshold selection, while preserving the hardware-efficient CATS runtime masking form.

3 Method↩︎

3.1 SATS: Sensitivity-Aware Thresholding for Sparsity↩︎

To quantify the impact of zeroing out neurons under a specific threshold, we define a local MLP distortion metric inspired by the Cumulative Errors of Tail Truncation (CETT) introduced in \(\mathrm{ReLU}^2\) [2]. Their work demonstrated that a larger MLP truncation error is associated with worse model performance, which motivates our method of using local distortion as a calibration signal. Unlike \(\mathrm{ReLU}^2\), which studies sparsification by thresholding MLP outputs directly, we use this signal only to calibrate the per-layer gate thresholds for CATS-style activation masking.

The CATS method selects thresholds from the empirical distribution of post-gate activations, which directly models sparsity but does not model the distortion caused by thresholding. Applying adjacent thresholds on the gate layer can contribute differently to the final MLP output depending on the up-projection value and the down-projection weights. As shown in Figure 1 (a), similar layer sparsity can introduce substantially different local distortion across layers. Since the sensitivity of the layers to distortion (distortion measured using Equation 1 ) due to thresholding can vary widely, we replace the percentile-based threshold calibration with a sensitivity-aware calibration method while preserving the CATS runtime masking method.

For a candidate threshold \(t\), let \(y_\ell(x)\) denote the dense MLP output at layer \(\ell\), and let \(\tilde{y}_{\ell,t}(x)\) denote the thresholded MLP output obtained after removing values whose post-gate activation magnitude falls below \(t\). We define the local thresholding error as: \[\label{eq:95err} \mathrm{Err}_\ell(t) = \mathbb{E}_{x}\left[\frac{||y_\ell(x) - \tilde{y}_{\ell,t}(x)||_2}{||y_\ell(x)||_2 + \epsilon}\right]\tag{1}\]

where \(\epsilon\) is a small constant for numerical stability. This score measures how much thresholding at a specific \(t\) distorts the MLP output at layer \(\ell\).

As in CATS, we first build a candidate threshold set \(\mathcal{T}\) from a calibration dataset using the post-gate activation magnitudes. This maintains the inference-time gate masking method from CATS, however the threshold selection method is modified. For each candidate threshold, we estimate its local error score \(\mathrm{Err}_\ell(t)\) on a calibration dataset. We then introduce a layer-wise error budget \(b\) and for each layer \(\ell\), we select the value of the largest threshold \(t_\ell\) whose local error does not exceed that budget.

\[t_\ell^\star(b) = \max\{t \in \mathcal{T}_\ell : \mathrm{Err}_\ell(t) \le b\}\]

For any fixed budget \(b\), applying this rule independently across all layers yields a model-wide threshold assignment. \[T^\star(b) = [t_1^\star(b),\dots,t_L^\star(b)]\]

This defines one sparse model for each choice of \(b\). Larger budgets permit more aggressive thresholding and generally lead to higher sparsity. Therefore, the choice of \(b\) is particularly important here. While \(b\) is selected to model the local MLP distortion, it remains a useful signal in modeling the end-to-end LLM performance. Figure 1 (b) demonstrates that local MLP distortion tracks end-to-end model performance consistently, which shows that it is informative enough to guide threshold calibration.

We treat \(b\) as a calibration parameter used to reach a desired sparsity level. Specifically, for a target sparsity level \(s_{\mathrm{tar}}\), each candidate budget \(b\) generates a threshold assignment \(T^\star(b)\), which yields a realized average sparsity S(b), defined as: \[S(b) = \frac{1}{L}\sum_{\ell=1}^{L} \mathrm{sp}_\ell(t_\ell^\star(b))\] where \(\mathrm{sp}_\ell(t)\) denotes the expected sparsity at layer \(\ell\) under a given threshold \(t\). We then select the budget whose realized sparsity is closest to the target: \[b^\star = \arg\min_b | S(b) - s_{\mathrm{tar}}|\] This makes the roles of \(b\) and \(s_{\mathrm{tar}}\) distinct. Budget \(b\) controls which threshold is admissible under the local error criterion, and target sparsity \(s_{\mathrm{tar}}\) is used only in an outer calibration loop to choose among the resulting sparse models.

In practice, we implement this calibration with a coarse-to-fine search. We first sweep a coarse set of budget values and identify the region whose realized sparsity is closest to our target sparsity. We then refine locally around that region by searching nearby threshold candidates more densely for each layer. Conceptually, this approach recasts threshold calibration as a target-sparsity search over a sensitivity-constrained threshold selection.

3.2 Token Routing↩︎

The activation thresholding methods discussed so far produce a single modified sparse model, which is then applied uniformly whenever the sparse configuration is used. While this static design is simple, it also makes the same sparse computation path for every token. Token routing asks whether this decision can be made more selectively. Instead of applying the sparse path uniformly, we allow each token to choose between the base model path, which preserves original model performance, and a sparse path, which applies activation thresholding for faster inference.

Considering practical model deployment scenarios, we only use the current token identity for the routing decision. This keeps the routing overhead minimal, since deployment requires only a lookup into a precomputed table. Under this constraint, the main goal is to assign each token a score that reflects how safe it is to use the sparse path on average.

To construct this score, we use a calibration corpus of tokenized sequences. For each token occurrence \(i\) in that corpus, we run both the base path and the sparse path on the same token and record the next-token cross-entropy loss for that position. It is important to note that these losses are not intrinsic properties of the token type. Rather, they are measured from actual occurrences in the calibration corpus and depend on the surrounding sequence context. Routing calibration, therefore, builds a statistical estimate of how risky the sparse path would be for a given token.

Let \(x_i\) denote the identity of the current token occurrence, and let \(\ell_{\mathrm{base}}\) and \(\ell_{\mathrm{sparse}}\) denote the per-token cross-entropy losses of the base path and sparse path, respectively. For the main routing target, we use the excess damage of the sparse path relative to the base path: \[d_i = \max(0, \ell_{\mathrm{sparse}} - \ell_{\mathrm{base}})\] This value is zero when the sparse path is no worse than the base path at occurrence \(i\), and positive when the sparse path has higher loss. Therefore, this value approximates how harmful routing that token through the sparse path would be.

Since the routing mechanism only observes the current token identity \(x_i\), it cannot predict the exact loss difference for each occurrence. Instead, it estimates a statistical average of this excess damage based on the calibration dataset: \[s(x) \approx \mathbb{E}[d_i \mid x_i = x]\] Lower scores indicate tokens that are, on average, safer to route through the sparse path.

We store these estimates in a lookup table over the vocabulary. Let \(\mathcal{I}_x = \{i : x_i = x\}\) be the set of positions at which token \(x\) appears in the routing corpus. The token score is estimated as \[\mathrm{route}(x_i; \tau) = \begin{cases} \text{sparse path} & \text{if } \hat{s}(x_i) < \tau, \\ \text{base path} & \text{otherwise} \end{cases}\] For our routing method, a token sent to the sparse path uses the activation-thresholded MLP computation across transformer blocks, while tokens sent to the base path use the base MLP blocks. The threshold \(\tau\) is used to control the routed-token fraction, and in turn, the quality-throughput trade-off.

As in SATS, calibration and operating-point selection are treated separately. We first estimate the token score table on a routing dataset, then calibrate the routing threshold on a held-out split. For a target routed fraction \(r_{\mathrm{tar}}\), each candidate threshold \(\tau\) induces a realized routed fraction \(R(\tau)\) on the calibration split, and we select \[\tau^\star = \arg\min_{\tau} |R(\tau) - r_{\mathrm{tar}}|\] This produces different routing policies at different operating points. In our experiments, we report the performance of 25%, 50%, and 75% target routing settings to validate the effectiveness of our approach. These routing settings are calibration operating points obtained by sweeping routing thresholds on a held-out calibration dataset and selecting the threshold whose realized routed fraction is closest to the corresponding target (e.g., 25%, 50%, or 75% target sparsity).

4 Experiments↩︎

Base Models: We use the llama 3.1 8B [3] and Qwen 3 8B  [4] models to demonstrate the effectiveness of our approaches.

Sparsity Levels: We evaluate the performance of CATS at 30% and 50% targeted sparsity levels, since at higher sparsity levels (e.g. 70% or 90%), too much information is removed, which makes them impractical for deployment. For proper comparison, we find the equivalent operating point that yields a similar realized sparsity level for SATS and compare the performance with CATS.

For token routing experiments, we pick SATS matched to CATS 50% for the two base models and observe their performance. Practically, at roughly 30% sparsity level, both the accuracy loss and speedup gain are far too small to benefit from the quality-throughput trade-off offered by token routing, so we only demonstrate the effectiveness of the method at SATS matched to CATS 50% sparsity.

For the final SATS configuration used in our experiments, we apply SATS to all MLP layers except the MLP in the final layer, where we retain the corresponding CATS threshold. Empirically, we found that thresholding this MLP layer had an outsized impact on text generation performance, and applying SATS there could disproportionately degrade language-modeling quality even when earlier layers might benefit from SATS. We therefore keep the CATS threshold in the final MLP layer as a pragmatic guardrail, while using SATS on the remaining layers to reach an equivalent CATS sparsity and demonstrate the effectiveness of our approach.

Metrics: We report the average accuracy over seven downstream tasks and perplexity on the Wiki-Text2 [8] test dataset and a held-out test set from the RefinedWeb [9] dataset. For throughput, we follow CATS and report the FP32 end-to-end throughput. We further conducted experiments to observe the variation in performance and latency between FP32 (full precision) and BF16 (half-precision).

Datasets: For downstream task evaluation, we use PIQA [10], OpenBookQA [11], ARC-Easy [12], Winogrande  [13], Hellaswag [14], SciQ [15] and BoolQ  [16] from LM Eval Harness [17]. For fine-tuning base models, we used a small subset of the RefinedWeb dataset [9]. Language modeling quality is measured with perplexity on a held-out split of RefinedWeb and Wiki-Text2 [8] test set.

Computational Environment: For evaluating the quality and end-to-end throughput of the models, we utilized NVIDIA A100 40 GB. For fine-tuning base models using LoRA [18], we used NVIDIA RTX Pro 6000 96 GB. During inference, we used the transformers v4.57.6 Huggingface library, PyTorch v2.8, and CUDA v12.8. We used Triton for our GPU kernels, while torch.compile was used for evaluating end-to-end inference speedup.

4.1 Comparison of SATS and CATS↩︎

4pt

Training-free Performance: The comparative performance of CATS [1] and the proposed SATS is provided in Table 1 and Table 2, which demonstrate the downstream task performance and generation performance, respectively.

The tables show that at the matched 30% operating point, SATS is broadly competitive with CATS on both models. On llama 3.1 8B, SATS has a slight improvement in average downstream accuracy relative to CATS, but with a small perplexity cost. On Qwen 3 8B, SATS improves average accuracy and Wiki-Text2 perplexity, while RefinedWeb perplexity remains slightly worse.

The clearer gains appear at the matched 50% operating point. On llama 3.1 8B, SATS improves average downstream accuracy from 75.44 to 76.2 at an identical sparsity level, while providing better perplexity scores on both RefinedWeb and Wiki-Text2. On Qwen 3 8B, SATS improves average downstream accuracy from 73.99 to 74.31 while improving perplexity metrics. This demonstrates that at similar realized sparsity levels, the calibration method of SATS outperforms CATS in terms of performance. While we reported our accuracy and perplexity results in BF16, we also conducted experiments on the difference in performance between FP32 and BF16 on llama 3.1 8B on Table 5, which demonstrates that the difference between the performance of the model in these data formats are minimal.

4pt

Fine-tuned model performance: We compare the performance of the fine-tuned CATS and SATS on Table 3 and Table 4. While Table 3 indicates that the difference between CATS and SATS is reduced in downstream task average accuracy, Table 4 highlights that for the generation task performance, SATS still performs better.

While llama 3.1 8B models show little difference in downstream task performance compared to training-free methods, the effect of fine-tuning becomes more evident in language modeling tasks, where these models have lower levels of perplexity in general. It is also interesting to note that the fine-tuned Qwen 3 models outperformed even the corresponding base model.

4pt

5pt

5pt

4.2 Evaluation of Token Routing↩︎

We use the SATS matched to CATS-50% operating point and evaluate the effectiveness of the token routing method on both training-free and fine-tuned settings.

Training-free Methods: The results in Table 6 indicate that token routing improves the quality relative to the static SATS method while preserving most of the throughput gain over dense baseline inference. On llama 3.1 8B, the 50% target routing provides the best balance between quality and speed in terms of accuracy and perplexity.

On Qwen 3 8B, the 50% target routing improves average accuracy from 74.31 to 75.11, while improving the perplexity metrics. Though static SATS provides the best throughput gain, routed SATS remain fast while boosting average accuracy. While we have measured our latency in FP32 throughout the experiments, we also observed the BF16 performance of the proposed token routing method on Table 8 for llama 3.1 8B, which follows similar pattern as the FP32 latency performance of the model.

Fine-tuning Methods: The routing results for the fine-tuned models are shown in Table 7. The speedup gain in the fine-tuned routed models is less pronounced compared to the training-free setting, since the sparse path needs to perform both the thresholded base MLP computation and the LoRA update. In our initial experiments, the naive implementation introduced substantial extra overhead from separate low-rank operations. To mitigate this cost, we use LoRA-aware sparse kernels that combine selected LoRA computations with the sparse MLP path and reduce kernel-launch and small-matrix overhead through custom Triton kernels. Though this additional computation narrows the relative gain, the routed fine-tuned models still retain a clear throughput advantage over dense inference.

After fine-tuning the models, the sparse model performance improves notably for Qwen 3. It is interesting to note that with the improved performance of the fine-tuned Qwen 3 model, routing the tokens to the SATS path more (e.g., 50% or 75% route target) improves the accuracy, even surpassing the performance of both static dense and sparse models in terms of average accuracy.

5pt

5pt

5pt

5 Discussion and Conclusion↩︎

We studied how to utilize sparsity in LLMs through sensitivity-aware activation thresholding and token-level routing. Our results indicate that the proposed SATS improves over percentile-based calibration at similar sparsity levels. We also demonstrated that token routing improves downstream accuracy and language-modeling quality while remaining faster than dense inference.

These results indicate that both activation-level and token-level sparsity can be utilized for efficient LLM inference. Better threshold calibration improves the static sparse model quality, while token routing allows selectively applying the sparse model for a better quality-throughput trade-off. Overall, the proposed SATS and token routing improve practical LLM inference efficiency.

References↩︎