1 Introduction↩︎

The emergence of large language models capable of code generation—commonly termed CodeLLMs—has fundamentally transformed numerous domains within computer science. These models offer substantial benefits for accelerating software development and streamlining maintenance workflows; however, they simultaneously present significant security concerns. Threat actors can exploit the same generative capabilities to author malware, craft software exploits, and automate vulnerability discovery. Recent threat intelligence reports document adversaries employing LLMs to obfuscate malicious code, generate context-aware payloads, and orchestrate sophisticated intrusion campaigns, confirming that AI-augmented offensive tooling has transitioned from theoretical concern to operational reality [1], [2]. Of particular concern is the potential for adversaries to leverage CodeLLMs’ rapid generation capabilities to systematically mutate existing malware, thereby evading signature-based detection mechanisms while continuously reintroducing new variants into the threat landscape.

This emerging application of CodeLLMs reflects a well-established lineage of code mutation techniques in sophisticated malware [3]–[6]. Because many detection systems continue to rely on static signatures—whether binary patterns, syntactic structures, or system call sequences—malware authors are strongly incentivized to transform their code’s representation while preserving its underlying functionality [3], [6], [7]. Historically, such transformations have been achieved through automated, rule-based mutation engines compact enough to be embedded directly within the malicious payload. This architecture enables real-time polymorphism as the malware propagates through operational environments. Metamorphic malware exemplifies this approach: by rewriting its own code with each replication cycle, it has proven remarkably effective at circumventing static analysis methods [4], [8].

Contemporary CodeLLMs already facilitate offline variant generation, wherein attackers leverage separate infrastructure to produce novel malware strains prior to deployment. However, a more consequential threat lies in the prospect of miniaturization—compressing these models sufficiently to embed directly within malware payloads or execute rapidly on consumer-grade hardware [8], [9]. The central technical question is whether modern CodeLLMs can be reduced to a minimal footprint through pruning, distillation, or quantization while retaining their capacity for semantics-preserving code transformation/mutation [10]–[12]. Crucially, many auxiliary capabilities—such as documentation generation, test synthesis, and general-purpose prompt-based programming—become superfluous when the model’s sole objective is code mutation. This observation motivates the hypothesis that specialized, highly compressed models optimized exclusively for code mutation are not only feasible but represent a significant and understudied attack vector [8], [9]. Our work directly addresses this gap by systematically investigating the compressibility limits of CodeLLMs with respect to their code mutation fidelity.

In this work, we take a first step toward rigorously evaluating this hypothesis. Our key contributions are as follows:

1. We propose SecRL-Prune, an RL-based structured pruning framework for CodeLLMs that uses a KL-guided pruning objective to preserve teacher behavior under compression.

2. We introduce a caching mechanism that accelerates RL training and reduces the resource footprint of pruning: by precomputing teacher outputs on a calibration set, SecRL-Prune can train its policy with only the student model in memory, yielding lower peak GPU usage than prior structured pruning techniques.

3. We conduct a systematic empirical study of how structured compression affects CodeLLMs’ ability to perform functionality-preserving, syntax-diverse code mutation.

4. We conduct a case study using real malware samples to assess how semantics-preserving code mutations (using pruned CodeLLMs) affect malware detection across more than \(62\) industry-standard malware detection engines.

The rest of this paper is structured as follows: In Section 2, we provide a survey of the existing body of work on CodeLLMs and code mutation literature. In Section 3, we provide the necessary background on the structure of LLMs that we leverage in our pruning technique and discuss the fundamentals of how our reward function is designed. In Section 4, we present SecRL-Prune, our novel approach to pruning CodeLLMs. In Section 5, we provide a detailed description of our experimental setup, including the datasets used. In Section 6, we discuss the results, and in Section 7, we provide open questions and a path forward for this research.

2 Related Work↩︎

Malicious use of CodeLLMs, particularly for malware code mutation, has been investigated in recent work on LLM-driven code mutation and metamorphic malware evolution. These studies [8], [9] provide both survey-style overviews and empirical analyses of the code-mutation capabilities of open- and closed-source CodeLLMs. In contrast, our work focuses on recent advances in LLM compression, with an emphasis on preserving functionality relevant to adversarial code generation/mutation [11]–[13].

Model compression aims to reduce model size and improve transformer efficiency while preserving task performance (e.g., code mutation capabilities), using techniques such as knowledge distillation [13]–[15], quantization [10], and pruning [11], [16].

In knowledge distillation, a smaller student LLM is trained to mimic the predictions or internal signals of a larger teacher, typically using a Kullback-Leibler(KL) divergence–based loss between teacher and student distributions [17]. DistilBERT compresses BERT via such an approach, achieving roughly half the parameter count while retaining most of BERT’s language understanding, by combining supervised cross-entropy with a KL-divergence loss on softened logits [13]. MiniLM focuses on distilling self-attention knowledge, training the student to match the teacher’s attention distributions and value relations via KL divergence [14], and MiniLLM transfers the behaviour of large generative LLMs into smaller students using an objective tailored for generation [15].

Quantization techniques, by contrast, reduce the numerical precision used to represent LLM weights to save memory and accelerate inference. For example, QLoRA combines 4-bit quantization with low-rank adapters to enable memory-efficient fine-tuning of large models on standard hardware [10].

Pruning directly removes parameters from an already trained network. In unstructured pruning, individual weights are zeroed out rather than physically removed from the model. This produces fine-grained sparsity patterns that can be highly parameter-efficient in theory, and techniques like SparseGPT use blockwise approximations of the Hessian to decide which weights can be zeroed in LLMs [11]. However, because the original tensor shapes are preserved and zeros are scattered irregularly, practical speedups often depend on specialized sparse kernels or hardware support. Without such support, the compressed model may still execute as a dense network.

In structured pruning, the basic units being removed are larger groups of weights—such as channels, neurons, attention heads, or even whole layers—so that tensor dimensions themselves shrink, yielding architectures that are inherently easier to deploy. Early work in this direction, such as LLM-Pruner, scores "coupled" structures (e.g., head–MLP groups) using gradient and curvature information, removes low-importance groups, and applies lightweight fine-tuning (e.g., LoRA-style adapters) to recover accuracy [16]. Subsequent methods have pushed structured pruning further in two directions. One line of work explores runtime adaptivity, dynamically adjusting which structures are pruned during inference to trade off cost and quality on the fly [18]. Another formulates pruning as policy learning, training a reusable pruning policy that generalizes across compression ratios without requiring additional calibration data [19].

Overall, these techniques show that both unstructured and structured pruning can substantially reduce the effective parameter count of large language models, and that recent post-training methods can achieve high compression ratios with modest fine-tuning. However, most of this work targets natural-language LLMs and is evaluated on generic language understanding or generation benchmarks. Their behaviour on code-generation models and, in particular, their impact on execution-based metrics and mutation/diversity properties of CodeLLMs remains comparatively underexplored. Recall that such compression of CodeLLMs can be an important enabling factor in the development of next-generation metamorphic malware, which is the core thesis of this manuscript.

3 Background↩︎

Before introducing our novel approach for CodeLLMs pruning, we first review the transformer architecture underlying modern CodeLLMs and introduce the notation and Kullback–Leibler (KL) divergence used in our work.

3.1 Transformer Structure in CodeLLMs↩︎

In this work, our pruning actions operate directly on the intermediate channels of the feed-forward network (FFN) inside each transformer block. We therefore focus our discussion on this component. Figure 1 shows the standard two-layer FFN used in decoder-only transformers.

Consider transformer block (layer) \(l\). For a token position \(t\), let \(h_t^{\,l-1}\in\mathbb{R}^d\) be the input hidden state entering the MLP (the \(d\)-dimensional residual stream). The MLP applies two linear layers with a pointwise nonlinearity in between: \[u_t^l = W_1^l\, h_t^{\,l-1}, \qquad z_t^l = \sigma(u_t^l), \qquad y_t^l = W_2^l\, z_t^l,\] where \(W_1^l \in \mathbb{R}^{d_{\mathrm{ff}}^l \times d}\) expands the representation from hidden size \(d\) to an intermediate width \(d_{\mathrm{ff}}^l\), \(\sigma(\cdot)\) is a pointwise activation function (e.g., GELU), and \(W_2^l \in \mathbb{R}^{d \times d_{\mathrm{ff}}^l}\) projects back to \(\mathbb{R}^d\). The output \(y_t^l\) is then added back to the residual stream via the standard residual connection.

The intermediate activation \(z_t^l \in \mathbb{R}^{d_{\mathrm{ff}}^l}\) has \(d_{\mathrm{ff}}^l\) coordinates, and we refer to each coordinate \(z_{t,i}^l\) as an MLP channel (the middle “neurons” in Figure. 1). Channel \(i\) is formed by (a) the \(i\)-th row of \(W_1^l\), which computes \(u_{t,i}^l\), and (b) the \(i\)-th column of \(W_2^l\), which determines how \(z_{t,i}^l\) contributes to the output \(y_t^l\). This channel-level view is exactly the structural unit we later keep or remove in SecRL-Prune.

The final output of the model depends on these MLP computations across all layers. At each time step \(t\), after processing all tokens up to position \(t\), the model produces a probability distribution over the vocabulary \(V\) for the next token. We denote this distribution by \[p(\cdot \mid x_{<t}) = \mathrm{softmax}\!\bigl(W_{\text{out}}\, h_t^{N}\bigr),\] where \(h_t^{N}\in\mathbb{R}^d\) is the final-layer hidden state at position \(t\) and \(W_{\text{out}}\in\mathbb{R}^{|V|\times d}\) is the output projection.

In practice, this distribution is typically peaked: a small set of tokens receives most of the probability mass (labeled “Top Predictions” in Fig. 3), while the remaining tokens share the leftover mass (“Others”).

3.2 KL Divergence and Knowledge Distillation↩︎

Our pruning method (presented in Section 4) aims to compress a CodeLLM while preserving its behaviour on code-generation and mutation tasks. We therefore need a simple token-level measure of how close the pruned model’s next-token predictions are to the original model’s predictions. We use the Kullback–Leibler (KL) divergence [17] in a standard knowledge-distillation setup, where the original model is the teacher and the pruned model is the student.

KL divergence measures the difference between two probability distributions over the same set \(V\). For distributions \(p\) and \(q\) over \(V\), it is defined as \[\label{eq:kl} \mathrm{KL}(p \,\|\, q) = \sum_{v \in V} p(v)\,\log\!\frac{p(v)}{q(v)} .\tag{1}\]

If \(\mathrm{KL}(p\,\|\,q)=0\), then \(p\) and \(q\) are exactly the same distribution (they assign the same probability to every token). As \(\mathrm{KL}(p\,\|\,q)\) increases, \(q\) deviates more from \(p\), meaning the two distributions become less similar.

In our setting, at each time step \(t\) (given the prefix up to \(t\)), the teacher defines a next-token distribution \(p_T(\cdot \mid \text{prefix up to }t)\) and the student defines \(p_S(\cdot \mid \text{prefix up to }t)\). The standard distillation objective averages the teacher-to-student divergence over time steps: \[\label{eq:avg} \begin{align} \mathcal{L}_{\mathrm{KD}} &= \frac{1}{T}\sum_{t=1}^{T} \mathrm{KL}\!\Big( p_T(\cdot \mid \text{pref } t) \,\big\|\, p_S(\cdot \mid \text{pref } t) \Big). \end{align}\tag{2}\]

KL-based distillation is a standard technique for training compact student models to match a larger teacher [13]–[15]. In contrast,SecRL-Prune reuses this teacher-to-student KL divergence as a similarity signal that later guides our pruning policy (discussed in Section 4.3).

4 The Proposed Approach: SecRL-Prune↩︎

Our overarching objective is to compress a given CodeLLM while preserving its ability to perform functionality-preserving code mutation and related adversarial code-generation tasks. To this end, we introduce SecRL-Prune, an RL-based structured pruning framework that operates at the level of MLP channels (defined in Section 3.1) within transformer blocks. Starting from a fully trained CodeLLM (Teacher), SecRL-Prune learns a pruning policy that retains channels most important for code mutation while pruning others, guided by a KL-based distillation reward. We focus on pruning the feed-forward (FFN) layers because they contain the majority of model parameters (roughly 60–70%) compared to self-attention, embeddings, and output projection layers.

As depicted in Figure 2, SecRL-Prune takes two inputs: a calibration set of code prompts and a pretrained CodeLLM, which serves as the teacher. The core problem is determining, for each MLP channel (in the teacher model) across all layers, whether to keep it or remove it. This defines a combinatorially large search space (i.e., exponential in the total number of channels) that makes exhaustive or random search intractable. We formulate this as a reinforcement learning problem: rather than searching blindly, we train a policy that learns to identify which channels are important and which can be safely pruned, guided by a reward signal that measures how well each candidate mask (a binary vector keep/remove decision for every channel) preserves the teacher’s behavior.

A key practical challenge is that naively comparing teacher and student outputs during training would require keeping both models in GPU memory simultaneously. We address this by running the teacher once on the calibration set and caching only its top-\(P\) token predictions at each time step, reducing memory usage significantly.

With the teacher outputs cached, we train a lightweight pruning policy that proposes, for each layer, which MLP channels to keep and which to remove (Section 4.2). Each decision produces a candidate pruned model (the student), which we evaluate by comparing its predictions to the cached teacher distribution via KL divergence (Section 4.3). This reward drives the policy toward masks that best preserve the teacher’s output distribution. After training, we select the student achieving the highest reward as the final compressed CodeLLM (Section 4.4)—the policy itself is discarded, having served its purpose as a learned search mechanism.

In the following section, we give details on Top-P logit caching, the reward function, policy design, and the final output of SecRL-Prune.

4.1 Compute Top-\(P\) Logits↩︎

Standard knowledge distillation requires running teacher and student models in parallel, which is memory-prohibitive for large CodeLLMs. We eliminate this bottleneck by precomputing and caching the teacher’s outputs before policy training begins. Furthermore, rather than storing the full \(|V|\)-way distribution at each time step, we cache only the top-\(P\) most likely tokens—prior work on logit-based distillation shows that this is often sufficient to match full-vocabulary distillation while being much more memory- and compute- efficient [20]–[22].

Concretely, at each time step \(t\), we obtain the teacher’s next-token distribution \(p_T(\cdot \mid x_{<t})\), sort tokens by probability, and retain the \(P\) most likely tokens, denoted \(S_t\). We cache the token IDs in \(S_t\) along with their teacher logits. All remaining probability mass is grouped into a single Other bucket: \[p_T(\mathrm{\small Other} \mid x_{<t}) = 1 - \sum_{v \in S_t} p_T(v \mid x_{<t}).\]

Each time step is thus represented by a compact \((P+1)\)-way target: the top-\(P\) tokens plus Other (refer to Figure 3). For the student, we evaluate \(p_S(\cdot \mid x_{<t}, m)\) on the teacher-cached top-\(P\) token set \(S_t\) and define the aggregated remainder mass as \(p_S(\mathrm{\small Other}\mid x_{<t}, m) \;=\; 1 - \sum_{v \in S_t} p_S(v \mid x_{<t}, m)\).

4.2 Policy Architecture↩︎

The goal of the policy is to learn a probability distribution over pruning masks—that is, which MLP channels to keep and which to remove across all layers. Rather than enumerating pruning masks directly, we allow the system to learn per-channel scores that define this distribution: higher scores make a channel more likely to be retained. During training, the system samples masks from this distribution, evaluates the resulting student via the KL-based reward, and uses the reward signal to update the scores toward masks that better preserve the teacher’s behaviour.

A standard RL approach would train a neural network that observes some representation of the model and outputs pruning decisions. We opt for a simpler design: we directly learn one parameter vector per transformer block, where each entry scores the importance of a corresponding MLP channel. Recall from Section 3.1 that in layer \(l\) the MLP computes an intermediate activation \(z_t^l\in\mathbb{R}^{d_{\mathrm{ff}}^l}\), where each coordinate corresponds to an MLP channel.

For each layer \(l\), we maintain a learnable vector \(\pi^l\in\mathbb{R}^{d_{\mathrm{ff}}^l}\) with one score per channel. At each policy step, we convert these scores into a probability distribution (via softmax) and sample a fixed number of channels to keep, matching the target pruning ratio. The selected channels form a binary mask \(m^l\in\{0,1\}^{d_{\mathrm{ff}}^l}\).

During a forward pass, we apply the mask after the activation and before the second linear projection (see Figure 1). That is, after computing \(z_t^l=\sigma(u_t^l)\), we gate channels by \[\begin{align} z_t^l &\leftarrow m^l \odot z_t^l,\\ y_t^l &= W_2^l\, z_t^l, \end{align}\] where \(\odot\) denotes element-wise multiplication.

If \(m_i^l = 0\), channel \(i\) contributes nothing to the MLP output (its activation is suppressed), so the corresponding structural components can be removed: the \(i\)-th row of \(W_1^l\) and the \(i\)-th column of \(W_2^l\). If \(m_i^l = 1\), the channel is preserved. Collecting \(\{\pi^l\}_{l=1}^{N}\) defines the full pruning policy across the model.

4.3 Reward Function↩︎

The reward should encourage pruning masks that preserve the teacher’s output distribution. Given a mask \(m\), let \(p_T(\cdot \mid x_{<t})\) and \(p_S(\cdot \mid x_{<t}, m)\) denote the teacher and student next-token distributions at time step \(t\). We measure their divergence using KL, but rather than summing over the full vocabulary \(V\) (Eq. 1 ), we sum over the cached Top-\(P\) set \(S_t\) and add a term for the Other bucket (Section 4.1). This yields a per-step distillation loss over a compact \((P{+}1)\) distribution: \[\begin{align} \ell_t(m) &= \sum_{v\in S_t} p_T(v\mid x_{<t})\, \log\frac{p_T(v\mid x_{<t})}{p_S(v\mid x_{<t},m)} \nonumber\\ &\quad+\; p_T(\mathrm{\small Other}\mid x_{<t})\, \log\frac{p_T(\mathrm{\small Other}\mid x_{<t})}{p_S(\mathrm{\small Other}\mid x_{<t},m)}. \label{eq:lt} \end{align}\tag{3}\]

We aggregate this loss across all time steps (Eq. 2 ) to obtain \(\mathcal{L}_{\mathrm{KD}}(m)\) for a single code prompt, then define the reward as the negative expected distillation loss over the calibration set: \[\label{eq:reward95expectation} R(m) = -\mathbb{E}_{\text{prompt} \in \mathcal{D}_{\text{cal}}}\!\left[\mathcal{L}_{\mathrm{KD}}(m)\right].\tag{4}\]

Higher reward corresponds to masks that better preserve the teacher’s distribution. We update the policy parameters (i.e., per-channel scores learned by the RL agent, discussed in Section 4.2) using REINFORCE [23], a policy-gradient algorithm that adjusts the per-channel scores based on the reward signal from sampled masks.

4.4 The output of SecRL-Prune↩︎

After the policy-training loop terminates (refer to Figure 2), we select the pruning mask \(m^\star\) that achieved the highest reward on the calibration set. We then apply this mask to the teacher model, converting it into a structurally pruned CodeLLM by retaining only the MLP channels marked active by \(m^\star\) and physically removing the corresponding rows of \(W_1^l\) and columns of \(W_2^l\) in each layer (as discussed in Section 4.2). The resulting student is a smaller, faster CodeLLM that requires no teacher cache at inference time. This pruned model (not the policy learned by the RL agent) is the sole output of SecRL-Prune and serves as the final compressed CodeLLM.

5 Experimental Setup↩︎

5.1 Models and Baselines↩︎

We evaluate SecRL-Prune on three widely used CodeLLMs: CodeLlama-7B-Instruct, CodeLlama-7B-Python [24], and Qwen2.5-Coder-7B-Instruct [25]. For all models, we use the official released checkpoints without additional finetuning.

SecRL-Prune operates exclusively on the FFN/MLP sub-layers, which contain the majority of model parameters. For compression ratio \(\rho \in \{0.10, 0.20, 0.25, 0.30\}\), we prune a \(\rho\) fraction of MLP channels in each layer while keeping attention, embeddings, and the output projection unchanged. The resulting student retains the teacher’s depth and attention configuration but has reduced FFN widths, keeping a \((1-\rho)\) fraction of FFN channels per layer.

We compare against two reference points: (i) the uncompressed teacher as an upper bound, and (ii) PruneNet [19], a recent state-of-the-art structured pruning method based on policy learning. We select PruneNet as our primary baseline because it has been shown to outperform earlier structured pruning methods, making it a competitive reference for evaluating structured pruning on CodeLLMs  [19].

5.2 Evaluation Protocol↩︎

Our primary objective is to evaluate how well SecRL-Prune preserves the code-generation and mutation capabilities of a teacher CodeLLM after structured pruning. We use the HumanEval benchmark [26], which contains Python programming problems with unit tests, and evaluate pruned student models against their corresponding teachers.

We report three families of metrics. First, pass@k [26] measures execution accuracy using the standard protocol and estimator for \(k \in \{1, 10, 100\}\). Second, var@k [8] measures diversity among correct outputs on problems where the model produces at least one correct solution. For each HumanEval task, we consider the first \(k\) generations, keep only those that pass the unit tests, and then count the number of unique correct solutions using hashes of the normalized code. While pass@k captures whether the model finds any correct solution, var@k captures how diverse its correct solutions are; the two should be interpreted together. Third, Preserved correctness (Pres.) reports the percentage of the teacher’s pass@k retained by the pruned student, averaged over \(k \in \{1, 10, 100\}\).

The calibration set used during pruning contains a total of 300 prompts, sampled from MBPP  [27] and a small collection of GitHub functions. During pruning, both teacher and student are prompted only with the natural-language description and function signature; unit tests are never used in the RL loop.

5.3 Implementation Details↩︎

All experiments are implemented in PyTorch using HuggingFace Transformers. SecRL-Prune is trained with REINFORCE using a Top-\(P\) cache size of \(P = 128\), batch size of 300, learning rate of \(3 \times 10^{-4}\), and 10 policy updates per compression ratio. Experiments run on NVIDIA A100 40 GB GPUs, and we report the mean over five random seeds for each configuration.

To quantify training cost, we measure peak GPU memory usage during RL optimization (Figure 5). Because SecRL-Prune caches the teacher once and trains with only the student in memory, it uses substantially less peak memory than approaches that require both models to remain loaded throughout training.

5.4 Real-World Malware Mutation Against VirusTotal Detectors↩︎

To evaluate whether semantics-preserving code mutation affects malware detectability, we conducted a small case study using three real-world samples from MalwareBazaar [28] (two Python and one JavaScript). For each sample, we first uploaded the original file to VirusTotal [29] and recorded the point-in-time detection score as “flagging engines / total engines.” Next, we randomly selected a single function in each sample and generated a syntactically different but semantically equivalent rewrite using a pruned CodeLLM at 20% compression (CodeLlama-7B-Instruct or Qwen2.5-Coder-7B-Instruct), leaving the rest of the file unchanged. We then uploaded the mutated variant to VirusTotal and recorded the updated detection score under the same reporting format. We report before/after detections for each sample and mutator; results are point-in-time and intended as an illustrative case study rather than a large-scale statistical evaluation.

6 Results and Discussion↩︎

We evaluate SecRL-Prune across three CodeLLMs and four compression ratios, comparing against PruneNet and the uncompressed teacher. Our results demonstrate that SecRL-Prune consistently preserves more of the teacher’s coding correctness and code mutation than the baseline, while requiring substantially less memory during training. We discuss each finding in turn, then address limitations and directions for future work.

6.1 Preserved Coding Capability↩︎

Table 1 summarizes HumanEval pass@k scores across all configurations. As expected, the uncompressed teacher attains the strongest performance, and all pruning methods degrade as compression increases. However, SecRL-Prune degrades more gracefully than the current state-of-the-art PruneNet [19] across the board.

At 10% compression, SecRL-Prune retains roughly 80–85% of the teacher’s average pass@k, compared to 45–72% for PruneNet. The gap widens as pruning becomes more aggressive: at 30% compression, SecRL-Prune roughly doubles PruneNet’s preserved correctness on Qwen2.5-Coder-7B-Instruct (63.7% vs.%) and maintains clear advantages on both CodeLlama variants. Figure 4 visualizes this trend: SecRL-Prune follows a flatter, higher preservation curve at every compression level, with the advantage most pronounced at aggressive pruning ratios.

These results indicate that KL-guided channel selection effectively identifies which FFN capacity is essential for code generation. Even at 30% compression—where absolute pass@1 drops substantially—the pruned models still solve a non-trivial fraction of HumanEval tasks, suggesting that CodeLLMs retain considerable capability even under aggressive structured pruning.

6.2 Preserved Code Mutation Capability↩︎

Beyond correctness, a mutation-capable CodeLLM must generate diverse correct variants. Table 2 reports \(\mathrm{var}@k\), which measures diversity (i.e., producing correct semantics with different syntactical structures) among correct outputs on tasks where the model produces at least one solution.

Across all three CodeLLMs, SecRL-Prune preserves \(\mathrm{var}@k\) substantially better than PruneNet, especially at higher compression ratios. On Qwen2.5-Coder-7B-Instruct, light pruning even slightly improves solution diversity (var@10 increases from 34.71% to 35.72% at 10% compression), and var@100 remains close to the dense baseline across compression levels. In contrast, PruneNet degrades more sharply: at 30% compression, SecRL-Prune retains var@100 of 7–8% on the CodeLlama variants, whereas PruneNet collapses below 3%.

These findings suggest that our channel selection retains not just the capacity to produce correct code, but also the variability needed for effective code mutation. However, \(\mathrm{var}@k\) must be interpreted alongside pass@k: since \(\mathrm{var}@k\) is computed only over solved problems, a model with low pass@k can still exhibit relatively high \(\mathrm{var}@k\) on the few tasks it solves. The two metrics together provide a more complete picture of both correctness and diversity.

6.3 Resource Efficiency and Security Implications↩︎

Figure 5 compares peak GPU memory during policy training at 20% compression. Across all three teachers, SecRL-Prune uses less than half the memory of PruneNet: peak usage drops from 33–37 GiB to 14–16 GiB, a consistent 55–60% reduction. This efficiency stems directly from our caching design: SecRL-Prune runs policy optimization with only the student in memory, whereas PruneNet requires both teacher and student to remain loaded throughout training.

From a security perspective, this lower resource footprint carries important implications. If a mutation-capable CodeLLM can be trained on a single commodity GPU and compressed to run on endpoint-class devices—exactly where many intrusions originate—it becomes more realistic to embed such models into malware or operate them on cheap, disposable infrastructure. Our results suggest that RL-based structured pruning not only preserves mutation capability, but may also lower the barrier to obtaining and deploying specialized code-mutation engines. This dual-use concern motivates continued research into detection and defense mechanisms alongside compression techniques.

6.4 Real-World Impact: Malware Detection Evasion↩︎

The metrics reported thus far—pass@k and \(\mathrm{var}@k\)—measure correctness and diversity on programming benchmarks, but they do not directly demonstrate whether pruned CodeLLMs retain practical utility for adversarial tasks. To bridge this gap, we evaluate whether mutations generated by our compressed models can evade real-world malware detection systems.

Figure 6 illustrates our experimental setup. We selected three malware samples—two written in Python and one in JavaScript—to test generalization across languages. For each sample, we used our pruned CodeLLMs (at 20% compression) to generate a single semantics-preserving mutation of one function, then submitted both the original and mutated versions to VirusTotal [29], which aggregates verdicts from over 60 antivirus and anti-malware engines. We compare the detection rates before and after mutation to quantify evasion effectiveness. Table [tab:virustotal] reports the results, and also includes a coarse mutation magnitude measure: Mutation Level (%), defined as the line-diff ratio between the mutated file and the original file (i.e., the percentage of lines added, removed, or modified relative to the original file). In our experiments, we also verified that the mutated samples remained executable and functionally correct.

The results are striking. For the first Python sample, detections drop from 28/63 to 12/63 (mutated using pruned CodeLlama) and 9/63 (mutated using pruned Qwen)—a 57–68% reduction in detection rate from a single function rewrite. The second Python sample shows an even larger effect, falling from 25/63 to 7/63 and 6/63 (72–76% reduction). Most dramatic is the JavaScript sample, where detections plummet from 9/62 to just 1/62 for both mutators—an 88% reduction. Because files are small and the mutated function can occupy a large fraction of total lines, this coarse line-diff ratio can appear high even when only one function is rewritten.

These findings carry two important implications. First, they validate that our pruned CodeLLMs retain meaningful mutation capability beyond what benchmark metrics alone can capture: a model compressed by 20% can still generate rewrites (i.e., mutate) that substantially degrade signature- and pattern-based detection. Second, the non-zero residual detections indicate that some engines employ more robust features beyond superficial syntax, suggesting a path forward for defenders. Overall, this experiment demonstrates that the security concerns motivating our work are not hypothetical—compressed CodeLLMs pose a realistic threat to static malware defenses.

10pt

7 Conclusion and Future Work↩︎

This paper studied an emerging security risk: whether CodeLLMs can be structurally compressed while retaining the functionality-preserving code mutation behavior that enables rapid generation of malware variants and evasion of static defenses. We introduced SecRL-Prune, an RL-based structured pruning framework that treats channel selection as a learned search problem using a KL-guided teacher–student reward, together with a Top-\(P\) caching mechanism that reduces peak GPU memory by over 50% compared to prior methods.

Across three real-world CodeLLMs and four compression ratios, SecRL-Prune consistently preserved more execution-level correctness (pass@k) and mutation diversity (\(\mathrm{var}@k\)) than a strong policy-learning baseline. These results show that code-mutation capability can survive significant structured compression, increasing the feasibility of compact mutation-focused models while helping defenders assess the risk posed by miniaturized mutation engines.

Several limitations point toward future work: the reward is defined over a calibration distribution and truncated Top-\(P\) support, which may under-emphasize rare or long-tail behaviors; \(\mathrm{var}@k\) captures diversity only among correct outputs; and we prune only FFN channels while leaving attention untouched. Future work should explore broader calibration data, alternative diversity metrics, joint pruning of FFN and attention components, and staged prune-and-recover pipelines with intermediate fine-tuning.

To our knowledge, this is the first structured pruning framework tailored specifically to CodeLLMs. While the pruned models remain meaningfully capable, a noticeable gap to the dense teacher persists, motivating broader validation across additional models, benchmarks, and programming languages.

References↩︎