1 Introduction↩︎

Fine-tuning large language models (LLMs) [1]–[4] has become a standard technique for adapting pretrained models to downstream tasks. However, as LLMs scale to billions of parameters, conventional first-order fine-tuning becomes increasingly memory-intensive due to backpropagation, activation storage, and optimizer states. Zeroth-order (ZO) optimization provides a promising forward-only alternative, estimating update directions using only loss evaluations and avoiding backpropagation [5]. Despite its memory advantage, ZO fine-tuning suffers from high gradient-estimation variance when perturbations are sampled in the full parameter space. MeZO [5] showed that in-place ZO-SGD can fine-tune very large models with inference-level memory, but later studies identified slow convergence and instability as major limitations.

While ZO optimization offers an appealing alternative, matching FO optimization in terms of convergence and accuracy is still challenging. The core issue is that gradients estimated from function queries often have high variance, and this variance worsens in higher-dimensional problems [6]–[8]. Subsequent work has improved ZO fine-tuning through better optimizers and more structured perturbations, including ZO-Adam and momentum variants [9], ZO-Muon [10], curvature-aware preconditioning [11], low-rank perturbations [12]–[14], and sparse perturbations [15]. A particularly effective direction is low-rank or subspace-based ZO perturbation, which reduces the perturbation dimension to improve estimator stability. LOZO [13] and SubZero [12] restrict perturbations to low-rank matrices to reduce variance. AGZO [14] further incorporates activation information into low-rank ZO perturbations, showing that activation-aware directions can provide stronger update signals. ZO-Act uses activation information in a different way. Instead of only guiding the construction of perturbation directions, it uses activations to define a fixed low-dimensional parameterization. Specifically, each adapted weight update is represented by a frozen activation-informed basis and a small trainable coefficient matrix. This turns ZO fine-tuning into explicit subspace optimization, reducing the perturbation dimension, enabling standard momentum-based optimizers such as Adam, avoiding full-weight perturbation materialization, and naturally supporting quantized backbones by keeping the original weights frozen.

We therefore propose ZO-Act, a one-shot activation-informed ZO fine-tuning method. For each adapted linear layer, ZO-Act computes a low-rank basis from input activations using a calibration batch, freezes this basis throughout fine-tuning, and optimizes only the corresponding coefficient matrix. This one-shot fixed-parameterization design provides a simple forward-only alternative to perturbation-based low-rank ZO methods while preserving the memory advantages needed for large and quantized LLM adaptation.

• We propose ZO-Act, a one-shot activation-informed ZO fine-tuning method that uses input activations to define a fixed low-dimensional subspace. ZO-Act freezes the activation basis and optimizes only lightweight coefficient matrices, turning full-weight perturbation into explicit coefficient-space optimization. This reduces the effective perturbation dimension, exposes explicit trainable variables compatible with standard momentum-based optimizers such as Adam, and naturally supports quantized LLM fine-tuning by keeping the low-bit weights frozen.

• We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space and show that, by perturbing the \(k\)-dimensional coefficients instead of the \(d\)-dimensional weights, it reduces both the variance-dependent convergence term and the finite-difference error of the ZO estimator. The analysis also makes the resulting trade-off explicit: ZO-Act obtains lower-variance estimation at the cost of a controlled subspace approximation bias, which is mitigated by the low-rank structures of LLM activations and gradients.

• We demonstrate that ZO-Act consistently improves over strong ZO baselines on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B across language understanding, question answering, and commonsense reasoning tasks, showing its effectiveness for both full-precision and quantized forward-only fine-tuning.

2 Related Work↩︎

2.0.0.1 Zeroth-order fine-tuning.

MeZO [5] adapts simultaneous perturbation stochastic approximation to language model fine-tuning by estimating an update direction from two forward losses evaluated under opposite random perturbations. Its in-place implementation avoids storing explicit perturbation vectors and therefore substantially reduces training memory. Nevertheless, MeZO samples perturbations in the full parameter space, so the resulting estimator remains highly noisy in billion-dimensional models.

A number of recent works improve this basic ZO fine-tuning pipeline through optimizer design, curvature information, and structured parameter selection. Zhang et al. [9] provide a systematic study of ZO variants for LLM adaptation, including momentum, Adam-style updates, conservative updates, block-wise descent, and hybrid ZO-FO training. HiZOO [11] estimates diagonal Hessian information and uses it to precondition ZO updates, addressing the heterogeneous curvature of LLM loss landscapes. ZO-Muon [10] combines matrix-structured ZO updates with Muon-style optimization to improve the stability and effectiveness of LLM fine-tuning. These methods demonstrate that reducing estimator noise and improving update geometry are crucial for making ZO fine-tuning competitive.

2.0.0.2 Structured ZO perturbations.

Another closely related line of work reduces the perturbation dimension by imposing structured perturbations. S-MeZO [15] reduces the update dimension by selecting a sparse set of sensitive parameters. LOZO [13] and SubZero [12] perform ZO optimization in randomly generated low-dimensional subspaces. ZO-Muon [10] further exploits low-rank structure by combining subspace gradient orthogonalization with Muon-style matrix updates, improving the stability and effectiveness of ZO fine-tuning. AGZO [14] incorporates activation information into the construction of low-rank ZO perturbations, showing that activation-aware directions can provide more informative update signals than purely random perturbations. ZO-Act is most closely related to this subspace-based line of work, but differs in how the subspace is used. Existing methods primarily use low-dimensional subspaces to construct more effective perturbation directions, whereas ZO-Act uses input activations to define a fixed update parameterization. The activation-informed basis is computed once and kept frozen, and fine-tuning is performed by optimizing only the lightweight coefficient matrices.

3 Methodology↩︎

Motivation. Consider a standard linear layer with input dimension \(m\) and output dimension \(n\), \(\mathbf{Y} = \mathbf{X}\mathbf{W},\) where \(\mathbf{X}\in \mathbb{R}^{b\times m}\), \(\mathbf{W}\in \mathbb{R}^{m\times n}\), and \(\mathbf{Y}\in \mathbb{R}^{b\times n}\). Let \(\mathbf{g}_{\mathbf{Y}}\in \mathbb{R}^{b\times n}\) denote the gradient of the final loss with respect to the layer output \(\mathbf{Y}\). Then the gradient with respect to the weight matrix \(\mathbf{W}\) is \(\mathbf{g}_{\mathbf{W}} = \mathbf{X}^\top \mathbf{g}_{\mathbf{Y}}.\) Let \(\mathbf{X} = \mathbf{U}\mathbf{D}\mathbf{V}^\top\) be the thin SVD of \(\mathbf{X}\), where the diagonal entries of \(\mathbf{D}\) are the singular values in descending order. Denote the top-\(r\) components by \[\mathbf{U}_r := \mathbf{U}_{:,:r}, \quad \mathbf{V}_r := \mathbf{V}_{:,:r}, \quad \mathbf{D}_r := \mathbf{D}_{:r,:r}.\] The weight gradient can be written as \[\mathbf{g}_{\mathbf{W}} = \mathbf{V}\mathbf{D}\mathbf{U}^\top \mathbf{g}_{\mathbf{Y}}.\] If the singular values of \(\mathbf{X}\) decay rapidly, then \(\mathbf{g}_{\mathbf{W}}\) can be well approximated by its rank-\(r\) truncation: \[\mathbf{g}_{\mathbf{W}} \approx \mathbf{V}_r\mathbf{D}_r\mathbf{U}_r^\top \mathbf{g}_{\mathbf{Y}}.\] This suggests that the dominant components of the weight gradient lie in the subspace spanned by the top right singular vectors of the input activation matrix, namely \(\mathbf{V}_r\).

The same intuition can also be understood from the perspective of weight perturbation. Suppose we perturb the weight matrix along the activation-informed subspace: \(\Delta \mathbf{W} = \mathbf{V}_r \mathbf{R}, \qquad \mathbf{R}\sim \mathcal{N}(0, I).\) Then the induced perturbation on the layer output is \[\Delta \mathbf{Y} = \mathbf{X}\Delta \mathbf{W} = \mathbf{U}\mathbf{D}\mathbf{V}^\top \mathbf{V}_r \mathbf{R} = \mathbf{U}_r\mathbf{D}_r \mathbf{R}.\] Thus, perturbing \(\mathbf{W}\) within the span of \(\mathbf{V}_r\) directly targets the dominant activation directions of the layer. In contrast, perturbations outside this subspace are largely suppressed by the small singular values of \(\mathbf{X}\) and have limited effect on the layer output. Therefore, restricting zeroth-order perturbations to the \(\mathbf{V}_r\) subspace preserves the most effective perturbation directions while substantially reducing the perturbation dimension.

ZO-Act. Motivated by the above observation, we restrict zeroth-order perturbations to the dominant right singular subspace of the input activations. For each linear layer, we first compute a fixed activation-informed basis from a calibration batch, and then perform zeroth-order optimization only over a low-dimensional coefficient matrix.

Consider a linear layer with pretrained weight \(\mathbf{W}\in\mathbb{R}^{m\times n}\). Let \(\mathbf{V}_r\in\mathbb{R}^{m\times r}\) be the top-\(r\) right singular vectors of the input activation matrix \(\mathbf{X}\). We parameterize the weight update as \[\Delta \mathbf{W} = \mathbf{V}_r \mathbf{B},\] where \(\mathbf{B}\in\mathbb{R}^{r\times n}\) is the only trainable parameter for this layer. The effective weight is therefore \[\mathbf{W}_{\mathrm{eff}} = \mathbf{W} + \mathbf{V}_r \mathbf{B}.\] The pretrained weight \(\mathbf{W}\) and the activation basis \(\mathbf{V}_r\) are both frozen throughout training, while \(\mathbf{B}\) is initialized as zero.

At each training step, we sample a random perturbation in the low-dimensional coefficient space: \[\mathbf{Z}\sim \mathcal{N}(0,I), \qquad \mathbf{Z}\in\mathbb{R}^{r\times n}.\] The corresponding weight-space perturbation is \[\Delta \mathbf{W}_{\mathrm{ZO}} = \mathbf{V}_r \mathbf{Z}.\] Given a perturbation magnitude \(\mu>0\), the forward-perturbed effective weight is \[\mathbf{W}_{\mathrm{eff}}^{+} = \mathbf{W} + \mathbf{V}_r(\mathbf{B}+\mu \mathbf{Z}).\] For an input activation \(\mathbf{X}\), the perturbed forward pass can be written as \[\mathbf{Y}^{+} = \mathbf{X}\mathbf{W} + (\mathbf{X}\mathbf{V}_r)(\mathbf{B}+\mu\mathbf{Z}).\]

Let \(\mathcal{L}^{+}\) denote the loss from the forward-perturbed model and let \(\mathcal{L}\) denote the unperturbed loss on the same mini-batch. The forward-difference zeroth-order estimator for \(\mathbf{B}\) is \[\widehat{\mathbf{g}}_{\mathbf{B}} = \frac{\mathcal{L}^{+}-\mathcal{L}}{\mu}\mathbf{Z}.\] With \(q\) independent perturbation directions, we average the estimates: \[\widehat{\mathbf{g}}_{\mathbf{B}} = \frac{1}{q} \sum_{j=1}^{q} \frac{\mathcal{L}^{(j,+)}-\mathcal{L}}{\mu} \mathbf{Z}^{(j)}.\] We then update \(\mathbf{B}\) using a first-order optimizer such as Adam with the estimated gradient \(\widehat{\mathbf{g}}_{\mathbf{B}}\). Since the perturbation is sampled in \(\mathbb{R}^{r\times n}\) rather than \(\mathbb{R}^{m\times n}\), the effective perturbation dimension is reduced from \(mn\) to \(rn\), where \(r\ll m\). The whole ZO-Act algorithm is summarized in Algorithm 1 and Algorithm 2.

4 Theoretical Analysis↩︎

We analyze ZO-Act as zeroth-order optimization over a restricted coefficient space. Let \(F(\theta)\) denote the fine-tuning objective, where \(\theta\in\mathbb{R}^{d}\) collects the weight parameters of all adapted linear layers. As described in Section 3, ZO-Act freezes the pretrained weights and the activation-informed bases, and optimizes only the coefficient matrices \(\{\mathbf{B}_\ell\}_{\ell=1}^{L}\). Let \(\beta\in\mathbb{R}^{k}\) collect all trainable coefficient matrices, where \(k=\sum_{\ell=1}^{L} r n_\ell\). Equivalently, there exists a fixed activation-informed embedding \(U\) such that \(\theta(\beta)=\theta_0+U\beta\), where \(\theta_0\) denotes the frozen pretrained weights. Thus, ZO-Act optimizes the restricted objective \(\phi(\beta)=F(\theta_0+U\beta)\), rather than the full objective \(F(\theta)\) over all weight entries.

This restricted formulation creates a variance–bias trade-off. Because ZO-Act samples perturbations only in the \(k\)-dimensional coefficient space, the variance-dependent term in the ZO estimator is governed by \(k\), rather than the full weight dimension \(d\). At the same time, restricting updates to the range of \(U\) introduces a subspace approximation bias: ZO-Act can only reduce the component of the gradient captured by the activation-informed update subspace.

We first examine the variance of the estimated gradient in the coefficient space. Let \(g_t=\nabla\phi(\beta_t)\). For a single Gaussian direction \(z_t\sim\mathcal{N}(0,I_k)\), the one-sided estimator is \[\widehat g_t = \frac{\phi(\beta_t+\mu z_t)-\phi(\beta_t)}{\mu}z_t .\] For small \(\mu\), the leading term is \(a_t=\langle g_t,z_t\rangle z_t .\) This term is unbiased, since \(\mathbb{E}[a_t]=g_t\). Its second moment is \[\mathbb{E}\|a_t\|^2 = \mathbb{E}[\langle g_t,z_t\rangle^2\|z_t\|^2] = (k+2)\|g_t\|^2 .\] Therefore, \[\mathbb{E}\|a_t-g_t\|^2 = \mathbb{E}\|a_t\|^2-\|g_t\|^2 = (k+1)\|g_t\|^2 .\] With \(q\) independent perturbation directions, define \(\bar a_t=q^{-1}\sum_{j=1}^q a_t^{(j)}\). Then \[\mathbb{E}[\bar a_t]=g_t, \qquad \mathbb{E}\|\bar a_t-g_t\|^2 = \frac{k+1}{q}\|g_t\|^2 .\] Thus, the leading variance of the ZO gradient estimator scales linearly with the perturbation dimension and decreases as \(1/q\) with multi-query averaging.

We now analyze the convergence of ZO-Act on the objective \(\phi(\beta)=F(\theta_0+U\beta)\). The analysis follows the standard analysis framework for gradient descent with a zeroth-order gradient estimator.

The proof is shown in Appendix 9.1. Theorem 1 highlights the main optimization benefit of ZO-Act. Full-weight ZO corresponds to the special case \(k=d\), where \(d=\sum_{\ell=1}^{L}m_\ell n_\ell\) is the full adapted weight dimension. ZO-Act instead perturbs only the coefficient matrices, giving \(k=\sum_{\ell=1}^{L}r n_\ell\). Since \(r\ll m_\ell\), we have \(k\ll d\), so the variance-dependent term that governs the convergence of \(\phi\) is substantially reduced, yielding a lower-variance and more stable ZO estimator.

The second term is the finite-difference error induced by the one-sided estimator with nonzero perturbation scale \(\mu\). Under smoothness-only assumptions, this term scales as \(O(L_F^2\mu^2 k^3)\). For full-weight ZO, the same term scales as \(O(L_F^2\mu^2 d^3)\). Since ZO-Act uses the much smaller coefficient dimension \(k\ll d\), it can dramatically reduce this finite-difference noise. Thus, the low-dimensional activation-informed parameterization reduces both the leading variance term and the higher-order finite-difference error, making the ZO estimator substantially more stable.

This improvement comes with a subspace approximation bias. ZO-Act optimizes the restricted objective \(\phi(\beta)=F(\theta_0+U\beta)\), rather than directly optimizing \(F(\theta)\) over the full adapted weight space. Therefore, ZO-Act can only reduce the component of the full gradient that lies in the activation-informed update subspace. Let \(P_U=UU^\top\) be the projector onto this subspace. The theorem controls the projected component through \(\nabla\phi(\beta_t)=U^\top\nabla F(\theta_t)\), while the residual term \(\|(I-P_U)\nabla F(\theta_t)\|^2\) measures the bias introduced by restricting updates to the activation-informed subspace. Thus, ZO-Act trades a controlled subspace bias for a much lower-variance zeroth-order estimator. In large language model fine-tuning, this subspace bias is often mitigated by the low-rank structure of adaptation. Effective task-specific updates are known to be highly structured [16] and are often well captured by low-dimensional parameterizations such as LoRA [2], [3]. ZO-Act leverages this phenomenon in a data-informed way by choosing the subspace from dominant input activation directions, where layerwise gradients are expected to concentrate; we verify this gradient concentration empirically in Section 5.3. The method can thus preserve important update directions while greatly reducing ZO estimation noise.

To further understand the subspace bias, consider a least-squares layerwise approximation. Let \(\mathbf{R}=\mathbf{Y}-\mathbf{X}\mathbf{W}_0\) be the residual target. The full update problem minimizes \(\|\mathbf{X}\Delta\mathbf{W}-\mathbf{R}\|_F^2\), while ZO-Act restricts the update to \(\Delta\mathbf{W}=\mathbf{V}\mathbf{B}\) and minimizes \(\|\mathbf{X}\mathbf{V}\mathbf{B}-\mathbf{R}\|_F^2\). If \(P_X\) and \(P_{XV}\) denote the projectors onto \(\operatorname{col}(\mathbf{X})\) and \(\operatorname{col}(\mathbf{X}\mathbf{V})\), respectively, then the restricted optimality gap is \[\phi_V^\star-f^\star = \|(P_X-P_{XV})\mathbf{R}\|_F^2 .\] If \(\mathbf{X}=\sum_i\sigma_i u_i v_i^\top\) and \(\mathbf{V}=\mathbf{V}_r\) contains the top \(r\) right singular vectors of \(\mathbf{X}\), then \(\operatorname{col}(\mathbf{X}\mathbf{V}_r)=\operatorname{span}\{u_1,\ldots,u_r\}\). Therefore, \[\phi_V^\star-f^\star = \sum_{i>r} \|u_i^\top \mathbf{R}\|_2^2 .\] Thus, the approximation bias is small when the residual target \(\mathbf{R}\) has little energy along the discarded left singular directions of the activation matrix. When the activation spectrum is concentrated, the dominant left singular directions often capture the most influential output variations induced by input activations, making this residual energy small in practice. This supports the use of activation-informed low-rank subspaces: a small rank can preserve the main update directions while substantially reducing the variance of zeroth-order perturbations.

5 Experiments↩︎

We evaluate ZO-Act on forward-only fine-tuning tasks for large language models. Our experiments cover three settings: language understanding and question answering, commonsense reasoning, and quantized LLM fine-tuning. For full-precision models, we consider Llama-3-8B [17] and OPT-13B [18]. For quantized fine-tuning, we evaluate an INT4 quantized Llama-3-8B model, denoted as Llama-3-8B-w4, which is quantized by MagR [19].

To ensure a fair comparison with existing ZO baselines, we match or slightly reduce the total number of forward passes used by ZO-Act compared with the corresponding baseline methods whenever possible. Unless otherwise specified, ZO-Act uses Gaussian perturbations with \(q=8\) perturbation directions per update step and perturbation magnitude \(\mu=10^{-3}\). We use rank \(r=1\) for full-precision models and rank \(r=32\) for the INT4 quantized model. The rank-one subspace is sufficient for full-precision fine-tuning and gives the lowest perturbation dimension, while the quantized model benefits from a moderately larger subspace to compensate for the reduced capacity. We use the forward-difference estimator and optimize the low-rank coefficient matrices with Adam. Full details are shown in Appendix 10.

5.1 Language Understanding↩︎

2.8pt

5.1.0.1 Models and Datasets.

Following the setting of [10], ZO-Act is evaluated on standard language understanding and question answering tasks. The evaluation includes six benchmarks: SST-2, RTE, CB, BoolQ, WiC, and SQuAD in SuperGLUE [20]. F1 score is reported for SQuAD, and accuracy is reported for the remaining tasks. Experiments are conducted on two full-precision LLMs, Llama-3-8B and OPT-13B, as well as an INT4 quantized Llama-3-8B model.

5.1.0.2 Baselines.

We compare ZO-Act with representative zeroth-order fine-tuning methods, including MeZO [5], S-MeZO [15], HiZOO [11], LOZO [13], SubZero [12], Subspace-MeZO [10], AGZO [14], and ZO-Muon [10]. The baseline results, except for AGZO, are taken from the ZO-Muon paper. The AGZO results are obtained using our implementation.

5.1.0.3 Results.

Table 1 reports the language understanding and question answering results. ZO-Act with Adam achieves strong performance across both model families. On Llama-3-8B, ZO-Act obtains the best full-precision ZO result on SST-2, RTE, BoolQ, and WiC, and achieves especially large gains on RTE and CB compared with prior ZO baselines. For example, compared with ZO-Muon, ZO-Act improves RTE from 81.2 to 87.0 and CB from 69.6 to 89.3. On OPT-13B, ZO-Act also achieves the best full-precision ZO result on SST-2, RTE, CB, and BoolQ, and remains competitive on WiC and SQuAD. These results show that the activation-informed space provides effective update directions and leads to robust forward-only fine-tuning performance across different model architectures.

ZO-Act also remains effective when applied to the INT4 quantized Llama-3-8B model. Although the INT4 results are generally lower than the full-precision ZO-Act results, the performance remains comparable to or stronger than many full-precision ZO baselines. These results suggest that ZO-Act can preserve strong task adaptation ability even when the pretrained model is quantized, supporting its suitability for memory-constrained fine-tuning.

Finally, comparing ZO-Act with Adam and SGD highlights the importance of optimizing the explicit coefficient matrices with a momentum-based optimizer. Replacing Adam with SGD leads to noticeably worse performance on most tasks, especially on OPT-13B. This confirms one practical advantage of the ZO-Act parameterization: by exposing lightweight trainable coefficient matrices, it allows standard optimizers such as Adam to be directly applied to ZO gradient estimates.

5.2 Commonsense Reasoning↩︎

4.2pt

5.2.0.1 Models and Datasets.

We further evaluate ZO-Act on full-data commonsense reasoning tasks. The benchmark consists of eight tasks: ARC-Challenge, ARC-Easy [21], BoolQ [22], HellaSwag [23], OpenBookQA [24], PIQA [25], SIQA [26], and WinoGrande [27]. We fine-tune Llama-3-8B on the combined training data and report accuracy on each evaluation task individually, as well as the average accuracy. All tasks are evaluated under an open-ended generation protocol with answer-match scoring.

5.2.0.2 Baselines.

We compare ZO-Act with representative ZO baselines, including LOZO, AGZO, SubZero, HiZOO, and ZO-Muon. ZO-Act and ZO-Muon are trained for 20k steps using forward differences with \(q=8\), corresponding to 9 forward passes per step, including one unperturbed forward pass and eight perturbed forward passes. The remaining baselines are trained for 100k steps with 2 forward passes per step. Thus, ZO-Act uses no more total forward evaluations than these baselines.

5.2.0.3 Results.

Table 2 reports the commonsense reasoning results on Llama-3-8B. ZO-Act achieves the best average performance among all evaluated ZO methods and obtains the top result on nearly all sub-tasks. Compared with prior baselines, ZO-Act shows consistent improvements across the benchmark, indicating that the activation-informed coefficient space provides effective update directions beyond the tasks considered in Table 1. When ZO-Act is applied to the INT4 quantized model, it remains competitive with strong full-precision ZO baselines, despite some degradation relative to the full-precision ZO-Act model. This suggests that the same ZO-Act algorithm can be effectively used on quantized models while retaining strong adaptation performance.

5.3 Stability and Gradient Alignment of Activation Subspaces↩︎

ZO-Act computes the activation-informed basis once at initialization and keeps it fixed during fine-tuning. To validate this design choice, we perform a diagnostic study using a separate full first-order fine-tuning run of Llama-3-8B on RTE for three epochs. We inspect the query projection layers at depths 0, 15, and 31.

For each layer \(\ell\), let \(\mathbf{V}_{\ell,0}\in\mathbb{R}^{m_\ell\times r}\) denote the top-\(r\) right singular vectors of the input activation matrix at initialization, and let \(\mathbf{V}_{\ell,t}\) denote the corresponding activation basis recomputed at checkpoint \(t\). We measure the stability of the activation subspace by \(\frac{\|\mathbf{V}_{\ell,t}^{\top}\mathbf{V}_{\ell,0}\|_F^2}{r}.\) We also measure the fraction of full-gradient energy captured by the fixed initialization subspace: \(\frac{\|\mathbf{V}_{\ell,0}^T\mathbf{G}_{\ell,t}\|_F^2}{\|\mathbf{G}_{\ell,t}\|_F^2},\) where \(\mathbf{G}_{\ell,t}\) is the full weight gradient of layer \(\ell\) at checkpoint \(t\).

Figure 3 (a) shows the results at the final checkpoint. The activation subspace remains stable across all inspected q-projection layers. For \(r=1\), the overlap is close to one at all three depths, indicating that the dominant activation direction changes little during fine-tuning. For \(r=32\), the overlap is slightly lower but still remains high.

The fixed activation subspace also captures meaningful gradient energy. The results are shown in Figure 3 (b). Even with \(r=1\), the initialization subspace captures a clear portion of the full-gradient energy, showing that the dominant activation direction is already gradient-aligned. This is important for ZO-Act, since the rank-one setting uses the smallest perturbation dimension and therefore benefits most from variance reduction. When the rank is increased to \(r=32\), the captured gradient energy further increases, indicating that additional activation directions provide broader gradient coverage. Together, these results demonstrate that a very small activation subspace can already identify useful update directions.

5.4 Effect of Subspace Rank↩︎

We further study the effect of the activation subspace rank on RTE. Table 3 reports the test accuracy of ZO-Act with different ranks on Llama-3-8B and OPT-13B. Interestingly, the rank-one subspace achieves the best performance on both models. Increasing the rank does not necessarily improve the final test accuracy. These results suggest that the dominant activation direction already captures a highly effective update subspace for ZO fine-tuning. The effect of rank is governed by two opposing forces: larger ranks improve subspace coverage and expressiveness, but also increase the coefficient-space perturbation dimension and hence the variance and difficulty of zeroth-order optimization. This explains why the accuracy is non-monotonic in the rank rather than uniformly decreasing: very small ranks may underfit the update subspace, intermediate ranks can be variance-dominated, and the adaptive scaling of Adam partially compensates for the added variance at larger ranks. The rank-one subspace nonetheless attains the best accuracy on both models while using the smallest perturbation dimension. In our main experiments, we therefore use a small activation-informed rank, which provides strong performance while preserving the variance-reduction advantage of ZO-Act.

5.5 Efficiency Comparison.↩︎

7.0pt

We compare the practical efficiency of ZO-Act with representative ZO fine-tuning baselines in Table 4. The comparison is conducted on the RTE task using Llama-3-8B, and all methods are evaluated under the same total forward-query budget of 4500 forward passes.

ZO-Act achieves the fastest runtime among the evaluated methods while matching the lowest memory usage. Its memory is the same as LOZO, SubZero, and AGZO, and lower than ZO-Muon and HiZOO. The one-shot subspace initialization adds negligible overhead, taking only 1.5 seconds before fine-tuning. This indicates that the activation-informed coefficient-space design introduces little practical overhead while retaining the memory efficiency of forward-only fine-tuning. Moreover, when ZO-Act is applied to the INT4 quantized model, the memory usage is further reduced to 5.8GB, demonstrating its advantage for low-bit fine-tuning.

6 Conclusion↩︎

We proposed ZO-Act, a one-shot activation-informed zeroth-order fine-tuning method for large language models. ZO-Act uses input activations to construct a fixed low-dimensional subspace and optimizes only lightweight coefficient matrices within this subspace. By reducing the perturbation dimension, this design lowers the variance of ZO gradient estimation, enables standard momentum-based optimizers, improves convergence, and naturally supports quantized LLM fine-tuning.

Our theoretical analysis shows that ZO-Act improves the stability of zeroth-order optimization by reducing the variance-dependent convergence term and the finite-difference error. It also clarifies the main trade-off introduced by the activation-informed subspace: ZO-Act gains more stable coefficient-space optimization while relying on the selected subspace to capture useful update directions. Empirically, ZO-Act achieves strong performance on Llama-3-8B, OPT-13B, and INT4 Llama-3-8B across language understanding, question answering, and commonsense reasoning tasks, consistently improving over strong ZO fine-tuning baselines. These results suggest that activation-informed coefficient-space optimization is an effective and practical approach for forward-only fine-tuning of both full-precision and quantized LLMs.

7 Limitations↩︎

Although ZO-Act substantially improves over existing ZO fine-tuning baselines, it still does not fully close the gap to first-order fine-tuning. In particular, first-order methods can often achieve stronger final performance because they use exact backpropagation gradients rather than noisy zeroth-order estimates. ZO-Act reduces the variance of ZO estimation by restricting perturbations to an activation-informed coefficient space, but the updates are still based only on forward loss evaluations and therefore remain less informative than full gradients.

ZO-Act also remains slower than standard first-order fine-tuning in terms of wall-clock training time. Each ZO update requires multiple forward evaluations to estimate a gradient direction, while first-order methods obtain gradients through a single forward-backward pass. As a result, ZO-Act is most useful in settings where backpropagation is memory-prohibitive, unavailable, or difficult to support, such as inference-oriented or quantized deployment environments. Improving the runtime efficiency and further narrowing the final performance gap between ZO and FO fine-tuning remain important directions for future work.

8 Ethical Considerations↩︎

This work focuses on improving the optimization efficiency of zeroth-order fine-tuning for large language models and does not introduce new ethical risks beyond those already associated with fine-tuning pretrained LLMs. All experiments are conducted on publicly available models (Llama-3-8B, OPT-13B) and standard public benchmarks (SuperGLUE and commonsense reasoning datasets), used in accordance with their respective licenses and intended research use. We do not collect any new data or involve human subjects.

ZO-Act is a general-purpose fine-tuning method and inherits the limitations and potential harms of its underlying pretrained models, including possible biases, factual errors, and harmful generations. Because ZO-Act adapts models using only forward loss evaluations, it does not mitigate or amplify these issues by design; practitioners should apply standard safety and bias evaluations before deploying any adapted model. By lowering the memory cost of fine-tuning and supporting quantized backbones, ZO-Act may broaden access to LLM adaptation on resource-constrained hardware. We view this increased accessibility as largely beneficial, but note that, as with any fine-tuning technique, it could in principle be used to adapt models toward harmful ends.

9 Proofs and Analysis↩︎

9.1 Formal Statement and Proof of Theorem 1↩︎

10 Experiment Setups↩︎

In all experiments, ZO-Act is applied to all linear layers except the embedding layers and task-specific linear heads.

10.0.0.1 Language understanding and question answering.

Following the setting of [10], ZO-Act is evaluated on six standard language understanding and question answering benchmarks: SST-2, RTE, CB, BoolQ, WiC, and SQuAD in SuperGLUE [20]. F1 score is reported for SQuAD, while accuracy is reported for all other tasks. Experiments are conducted on two full-precision LLMs, Llama-3-8B and OPT-13B, as well as an INT4 quantized Llama-3-8B model. Following prior ZO fine-tuning work [5], we randomly sample 1,000 training examples and 1,000 test examples for each task, and use the same prompts as MeZO [5].

10.0.0.2 Commonsense reasoning.

The commonsense reasoning benchmark consists of eight tasks: ARC-Challenge, ARC-Easy [21], BoolQ [22], HellaSwag [23], OpenBookQA [24], PIQA [25], SocialIQA [26], and WinoGrande [27]. ZO-Act is fine-tuned on the combined training data using Llama-3-8B, and performance is reported as accuracy on each individual task as well as the average accuracy across all tasks.

The hyperparameters are reported in Table 5, Table 6, and Table 7.

4pt

4pt

4pt

References↩︎