1 Introduction↩︎

Nowadays, fine-tuning pre-trained foundation models on downstream tasks has become a standard paradigm. However, as model sizes grow, traditional first-order optimizers such as Adam become increasingly expensive. In particular, these methods impose significant memory overhead, up to 12 times [1] more than inference. Even with parameter-efficient fine-tuning (PEFT) methods such as LoRA [2] and Prefix-Tuning [3], training can still incur nontrivial memory needs due to backpropagation and optimizer states.

To address these challenges, memory-efficient zeroth-order (MeZO) optimizer [1] has been proposed. This approach only requires two forward passes per step and achieves competitive performance to first-order methods while maintaining memory usage comparable to inference. Many subsequent methods, such as HIZOO [4], LOZO [5], MeZO-SVRG [6], ZO-AdamU [7], and ZO-DAP [8] attempt to improve upon MeZO by manually designing more sophisticated parameter-updating rules. However, these designs are often based on intuition or mathematical approximations, and still typically require extensive hyperparameter tuning beyond learning rates to perform well in practice.

We argue that prior works have largely overlooked the potential of learning to learn (L2L) techniques [9] for improving zeroth-order optimization in this setting. Unlike hand-designed optimizers, L2L provides a data-driven approach to learn effective optimization strategies. Rather than manually tuning update rules and hyperparameters, L2L leverages auxiliary neural networks that adaptively guide the optimization process. These learned optimizers typically rely on the same information accessible to conventional optimizers, such as gradient signals or their approximations. By leveraging such inputs, they often outperform manually designed counterparts in both convergence speed and final performance, as they are able to explore the loss landscape more effectively [10]. For example, learned optimizers have been shown to surpass SGD and even Adam across a variety of models and tasks [11]. Similar improvements have also been observed in zeroth-order optimization settings on small-scale models [12].

While L2L methods have shown promise on small-scale models [13], we believe their potential is even greater in the era of foundation models. In the small-model regime, different tasks typically require different models, and L2L optimizers often exhibit limited transferability across model architectures [10]. As a result, a separate optimizer must be trained for each model-task pair, leading to substantial additional costs. In contrast, a recent LLM supply chain study shows that while there are many specialized checkpoints on platforms like Huggingface, most are derivatives of a handful of core base models like Llama and Qwen [14]. Moreover, for a given LLM, the structure or properties leveraged by certain optimizers are often consistent across tasks [15]. This provides a great opportunity for L2L methods, where a learned optimizer trained once for a base LLM can be potentially reused across diverse derivative models and tasks. Toward practical adoption, if model creators were to ship a pretrained learned finetuner alongside each base model, it would unlock a memory-efficient fine-tuning path with competitive performance for downstream users.

In the context of zeroth-order optimization for LLMs, learning a perturbation distribution with non-uniform and adaptive variance scales, instead of a standard normal distribution, could be beneficial [4], [16], [17]. However, the sheer number of parameters of LLMs introduces new challenges when applying L2L as it requires differentiating through the optimization process, which demands storing a substantial number of activations for backpropagation. Moreover, naively applying coordinate-wise auxiliary networks at the LLM scale can result in prohibitive memory overhead. To address this, we draw inspiration from recent analyses of Transformer curvature [18], which suggest that Transformer Hessians often exhibit an approximately block-diagonal structure. This inspires us to adopt a block-wise parameterization and share a single perturbation variance within each parameter block. We thus propose ZO Fine-tuner, which consists of highly compact and memory-efficient per-parameter-block auxiliary networks that learn shared effective perturbation variances. As a result, the memory cost is minimal: for OPT-30B, the storage required for all auxiliary networks is less than 2MB under FP16 precision, which is negligible compared to the 60GB model itself. In the meantime, through extensive experiments, we demonstrate that our ZO Fine-tunertrained on a single dataset is highly generalizable across model derivatives and datasets, which strongly underscores the potential of the “train once, reuse widely” goal. Our contributions are summarized as follows:

• We extend the L2L framework to LLMs and show that a single learned optimizer trained on a base model can generalize across downstream tasks and derivative checkpoints.

• We propose block-wise perturbations with a shared variance per parameter block, enabling L2L at LLM scale.

• Across four models and seven datasets (28 task-model pairs), ZO Fine-tuner outperforms the strongest baseline (lower training loss) in 82.1% of the combinations, achieving an average of 2.5% improvement in accuracy with tiny memory and time overhead.

2 Related Work↩︎

Zeroth-order optimization. Zeroth-order optimization appears in a wide range of applications where either the objective function is implicit, or its gradient is impossible or too expensive to compute. For example, methods such as [19], [20] consider derivative-free distributed algorithms for non-convex multi-agent optimization. ZO-BCD [21], ZOO [22], ZO-signSGD [23], and ZO-HessAware [24] utilize zeroth-order stochastic optimization to generate black-box adversarial examples in deep learning. Beyond that, MeZO [1] firstly adapted the classical ZO-SGD method to fine-tune LLMs, while achieving comparable performance with extremely great memory reduction. Subsequently, ZO-AdaMU [7] improved ZO-SGD by incorporating momentum into its stochastic approximation process. HIZOO [4] leverages Hessian information to enhance performance in a memory-efficient manner. Other works explore structural properties of the gradient to improve MeZO, such as utilizing low-rank approximations [5], [25] or exploiting gradient sparsity [15], [26].

Learning to learn. Previous studies have investigated using neural networks to improve optimization update rules, replacing manually crafted algorithms such as Adam [27]. [28] tried to use recurrent neural networks (RNNs) to model the optimization process to learn adaptively. After that, [29] gave an overview of the idea and techniques of learning to learn; for example, they proposed to train RNNs to optimize basic convex functions. Then [9], [30]–[33] introduced a variety of sophisticated strategies to enhance the performance of optimizers in deep learning. Additionally, [34] and [35] adopted reinforcement learning (RL) policy search techniques into the L2L framework. In the context of zeroth-order optimization, [12] applied L2L techniques to enhance performance on small-scale models.

3 Method↩︎

3.1 Motivation↩︎

The foundational work in zeroth-order optimization for LLM fine-tuning, MeZO [1], simply estimates the directional derivative as the step size on a certain sampled direction by evaluating the model at two perturbed parameter points. This approach only requires two forward passes and avoids backpropagation, making it attractive for memory-constrained training. Given a model with parameters \(\theta \in \mathbb{R}^d\) and loss function \(\mathcal{L}\), MeZO estimates the gradient on a mini-batch \(\mathcal{B}_t\) as: \[\label{eq:spsa} \begin{align} \hat{g}(\theta_t;\mathcal{B}_t) &=\frac{\mathcal{L}(\theta_t+\epsilon u_t;\mathcal{B}_t) -\mathcal{L}(\theta_t-\epsilon u_t;\mathcal{B}_t)}{2\epsilon}\,u_t, \\ u_t &\sim \mathcal{N}(0, I_d), \end{align}\tag{1}\]

We argue that a fixed sampling distribution such as \(\mathcal{N}(0,I)\) is suboptimal: the quality of zeroth-order gradient estimates depends on local properties of the landscape at each step [4], [16], [17]. Therefore, through learning, L2L approach has the potential to generate perturbations \(u_t\) that are informed by such local signals and thus allocate perturbation effort more effectively. However, naive implementations of this idea can incur prohibitive memory overhead. For example, learning a separate perturbation for each individual parameter using a fully connected auxiliary network would require at least \(O(d^2)\) parameters for a model with \(d\) parameters.

We thus turn to exploit geometric structure in LLMs. Recent empirical analyses suggest that Transformer curvature often exhibits an approximately block-structured pattern, where much of the Hessian mass concentrates within natural parameter groups (e.g., embeddings, attention Q/K/V matrices, and projections) [18]. This observation motivates a coarse-grained control of perturbations: instead of learning coordinate-wise perturbations, we target per-block adaptation. We now formalize this idea and show that even a simple, block-wise adjustment of perturbation variance can improve over MeZO.

The formal version and proof of this theorem can be found in appendix 9. At a high level, this theorem states that if the Hessian exhibits a block wise structure, then learning an adaptive per-block shared variance can improve convergence over MeZO. Crucially, the per-block parameterization yields this improvement without incurring prohibitive memory cost as the number of parameter blocks is far less than the number of parameters. For instance, in LLaMA-8B, the model contains only 291 parameter blocks, despite having over 8 billion individual parameters. This result thus motivates and justifies our design of ZO Fine-tuner, a per-block variance learner. Below, we first discuss the architecture of our ZO Fine-tuner and how to finetune downstream LLMs using a given ZO Fine-tuner. Then we introduce the training scheme for ZO Fine-tuner to enable generalizations.

3.2 ZO Fine-tuner↩︎

Architecture. As we discussed in the motivation section, we design ZO Fine-tuner to dynamically generate a block diagonal variance matrix \(\Sigma_t\) corresponding to each parameter group at each optimization step via lightweight neural networks named PertNN. To incorporate all the dynamic information, PertNN takes in model parameters \(\theta_t\), previously used perturbation variances \(\Sigma_{t-1}\), and their observed loss as inputs \(\boldsymbol{\ell}_{t-1}\). Intuitively, these inputs encourage PertNN to consider the effectiveness of past updates, where perturbations that lead to sharper loss changes might indicate more informative directions. However, we notice that the model parameters are still too memory-intensive as an input feature. Therefore, we further compress the memory usage by only feeding the summarizing statistics of the model parameters \(\theta_t\) into PertNN, such as \(\operatorname{Mean}(\theta_t)\) and \(\operatorname{Var}(\theta_t)\).

Formally, the perturbation variance at each step \(t\) is generated as follows. For parameter block \(i\), \(\sigma_{t-1}^{\scriptscriptstyle(i)}\) is the previous perturbation variance, \(d_i\) is the number of parameters in this block, and \(\text{Mean}_t^{\scriptscriptstyle(i)}\) and \(\text{Var}_t^{\scriptscriptstyle(i)}\) represent the current mean and variance of the block’s parameter values. \(\omega^{\scriptscriptstyle(i)}\) denotes the learnable parameters of the auxiliary neural network assigned to block \(i\). \[\label{eq:blockwise95pert} \begin{align} \sigma_t^{(i)} &= \mathrm{PertNN}^{(i)}\left(\boldsymbol{\ell}_{t-1}, \sigma_{t-1}^{(i)}, \text{Mean}_t^{(i)}, \text{Var}_t^{(i)}; \omega^{(i)} \right), \\ \Sigma_t &= \text{diag}(\sigma_t^{(1)} I_{d_1}, \sigma_t^{(2)} I_{d_2}, \dots, \sigma_t^{(n)} I_{d_n}). \end{align}\tag{2}\] With this variance, ZO Fine-tuner then updates model parameters with \[\label{eq:update} \begin{align} \hat{g}(\theta_t;\mathcal{B}_t;\omega) &= \frac{\mathcal{L}(\theta_t+\epsilon u_t;\mathcal{B}_t) -\mathcal{L}(\theta_t-\epsilon u_t;\mathcal{B}_t)}{2\epsilon}\,u_t, \\ u_t \sim \mathcal{N}(0,&\Sigma_t\Sigma_t^\top),\quad \theta_{t+1} = \theta_t - \eta\,\hat{g}(\theta_t;\mathcal{B}_t;\omega). \end{align}\tag{3}\]

Importantly, we should note that \(\hat{g}\) is inherently a function of \(u_t\), which is a function of \(\Sigma_t\), and thus a function of the parameter of PertNN \(\omega\). To enable gradient-based training of PertNN within the L2L framework, we adopt the reparameterization trick: instead of sampling \(u_t\) directly from \(\mathcal{N}(0, \Sigma_t)\), we sample \(z_t \sim \mathcal{N}(0, I_d)\) and compute \(u_t = \Sigma_t z_t\). This makes the entire perturbation process differentiable, allowing gradients to flow back through the perturbation generation module.

Normalization. Although effective, this non-uniform variance introduced a new challenge when using ZO Fine-tuner as an optimizer. From the two-point ZO estimator, we see \[\label{eq:two95point95estimate} \begin{align} \mathbb{E}[\hat{g}(\theta_t;\mathcal{B}_t)] =&\mathbb{E}[\frac{\mathcal{L}(\theta_t+\varepsilon u_t;\mathcal{B}_t) -\mathcal{L}(\theta_t-\varepsilon u_t;\mathcal{B}_t)}{2\varepsilon}] \\ \approx & \mathbb{E}[u_tu_t^\top]\nabla\mathcal{L}(\theta_t;\mathcal{B}_t). \end{align}\tag{4}\] Therefore, when fine-tuning downstream tasks, we note that the effective learning rate became \(\eta\!\cdot\! \frac{\Vert u_t\Vert^2}{d}\) on average. This makes controlling the effective learning rate difficult, and the learned variance \(\Sigma_t\) became a confounding variable in the update size. In reality, we wish \(\Sigma_t\) to only carry information about relative block-wise variance, and we could still use a single learning rate to control the overall step size to ensure stable training. Therefore, we introduce the following normalization, which ensures the decoupling of the variance and the learning rate. We note that if \(u_t\!=\!\Sigma_t z_t\) with \(z_t\!\sim\!\mathcal{N}(0,I)\), then \(\mathbb{E}\left\lVert u_t \right\rVert^2\!=\text{tr}(\Sigma_t\Sigma_t^\top)\!=\!\left\lVert \Sigma_t \right\rVert_{\!F}^2\). We then normalize by fixing the total variance budget and let \(\left\lVert \Sigma_t \right\rVert_{\!F}^2\!=\!\left\lVert I_d \right\rVert_{\!F}^2\!=\!d\). Thus, only the relative block-wise variances are learned. In practice, this keeps \(\left\lVert u_t \right\rVert\) approximately constant (by concentration in high dimensions). For example, with our generated \(\Sigma_t\), if \(u_t\!\sim\!\mathcal{N}(0,\Sigma_t\Sigma_t^\top)\), \(\left\lVert u_t \right\rVert\) concentrates around \(\left\lVert \Sigma_t \right\rVert_{\!F}\) and we achieve the desired control over the effective learning rate.

Complete Optimization Algorithm. As summarized in Algorithm 2 and Figure 1, ZO Fine-tuner first compute the block-wise non-uniform perturbation variance \(\Sigma_t\) using the learned neural network PertNN. Then it applies normalization to control the overall magnitude of the perturbation. Finally, it uses the normalized perturbation to update the LLM following 3 . We notice this incurs minimal overhead compared to MeZO, in terms of both memory and speed. In particular, the only memory overhead compared to MeZO is the light-weight per-block PertNN, whereas the only speed overhead is the query to PertNN.

3.3 Training ZO Fine-tuner↩︎

We now turn to training ZO Fine-tuner in a L2L fashion. The key idea is to treat the model’s own finetuning trajectory as supervision. After a single update by ZO Fine-tuner, we evaluate the post-update loss and adjust PertNN so as to reduce this quantity across tasks. We next formalize this meta-objective and outline several practical choices that make training stable.

Data Source and Objective Function. First, we need a source of training data for our ZO Fine-tuner. In our setting, this data corresponds to different model states with various losses. A key insight of us is to notice that the fine-tuning process of LLMs under a first-order optimizer naturally produces a trajectory of intermediate model states, and we can directly leverage this trajectory to optimize the perturbation variance generator.

Along the first order optimization trajectory with loss function \(\mathcal{L}\), we obtain a set of model parameters \(\{\theta^{k}_{0}\}_k\). We then attempt to perform a one-step zeroth-order update using our ZO Fine-tuner with update rule 3 to get \(\theta^{k}_{1}\) and use the resulting loss as a feedback signal to assess and optimize the effectiveness of the current perturbation strategy. Specifically, at each step we aim to minimize the post-update loss \(\mathcal{L}(\theta^{k}_{1})\). As we discussed in section [sec:finetuner], the estimated gradient \(\hat{g}\) is implicitly a differentiable function of the parameters \(\omega\) of PertNN per the reparametrization trick. Therefore, we can use a gradient-based method to update ZO Fine-tuner. Formally, the objective for training ZO Fine-tuner is therefore: \[\begin{align} \label{L95ZO} \min_\omega \mathcal{L}_{\text{ZO}}(\theta^{k}_0;\omega) &:= \min_\omega \mathcal{L}\left(\theta_0^k - \eta\hat{g}(\theta_{0}^k, \omega)\right) \end{align}\tag{5}\] After the update, we move the parameters \(\theta_0^k\) along the first-order trajectory to get \(\theta_0^{k+1}\) and continue learning. As the inputs to ZO Fine-tuner are task and model-agnostic state summaries, rather than task-specific features, the learned decisions are largely invariant to differences across datasets or nearby checkpoints. As we will demonstrate in experiments, our ZO Fine-tuner trained on one single dataset can be transferred to efficiently finetune other datasets and model derivatives.

Periodic Reset of Model Parameters. During the training of our ZO Fine-tuner, a lot of data needs to be generated. However, since the optimizer is trained along the fine-tuning trajectory of a model using a first-order optimizer, the auxiliary network tends to receive inputs that are chronologically ordered. In particular, it will get more data from the low-loss region. As a result, it may lead to overfitting to the low-loss region of the parameters while learning the crucial high-loss region insufficiently. To address this issue, we introduce a periodic re-initialization mechanism. After each complete optimization cycle or when the loss has sufficiently decreased by the first-order guidance, we reset the model parameters to their original pre-finetuning state and restart the fine-tuning process. This approach introduces diversity into the input distribution by exposing the optimizer to model states from multiple phases of training.

Complete Learning to Learn Framework for ZO Fine-tuner Training. Algorithm 3 illustrates the complete L2L framework for training ZO Fine-tuner. At each training step, we sample a training dataset and a batch from this dataset to perform a one-step update to ZO Fine-tuner as described above. Moreover, we periodically reset the model parameters to mitigate the bias discussed previously. Despite the complexity of this training algorithm, we would like to emphasize that it is a one-time cost: once ZO Fine-tuneris learned, deployment reduces to Algorithm 2.

4 Experiment↩︎

Following MeZO [1], we evaluate ZO Fine-tuner with four LLMs: LLaMA-3.2-1B [36], LLaMA-3.1-8B [36], Qwen2.5-14B [37], and OPT-30B [38] using seven diverse benchmark datasets including SST-2 [39], CB [40], COPA [41], BoolQ [42], WSC [43], SQuAD [44], and DROP [45].

We compare our approach against four representative zeroth-order optimization baselines for LLM fine-tuning: HIZOO [4], LOZO [5], MeZO and MeZO-Adam [1]. Due to computational resource constraints, we replace the expensive MeZO-Adam with a more efficient variant, MeZO-AdamU [7], for models larger than LLaMA-3.2-1B. To ensure a fair comparison, we perform the same grid search over learning rates for each method and pick the best learning rate when reporting. Moreover, we used the same prompt templates as in the original MeZO for all baselines. More details can be found in Appendix 7.2.

For our ZO Fine-tuner, we train it once using algorithm 3 on the COPA dataset. This choice is mainly due to COPA’s consistently smooth loss decrease during standard fine-tuning, and its small size can enable fast training cycles. Unless otherwise noted, the ZO Fine-tuner trained on COPA is reused as is across all other tasks and models in our main experiments. Thus, all reported results provide a direct test of out-of-distribution transfer. In section 4.2, we also provide ablations regarding the choice of meta-learning dataset, and in Appendix 8.5, we further discuss multi-dataset training. Other hyperparameters and training details can be found in section 7.3 and section 8.8.

4.1 Main Results↩︎

Generalization Across Datasets. Figure 4 compares convergence across all 28 dataset-model pairs using each method’s best learning rate. We observe that ZO Fine-tuner(red) consistently reaches lower loss faster. The effect is especially clear on SST-2, CB, COPA, SQuAD, and DROP, where curves descend more steeply early on and settle at a better plateau. In addition, we report the final loss and accuracy values for all 28 combinations in Table 1. On average, ZO Fine-tuner achieves an average accuracy improvement of 2.5% over MeZO. Overall, our method outperforms the baselines in 75.0% of the task-model combinations in accuracy and 82.1% in the converged loss. These results indicate the strong generalization capability of ZO Fine-tuner, as training ZO Fine-tuner on a single COPA dataset already yields consistent gains across datasets. To show an upper bound of the improvements when full gradient is available, we provide the FO Adam baseline in appendix 8.7.

Generalization Across Model Derivatives. We further evaluate whether ZO Fine-tuner transfers from a base model to its finetuned variants. Motivated by the fact that many checkpoints are obtained by further fine-tuning a small set of base LLMs, we train ZO Fine-tuner on LLaMA-3.1-8B and then apply it as is to fine-tune the derived checkpoint Llama-3.1-8B-Instruct. As shown in Table 4, ZO Fine-tuner consistently outperforms MeZO on both evaluated datasets in terms of average training loss and final accuracy. Practically, this supports a train once, reuse across derived checkpoints workflow: model developers could ship a pretrained finetuner with each base model, enabling downstream users to efficiently fine-tune derivative checkpoints with MeZO-style near-inference memory.

Generalization to Longer-Sequence Datasets. To assess whether our method generalizes to more commonly used longer sequence datasets, we also conduct experiments on a math-finetuning task, which has a long solution and is closer to modern reasoning training. Concretely, we fine-tuned Qwen2.5-14B on MetaMathQA [46] using the ZO Fine-tuner trained on COPA for 10000 steps, and compared it with MeZO under the same setup. The results can be found in Table 5, where we report the evaluation results on GSM8K [47] and Math-500 [48]. We observe that the COPA-trained ZO Fine-tuner still outperforms MeZO on this math dataset. This result suggests that the learned optimizer is not restricted to the GLUE/SuperGLUE-style tasks, and can transfer nontrivially to a substantially different task suite. This further validates our train once, reuse widely slogan.

4.2 Ablation Studies↩︎

Learning Rate. Figure 5 further demonstrates the sensitivity of different methods to the choice of learning rate. Notably, ZO Fine-tuner often achieves comparable loss at a learning rate of \(1\!\times\!10^{-8}\) to that of baseline methods operating at \(1\!\times\! 10^{-7}\). When the learning rate further increases to \(1 \times 10^{-6}\), many baseline methods suffer from instability and fail to converge, falling short of the performance that ZO Fine-tuner achieves at \(1 \times 10^{-7}\).

Normalization & Periodic Reset. We also conduct experiments to evaluate the effectiveness of our design choices, including normalization introduced in section [sec:normalize] and periodic reset in section [sec:periodic95reset]. From table 2, it is clear that both normalization and periodic reset help ZO Fine-tuner for achieving better performance. Moreover, combining the two strategies yields the best overall loss and accuracy.

Parameter Sharing Strategy. We also evaluate the granularity of sharing in ZO Fine-tuner, comparing our block-wise scheme to a simpler layer-wise sharing baseline. As shown in Table 3, block-wise sharing consistently achieves lower final loss and higher accuracy. Importantly, this choice is theory-driven: when the Hessian is (approximately) block-diagonal, theorem 1 indicates that the natural unit for variance sharing is the Hessian block itself.

Meta-Learning Dataset. Finally, we evaluate the robustness of ZO Fine-tuner under different meta-learning datasets. Specifically, we trained additional ZO Fine-tuners for LLaMA-3.2-1B using SQuAD as the meta-training dataset and for LLaMA-3.1-8B using SST-2 as the meta-training dataset. We then applied these learned optimizers to fine-tune the corresponding models on SST-2, COPA, and SQuAD. The results can be found in Table 6. It can be observed that the trained ZO Fine-tuner remains consistently better than MeZO. This demonstrates that the performance of ZO Fine-tuner is not attributed to some special properties of the COPA dataset, but rather our effective designs.

4.3 Memory Usage and Time Efficiency Analysis↩︎

Memory Usage. The memory overhead of ZO Fine-tuner when using to finetune LLMs mainly comes from the additional memory taken by PertNN. However, the parameter number of our ZO Fine-tuner is extremely small, even negligible, compared to the LLMs. Consequently, the memory footprint of our method remains essentially identical to that of MeZO. Under equivalent experimental settings, it requires only \(1/4\) of the memory overhead incurred by Adam. For example, MeZO and ZO Fine-tuner peak at 61GB and 62GB of GPU memory when fine-tuning OPT-30B, whereas first-order Adam reaches 312GB with FP16. More details can be found in Appendix 8.3.

Time Efficiency. Similarly, the time overhead comes from the query to PertNN. However, this overhead is typically minimal. For example, when fine-tuning on DROP using LLaMA-3.2-1B on an L40S GPU with a batch size of 16, the generation of perturbation takes only 0.025 seconds, while all other operations take approximately 0.70 seconds. This means our method introduces less than 3.4% additional overhead, demonstrating time efficiency. This overhead becomes even less significant with larger models. For instance, under the same setting as LLaMA-3.1-8B, perturbation generation takes only 0.052 seconds compared to a total runtime of 3.14 seconds. More details can be found in Appendix 8.2.

5 Conclusion↩︎

We introduced ZO Fine-tuner, a learned zeroth-order optimizer that learns adaptive per-block perturbation variances for efficient LLM fine-tuning. Experiments across four models and seven datasets show that a finetuner trained once on a single dataset transfers reliably to new tasks, finetuned derivative checkpoints, and broader task suites, supporting a practical “train once, reuse widely” workflow.

Our work focuses on improving optimization quality within the canonical MeZO-style zeroth-order regime. However, we note that techniques such as LoRA, quantization, and offloading are also important for reducing the memory footprint. Although not straightforward, we believe these methods can be potentially combined with ZO Fine-tuner to improve the memory-performance tradeoff in practice further. Systematically considering and evaluating these combinations is an important direction for future work.

Acknowledgments↩︎

This work used the Delta system at the National Center for Supercomputing Applications (award OAC 2005572) through allocation CIS250462 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296.

Impact Statement↩︎

This paper presents work whose goal is to advance the field of machine learning. In particular, our work aims to improve the practicality of LLM fine-tuning, which could have both beneficial and potentially harmful downstream uses. Other than this, there are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.”

6 Discussions↩︎

6.1 Limitations and Future Work↩︎

The coordinate-wise parameterization has been shown to be effective for learning-to-learn (L2L) zeroth-order optimization on small-scale models [12]. While our current design adopts a diagonal variance matrix \(\Sigma_t\) for memory efficiency and strong empirical performance, exploring richer (e.g., non-diagonal) covariance structures is a promising direction. Such extensions could capture cross-parameter correlations, but would likely require additional techniques (e.g., structured parameterizations or low-rank factorizations) to control the associated memory and compute overhead.

More broadly, additional properties of LLM gradients and curvature may be exploitable. For example, prior work explicitly leverages low-rank structure in LLM gradients [5], [25]. An interesting future direction is to incorporate such structure into the perturbation design—either to generate more informative perturbations or to further reduce fine-tuning cost.

Finally, our experiments focus on improving optimization quality within the canonical MeZO-style zeroth-order regime. Techniques such as LoRA, quantization, and offloading are largely orthogonal system- or parameterization-level choices, and we expect them to be complementary to ZO Fine-tuner. Systematically evaluating and optimizing these combinations is an important direction for future work.

7 Implementation Details↩︎

7.1 Datasets and Models↩︎

We evaluate all optimizers on seven NLP tasks spanning multiple formats, including natural language inference, question answering, and commonsense reasoning. SST-2 [39] is a binary sentiment classification benchmark from the GLUE suite. CB [40] and COPA [41] are low-resource natural language inference tasks from SuperGLUE, requiring models to recognize textual entailment or choose causal relationships. BoolQ [42] involves answering yes/no questions given short passages. WSC [43] tests pronoun resolution in challenging coreference contexts. SQuAD [44] and DROP [45] are span-based question answering datasets that require locating answer spans in context paragraphs. For most classification tasks, we report accuracy as the evaluation metric. For SQuAD and DROP, we follow standard practice and report F1 score to better capture partial match quality.

We evaluate our optimizers on four representative large language models with diverse architectures and scales: LLaMA-3.2-1B [36], LLaMA-3.1-8B [36], Qwen2.5-14B [37], and OPT-30B [38].

7.2 Hyperparameters↩︎

We use a two-layer MLP with 64 hidden units and a tanh activation function as the auxiliary neural network for each parameter block. Table 7 presents the hyperparameter search grids used in our experiments to facilitate reproducibility. We primarily perform a grid search over three learning rate values: \(10^{-4}\), \(10^{-5}\), and \(10^{-6}\) for MeZO-Adam, and \(10^{-6}\), \(10^{-7}\), and \(10^{-8}\) for all other methods. For the WSC task, we additionally include \(3 \times 10^{-7}\), as most methods exhibit slow loss decay at \(10^{-7}\) and become unstable when using \(10^{-6}\). We run 20,000 optimization steps on LLaMA-1B and LLaMA-8B, and 10,000 steps on Qwen-14B and OPT-30B due to resource limitation. A batch size of 16 is used for all models by default, except for OPT-30B, where we reduce it to 4 due to GPU memory constraints. Also due to computational resource constraints, we replace the expensive MeZO-Adam with its more efficient variant MeZO-AdamU [7] for models larger than LLaMA-1B. As shown in the hyperparameter table, our method, together with MeZO, requires the smallest number of tunable hyperparameters among all baselines.

7.3 Learning to Learn Details↩︎

In Section 3.3, we introduced our learning to learn framework. Here, we elaborate on additional implementation details. We use a two-layer MLP with 64 hidden units and a tanh activation function as the auxiliary neural network for each parameter block. Empirically, we set \(\epsilon = 10^{-3}\), \(\eta_1 = 10^{-6}\), and \(\eta_2 = 10^{-2}\) in Algorithm 3. When the task list \(\mathcal{T}_{\mathrm{list}}\) contains only a single task, the framework reduces to single-dataset training as a special case. We find that training on a single dataset can yield competitive performance with reduced cost. In our experiments, the optimizer is trained on COPA. A comparison between single-dataset and multi-dataset training results is provided in Section 8.5. We also block certain gradient flows to reduce memory consumption during learning-to-learn. Specifically, recall that \[\begin{align} \hat{g}(\theta_t; \omega) &= \frac{\mathcal{L}(\theta_t + \epsilon u_t) - \mathcal{L}(\theta_t - \epsilon u_t)}{2\epsilon} u_t,\\ u_t &=\mathrm{PertNN}(\theta_t, \Sigma_{t-1}, \boldsymbol{\ell}_{t-1};\omega)z_t, z_t \sim \mathcal{N}(0,I_d). \end{align}\] The gradient of the ZO loss, defined as \(\mathcal{L}_{\text{ZO}}(\theta; \omega) := \mathcal{L}(\theta - \eta \hat{g}(\theta; \omega))\), propagates first to \(\hat{g}(\theta_t; \omega)\), and then further through both components used to construct it: the perturbation direction \(u_t\) and the finite-difference estimator \(\frac{\mathcal{L}(\theta_t + \epsilon u_t) - \mathcal{L}(\theta_t - \epsilon u_t)}{2\epsilon}\). To save memory, we cut off the gradient flow through the finite-difference term, which eliminates the need to back-propagate through the inner loss evaluations and store their activations. Despite this approximation, we still observe strong empirical performance.

8 Additional Experimental Results↩︎

8.1 Additional Results on Learning Rate Sensitivity↩︎

We previously presented the sensitivity of different optimization methods to the learning rate in Section 4.2. Due to space constraints, only a subset of the results was shown. Here, we provide the complete loss curves across the three benchmarks SQuAD, SST-2, and COPA using LLaMA-1B, LLaMA-8B, Qwen-14B, and OPT-30B, as shown in Figure 6 and Figure 7.

Across the grid search over learning rates \(10^{-6}\), \(10^{-7}\), and \(10^{-8}\), the ZO Fine-tuner consistently achieves superior results compared to all baselines when comparing their best-performing settings. On LLaMA-1B, LLaMA-8B, Qwen-14B and OPT-30B, our method exhibits faster convergence and achieves lower final loss, particularly under the two learning rates \(10^{-7}\) and \(10^{-8}\). Notably, ZO Fine-tuner often matches or exceeds the best performance of other methods at \(10^{-7}\), even when operating at \(10^{-8}\) on LLaMA-1B, LLaMA-8B and Qwen-14B.

When increasing the learning rate from \(10^{-8}\) to \(10^{-7}\), ZO Fine-Tuner continues to improve. In contrast, baseline methods tend to suffer from instability at higher learning rates like \(10^{-6}\) when increasing from \(10^{-7}\), especially on LLaMA-1B and LLaMA-8B. At the low end (\(10^{-8}\)), many baselines exhibit stagnation, which means their loss decreases slowly or plateaus. This suggests limited adaptivity in low-gradient regimes. These observations underscore the robustness of ZO Fine-Tuner across a wide range of learning rates and tasks, highlighting its strong default behavior even without fine-tuned hyperparameters.

8.2 Time Analysis↩︎

We further break down the runtime of each component involved in the optimizer and summarize the results in Table 8. Among these, the variance generation step is extremely lightweight. It only accounts for 3.39% of total runtime on LLaMA-1B, and less than 2.06% on larger models such as LLaMA-8B, Qwen-14B, and OPT-30B. This highlights the efficiency of our design: although we introduce an additional learned component to control perturbation variance, it imposes almost no computational overhead.

This can be easily explained by the fact that we only employ a lightweight neural network for each parameter block, specifically a two-layer MLP with just 32 hidden units. In addition, both the input and output of these networks are compressed, further reducing the computational cost.

In contrast, the dominant cost comes from loss computation, which includes forward passes for both positive and negative perturbations. This accounts for over 77%–95% of total runtime and is intrinsic to all zeroth-order optimization frameworks. Overall, our method introduces minimal additional cost while achieving adaptive and effective optimization.

8.3 Memory Analysis↩︎

Table 9 reports the peak GPU memory usage of various optimization methods across different model sizes on the SST-2 dataset. We observe that all zeroth-order (ZO) methods, including MeZO, LOZO, and HiZOO, exhibit similar memory footprints. The only notable exception is ZO-AdaMU, which incurs higher memory usage due to its additional momentum tracking. Compared to the first-order method like Adam, all ZO methods consume significantly less memory, highlighting the efficiency of ZO-based optimization. Notably, our ZO Fine-Tuner achieves comparable memory usage to other ZO baselines, indicating that it introduces no additional memory overhead beyond standard ZO designs.

8.4 Cost of Learning to Learn↩︎

We also assess the time and memory overhead incurred during the training of the ZO Fine-Tuner in table 10. In general, L2L takes approximately 2.4× the time of standard first-order fine-tuning. This is expected, as the L2L process inherently includes a full fine-tuning phase using SGD. However, importantly, this cost is incurred only once, as a single ZO Fine-Tuner trained for a given model can be reused across diverse downstream tasks, effectively amortizing the training cost.

8.5 Comparison between Single-dataset and Multi-dataset Training↩︎

We also compare the performance of ZO Fine-tuner under single-dataset and multi-dataset training settings. In the multi-dataset setting, we construct a diverse training set by selecting one representative dataset from each task type: SST-2 for sentiment analysis, CB and COPA for natural language inference, and SQuAD for question answering. For the single-dataset setting, the optimizer is trained solely on COPA.

The multi-dataset setting could lead to better performance. However, as shown in Figure 8, in some cases, the ZO Fine-tuner trained on a single dataset can outperform its multi-dataset counterpart. Overall, the two settings yield comparable performance. Single-dataset training is also simpler to implement and tune, while still achieving competitive results. And that’s why we choose to use it throughout the main experiments.

8.6 Accuracy Curve↩︎

Although we believe that for our work, the loss curve is a more direct reflection of the optimizer’s effectiveness, we also provide accuracy curves for comprehensiveness. In particular, we plotted the accuracy curve for ZO Fine-tunerand MeZO on SST2 and SQuAD dataset with LLaMA-3.1-8B. It is clear that our ZO Fine-tunerachieves faster convergence and higher final accuracy compared to MeZO.

8.7 Comparison with First-Order Baselines↩︎

To contextualize the performance gap between zeroth-order (ZO) fine-tuning and gradient-based training, we report first-order (FO) fine-tuning results. FO baselines require backpropagation and optimizer states, and thus fall outside the MeZO-style ZO regime studied in the main paper; we include them here only to provide a practical upper bound on attainable accuracy when gradients are available.

Table 11 reports final accuracy across all datasets for LLaMA-3.2-1B when fine-tuned with Adam. Table 12 additionally reports SST-2 results across multiple models. Overall, FO methods generally outperform ZO methods, while ZO Fine-tuner remains competitive with a modest performance gap in most settings.

4pt

6pt

8.8 Additional L2L Implementation Details↩︎

Trajectory choices (SGD vs.Adam). Our L2L training uses a first-order trajectory to generate model states for optimizer learning. We provide an ablation of the trajectory optimizer (SGD vs.Adam) on SST-2 for LLaMA-3.2-1B and LLaMA-3.1-8B in Table 13 and Table 14. Across intervals, Adam and SGD lead to similar loss curves and final test accuracy; in some intervals, SGD is slightly better. Since SGD has a substantially smaller memory footprint than Adam, we use SGD to generate trajectories for all reported L2L runs.

Periodic reset schedule. We tune the reset schedule based on the first-order trajectory behavior. For example, for LLaMA-3.2-1B on COPA, after 5 epochs of SGD the loss typically drops to around 0.001; we therefore reset every 5 epochs and train ZO Fine-tuner for 15 epochs total (with data reshuffling). We apply analogous tuning for other models.

L2L learning rates. The L2L pipeline involves multiple learning rates (for the first-order trajectory, the zeroth-order updates, and updating ZO Fine-tuner), making exhaustive sweeps impractical. Our choices (Appendix C.3) are guided by earlier empirical testing; we observe that larger learning rates than those reported frequently lead to instability or failure. We therefore fix these hyperparameters across experiments.

One-time tuning and reproducibility. Although tuning the L2L process introduces additional design choices, this tuning is performed only once per base model, and the resulting ZO Fine-tuner is reused across datasets and derivative checkpoints. We also release our trained finetuner checkpoints and fixed-seed scripts to facilitate reproducibility.

4pt

4pt

9 Theoretical Analysis↩︎

In this section, we formally discuss our theoretical results and derive theorem 1. First, we set up the notations and definition we need and formally present the theorem.

Assumption 1 and Theorem 2 are directly from the theoretical analysis of MeZO [1]. MeZO states the Lipschitz condition alone does not guarantee convergence in high-dimensional settings and it is necessary to leverage the low-rank structure of the Hessian matrix. Assumption 2 is consistent with the actual situation, which has been deeply researched and checked by work like Adam-mini [18], [49].

In this section, we consider a setting where, at each iteration, we sample a perturbation block-by-block: for the \(i\)-th parameter block, we draw noise from the distribution \(N(0, \sigma_i I_{d_i})\), apply the perturbation to the \(i\)-th block of parameters, and perform a zeroth-order update accordingly. Let the total number of blocks be \(b\), and denote by \(\mathcal{L}(\theta_{t,j})\) the loss after perturbing the \(j\)-th block at iteration \(t\). We further denote the full loss after perturbing all \(b\) blocks as \(\mathcal{L}(\theta_{t+1})\), and let \(\nabla_j \mathcal{L}(\theta_{t,j})\) denote the \(j\)-th block of the gradient evaluated at \(\theta_{t,j}\). Here, we adopt the sphere (normalized-Gaussian) perturbation used in the original MeZO analysis for its built-in step-size control. An analogous convergence form also holds for Gaussian perturbations, as shown in prior work[1], when the probability of large updates \(\|\theta_{t+1} - \theta_t\|\) is kept small, which ensures the required local assumptions hold with high probability.

References↩︎