SAMPLe: A Sharpness Aware Minimization based Optimizer for Prompt Learning in Vision-Language Models
July 07, 2026
Abstract
Pre-trained Vision-Language Models (VLMs) like CLIP have proven highly effective as foundation models for various downstream applications. However, prompt learning in VLMs encounters a performance-generalization dilemma: while prompts can be tuned to achieve high accuracy on seen distributions, this tuning process often undermines their generalizability to unseen data. The limited set of learnable prompts, which contextualize and condition the input to steer it toward the task within the pretrained VLM, tends to overfit the training data, leading to a trade-off between task-specific performance and preserving generalization. To address this dilemma, we introduce SAMPLe (Sharpness-Aware Minimization Prompt Learning), a plug-in sharpness-aware optimizer that enhances prompt generalizability by accounting for loss landscape sharpness. Unlike conventional methods, SAMPLe balances exploration and exploitation by satisfying objective function constraints at each step, dynamically adapting to the current optimization state based on the local curvature and gradient properties. This approach reduces overfitting on seen distributions and improves adaptability to unseen data, preserving the generalization potential of pre-trained VLM models. We integrate SAMPLe into multiple prompt learning frameworks, including CoOp, CoCoOp, MaPLe, TCP, and Co-Prompt, demonstrating its effectiveness across diverse methods. Experiments show that SAMPLe elevates prompt learning frameworks and consistently outperforms existing optimizers across diverse settings, establishing itself as a robust, model-agnostic solution for prompt learning.
1 Introduction↩︎
![Figure 1: Normalized loss landscapes of CoOp[1] on ImageNet using F-SAM[2], SAGM[3], and the proposed SAMPLe, each scaled by the maximum absolute loss (among FSAM, SAGM, and SAMPLe) preserving relative depth and sharpness. The visualization demonstrates SAMPLe’s effectiveness in achieving both flatter minima and lower empirical risk.](/article/2607.05727/_pdflatex/cuplqxyd.png "fig:")
Vision-Language Models (VLMs) such as CLIP have become essential for a wide range of visual tasks due to their robust multimodal understanding, leveraging both vision and text to achieve notable zero-shot performance across domains [4], [5]. For downstream applications, prompt learning has emerged as an efficient alternative to traditional fine-tuning by introducing learnable prompts that adapt VLMs to specific tasks while keeping their parameters fixed [1], [6]. However, a core challenge persists in balancing high task-specific accuracy while retaining generalization to unseen classes [7], [8] under the constraint of a limited number of learnable parameters.
This restriction amplifies sensitivity in optimization, since updates are confined to a narrow parameter subspace of the learnable prompts. Authors in [9] showed that smaller capacity models often converge to solutions that are less robust to perturbations, leading to weaker generalization. On the other hand, [8] further linked such generalization gaps to sharp minima characterized by large positive eigenvalues of the Hessian, while [10] emphasized that even with low training error, a large train–test gap persists under sharp solutions. These findings highlight that the limited learnable space in prompt learning makes models especially vulnerable to sharp, high-curvature minima in the loss landscape. Although various techniques [1], [6], [11]–[13] have been proposed to preserve the generalizability of VLMs while adapting them to downstream tasks, just [14] considered role of gradient manipulation with respect to the loss landscape.
Incorporating model-agnostic loss landscape awareness into VLM prompt learning can substantially enhance generalization. By guiding optimization toward flatter and more stable minima, the model can produce prompts that are both robust and transferable across tasks. However, because learnable prompts involve only a small number of tunable parameters, they are especially prone to converging to sharp, non-robust minima. While flatter landscapes are crucial for generalization, prior work shows they may come at the cost of reduced accuracy on seen distributions [3], [15]. Thus, achieving effective prompts requires carefully balancing the minimization of training loss (exploitation) with the search for sufficiently flat regions of the landscape (exploration), ensuring strong performance on both seen and unseen classes.
To address these challenges, we propose SAMPLe (Sharpness-Aware Minimization Prompt Learning), a framework specifically designed to incorporate sharpness-aware minimization (SAM) into prompt learning. SAMPLe directly targets the trade-off between minimizing loss values and maintaining a flat loss landscape, ensuring both accuracy and generalization. It introduces three essential strategies to achieve this balance. First, SAMPLe optimizes for a loss minimum that is sufficiently low in training loss and flat in the loss landscape, providing both stability and robustness. Second, it ensures that SAM gradient updates, computed at the deviated point (see Sec.4), are coherently aligned with empirical risk minimization (ERM) gradients, as illustrated in Fig.2a and Fig.2b. This alignment addresses the instability caused by the limited parameter space of prompts. Finally, SAMPLe constrains the SAM gradient updates at the perturbed point to be orthogonal to the full-batch gradient (refer to Sec.3.1, Sec.4), effectively aligning it with a relaxed version of the ERM gradient that seeks the flattest minima. Satisfying both constraints of alignment to the ERM gradient and its relaxed version in each iteration is the central objective of the SAMPLe optimizer, ensuring a precise adaptive balance between exploration and exploitation.
These dual objectives are critical for effective prompt learning. However, sharpness-aware minimization optimizers, including SAM, F-SAM[2], and SAGM[3] (see Fig.2a and Fig.4), fail to balance both conditions. Beyond these optimizers, a recently published study [14] proposed GCSCoOP, a SAM-based method designed to address the generalization problem by considering both loss value and loss sharpness. However, it suffers from limited scalability, as it relies on a heuristic gradient computation with two hyperparameters (\(\beta_1\), \(\beta_2\)). In contrast, SAMPLe updates gradients by automatically balancing exploitation and exploration constraints considering the current state of optimization.
The key contributions of this work can be summarized as follows:
We introduce SAMPLe, a SAM-based optimizer specifically designed for VLM prompt learning. It comprehensively defines a dual-objective optimization framework that adaptively balances exploiting low ERM loss and exploring the flattest minima at every iteration.
We analyze the limitations of existing sharpness-aware optimization methods in VLM prompt learning and establish the necessity of an adaptive dual-objective approach, motivating the design of SAMPLe.
We demonstrate the superior performance of SAMPLe over SOTA in VLM prompt learning and provide an ablation study comparing it with related sharpness-aware optimizers.
2 Related Work↩︎
Prompt learning is a powerful approach for adapting large pre-trained VLMs like CLIP [4], ALIGN [5], LiT [16], and FILIP [17] to downstream tasks. These models, trained on large image-text datasets using contrastive learning [18], excel at general representation learning [19], but adapting them to specific tasks, especially in low-data settings like few-shot learning [1], remains challenging. Traditional fine-tuning methods are computationally expensive and prone to overfitting [20]. Prompt learning addresses these issues by introducing learnable prompts, enabling efficient task adaptation without retraining the entire model. CoOp [6] fine-tunes continuous text prompts for few-shot recognition but struggles with unseen classes. CoCoOp [1] improves zero-shot generalization by conditioning prompts on image features, while MaPLe [20] optimizes prompts across both vision and language branches. Methods like PromptSRC [20] and ProDA [21] further enhance generalization using regularization and diverse task-related representations. Recent works extend this line with textual-based class-aware prompt tuning, TCP [22]), decoupled embedding parameterization, DEPT [23]), overfitting-aware prompt regularization, LOBG [24]), and diversity covariance-aware prompt learning [25].
SAM is an effective optimization framework designed to improve model generalization by finding flatter minima in the loss landscape [26]. SAM has been successfully applied in language modeling [27], fluid dynamics [28], medical imaging [29], multimodal learning [30], and Reinforcement Learning (RL) [31]. Building on the original framework, several extensions have refined SAM’s adaptability. F-SAM [2] utelizes a less aggressive perturbation that leads to better generalization, while SAGM [3] enhances gradient alignment to balance loss minimization and flatness. GCSCoOP[14], a recent SAM-based approach, aims to address generalization by considering both loss sharpness and value but remains dependent on manually set hyperparameters. Adaptive SAM (ASAM) adjusts the sharpness radius dynamically [32], GSAM simplifies sharpness measurement using surrogate gap calculations [15], and Fisher SAM leverages Fisher information for sharper neighborhood estimations [33]. GAM further improves generalization by focusing on the maximal gradient norm [34]. In vision tasks, SAM has demonstrated notable success in models like Vision Transformers and MLP-Mixers [35], enhancing both accuracy and robustness. However, SAM’s potential remains insufficiently explored in multi-modal learning.
3 Preliminaries↩︎
Prompt Learning in VLM: Prompt learning aims to enhance the adaptability of VLMs by learning task-specific prompts. This approach involves introducing tunable token vectors \(\mathbf{v} = \{ v_i \}_{i=1}^{N}\), where the text inputs for each class are represented as \(t_m = {v_1, \dots, v_N, c_m}\), with \(c_m\) denoting the class name and where \(N+1\) defines number of tokens for text input. These token vectors are optimized using cross-entropy loss \(\mathcal{L}_{CE}\), while keeping the parameters of the underlying VLM model, such as CLIP, frozen: \[{L}_{CE}(\mathbf{v}) = - \sum_{m} y_m \log p(m|x), \label{eq}\tag{1}\] \[p(m|x) = \frac{\exp(\text{cos}(\mathcal{I}(x), \mathcal{T}(t_m)) / \tau)}{\sum_{j=1}^M \exp(\text{cos}(\mathcal{I}(x), \mathcal{T}(t_j)) / \tau)}. \label{emktvfhn}\tag{2}\] The learned token vectors enable the model to adapt to specific tasks by refining how class names are represented as input to the model, improving the prompt’s alignment with task requirements.
Sharpness-aware Prompt Learning: SAM is a recently proposed optimization framework designed to enhance generalization by avoiding sharp minima in the loss landscape [36].
Consider a deep neural network \(f(x; \theta)\) parameterized by \(\theta \in \mathbb{R}^d\), with a loss function \(\ell(f(x; \theta), y)\) that measures the discrepancy between the model’s prediction \(f(x; \theta)\) and the true label \(y\). For a dataset \(\mathcal{D} = {(x_i, y_i)}_{i=1}^N\), the empirical loss over the dataset is given by: \[{L}(\theta; \mathcal{D}) = \frac{1}{N} \sum_{i=1}^N \ell(f(x_i; \theta), y_i). \label{eq:empirical95loss}\tag{3}\]
Optimization methods like stochastic gradient descent (SGD) often converge to sharp minima that generalize poorly. SAM mitigates this by solving the following min-max optimization problem: \[\min_{\theta} \max_{\|\epsilon\|_2 \leq \rho} {L}(\theta + \epsilon; \mathcal{D}), \label{eq:minmax95sam}\tag{4}\] where \(\epsilon \in \mathbb{R}^d\) is a perturbation constrained by \(\|\epsilon\|_2 \leq \rho\), with \(\rho\) controlling the radius of the perturbation around the parameter \(\theta\). Because solving the inner maximization directly is computationally expensive, SAM uses a first-order Taylor expansion to approximate the perturbation \(\epsilon^\star\): \[\epsilon^\star \approx \rho \cdot \frac{\nabla {L}(\theta; \mathcal{D})}{\|\nabla {L}(\theta; \mathcal{D})\|_2}. \label{eq:sam95perturbation}\tag{5}\] The network parameters are updated as follows: \[\theta_{t+1} \leftarrow \theta_{t} - \eta \cdot \nabla_{} \mathcal{L}(\theta_{t} + \epsilon^\star; \mathcal{D}) \label{eq:sam95update95rule},\tag{6}\]
where \(\eta\) is the learning rate, and \(\epsilon^\star\) is the perturbation that guides the model towards flatter minima in Fig. 2a.
The limited number of learnable parameters in prompt learning often makes traditional approaches susceptible to overfitting or getting trapped in sharp loss surfaces [13]. SAM addresses this challenge by guiding the optimization process toward flatter minima in the loss landscape, enhancing the model’s ability to generalize to unseen distributions and domains. By integrating SAM into the VLM prompt learning frameworks, we effectively improve the model’s generalization across unseen domains while maintaining good performance on the training dataset.
3.1 Full-Batch Bias Hinders SAM↩︎
One of the core ideas behind SAM is its perturbation vector, \(\epsilon\), which plays a crucial role in guiding the model toward flatter minima, thereby enhancing generalization. When applied effectively, the perturbation vector steers the optimization process toward regions of the loss landscape that are less sensitive to sharp parameter changes. However, when the perturbation vector is computed as if all training samples were processed in a single batch (i.e., a full-batch perturbation), it can diminish the natural stochasticity of a mini batch based approach. This stochasticity is crucial for balancing exploration across the loss landscape. As noted by [2], this lack of stochastic variability in full-batch perturbation can limit generalization, even resulting in worse performance compared to SGD in some cases.
While the full gradient provides a global view of the entire dataset, this global smoothness often comes at the expense of exploration, particularly when the perturbation \(\epsilon\) is excessively aligned with the full gradient. This alignment suppresses the stochastic variability introduced by mini-batch gradients, leading to over-smoothing during training. As a result, the model may converge to sharper minima, reducing generalization. This phenomenon can cause SAM to underperform even compared to traditional optimization methods like SGD [2]. The full-gradient perturbation in the SAM context captures the overall trend of the loss function across the entire dataset. The projection of the mini-batch gradient onto the full batch gradient demonstrated in Fig. 5 (in Appendix) can be formulated as:
\[\begin{align} \text{Proj}_{\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})} \nabla_{} {L}_{}(\theta; \mathcal{D}) = \frac{\|\nabla {L}_{}(\theta; \mathcal{D})\|_2}{\|\nabla^{\mathcal{F}}_{} {L}_{}(\theta; \mathcal{D})\|_2} \cos\big(\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D}), \nabla_{} {L}_{}(\theta; \mathcal{D})\big) \nabla_{} {L}^{\mathcal{F}}_{\mathcal{}}(\theta; \mathcal{D}), \end{align} \label{eq:fg95proj}\tag{7}\] where \(\cos(\cdot)\) represents the cosine similarity between the mini-batch gradient, \(\nabla_{} {L}_{}(\theta; \mathcal{D})\), and the full batch gradient vector, \(\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\). The batch-specific gradient is then defined as:
\[\nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D}) = \nabla_{} {L}_{}(\theta; \mathcal{D}) - \sigma ~ \xi ~ \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D}), \label{eq:B95proj}\tag{8}\]
where \(\xi\) and \(\sigma = \frac{\|\nabla{L}_{}(\theta; \mathcal{D})\|_2}{\|\nabla^{\mathcal{F}}_{} {L}_{}(\theta; \mathcal{D})\|_2}\) are cosine similarity operator and normalization factor, respectively. This batch-specific gradient represents the component of the mini-batch gradient orthogonal to the full batch gradient. Due to the computational overhead of computing the full gradient over the entire dataset, it is typically approximated using an exponentially moving average (EMA) of the mini-batch gradients:
\[m_t = \lambda m_{t-1} + (1 - \lambda) \nabla_{} {L}(\theta; \mathcal{D}), \label{eq:EMA}\tag{9}\]
where \(m_t\) approximates \(\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) if \(t\) is large enough, and \(\lambda\) controls the influence of past gradients [2]. The full-gradient component biases the perturbation vector \(\epsilon\) based on the entire dataset’s loss landscape, leading to smoother updates but reducing the exploration necessary for finding flatter minima. This smoothing effect can hinder SAM’s generalization performance by diminishing the stochastic variability introduced by mini-batch gradients [2].
Therefore, balancing the stability offered by full-batch gradient updates with the exploration provided by batch-specific gradients is essential for maintaining SAM’s effectiveness.
4 SAMPLe↩︎
Given the challenges of VLM prompt learning, we introduce two pivotal challenges to ensure that the model performs robustly across both seen and unseen distributions: (i) Optimal minima: The learned prompts must not only identify a flat minimum in the loss surface but also ensure that this minimum corresponds to a sufficiently low loss value, guaranteeing well-optimized performance on the training samples. (ii) Generalization to unseen domains: While prompts may perform well on the training data, it is critical to ensure they generalize effectively to unseen distributions. This challenge is amplified by the significantly smaller number of parameters in prompts compared to overparameterized neural networks, making it more challenging to train models that generalize across diverse domains. To address this, the optimization strategy enforced by the objective function must inherently balance exploitation and exploration, dynamically adapting to the current state of the optimization. This ensures that the learning process acquires network parameters capable of achieving both a sufficiently low loss on the training data and robust generalization to unseen domains and distributions. Inspired by [3] and designed to satisfy these two conditions simultaneously, we propose the SAMPLe objective function to optimize the training of learnable prompts as follows: \[\begin{align} \min_{\theta} \mathcal{L}(\theta; \mathcal{D})~ = &\min_{\theta} \Big[{L}_{}(\theta; \mathcal{D}) + {L}_{}\left(\theta + {\epsilon}^{\star} - \alpha (\nabla^{\mathcal{B}}_{} \mathcal{L}(\theta; \mathcal{D}))\right)\Big] =\\ &\min_{\theta} ~ \Big[{L}_{}(\theta; \mathcal{D}) + {L}_{}\left(\theta + {\epsilon}^{\star} - \alpha (\nabla {L}_{}(\theta; \mathcal{D}) - \xi ~ \sigma ~ \nabla^{\mathcal{F}} {L}(\theta; \mathcal{D})); \mathcal{D}\right)\Big], \end{align} \label{eq:obj95fn951}\tag{10}\] where the optimal perturbation is defined by Eq. 5 as \({\epsilon}^{\star} = \rho \cdot \frac{\nabla {L}_{}(\theta; \mathcal{D})}{\|\nabla {L}_{}(\theta; \mathcal{D})\|_2}\).
4.1 SAMPLe Analysis and Algorithm:↩︎
This section presents an analysis to clarify the proposed SAMPLE method. Applying the Taylor expansion reveals that using \(\nabla^{\mathcal{B}} L(\theta; \mathcal{D})\) in the second term of the loss function implies two critical conditions that enhance the learning process: \[\begin{align} &\min_{\theta} {L}_{p}\left(\theta - \alpha \nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D})\right) = \\ &\min_{\theta} \Big[{L}_{p}(\theta; \mathcal{D}) - \alpha \big(\nabla_{} {L}_{p}(\theta; \mathcal{D}) \cdot \nabla_{} {L}_{}(\theta; \mathcal{D})\big)~+ \\ & \alpha \xi \sigma \big(\nabla_{} {L}_{p}(\theta; \mathcal{D}) \cdot \nabla_{}^{\mathcal{F}} {L}(\theta; \mathcal{D})\big) + R(\nabla_{}^{\mathcal{B}} {L}(\theta; \mathcal{D})^2) \Big]\approx \\ &\min_{\theta} \Big[{L}_{p}(\theta; \mathcal{D}) - \alpha \nabla_{} {L}_{p}(\theta; \mathcal{D}) \cdot \nabla_{} {L}_{}(\theta; \mathcal{D}) + \alpha \xi \sigma \nabla_{} {L}_{p}(\theta; \mathcal{D}) \cdot \nabla_{}^{\mathcal{F}} {L}(\theta; \mathcal{D})\Big] \end{align} \label{eq:taylor95expansion951}\tag{11}\]
where \(L_P\), represents the loss at the perturbed point in parameter space;
\[\begin{align} \min_{\theta} \Big[&{L}(\theta; \mathcal{D}) + {L}_{p}(\theta; \mathcal{D}) - \alpha \big(\nabla_{} {L}_{p}(\theta; \mathcal{D}) \cdot \nabla_{} {L}_{}(\theta; \mathcal{D})\big)~+\\ & \alpha \xi \sigma \big(\nabla_{} {L}_{p}(\theta; \mathcal{D}) \cdot \nabla_{}^{\mathcal{F}} {L}(\theta; \mathcal{D})\big)\Big]. \end{align} \label{eq:aligned95sam95objective}\tag{12}\]
The objective function can be interpreted as follows:
First, minimizing the loss at the current point (\(L\)) and the perturbed neighboring point (\(L_p\)). Second, considering the alignment between the
gradient at the perturbed point and the gradient at the current point, represented as \(-\alpha \big(\nabla L_{p}(\theta; \mathcal{D}) \cdot \nabla L(\theta; \mathcal{D})\big)\), which we refer to as exploitation.
Third, ensuring that the gradient at the perturbed point is orthogonal to the full-batch gradient or aligned with the batch-specific gradient (see Fig. 5 in Appendix and subsection 4.2), represented as \(+\big(\nabla L_{p}(\theta; \mathcal{D}) \cdot \nabla^{\mathcal{F}} L(\theta; \mathcal{D})\big)\), which we refer to as exploration.
4.2 Gradient Orthogonality Analysis:↩︎
The third term of Eq. 12 is designed to drive \(\nabla {L}_{p}(\theta; \mathcal{D})\) orthogonal to \(\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) at the original point \(\theta_t\), as we establish below. By Eq. 8 , the mini-batch gradient decomposes as \(\nabla {L}_{}(\theta; \mathcal{D}) = \nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D}) + \xi \sigma \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\), so the second term of Eq. 12 alone would pull \(\nabla {L}_{p}(\theta; \mathcal{D})\) toward both components. The third term introduces anti-alignment with \(\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) (Eq. 10 ): at every iteration, the second term drives \(\nabla {L}_{p}(\theta; \mathcal{D})\) toward \(\nabla {L}_{}(\theta; \mathcal{D})\) for performance on seen distributions, while the third term resists the full-batch component \(\xi \sigma \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) within that alignment. The equilibrium of these two forces steers \(\nabla {L}_{p}(\theta; \mathcal{D})\) exclusively toward \(\nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D})\). Since \(\nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D}) \perp \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) by construction (Eq. 8 and Fig. 5, Appendix), this equilibrium guarantees orthogonality of \(\nabla {L}_{p}(\theta; \mathcal{D})\) to \(\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) at \(\theta_t\).
Substituting \(\nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D}) = \nabla {L}_{}(\theta; \mathcal{D}) - \xi \sigma \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) (Eq. 8 ) into the joint second and third terms of Eq. 12 :
\[\begin{align} &-\alpha\big(\nabla {L}_{p}(\theta; \mathcal{D}) \cdot \nabla {L}_{}(\theta; \mathcal{D})\big) +\alpha\xi\sigma\big(\nabla {L}_{p}(\theta; \mathcal{D}) \cdot \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\big)\nonumber\\ &= -\alpha\big(\nabla {L}_{p}(\theta; \mathcal{D}) \cdot \nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D})\big) \\ &~~~-\underbrace{\alpha\xi\sigma\big(\nabla {L}_{p}(\theta; \mathcal{D}) \cdot \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\big) + \alpha\xi\sigma\big(\nabla {L}_{p}(\theta; \mathcal{D}) \cdot \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\big)}_{=\,0}\nonumber\\ &= -\alpha\big(\nabla {L}_{p}(\theta; \mathcal{D}) \cdot \nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D})\big). \label{eq:orthog95cancel} \end{align}\tag{13}\]
The \(\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) component cancels, and the objective reduces to maximizing \(\nabla {L}_{p}(\theta; \mathcal{D}) \cdot \nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D})\) exclusively. Since \(\nabla^{\mathcal{B}}_{} {L}(\theta; \mathcal{D}) \perp \nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) by construction (Eq. 8 ), \(\nabla {L}_{p}(\theta; \mathcal{D})\) is driven toward the subspace orthogonal to \(\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\) at \(\theta_t\). To confirm this is preserved under \(\epsilon^{\star}\), let \(\hat{\theta}_t = \theta_t + \epsilon^{\star}_t - \alpha_t \nabla^{\mathcal{B}}_{} {L}(\theta_t; \mathcal{D})\) denote the perturbed point. By the triangle inequality, \(\|\epsilon^{\star}_t\| \leq \rho_t\) (Eq. 5 ) and condition (i) in Sec. 4.3: \[\|\hat{\theta}_t - \theta_t\| \leq \rho_t + \alpha_t \nabla\mathcal{L}_{\max}. \label{eq:orthog95bound}\tag{14}\]
By K-Lipschitz gradient condition (ii) in Sec. 4.3 and Lemma 1 (Appendix, Proof of Theorem 1):
\[\begin{align} \bigl|(\nabla {L}_{p}(\theta; \mathcal{D}) {-} \nabla {L}_{}(\theta; \mathcal{D})) {\cdot}\nabla^{\mathcal{F}}_{} {L}(\theta; \mathcal{D})\bigr| &\leq K(\rho_t {+} \alpha_t\nabla\mathcal{L}_{\max}) \cdot\nabla {L}_{\max}(1{-}\lambda^T). \label{eq:orthog95vanish} \end{align}\tag{15}\]
As \(\rho_t, \alpha_t = \mathcal{O}(1/\sqrt{t})\) and \(\lambda^T \to 0\) as \(t \to \infty\), the right-hand side vanishes, confirming orthogonality is preserved throughout training.
4.3 Convergence of SAMPLe:↩︎
Assuming that the objective function defined in Eq. 10 satisfies the following conditions;
(i) The gradient of the loss function \(\nabla \mathcal{L}(\theta_t; \mathcal{D})\) is bounded, i.e., \[\|\nabla \mathcal{L}(\theta_t; \mathcal{D})\| \leq \nabla\mathcal{L}_{max}, \quad \forall t.\]
(ii) The stochastic gradient is K-Lipschitz, i.e., \[\|\nabla \mathcal{L}(\theta_t; \mathcal{D}) - \nabla \mathcal{L}(\theta_t'; \mathcal{D})\| \leq K \|\theta_t - \theta_t'\|, \forall (\theta_t,
\theta_t').\]
Let the learning rate and both perturbations radii be defined as \(\eta_t = \frac{\eta_0}{\sqrt{t}}\), \(\alpha_t = \frac{\alpha_0}{\sqrt{t}}\), and \(\rho_t =
\frac{\rho_0}{\sqrt{t}}\), respectively. Theorem. 1 in Appendix. 8.1 proves that the objective function satisfies following
inequality \[\frac{1}{T} \sum_{t=1}^{T} \left[ \|\nabla \mathcal{L}(\theta_t; \mathcal{D})\|^2 \right] \leq \mathcal{O}\left(\frac{\log T}{\sqrt{T}}\right),
\label{eq:convergense1}\tag{16}\] It means the objective function defined in Eq. 10 converges with a rate of \(\mathcal{O}\left(\frac{\log T}{\sqrt{T}}\right)\), which is
comparable with optimization methods such as SGD and Adam.
5 Experiments↩︎
In this section, we elaborate on the datasets, baselines, and implementation details utilized in our study. Next, we evaluate the proposed method on base-to-new class generalization (Sec. 5.2), cross-dataset generalization (Sec. 5.3), and cross-domain generalization (Sec. 5.4).
5.1 Datasets↩︎
In this work, we utilize \(11\) publicly available image classification datasets as downstream tasks: ImageNet [37], Caltech101 [38], OxfordPets [39], StanfordCars [40], Flowers102 [41], Food101 [42], FGVCAircraft [43], SUN397 [44], DTD [45], EuroSAT [46], and UCF101 [47]. Also, four datasets of ImageNetV2 [48], ImageNet-Sketch [49], ImageNet-A [50], and ImageNet-R [51] serve exclusively as target domains for cross-domain generalization. Detailed dataset descriptions are provided in Supplementary 8.3. For each baseline we strictly adhere to the configurations reported in the original papers, including learning rate schedules, backbone architecture, prompt length, prompt initialization, few-shot setup, and three random seeds. Detailed of hyperparameter and implementation descriptions are provided in Supplementary 8.6.
5.2 Base-to-New Generalization Setting↩︎
To evaluate the generalization capability of our method, we train the model on base classes and test its performance on both base and novel classes. This experimental setup is designed to evaluate the trade-off between maintaining strong performance on the base classes while adapting effectively to novel classes. The results are presented in Table 1, with the harmonic mean (HM) of base and novel accuracies serving as the primary metric for comparison. As shown in Table 1, integrating SAMPLe consistently improves the harmonic mean (HM) across different prompt-learning backbones and datasets, indicating a more balanced generalization between base and novel classes. On the averaged results (Table 1 (a)), SAMPLe achieves the highest HM for all methods. For example, CoOp improves from 77.65 (+SAGM) to 78.88 with SAMPLe, CoCoOp from 78.28 to 79.51, MAPLe from 79.96 to 80.28, CoPrompt from 81.49 to 82.02, and TCP from 81.40 to 81.95. These improvements are mainly associated with higher accuracy on the new classes while maintaining competitive performance on the base classes. A similar trend is observed across individual datasets. On Flowers102, SAMPLe provides the best HM across all backbones, for instance improving CoCoOp from 85.26 (+SAGM) to 87.15 and TCP from 86.48 to 87.02. On DTD, SAMPLe increases the HM from 65.46 to 69.77 for CoOp and from 70.93 to 71.37 for MAPLe. On EuroSAT, SAMPLe again achieves the highest HM values, reaching 82.44 for CoOp and 84.19 for TCP. Overall, the consistent improvements across methods and datasets suggest that SAMPLe leads to more stable optimization behavior and improved generalization to novel classes without drastically degrading base-class performance. Another highlighted fact is that, superiority of SAMPLe is almost clear over all mentioned methods specifically in HM, which means SAMPLe keeps both Base and New performance and does not sacrify one of them in favor of the other.
Comparison with State-of-the-Art Methods: In addition to comparisons with SAM-based optimizers, we evaluate CoOp+SAMPLe against several representative prompt-learning approaches, including CoOp, CoCoOp, MAPLe, and CoPrompt. As shown in Table 1, CoOp+SAMPLe achieves competitive and often superior harmonic mean (HM) performance across several datasets when compared with the corresponding naive prompt-learning methods. For example, on OxfordPets, CoOp+SAMPLe achieves an HM of 97.08, surpassing the naive variants of CoCoOp (96.43), MAPLe (96.58), and CoPrompt (96.87). This result suggests that SAMPLe can effectively improve generalization while maintaining strong base-class performance. On more challenging datasets such as FGVCAircraft, CoOp+SAMPLe obtains an HM of 37.58, outperforming MAPLe (36.50) and approaching the performance of CoPrompt (39.76). These results indicate that the proposed optimizer remains effective even in fine-grained recognition scenarios where intra-class variations are large and training samples are limited. Similarly, on StanfordCars, CoCoOp+SAMPLe achieves an HM of 75.51, outperforming CoCoOp+FSAM (74.85) and CoCoOp+SAGM (74.75). While maintaining comparable base accuracy, SAMPLe improves performance on the novel classes (77.89), leading to a higher harmonic mean. More broadly, the averaged results in Table 1 (a) reveal different behaviors among the optimizers. FSAM often improves generalization to novel classes but may reduce base-class accuracy. In contrast, SAMPLe maintains a stronger balance between base and novel performance, leading to consistently higher HM across different prompt-learning backbones. These observations suggest that SAMPLe provides a more stable trade-off between specialization on the base classes and transfer to unseen classes.
|
|
|
|
|
|
|
|
|
|
|
|
Insights and Ablation Study: Table 1 and Fig 4 demonstrate that SAMPLe consistently outperforms existing sharpness-aware optimizers by maintaining both generalization and stability across varying \(\rho\) and \(\lambda\). While F-SAM mitigates the excessive perturbation of traditional SAM by deviating from the mini-batch gradient, \(\nabla^{\mathcal{F}}\), it still enforces a rigid batch-specific gradient, \(\nabla^{\mathcal{B}}\), which remains inherently sensitive to the noisy approximation of \(\nabla^{\mathcal{F}}\). This structural limitation is evident in Fig 4, where F-SAM and SAMPLe exhibit comparable robustness to variations in \(\rho\), outperforming both SAM and SAGM. However, SAMPLe consistently achieves higher accuracy across all perturbation radii and prompt-learning methods, highlighting its optimization-aware perturbation alignment. Rather than rigidly enforcing batch-specific perturbations, SAMPLe dynamically aligns them with the current optimization state, striking an optimal balance between exploration and stability. This adaptability translates into significant performance gains, as confirmed by Figure 4.
5.3 Cross-Dataset Zero-Shot Generalization Setting↩︎
To evaluate cross-domain robustness, we train models on the ImageNet dataset containing \(1000\) classes and directly evaluate them on \(10\) unseen target datasets without any fine-tuning. This protocol measures the ability of prompt learning methods to transfer knowledge learned from a large-scale source domain to diverse downstream domains with varying visual characteristics.
Table 2 summarizes the cross-dataset performance of several prompt learning approaches with and without the proposed SAMPLe optimizer. Overall, incorporating SAMPLe consistently improves the transfer performance of prompt-based models across most datasets. For example, CoOp benefits significantly from SAMPLe, increasing its average accuracy from \(63.88\%\) to \(65.85\%\). The improvement is particularly notable on datasets such as Cars (\(66.35\%\) vs.\(64.51\%\)), Flowers (\(71.46\%\) vs.\(68.71\%\)), and Aircraft (\(23.61\%\) vs.\(18.47\%\)), which typically exhibit larger domain shifts from ImageNet.
0.3mm
A similar trend can be observed for CoCoOp. While the baseline CoCoOp achieves an average accuracy of \(65.74\%\), integrating SAMPLe increases the average performance to \(66.32\%\). The improvements are consistently observed across several datasets, including Cars (\(66.00\%\) vs.\(65.32\%\)), Flowers (\(72.84\%\) vs.\(71.88\%\)), Aircraft (\(23.40\%\) vs.\(22.94\%\)), SUN397 (\(67.58\%\) vs.\(67.36\%\)), and UCF101 (\(69.00\%\) vs.\(68.21\%\)), indicating that SAMPLe improves the robustness of prompt learning under domain shift. For more advanced prompt learning architectures, SAMPLe continues to provide consistent gains. When applied to MaPLe, the average performance increases from \(66.30\%\) to \(67.14\%\), with noticeable improvements on datasets such as Caltech (\(94.65\%\) vs.\(93.53\%\)), Pets (\(91.80\%\) vs.\(90.49\%\)), and EuroSAT (\(53.12\%\) vs.\(48.06\%\)). Similarly, TCP benefits from SAMPLe with the average accuracy increasing from \(66.29\%\) to \(67.53\%\), while achieving higher scores on datasets including Caltech (\(96.82\%\) vs.\(93.97\%\)), Cars (\(67.43\%\) vs.\(64.69\%\)), SUN397 (\(68.20\%\) vs.\(67.15\%\)), and DTD (\(47.79\%\) vs.\(44.35\%\)).
Even for strong baselines such as CoPrompt, which already achieves a relatively high average accuracy of \(67.00\%\), SAMPLe further improves the overall performance to \(67.95\%\). Improvements can be observed across multiple datasets including Caltech (\(96.24\%\) vs.\(94.50\%\)), Pets (\(93.19\%\) vs.\(90.73\%\)), and Flowers (\(73.59\%\) vs.\(72.30\%\)), suggesting that the proposed optimizer provides benefits that are complementary to architectural advances in prompt learning. Across all evaluated methods, SAMPLe consistently improves the average performance while maintaining stable accuracy on the source domain (ImageNet). This behavior indicates that the proposed optimization strategy improves the generalization capability of learned prompts without sacrificing source-domain alignment. We attribute these gains to SAMPLe’s ability to explore flatter regions of the loss landscape through its gradient component neutralization mechanism, which helps avoid overfitting to the source distribution and enables more robust transfer to unseen datasets.
5.4 Cross-Domain Zero-Shot generalization setting↩︎
Table 3 presents the results for domain generalization. The original ImageNet dataset trains learnable prompts to contextualize the model input in this setup. The evaluation is performed on four diverse ImageNet variants (-V2, -Sketch, -Adversarial, and -Rendition), each representing different types of distribution shifts. This evaluation tests how well the model generalizes to unseen domains. Integrating the proposed SAMPLe optimizer with CoOp and CoCoOp leads to consistent improvements in generalization performance. For example, CoOp+SAMPLe raises the average accuracy from \(59.28\%\) (CoOp) to \(60.41\%\), with notable gains on datasets such as ImageNet-S (1.32%), ImageNet-A (1.26%), and ImageNet-R (1.34%). Similarly, CoCoOp+SAMPLe increases the average accuracy of CoCoOp from \(59.91\%\) to \(60.47\%\), showing significant improvements on ImageNet-A (0.48%) and ImageNet-R (0.56%). Compared to other methods like ProGrad, KgCoOp, and MAPLe, CoCoOp+SAMPLe achieves the highest average accuracy of \(60.47\%\), surpassing MAPLe (\(60.27\%\)). This demonstrates SAMPLe’s ability to handle domain shifts more effectively.
The results show that SAMPLe enhances the model’s exploration during optimization, allowing it to adapt better to challenging datasets such as ImageNet-A and ImageNet-R.
0.3mm
6 Conclusion↩︎
This paper introduces SAMPLe, a novel, model-agnostic optimizer that enhances prompt learning across all modalities by integrating sharpness-aware objectives into the optimization process. Inspired by recent advancements in sharpness-aware minimization (SAM), SAMPLe balances exploitation and exploration by considering the optimization state, achieving a harmonious trade-off between accuracy and generalization. A key innovation of SAMPLe lies in its ability to dynamically adapt gradients for prompt learning tasks, ensuring flatter minima that promote robust generalization while maintaining strong performance on seen distributions. Through extensive analysis and rigorous experiments, we demonstrated that SAMPLe significantly improves the generalization capabilities of prompt-based vision-language models (VLMs) and broadly applies to modality-specific prompt-learning tasks. It consistently outperforms counterpart methods across various benchmark datasets, underscoring its effectiveness and robustness in addressing the challenges of prompt learning. Further details are provided in the supplementary material.
7 Acknowledgments↩︎
This material is based upon work supported by the National Science Foundation under Grant Number CNS-2232048, CNS-2204445, and CCF-2553684.
8 Appendix↩︎
In this supplementary material, we begin with the proof of Theorem \(1\), followed by a comprehensive comparison of various optimizers, including SAM, FSAM, and SAGM, against the proposed SAMPLe across all \(11\) datasets using CoOp, CoCoOp, MaPLe, CoPrompt, and TCP. We then present the ablation study and elaborate on deployment details in the subsequent sections.
8.1 Proof of Theorem 1↩︎
Lemma 1. Let \(\nabla {L}(\theta_t; \mathcal{D})\) be the mini-batch gradient at iteration \(t\) and assume that \(\nabla {L}(\theta_t; \mathcal{D})\) is bounded by \({L}_{max}\), i.e., \(\nabla {L}(\theta_t; \mathcal{D}) \leq {L}_{max}\). Suppose the exponentially moving average (EMA) of the gradients is given by: \[\begin{align} m_t = \lambda m_{t-1} + (1 - \lambda) \nabla {L}(\theta_t; \mathcal{D}), \end{align} \label{eq:EMA2}\qquad{(1)}\] where \(\lambda \in (0, 1)\). Then the EMA approximation \(m_T\) at iteration \(T\) is bounded by: \[\begin{align} m_T \leq \nabla{L}_{max} (1 - \lambda^T). \end{align} \label{eq:upper95bound}\qquad{(2)}\]
Proof: We proceed by induction and using properties of the geometric series.
At \(t = 1\), we have: \[\begin{align} m_1 = (1 - \lambda) \nabla {L}(\theta_1; \mathcal{D}), \end{align} \label{eq:m1}\tag{17}\] so, \[\begin{align} m_1 \leq (1 - \lambda) \nabla{L}_{max}. \end{align} \label{eq:m195bound}\tag{18}\]
For \(t \geq 2\), the EMA update is given by: \[\begin{align} m_t = \lambda m_{t-1} + (1 - \lambda) \nabla {L}(\theta_{t-1}; \mathcal{D}), \end{align} \label{eq:mt}\tag{19}\] and since \(\nabla {L}(\theta_t; \mathcal{D}) \leq \nabla{L}_{max}\), we can bound: \[\begin{align} m_t \leq (1 - \lambda) \sum_{i=1}^{t} \lambda^{t-i} \nabla{L}_{max}. \end{align} \label{eq:mt95bound}\tag{20}\]
The sum \(\sum_{i=1}^{t} \lambda^{t-i}\) is a geometric series and simplifies to: \[\begin{align} \sum_{i=1}^{t} \lambda^{t-i} = \frac{1 - \lambda^t}{1 - \lambda}. \end{align} \label{eq:geo95series}\tag{21}\]
Thus, the upper bound becomes: \[\begin{align} m_T \leq (1 - \lambda) \nabla{L}_{max} \frac{1 - \lambda^T}{1 - \lambda}. \end{align} \label{eq:mT95bound95step}\tag{22}\]
Simplifying: \[\begin{align} m_T \leq \nabla{L}_{max} (1 - \lambda^T). \end{align} \label{eq:mT95final}\tag{23}\]
Hence, as \(T \to \infty\), \(m_T\) asymptotically approaches \({L}_{max}\).
Theorem 1 (definition). Assuming the loss function defined as Eq. 10 , we have: \[\begin{align} \mathcal{L}(\theta_t; \mathcal{D}) = & {L}_{}(\theta_{t}; \mathcal{D}) + {L}_{}\big(\theta_{t} + {\epsilon}^{\star}_{t} - \alpha_{t} \nabla {L}(\theta_{t}; \mathcal{D}) ~+\\ &\alpha_{t} ~ \xi_{t} ~ \sigma_{t} ~ \nabla^{\mathcal{F}} {L}(\theta; \mathcal{D})\big) \end{align}\] and \(\mathcal{L}(\theta_t; \mathcal{D})\) satisfies the following assumptions:
its gradient \(\nabla \mathcal{L}(\theta_t; \mathcal{D})\) is bounded, i.e., \(\|\nabla \mathcal{L}(\theta_t; \mathcal{D})\| \leq \nabla\mathcal{L}_{max}\), \(\forall t\),
The stochastic gradient is K-Lipschitz gradient, i.e., \(\|\nabla \mathcal{L}(\theta_t; \mathcal{D}) - \nabla \mathcal{L}(\theta^{'}_{t}; \mathcal{D})\| \leq K \|\theta_t - \theta^{'}_{t}\|\), \(\forall \theta_t, \theta^{'}_{t}\).
Let the learning rate \(\eta_t\) be \(\eta_t = \frac{\eta_0}{\sqrt{t}}\), and let the perturbations decrease with the same rate as the learning rate, i.e., \(\rho_t = \frac{\rho_0}{\sqrt{t}}\) and \(\alpha_t = \frac{\alpha_0}{\sqrt{t}}\). If \(\hat{\theta}_{t}\) be defined as follows: \[\begin{align} &\hat{\theta}_{t} = ~\theta_{t} + \epsilon^{*} - \alpha_{t} \sum\limits_{(x,y)\in \mathcal{D}} \nabla {L}(\theta_t, \mathcal{D}) ~+\\ &\alpha_{t} ~ \sum\limits_{(x,y)\in \mathcal{D}} \xi_{t} ~ \sigma_{t} ~ \nabla^{\mathcal{F}} {L}(\theta_{t}; \mathcal{D})\\ \approx& ~\theta_{t} + \epsilon^{*} - \alpha_{t} \sum\limits_{(x,y)\in \mathcal{D}} \nabla {L}(\theta_t, \mathcal{D}) ~+\alpha_{t} ~ \sum\limits_{(x,y)\in \mathcal{D}} \xi_{t} ~ \sigma_{t} ~ m_{t} \end{align} \label{eq:32theta95hat}\qquad{(3)}\] where \(m_t\) approximates full-gradient as it is defined in Eq. 9 .
\[\frac{1}{T} \sum_{t=1}^{T} \left[ \|\nabla \mathcal{L}(\theta_t; \mathcal{D})\|^2 \right] \leq \mathcal{O}\left(\frac{\log T}{\sqrt{T}}\right),\]
Proof: For simplicity of equations, we define \(d_t\) as: \[d_{t} = \theta_{t+1} - \theta_{t} = -\eta_{t} \nabla \mathcal{L} (\theta_{t}) - \eta_{t} \nabla \mathcal{L} (\hat{\theta}_{t}) \label{eq:d95t}\tag{24}\] where \(\eta_t\) represents the learning rate. By K-Lipschitz gradient of \(\mathcal{L}\), the definition of \(d_t\) in Eq. 24 , and the inequality of \(\|\nabla \mathcal{L} (\theta_{t}) + \nabla \mathcal{L} (\hat{\theta}_{t})\|^2 \leq 2(\|\nabla \mathcal{L} (\theta_{t})\|^2 + \|\nabla \mathcal{L} (\hat{\theta}_{t})\|^2)\) we need to define an upper bound for loss update difference between every consequent iteration as follows:
\[\begin{align} &\mathcal{L}(\theta_{t+1}) - \mathcal{L}(\theta_t) \leq \langle \nabla \mathcal{L}(\theta_t), \theta_{t+1} - \theta_t \rangle + \frac{K}{2} \|\theta_{t+1} - \theta_t\|^2 =\\ & \langle \nabla \mathcal{L}(\theta_t), d_t \rangle + \frac{K}{2} \|d_t\|^2 = -\eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\theta_{t}) + \nabla \mathcal{L} (\hat{\theta}_{t})) \rangle \\ &\quad \quad + \frac{K \eta_t^2}{2} \|\nabla \mathcal{L} (\theta_{t}) + \nabla \mathcal{L} (\hat{\theta}_{t})\|^2 =\\ & -\eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L}(\theta_t) + \nabla \mathcal{L} (\theta_{t}) -\nabla \mathcal{L}(\theta_t) + \nabla \mathcal{L} (\hat{\theta}_{t}) \rangle \\ &\quad\quad + \frac{K \eta_t^2}{2} \|\nabla \mathcal{L} (\theta_{t}) + \nabla \mathcal{L} (\hat{\theta}_{t})\|^2 = \\ & -\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 - \eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L}(\theta_t) - \nabla \mathcal{L}(\theta_t) + \nabla \mathcal{L} (\hat{\theta}_{t}) \rangle \\ &\quad \quad + \frac{K \eta_t^2}{2} \|\nabla \mathcal{L} (\theta_{t}) + \nabla \mathcal{L} (\hat{\theta}_{t})\|^2 = \\ & -2 \eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 - \eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\hat{\theta}_{t}) - \nabla \mathcal{L}(\theta_t) \rangle \\ &\quad \quad + K \eta_t^2 (\|\nabla \mathcal{L} (\theta_{t})\|^2 + \|\nabla \mathcal{L} (\hat{\theta}_{t})\|^2)\\ &\leq -2 \eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 + \eta_t \langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\theta_{t}) - \nabla \mathcal{L}(\hat{\theta}_t) \rangle \\ & \quad \quad + K \eta_t^2 \nabla\mathcal{L}_{max}^2 \end{align} \label{eq:prelim}\tag{25}\]
in this step, we should show that there is an upper bound for the expression \(\langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\hat{\theta}_{t}) - \nabla \mathcal{L}(\theta_t) \rangle\), it is done as follows;
\[\begin{align} &\langle \nabla \mathcal{L}(\theta_t), \nabla \mathcal{L} (\hat{\theta}_{t}) - \nabla \mathcal{L}(\theta_t) \rangle =\\ &\leq \|\nabla \mathcal{L}(\theta_t)\|~\|\nabla \mathcal{L} (\hat{\theta}_{t}) - \nabla \mathcal{L}(\theta_t)\| \\ &\leq K ~\|\nabla \mathcal{L}(\theta_t)\|~\|\hat{\theta}_{t} - \theta_t\| \quad \small{\textit{(K-Lipschitz condition.)}} \\ & = K~\|\nabla \mathcal{L}(\theta_t)\| \cdot \|\epsilon^{*}_{t} - \alpha_{t} \sum\limits_{(x,y)\in \mathcal{D}} \nabla {L}(\theta_t, \mathcal{D}) +\\ &\quad \quad \alpha_{t} ~ \sum\limits_{(x,y)\in \mathcal{D}} \xi_{t} ~ \sigma_{t} ~ m_{t}\| \quad \small{\textit{(using Eq.~\ref{eq:32theta95hat})}} \\ &\leq K~\|\nabla \mathcal{L}(\theta_t)\| \cdot \|\epsilon^{*}_{t}\| ~+ \\ & \quad ~K\alpha_{t}~\|\nabla \mathcal{L}(\theta_t)\| \cdot \|\sum\limits_{(x,y)\in \mathcal{D}} (\nabla {L}(\theta_t, \mathcal{D}) - \xi_{t} ~ \sigma_{t} ~ m_{t})\| \\ & \text{\textit{(Lemma.~\ref{lemma1} and the fact that \nabla L_{max} \leq \nabla \mathcal{L}_{max}, given}} \\ &\text{\textit{\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N together would result in:)}}\\ &\leq K~\rho_{t}~\mathcal{\nabla{L}}_{max}~+ K ~N~\alpha_{t}~\mathcal{\nabla{L}}_{max}~\nabla{L}_{max}~ - \\ & K \alpha_{t}~\nabla\mathcal{L}_{max}~\xi_{t} \leq K(\rho_t+\alpha_{t}) \nabla\mathcal{L}_{max} + ~K~N~\alpha_{t}~\nabla\mathcal{L}_{max}^2 \end{align} \label{eq:cos95UpBound}\tag{26}\]
by replacing the upper bound in Eq. 26 in Eq. 25 we have: \[\begin{align} &\mathcal{L}(\theta_{t+1}) - \mathcal{L}(\theta_t) \leq -2 \eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 ~+\\ &\quad \quad \eta_t K(\rho_t+\alpha_{t}) \nabla\mathcal{L}_{max} + ~K~N~\eta_t~\alpha_{t}~\nabla\mathcal{L}_{max}^2 +\\ &\quad \quad K \eta_t^2 \nabla\mathcal{L}_{max}^2 \end{align} \label{eq:replace}\tag{27}\]
by rearranging the inequality, it will be as:
\[\begin{align} & 2 \sum^{T}_{t=1}\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \mathcal{L}(\theta_{t}) - \mathcal{L}(\theta_{t+1}) + K \eta_t^2 \nabla\mathcal{L}_{max}^2~+\\ &\quad \quad \eta_t K(\rho_t+\alpha_{t}) \nabla\mathcal{L}_{max} + ~K~N~\eta_t~\alpha_{t}~\nabla\mathcal{L}_{max}^2 \end{align} \label{eq:rearrange}\tag{28}\]
considering definition of \(\eta_{t} = \frac{\eta_{0}}{\sqrt{t}}\) and taking summation over all iterations on the left side of Eq. 28 , we can define a lower bound as follows:
\[\begin{align} 2 \frac{\eta_{0}}{\sqrt{T}}\sum^{T}_{t=1} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq 2 \sum^{T}_{t=1}\eta_t \|\nabla \mathcal{L}(\theta_t)\|^2 \end{align} \label{eq:L95LB}\tag{29}\]
using telescope sum properties and considering \(0 \leq \mathcal{L}_{t} \leq \mathcal{L}_{max} ~\forall t :\) \[\begin{align} \sum^{T}_{t=1}\big(\mathcal{L}(\theta_{t}) - \mathcal{L}(\theta_{t+1}) \big) = \mathcal{L}_{1} - \mathcal{L}_{T} \leq \mathcal{L}_{1} \end{align} \label{eq:R95UB}\tag{30}\]
by taking summation over all iterations on the right side of Eq. 28 and using Eq. 30 we will have: \[\begin{align} & \sum^{T}_{t=1} \big(\mathcal{L}(\theta_{t}) - \mathcal{L}(\theta_{t+1}) \big) + K \sum^{T}_{t=1} \big(\eta_t(\rho_t+\alpha_{t})\big) \mathcal{L}_{max} ~+\\ & \quad K~N~\sum^{T}_{t=1}(\eta_t~\alpha_{t})~\mathcal{L}_{max}^2 + K \sum^{T}_{t=1}(\eta_t^2) \mathcal{L}_{max}^2 ~\leq \mathcal{L}_{1}~+\\ & K \sum^{T}_{t=1} \big(\eta_t(\rho_t+\alpha_{t})\big) \nabla\mathcal{L}_{max} + KN\sum^{T}_{t=1}(\eta_t~\alpha_{t})~\nabla\mathcal{L}_{max}^2~+\\ & \quad \quad K \sum^{T}_{t=1}(\eta_t^2) \nabla\mathcal{L}_{max}^2 \end{align} \label{eq:rearrange2}\tag{31}\]
using these lower and upper bounds, we can write the following inequality: \[\begin{align} & 2 \frac{\eta_{0}}{\sqrt{T}}\sum^{T}_{t=1} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \mathcal{L}_{1}+K \sum^{T}_{t=1} \big(\eta_t(\rho_t+\alpha_{t})\big) \nabla\mathcal{L}_{max} +\\ & \quad \quad \quad \quad KN\sum^{T}_{t=1}(\eta_t~\alpha_{t})~\nabla\mathcal{L}_{max}^2+ K \sum^{T}_{t=1}(\eta_t^2) \nabla\mathcal{L}_{max}^2~ = \\ & \quad \quad \quad \quad \mathcal{L}_{1} + K\eta_{0}(\rho_{0}+\alpha_{0}) \nabla\mathcal{L}_{max}\sum^{T}_{t=1} (\frac{1}{t}) +\\ & \quad \quad \quad \quad KN\eta_{0}~\alpha_{0}\sum^{T}_{t=1}(\frac{1}{t})~\nabla\mathcal{L}_{max}^2 + K\eta_{0}^2\sum^{T}_{t=1}(\frac{1}{t})~\nabla\mathcal{L}_{max}^2 \end{align} \label{ftnjbxzq}\tag{32}\]
considering \(\sum^{T}_{t=1}(\frac{1}{t}) \leq 1+\log(T)\):
\[\begin{align} & \frac{1}{T}\sum^{T}_{t=1} \|\nabla \mathcal{L}(\theta_t)\|^2 \leq \frac{\mathcal{L}_{1}}{2\eta_{0}} + \\ &\frac{K\mathcal{L}_{max}\big(\rho_{0}+ \eta_{0}\mathcal{L}_{max}+(1+N\mathcal{L}_{max})\alpha_{0}\big)}{2} (\frac{1+\log(T)}{\sqrt{T}}) \end{align} \label{aebyjtdv}\tag{33}\]
which means; \[\frac{1}{T} \sum_{t=1}^{T} \left[ \|\nabla \mathcal{L}(\theta_t; \mathcal{D})\|^2 \right] \leq \mathcal{O}\left(\frac{\log T}{\sqrt{T}}\right), \label{eq:convergense}\tag{34}\] Hence, we conclude that the proposed loss function \(\mathcal{L}(\theta_t; \mathcal{D})\) converges with a rate comparable to first-order gradient-based optimization methods such as Adam and RMSProp, provided that the model is trained for sufficiently large iterations \(T\).
0.3mm
8.2 Optimizer Impact on CoOp, CoCoOp, MaPLe, and CoPrompt↩︎
Table 1 provides a comprehensive comparison of different prompt learning methods enhanced with various optimizers, including SAM, FSAM, SAGM, and the proposed SAMPLe, evaluated across all 11 datasets.
8.3 Datasets Details↩︎
We describe the \(11\) datasets and \(4\) variations of ImageNet used for evaluation, providing details on the number of classes, along with the training and testing sample sizes, as summarized in Table. 4.
8.4 Robustness to Perturbation Radius↩︎
Unlike SAM and SAGM, which maximize the worst-case perturbation per mini-batch, SAMPLe and F-SAM mitigate instability by constraining perturbations to align with the mini-batch gradient rather than strictly enforcing adversarial maximization. While F-SAM explicitly computes a batch-specific gradient perturbation, SAMPLe encourages alignment with the mini-batch gradient direction without rigidly restricting updates. This distinction allows SAMPLe to balance batch-specific adaptation and generalization, enhancing stability across perturbation scales while achieving superior accuracy without excessive sensitivity to \(\rho\) in Fig. 4.
8.5 Robustness to Whole-Batch Gradient Approximation↩︎
Figure 4 demonstrates that SAMPLe is significantly more robust to variations in \(\lambda\), which controls the effect of previous gradients in approximating the full-batch gradient. Unlike F-SAM, which strictly enforces the batch-specific gradient in its perturbation update, SAMPLe only encourages alignment, allowing for greater adaptability. This flexibility enables SAMPLe to dynamically compensate for changes in \(\lambda\) based on the optimization state, leading to consistently higher accuracy and lower variance.
8.6 Deployment Details↩︎
8.6.1 Hyperparameter selection↩︎
0.2mm
Table 5 presents the tuned hyperparameters for SAM, SAGM, and SAMPLe applied to CoOp, CoCoOp, MaPLe, Co-Prompt, and TCP across 11 datasets, focusing on the perturbation radius, \(\rho\), alignment parameter, \(\alpha\), and previous gradient effective factor, \(\lambda\). The results in Table 1 are well aligned with the hyperparameter choices in Table 5. In particular, SAMPLe preserves the same perturbation radius used by the sharpness-aware baselines (\(\rho=0.05\) or \(0.10\), depending on the prompt learner), retains the regularization strength of F-SAM (\(\lambda=0.15\)), and adopts a slightly stronger correction factor than SAGM (\(\alpha=0.0015\) vs.\(0.0010\)). This combination yields a more favorable balance between optimization stability and generalization, which is reflected in the results: across all five prompt learning methods, SAMPLe achieves the best average harmonic mean and, in most cases, the strongest new-class accuracy, while maintaining competitive or improved base-class performance. The gains are especially pronounced on datasets with larger base-to-new generalization gaps, such as Flowers102, DTD, EuroSAT, StanfordCars, UCF101, and FGVCAircraft, where SAMPLe consistently improves the balance between memorization of base classes and transfer to unseen categories. Importantly, these improvements are obtained with fixed hyperparameter settings for each method across all 11 datasets, indicating that the performance gains are not due to dataset-specific tuning, but rather to the optimizer itself. Overall, the evidence suggests that SAMPLe more effectively controls loss sharpness and reduces over-specialization, leading to stronger and more consistent base-to-new generalization than SAM, F-SAM, and SAGM.
8.6.2 MaPLe↩︎
We train MaPLe using a staged training strategy to improve generalization and stability. The training begins with optimizing the learnable prompts and the projection layers using standard optimization methods such as SGD or Adam. Once the initial training phase converges, we freeze the projection layers and continue training only the learnable prompts using SAMPLe, which is designed for prompt learning. This step ensures that the prompts adapt to both flatter minima and low values of loss, which enhances generalization while preserving the alignment between vision and text representations. By decoupling the optimization of prompts and projections, our method allows each component to specialize without disrupting the overall model convergence. The results in Table. 1 demonstrates that the SAMPLe optimizer performs well in staged strategy, leading to significantly better performance than the benchmark and other optimization methods, including SAM, F-SAM, and SAGM. Specifically, our approach achieves higher accuracy across the HM value,
8.6.3 Co-Prompt↩︎
Following the staged training strategy introduced in MaPLe, we apply SAMPLe to CoPrompt by first training the model using standard optimization and jointly updating the learnable prompts, projection layers, and adapter networks. After convergence, we freeze the adapters and projection layers and then optimize only the learnable prompts using SAMPLe. This allows them to refine their representations while preserving the alignment learned in earlier stages. This staged optimization ensures that the learned prompts remain discriminative and generalize well across unseen class distributions.
| Base | New | HM | |
|---|---|---|---|
| \(L\) | 84.00 | 77.23 | 80.48 |
| \(L_p\) | 84.61 | 77.98 | 80.99 |
| \(L + L_p - \alpha(\nabla L \cdot \nabla L_p)\) | 85.12 | 78.47 | 81.49 |
| \(L + L_p - \alpha(\nabla L \cdot \nabla L_p) + \alpha \xi \sigma(\nabla L_p \cdot \nabla^{\mathcal{F}} L)\) | 85.62 | 79.03 | 82.02 |
3pt
9 Term-wise ablation study↩︎
To assess the effectiveness of the objective function presented in Eq. 12 , we perform an ablation study against each term. In particular, we compare ERM, \(L(\theta, \mathcal{D})\), SAM, \({L}_{p}(\theta; \mathcal{D})\), SAGM, \({L}(\theta; \mathcal{D}) + {L}_{p}(\theta; \mathcal{D}) - \alpha \big(\nabla_{} {L}_{p}(\theta; \mathcal{D}) \cdot \nabla_{} {L}_{}(\theta; \mathcal{D})\big)\) , and SAMPLe, that consists of all terms. All items outperform ERM in the HM column, underscoring the effectiveness of SAM-based methods for prompt learning. Moreover, the exploration term in SAMPLe consistently improves performance across Base, New, and HM, highlighting the importance of balanced learning that accounts for both exploitation and exploration in prompt learning.