1 Introduction↩︎

Vision-Language Models (VLMs) [1]–[8] such as CLIP [1] enable zero-shot image classification using natural-language text prompts that describe each class. In practice, the classification accuracy is sensitive to the quality of the prompts. To avoid labor-intensive manual prompt engineering, prior studies [9]–[13] have explored LLM-based automated prompt generation. A common practice is to query the Large Language Model with a generic instruction (, “What does a {class} look like?”) [10], resulting in prompts based solely on the LLM’s prior knowledge. However, such prompts often lack proper grounding in the target visual distribution and suffer from inaccuracies due to class-name ambiguity or hallucinations.

Recent studies [14], [15] address this lack of visual grounding by leveraging a small number of labeled images to facilitate the discovery of task-optimized prompts. To exploit these few-shot examples, these methods typically optimize a single aggregate metric (, overall validation accuracy) as their primary objective for prompt generation. However, the fundamental limitation is that such a metric can be dominated by a subset of easily recognizable samples. Once a candidate prompt correctly classifies these trivial instances, validation accuracy saturates prematurely, failing to capture the broader visual diversity of the target domain. As a result, these methods often overfit to the few-shot set and yield only marginal gains even as more labeled shots become available.

To address these limitations and effectively exploit few-shot supervision, we propose Text Prompt Boosting (TPB), drawing inspiration from the classic AdaBoost algorithm. Rather than optimizing a single prompt that quickly plateaus on easy samples, TPB iteratively builds an ensemble of natural-language prompts (as shown in [fig:fig1a]) by explicitly redirecting focus onto “hard examples.” At each boosting round, TPB reweights misclassified training examples, effectively forcing the selection of new prompts from an LLM-generated pool that resolve these specific errors. By systematically addressing failure modes, TPB maximizes the utility of few-shot examples to cover broader visual variations, successfully capturing model-agnostic task knowledge.

Extensive evaluations across diverse image classification benchmarks show that TPB consistently outperforms state-of-the-art baselines. [fig:fig1b] previews this transfer-robust shot scalability across five ViT-L-scale VLMs. By effectively exploiting few-shot supervision, TPB exhibits superior shot scalability on the source model. Furthermore, although natural-language prompts are inherently transferable across heterogeneous VLMs through simple re-embedding, prior methods often fail to retain shot-driven gains after transfer. In contrast, TPB preserves the shot-driven improvements observed on the source model when transferred to larger, more capable VLMs, maintaining clear performance gains even at higher shot counts.

Our main contributions are summarized as follows:

• We propose a few-shot prompting framework that combines AdaBoost with natural-language text prompts, yielding a boosted ensemble with significantly improved shot scalability over prior text-based methods.

• We show that the prompt ensembles transfer seamlessly across heterogeneous VLMs by simple re-embedding, achieving superior cross-model performance compared to existing transfer baselines.

• We conduct extensive experiments across standard few-shot benchmarks and multiple VLM backbones, accompanied by detailed component analyses.

2 Related Work↩︎

2.1 Prompt Adaptation of VLMs.↩︎

Prompting strategies for vision-language models broadly fall into soft (continuous) and hard (discrete) paradigms. While soft prompting, often referred to as prompt-learning, which attaches learnable continuous context vectors to class names for parameter-efficient adaptation [16]–[20], the learned prompts lie in a model-specific representation space, making them unreadable and difficult to transfer directly across models. In contrast, hard prompt optimization searches over discrete natural-language tokens, ensuring that the resulting prompts remain human-readable and reusable across heterogeneous CLIP-style encoders. To efficiently navigate the enormous language space, recent methods leverage LLMs to generate and refine prompts. For instance, DCLIP [9] and CuPL [10] construct task-specific descriptions, whereas further methods incorporate few-shot adaptation via conversational feedback in LLMbo [14] and evolution-based search in ProAPO [15]. Alternatively, PEZ [21] employs a gradient-based hard-prompt optimization technique. However, despite the advantages of hard prompting, most existing methods lack mechanisms to effectively leverage more than a single labeled example per class. As a result, their performance tends to saturate and improves only marginally, even when more labeled examples are provided. Our work follows the hard-prompt paradigm but introduces an AdaBoost-based framework to effectively exploit additional supervision while preserving interpretability and cross-model transfer.

2.2 Diverse Applications of AdaBoost.↩︎

Adaptive Boosting (AdaBoost) [22]–[24] sequentially combines many weak classifiers by iteratively reweighting training examples, emphasizing those misclassified by earlier classifiers. This general framework has been applied across diverse domains. In the computer vision domain, AdaBoost has been widely used in fast detection pipelines, such as cascaded detectors [25] and integral channel feature-based detectors [26]. In text classification and information retrieval, boosting aggregates weak lexical predicates into confidence-rated document classifiers [27] and optimizes pairwise orderings for ranking [28]. Unlike these applications, we address cross-modal image recognition with VLMs, treating class-wise text prompts as weak classifiers that map an image to class-wise similarity scores.

3 Method↩︎

We propose Text Prompt Boosting (TPB), a prompt-ensemble framework for shot-scalable and transfer-robust few-shot adaptation of pretrained VLMs. TPB treats a class-wise prompt collection as a text-based weak classifier and builds a strong classifier via an AdaBoost-style ensemble over multiple boosting rounds. We begin with preliminaries in 3.1, then introduce large prompt pools in 3.2 and our weak learner, Greedy Prompt Composition (GPC), in 3.3. Finally, we describe how GPC is integrated into the TPB boosting loop in 3.4. An overview of TPB is shown in 1.

3.1 Preliminaries↩︎

3.1.1 VLM Classification with Text Prompts.↩︎

We consider \(K\)-way classification for an image \(\mathbf{x}\). Given a pretrained VLM with image encoder \(E_{\mathrm{img}}\) and text encoder \(E_{\mathrm{text}}\), we form a CLIP-style linear classifier whose weights are text features derived from class prompts \(t_k\). Specifically, the logit for class \(k\) is computed by cosine similarity between \(E_{\mathrm{img}}(\mathbf{x})\) and \(E_{\mathrm{text}}(t_k)\): \[s(\mathbf{x}, t_k) := \cos \bigl( E_{\mathrm{img}}(\mathbf{x}) , E_{\mathrm{text}}(t_k) \bigr) \, . \label{eq:single95sim}\tag{1}\] Instead of relying on a single prompt, multiple prompts can be grouped into a class-wise prompt bank \(B_k = \{ t_{k,1}, \dots, t_{k,n_k} \}\), and compute the class score by simple averaging: \[s(\mathbf{x}, B_k) := \frac{1}{\lvert B_k \rvert} \sum_{t_k \in B_k} s(\mathbf{x}, t_k) \, . \label{eq:avg95sim}\tag{2}\] Let \(\mathcal{B} := \{B_1, \ldots, B_K\}\) denote the collection of class-wise prompt banks. Inference on an image \(\mathbf{x}\) is then performed using this prompt collection with a fixed temperature \(\tau > 0\): \[p(y = k \mid \mathbf{x}; \mathcal{B}) := \frac{\exp \bigl(s(\mathbf{x}, B_k)/\tau\bigr)}{\sum_{i=1}^{K}\exp\bigl(s(\mathbf{x}, B_i)/\tau\bigr)} \, . \label{eq:prob95scalar}\tag{3}\] Based on these probabilities, the predicted class label is determined by selecting the class with the highest probability: \[\hat{y}(\mathbf{x}; \mathcal{B}) := \mathop{\mathrm{argmax}}_{k\in\{1,\dots,K\}} p(y = k \mid \mathbf{x}; \mathcal{B}) \, . \label{eq:vlm95pred}\tag{4}\]

While prior text-prompting studies [9]–[11], [14], [15] construct only a single collection \(\mathcal{B}\), our TPB framework sequentially constructs multiple diverse collections \(\{\mathcal{B}^{(m)}\}_{m=1}^{M}\). Each collection defines a weak classifier in an AdaBoost-style ensemble, enabling us to explicitly target hard examples while preserving the model transferability of discrete text prompts.

3.1.2 AdaBoost Overview.↩︎

AdaBoost [22] is a prominent ensemble algorithm that combines weak learners into a strong learner by iteratively training weak classifiers on reweighted data. Unlike standard AdaBoost, which typically uses decision trees as weak learners, our framework uses the VLM classifier induced by a prompt collection \(\mathcal{B}\) as the weak learner. At each boosting round \(m \in \{1,\ldots,M\}\), we construct a prompt collection \(\mathcal{B}^{(m)}\). Since the weak classifier is fully specified by the VLM predictions induced by \(\mathcal{B}^{(m)}\), the objective at round \(m\) is to minimize the weighted classification error on the training set \(\mathcal{D}=\{(\mathbf{x}_i,y_i)\}_{i=1}^{n}\): \[\mathcal{L}^{(m)} := \sum_{i=1}^{n} w_i^{(m)} \,\mathbb{I}\!\left[ y_i \neq \hat{y}(\mathbf{x}_i; \mathcal{B}^{(m)}) \right] \, , \label{eq:weighted95error}\tag{5}\] where \(w_i^{(m)}\) is the weight assigned to the \(i\)-th image at round \(m\), indicating how much the current prompt collection should focus on it, and \(\hat{y}\) denotes the predicted class defined in 4 .

After obtaining \(\mathcal{B}^{(m)}\), we update the sample weights for the next round by increasing the weights of misclassified (hard) samples and decreasing those of correctly classified ones. This fitting and reweighting process repeats, encouraging subsequent prompt collections to prioritize the remaining difficult examples. The final prediction is obtained by aggregating the outputs from all \(M\) rounds. For example, aggregation can be performed via weighted majority voting: \[\hat{y}_{\mathrm{final}} = \operatorname*{arg\,max}_{k} \sum_{m=1}^{M} \alpha^{(m)} \cdot \mathbb{I}\!\left[ \hat{y}(\mathbf{x}; \mathcal{B}^{(m)}) = k \right] \, , \label{eq:final95pred95hard95voting}\tag{6}\] where \(\alpha^{(m)}\) is the weight for the \(m\)-th weak learner. The specific reweighting and aggregation rules depend on the AdaBoost variant; we adopt SAMME.R [23], a multi-class real-valued extension, which we refer to simply as AdaBoost in the rest of this paper.

3.2 Large Prompt Pools↩︎

Before initiating the boosting loop, we first construct a comprehensive prompt pool \(\mathcal{P}=\{P_1,\dots,P_K\}\) for each class \(k\in\{1,\ldots,K\}\). To ensure sufficient expressive power, these prompt pools encompass both simple template-based prompts (, “a photo of a beagle.”) and richer, more detailed sentences generated by LLMs (, “beagles have large, floppy ears that hang down to the sides of their face.”). Concretely, we utilize the 80 hand-crafted templates provided by CLIP [1], alongside LLM-generated sentences released by several prior studies [9], [10], [12], [29], [30]. To further expand the discrete hypothesis space and enrich semantic diversity, we also append concatenations of these templates and descriptive sentences to each pool.

3.3 Greedy Prompt Composition↩︎

At each boosting round \(m\in\{1,\dots,M\}\), our framework constructs a collection \(\mathcal{B}^{(m)} = \{B_k^{(m)}\}_{k=1}^K\) of class-specific prompt banks, where each \(B_k^{(m)}\) is selected from the class-specific prompt pool \(P_k\). Exhaustively evaluating all joint prompt combinations over the \(K\) classes is computationally infeasible; thus, we adopt a two-stage greedy strategy, which we refer to as Greedy Prompt Composition (GPC). For notational simplicity, we omit the round index in this subsection and write \(B_k\) and \(w_i\) instead of \(B_k^{(m)}\) and \(w_i^{(m)}\).

3.3.1 Stage 1: Single-Template Initialization.↩︎

We initialize the prompt collection \(\mathcal{B}=\{B_k\}_{k=1}^K\) with a single shared template. Among the 80 standard templates \(\Phi_{\mathrm{template}}=\{\phi_1,\dots,\phi_{80}\}\), we select \(\phi^\star\) that minimizes the weighted classification error. Concretely, for each candidate template \(\phi\in\Phi_{\mathrm{template}}\), we form a template-induced collection \(\mathcal{B}(\phi)=\{B_k(\phi)\}_{k=1}^K\) by setting \(B_k(\phi)=[\,\phi(c_k)\,]\), where \(c_k\) denotes the label text of class \(k\). We then compute the weighted error of the resulting classifier (under the current example weights) and choose the best template. If multiple templates attain the same minimum error, we break ties randomly. Accordingly, we define \[\mathcal{B}_{\mathrm{init}} := \{B_k\}_{k=1}^K, \quad B_k := [\, \phi^\star(c_k) \,] \;\;\forall k \in \{1,\dots,K\}.\] We adopt this single-template initialization for two reasons. First, any classifier requires at least one prompt per class, and a shared template provides this with minimal overhead by simply inserting the class name. Second, using one template consistently across classes already yields a reasonably strong baseline, which serves as a stable starting point for greedy refinement and promotes fast convergence.

3.3.2 Stage 2: Class-wise Greedy Prompt Insertion.↩︎

Starting from the single-template initialization \(\mathcal{B}_{\mathrm{init}}\), we greedily expand the prompt banks by iteratively adding prompts to each class. Specifically, given the current prompt bank collection \(\mathcal{B}\) and class \(k\), we evaluate each candidate prompt \(t \in P_k\) by temporarily appending it to the bank, , \(\widetilde{B}_k(t) = B_k \| [t]\), and computing the resulting weighted error. Note that the class score is computed as an unweighted average over the prompts in a bank as defined in 2 . Under this formulation, adding a single prompt to an already populated bank may induce only a small change in the class score, which can be insufficient to resolve the remaining hard examples. Rather than introducing learnable continuous weights, which would require gradient updates, we implicitly approximate continuous weighting within a strictly discrete space by permitting duplicate selections of the same prompt; since the score is an unweighted average, duplicating a prompt increases its effective contribution. Ultimately, we select the candidate \(t_k^\star\) that yields the largest reduction in the weighted error, updating \(B_k \leftarrow B_k \| [t_k^\star]\) if such a candidate exists, or leaving \(B_k\) unchanged otherwise. We repeat this procedure cyclically across all classes \(k = 1, \dots, K\) until a full pass over all classes yields no further updates, resulting in the optimized bank collection \(\mathcal{B}^\star\).

3.4 Integration into the TPB Framework↩︎

3.4.1 Boosting-Loop Image Augmentation.↩︎

To induce informative variations and prevent overfitting, which occurs when the ensemble quickly memorizes the limited samples, we apply image augmentation to the few-shot dataset \(\mathcal{D}=\bigl\{(\mathbf{x}_i,y_i)\bigr\}_{i=1}^n\) before executing the GPC algorithm. With an augmentation factor \(a \in \mathbb{N}\), we generate randomly transformed copies for each training image \(\mathbf{x}_i\) using simple operations (, random resized crop and horizontal flip), resulting in an augmented dataset \(\mathcal{D}_{\mathrm{aug}}\) of size \(an\). We replicate the original sample weight \(w_i\) for each copy and renormalize the weight vector \(\mathbf{w}_{\mathrm{aug}}\) to sum to one. Consequently, the weighted errors evaluated during the GPC stages are computed over this augmented dataset \(\mathcal{D}_{\mathrm{aug}}\) and its corresponding weights \(\mathbf{w}_{\mathrm{aug}}\).

3.4.2 Iterative Weight Update and Final Ensemble.↩︎

In the overall TPB framework, the prompt bank collection \(\mathcal{B}^{\star(m)}\) obtained from GPC serves as a weak classifier for boosting round \(m\). We then apply this weak classifier to the original (non-augmented) few-shot dataset \(\mathcal{D}\) to update the sample weights \(\mathbf{w}^{(m)}\) to \(\mathbf{w}^{(m+1)}\), increasing the weights of misclassified examples and decreasing those of correctly classified ones. Using these updated weights, the next boosting round constructs a new weak classifier that focuses more intensely on the previously misclassified samples. Note that we initialize the weights uniformly in the first round, i.e., \(w_i^{(1)} = 1/|\mathcal{D}|\) for all \((\mathbf{x}_i,y_i)\in\mathcal{D}\). After the final round \(M\), all weak classifiers are aggregated into a strong classifier. For the reweighting and aggregation rules, we adopt SAMME.R [23], a multi-class variant of AdaBoost. Detailed pseudocodes are provided in the supplementary material.

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.1 Datasets.↩︎

Following prior text prompting  [9], [10], [14], [15] and prompt-learning studies [16], [19], we consider eleven image classification datasets that cover a wide range of domains and granularity: ImageNet-1K [31] and Caltech101 [32] for generic object classification, OxfordPets [33], StanfordCars [34], Flowers102 [35], Food101 [36], FGVCAircraft [37] for fine-grained image classification, SUN397 [38] for scene recognition, DTD [39] for texture classification, EuroSAT [40] for satellite image classification, and UCF101 [41] for action classification.

4.1.2 Implementation Details.↩︎

The number of boosting rounds \(M\) is set to \(50\), except for ImageNet, where it is set to \(30\), and the augmentation factor \(a\) is set to \(4\). Our framework is implemented based on the AdaBoost implementation in scikit-learn [42]. For the temperature parameter \(\tau\) in 3 , we use a fixed value \(\tau = 1\). Note that during inference, the text embeddings of the ensembled prompts can be pre-computed. Thus, for any new image, TPB only requires a single forward pass, ensuring that our method does not incur significant latency overhead compared to other baselines. All results, except for LLMbo [14] as described in 4.2.1, are averaged over three random seeds; full shot-wise and target-wise tables with standard deviations are in the supplementary material.

4.2 Few-Shot Adaptation and Shot Scalability↩︎

2pt

4.2.1 Text-based Baselines.↩︎

For zero-shot baselines, we employ the standard template “a photo of a {class}.” (denoted as CLIP), along with DCLIP [9] and CuPL [10]. For few-shot baselines, we compare our method against LLMbo [14], ProAPO [15], and a gradient-based hard-prompt optimization method, PEZ [21]. Except for LLMbo, all baseline results are reproduced using the authors’ official implementations. Since PEZ was not originally designed for our task, we reimplement it for image classification; implementation details are provided in the supplementary material.

4.2.2 Shot Scalability.↩︎

1 compares our TPB with existing text-based methods on eleven datasets using OpenAI CLIP [1] with a ResNet-50 backbone [43]. While other text-based baselines show only marginal or no gains as more shots become available, our TPB achieves both strong few-shot performance and clear improvements with additional labeled data. At one shot, TPB slightly outperforms ProAPO by about 1.1 percentage points (pp) in average accuracy. When the number of shots increases from one to sixteen, TPB gains 6.5 pp and outperforms ProAPO and other text-based baselines by a substantial margin. PEZ benefits from additional shots and improves by 18.7 pp between one and sixteen shots, but it remains clearly worse than other text-based baselines at one shot and only approaches their performance at sixteen shots.

We also compare TPB with the continuous prompt-learning method CoOp [16]. While CoOp achieves higher accuracy at larger shot counts, this comes at the cost of being coupled to the model’s specific representation space. In contrast, TPB preserves the universal nature of language, unlocking robust cross-model transferability where continuous prompts fail as further discussed in 4.3.

4.2.3 Qualitative Analysis: Targeting Hard Samples.↩︎

To visualize how TPB iteratively builds an ensemble by explicitly targeting misclassified images, we present a qualitative example of the GPC algorithm on a two-class task distinguishing beagle from basset hound. As shown in 3 (a), GPC constructs weak classifiers by selecting prompts from the pool that capture dominant, easily recognizable features. For the beagle, it highlights “large”, “floppy ears,” while for the basset hound, it emphasizes “low stature” and “front legs slightly splayed apart.”

While this prompt bank correctly classifies standard images as depicted in 3 (b), it fails on a challenging hard sample, a basset hound lying down where primary discriminative visual cues like legs and overall stature are heavily occluded. In a standard text-prompting paradigm, this error would likely remain unresolved. However, within the TPB framework, this specific misclassification is clearly penalized, assigning the image a higher sample weight in the subsequent boosting round. Consequently, the next iteration of GPC is explicitly forced to search for alternative textual cues, such as facial wrinkles or distinct snout shapes, specifically to correct this unresolved error. This highlights the underlying mechanism of TPB: rather than relying on a single static validation objective, the framework adapts to challenging samples by incrementally assembling diverse natural-language prompts.

4.3 Transfer Robustness Across Heterogeneous VLMs↩︎

4pt

4.3.1 Source and Target VLMs.↩︎

In our cross-model transfer experiments, we distinguish between ‘source models’ (used for prompt optimization) and ‘target models’ (used for evaluation). Note that the concept of a source model is not applicable to zero-shot methods, as they do not involve model-specific optimization. As sources, we use OpenAI CLIP [1] with two backbones: ResNet-50 and ViT-B/32. For target models, we use OpenCLIP [3], EVA-02-CLIP [8], CLIPA-v2 [4], DFN [7], SigLIP2 [5], and MetaCLIP2 [6]. These VLMs are released with various backbones, which we group by their size (, ViT-L, ViT-H) and report averaged results for each group. This setup covers a range of architectures, scales, and training recipes, allowing us to assess model-transfer performance across heterogeneous VLMs.

4.3.2 Transfer Baselines.↩︎

For model-transfer experiments, we evaluate both the text-based methods and prompt-learning baselines. As a representative prompt-learning method, we use PromptSRC [19], which adapts VLMs by learning visual and textual prompt tokens while keeping the backbone parameters frozen. Since these learned vectors cannot be directly reused in other VLMs with different architectures or representation spaces, we additionally employ emulated fine-tuning (EFT) [44] proposed in the language domain. EFT combines a large pretrained model with a smaller fine-tuned model by reweighting logits, effectively emulating the result of fine-tuning the large model without updating its parameters. This allows us to transfer the adaptation learned by PromptSRC to other VLMs in a training-free manner, but the inference overhead increases due to multiple forward passes (see the supplementary material for details).

4.3.3 Transfer-robust Shot Scalability.↩︎

2 reports cross-model transfer results when all methods operate on OpenAI CLIP ViT-B/32 with one to sixteen shot supervision and then evaluated on larger target models (ViT-L and ViT-H). On ViT-L targets, TPB improves the average accuracy from 76.76% (zero-shot CLIP) to 82.07%; on ViT-H targets, it improves from 78.75% to 84.24%. TPB also outperforms ProAPO, the strongest text-based baseline, by 2.06 pp on ViT-L targets and 1.85 pp on ViT-H targets. Notably, TPB also maintains a performance margin over CoOp+EFT and PromptSRC+EFT. While continuous methods like CoOp can achieve higher accuracy on the source model by overfitting to its specific representation space, they fail to preserve this advantage during transfer. In contrast, TPB achieves superior transferability and scalability while requiring only a single forward pass during inference, whereas EFT-based transfer necessitates multiple inference passes. Moreover, TPB exhibits superior transfer-robust shot scalability. While ProAPO’s performance gains from one-shot to sixteen-shot are limited to 0.42 pp on ViT-L and 0.32 pp on ViT-H, TPB achieves significantly higher improvements of 2.41 pp and 2.51 pp, respectively. This contrast demonstrates that TPB consistently preserves shot-driven gains where other text-based baselines saturate, capturing architecture-agnostic task knowledge that remains robust during cross-model transfer. Conversely, PEZ, which optimizes discrete prompts via gradients, transfers poorly and falls even below zero-shot performance, as we discuss in the following analysis.

Additional shot-wise transfer results on larger target VLMs (, ViT-E and ViT-G) are provided in the supplementary material. Consistent with 2, they show that TPB preserves shot-driven gains after transfer.

4.3.4 Failure Analysis of Gradient-optimized Prompts.↩︎

4pt

3 reports the sixteen-shot average accuracy over eleven datasets when adapting OpenAI CLIP RN50 and transferring the resulting prompts to OpenAI CLIP ViT-L/14. On the source model, PEZ improves zero-shot CLIP from \(55.98\%\) to \(61.09\%\), indicating that it successfully adapts the RN50 backbone to the training tasks. However, the learned prompts transfer poorly: on the ViT-L/14 target, accuracy drops from \(71.07\%\) (zero-shot CLIP) to \(53.02\%\), well below even the zero-shot baseline. To understand why PEZ transfers so poorly despite using text prompts, we inspect the optimized context tokens. For example, for the pug class in OxfordPets, PEZ yields the following prompts:

“hey darby pls violets kissestc accept liza any,,
adorable gorgeous ♡♡ :-) life behaved pug”

Here, every token except the final class name “pug” is obtained by gradient updates during optimization. The resulting context is a sequence of seemingly arbitrary words, symbols, and non-word fragments with no coherent natural-language meaning. Such prompts behave like model-specific codes in the RN50 representation space, rather than semantically meaningful descriptions that a human could interpret.

This observation suggests that the ability of a prompt to transfer across models does not come simply from it being text. Instead, prompts are more transferable when they stay close to fluent, human-readable natural-language. Prompts that drift too far from human-interpretable language may overfit a particular model and therefore fail to transfer.

4.4 Ablation Studies↩︎

4.4.1 Effect of Augmentation and Prompt Pool Diversity.↩︎

In [fig:ablation95augmentation], increasing the augmentation factor \(a\) consistently improves accuracy over using no augmentation, with the largest gains in the low-shot setting. Here, \(a\) controls the number of random transformations per training example, and \(a=0\) corresponds to using only the fixed few-shot set without any augmentation. Without augmentation, AdaBoost quickly fits the small training set within a few rounds, leaving little signal for subsequent boosting rounds, whereas randomly transformed views of the same images keep introducing informative errors, enabling later weak classifiers to correct them and yielding a stronger ensemble. [fig:ablation95bank] then examines the impact of the diversity of the prompt pool, constructed by combining several groups of LLM-generated descriptions. Using both CuPL and DCLIP descriptions yields better performance than using CuPL alone, even though DCLIP is individually weaker, as reflected by its lower accuracy in 1. Incorporating all available descriptions further improves accuracy, especially at higher shot counts, suggesting that a more diverse prompt pool helps TPB better exploit additional labeled examples as they become available.

4.4.2 Dynamics of Augmentation Across Boosting Rounds.↩︎

5 compares AdaBoost with and without image augmentation. Without augmentation ([fig:augmenttesta]), training the ensemble rapidly memorizes the few available examples, pushing training accuracy close to \(100\%\) within only a few rounds. Once training accuracy saturates, the reweighted distribution no longer highlights informative mistakes, preventing AdaBoost from meaningfully updating its weak learners; consequently, test accuracy remains almost unchanged across subsequent rounds. In contrast, when applying image augmentation with a factor of \(a{=}4\) ([fig:augmenttestb]), each boosting round observes newly transformed views of the same images. These variations prevent the model from perfectly fitting the training set, keeping training accuracy below saturation and ensuring that each round still produces informative errors for reweighting. This continual supply of hard examples stabilizes the boosting process and allows test accuracy to improve steadily over many rounds, demonstrating that augmentation is essential for effective boosting under low-shot supervision.

4.4.3 Effect of the Number of Weak Classifiers.↩︎

4pt

4 analyzes how the performance of TPB varies with the number of weak classifiers \(M\). With only a single weak classifier (\(M{=}1\)), TPB is slightly weaker than ProAPO in the one-shot setting. However, once \(M\) reaches \(10\), TPB already matches or exceeds ProAPO at both one-shot and sixteen-shot. Increasing \(M\) further to \(30\) or \(50\) provides additional but diminishing improvements, suggesting that performance effectively saturates in the range of \(M{=}30\)-\(50\).

4.4.4 Boosting rounds . augmentation.↩︎

4pt

To isolate boosting from image augmentation, we fix the total exposure budget \(M{\cdot}a{=}200\) and sweep \((M, a)\) as shown in 5. This sweep is conducted in the 16-shot setting over ten datasets excluding ImageNet, using OpenAI CLIP ViT-B/32 as the source model and averaging transfer accuracy over the same target groups as in 2: five ViT-L-scale and four ViT-H-scale VLMs. Under this fixed exposure budget, increasing \(a\) while reducing \(M\) improves source accuracy but degrades transfer accuracy. This comparison shows that transfer robustness is driven by iterative boosting rather than by augmented exposure alone.

5 Conclusion↩︎

We introduce Text Prompt Boosting (TPB), a boosting-based framework that ensembles natural-language prompts for shot-scalable and transfer-robust few-shot adaptation of vision-language models. By treating a class-wise prompt bank collection as a weak classifier and employing the Greedy Prompt Composition (GPC) procedure as the weak learner, TPB incrementally builds an ensemble of prompt-based classifiers that explicitly focus on hard examples while preserving interpretability and transferability. Consequently, the resulting strong classifier exhibits improved shot scalability on the source model, while simultaneously achieving transfer robustness, successfully preserving its shot-driven performance gains even when evaluated on various larger heterogeneous VLMs.

5.0.1 Limitations:↩︎

Despite its effectiveness, TPB has limitations. First, its construction cost can be non-negligible on large-scale datasets, where the candidate space is substantially larger. Second, AdaBoost-style reweighting may overemphasize noisy or atypical few-shot samples. Finally, TPB selects prompts from a fixed pool, which may limit its flexibility for fine-grained or ambiguous classes.

Acknowledgements↩︎

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (Nos. RS-2019-II191906, Artificial Intelligence Graduate School Program; RS-2024-00457882, AI Research Hub Project; RS-2026-25511821, Development of Personalized Media Service Recommendation and Generative Technology), by a grant (RS-2025-00564342) from the Korea Institute for Advancement of Technology (KIAT), funded by the Ministry of Trade, Industry and Energy (MOTIE), and by Seoul R&D Program (SP240008) through the Seoul Business Agency (SBA) funded by the Seoul Metropolitan Government.

Supplementary Materials↩︎

This supplementary material provides (A) further details of the Greedy Prompt Composition (GPC); (B) a summary of the SAMME.R algorithm used in our boosting framework; (C) experimental settings, including hardware setup and baseline implementation details; and (D) additional experiments and analyses.

6 Methodology Details↩︎

In 3.3, we introduce Greedy Prompt Composition (GPC) as the weak learning algorithm used in our AdaBoost framework. At boosting round \(m\), GPC constructs a weak classifier by composing class-specific prompt banks \(B_k^{(m)}\) drawn from the large prompt pool \(P_k\) for each class \(k \in \{1,\dots,K\}\). The procedure consists of two stages: (i) single-template initialization and (ii) class-wise greedy prompt insertion, which repeatedly selects the prompt that most reduces the weighted error in 5 . 6 summarizes the overall flow. The remainder of this section details Stage 2, , the class-wise greedy prompt insertion.

6.1 Prompt Duplication↩︎

Given the current prompt-bank collection \(\mathcal{B} = \{B_1, \dots, B_K\}\) and a target class \(k\), each iteration of Stage 2 selects a prompt \(t^\star\) from the pool \(P_k\) and appends \(d \in \mathbb{N}\) copies of it to the current bank \(B_k\). Let \(n_k = |B_k|\) denote the current size of \(B_k\), and let \(B_k^{(t,d)}\) denote the bank obtained by adding \(d\) copies of a candidate prompt \(t \in P_k\) to \(B_k\). Since the class score \(s(\mathbf{x}, B_k)\) is defined as the unweighted average similarity over the prompts in a bank, adding \(d\) copies of \(t\) changes the score for class \(k\) on an input \(\mathbf{x}\) to: \[s(\mathbf{x}, B_k^{(t,d)}) = \frac{n_k\,s(\mathbf{x}, B_k) + d\,s(\mathbf{x}, t)}{n_k + d}. \label{eq:score-update}\tag{7}\] While we can set \(d=1\) (no duplication), adding only a single copy to an already populated bank often yields a negligible change in the class score and fails to improve the weighted error. To better capture the influence of each candidate, we simply allow for duplicate selections of the same prompt. This approach serves as a discrete approximation of continuous weighting, allowing us to adjust the effective contribution of a prompt without introducing learnable weights or gradient updates.

6.2 Prompt Selection with Duplication↩︎

We now derive a closed-form expression for the change in weighted error when \(d\) copies of a candidate prompt \(t\in P_k\) are appended to \(B_k\). Let \(y_i\) denote the ground-truth label of \(\mathbf{x}_i\), \(\hat{y}_i\) denote the prediction of the current classifier, and let \(\hat{y}_i^{\mathrm{2nd}}\) denote the class with the second largest class score: \[\begin{align} \hat{y}_i &:= \mathop{\mathrm{argmax}}_{j \in \{1,\dots,K\}} s(\mathbf{x}_i, B_j)\;, \\ \hat{y}_i^{\mathrm{2nd}} &:= \mathop{\mathrm{argmax}}_{j \in \{1,\dots,K\} \setminus \{\hat{y}_i\}} s(\mathbf{x}_i, B_j)\;. \end{align}\] Since modifying \(B_k\) only changes the score of class \(k\), the prediction for each \(\mathbf{x}_i\) can change in only three ways: (i) no change, (ii) an upside flip (\(k\) overtakes \(\hat{y}_i\)), or (iii) a downside flip (\(k = \hat{y}_i\) and \(k\) is overtaken by \(\hat{y}_i^{\mathrm{2nd}}\)).

6.2.1 Upside flips.↩︎

An upside flip for sample \(i\) occurs when \(\hat{y}_i \neq k\) and the updated score of class \(k\) overtakes that of the current winner: \[\begin{align} s(\mathbf{x}_i, B_k^{(t,d)}) &= \frac{n_k s(\mathbf{x}_i, B_k) + d\,s(\mathbf{x}_i, t)}{n_k + d} > s(\mathbf{x}_i, B_{\hat{y}_i}) \\ &\iff \frac{n_k\big(s(\mathbf{x}_i, B_k)-s(\mathbf{x}_i, B_{\hat{y}_i})\big)}{s(\mathbf{x}_i, B_{\hat{y}_i})-s(\mathbf{x}_i,t)} < d, \end{align}\] which is meaningful only when \(s(\mathbf{x}_i, B_{\hat{y}_i}) < s(\mathbf{x}_i, t)\); otherwise, no finite \(d\) can make class \(k\) overtake \(\hat{y}_i\). The minimum integer \(d\) that causes an upside flip for \(\mathbf{x}_i\) is therefore \[d_i^{\uparrow}(t) := \begin{cases} \begin{align} &\left\lfloor \dfrac{n_k\big(s(\mathbf{x}_i, B_k)-s(\mathbf{x}_i, B_{\hat{y}_i})\big)}{s(\mathbf{x}_i, B_{\hat{y}_i})-s(\mathbf{x}_i,t)}\right\rfloor + 1\;, \\ &\qquad \text{if } \hat{y}_i\neq k \text{ and } s(\mathbf{x}_i,B_{\hat{y}_i}) < s(\mathbf{x}_i, t)\;, \end{align} \\\\ \begin{align} &+\infty\;, \\ &\qquad \text{otherwise\;,} \end{align} \end{cases}\]

where \(+\infty\) indicates that upside flips for sample \(i\) never occur, so that \(\mathbb{I}[d \ge d_i^{\uparrow}(t)] = 0\) for all finite \(d\).

Upside flips are beneficial when they correct a misclassified sample (, when \(y_i = k\)), and harmful when they change a previously correct prediction into an incorrect one (, when \(y_i = \hat{y}_i\) and \(y_i \neq k\)). We therefore define the potentially beneficial and harmful index sets for upside flips as \[\mathcal{B}^\uparrow_k := \{i : y_i = k\}, \qquad \mathcal{H}^\uparrow_k := \{i : y_i = \hat{y}_i,\; y_i \neq k\}.\]

6.2.2 Downside flips.↩︎

A downside flip for sample \(i\) occurs when \(\hat{y}_i = k\) and the updated score of class \(k\) falls below that of the second-best class: \[\begin{align} s(\mathbf{x}_i, B_k^{(t,d)}) &= \frac{n_k s(\mathbf{x}_i, B_k) + d\,s(\mathbf{x}_i, t)}{n_k + d} < s(\mathbf{x}_i, B_{\hat{y}_i^\mathrm{2nd}}) \\ &\iff \frac{n_k\big(s(\mathbf{x}_i, B_k)-s(\mathbf{x}_i, B_{\hat{y}_i^\mathrm{2nd}})\big)}{s(\mathbf{x}_i, B_{\hat{y}_i^\mathrm{2nd}})-s(\mathbf{x}_i,t)} < d, \end{align}\] which is meaningful only when \(s(\mathbf{x}_i, t) < s(\mathbf{x}_i,B_{\hat{y}_i^\mathrm{2nd}})\). The minimum integer \(d\) that causes a downside flip for \(\mathbf{x}_i\) is \[d_i^{\downarrow}(t) := \begin{cases} \begin{align} &\left\lfloor \dfrac{n_k\big(s(\mathbf{x}_i, B_k)-s(\mathbf{x}_i, B_{\hat{y}_i^\mathrm{2nd}})\big)}{s(\mathbf{x}_i, B_{\hat{y}_i^\mathrm{2nd}})-s(\mathbf{x}_i,t)}\right\rfloor + 1\;, \\ &\qquad \text{if } \hat{y}_i= k \text{ and } s(\mathbf{x}_i, t) < s(\mathbf{x}_i,B_{\hat{y}_i^\mathrm{2nd}})\;, \end{align} \\\\ \begin{align} &+\infty\;, \\ &\qquad \text{otherwise\;.} \end{align} \end{cases}\] Downside flips are beneficial when they fix a previously incorrect prediction (, when the new prediction \(\hat{y}_i^\mathrm{2nd}\) equals \(y_i\)) and harmful when they corrupt a previously correct prediction (, when \(y_i = k\)). Accordingly, we define \[\mathcal{B}^\downarrow_k := \{i : y_i = \hat{y}_i^\mathrm{2nd}\}, \qquad \mathcal{H}^\downarrow_k := \{i : y_i = k\}.\]

6.2.3 Change in weighted error.↩︎

Let \(\varepsilon(\mathcal{B})\) denote the weighted error in 5 of the classifier defined by the prompt collection \(\mathcal{B}\), and let \(\mathcal{B}^{(k,t,d)}\) denote the collection obtained by replacing \(B_k\) with \(B_k^{(t,d)}\). The change in weighted error when adding \(d\) copies of \(t\) to \(B_k\) is \[\Delta \varepsilon(k,t,d) := \varepsilon\big(\mathcal{B}^{(k,t,d)}\big) - \varepsilon(\mathcal{B})\;.\] Recall that adding copies of \(t\) only modifies the score of class \(k\). Thus, as we vary the candidate prompt \(t\) and the duplication factor \(d\), the only way the weighted error can change is when the prediction for some training example flips to a different class. As described above, every such change in prediction is either an upside flip (from \(\hat{y}_i\) to \(k\)) or a downside flip (from \(k\) to \(\hat{y}_i^\mathrm{2nd}\)). Consequently, the change in weighted error can be decomposed into contributions from these upside and downside flips: \[\begin{align} \Delta \varepsilon(k,t,d) &= \sum_{i\in \mathcal{H}^\uparrow_k} w_i\,\mathbb{I}\big[d\geq d^\uparrow_i(t)\big] + \sum_{i\in \mathcal{H}^\downarrow_k} w_i\,\mathbb{I}\big[d\geq d^\downarrow_i(t)\big] \\ &- \sum_{i\in \mathcal{B}^\uparrow_k} w_i\,\mathbb{I}\big[d\geq d^\uparrow_i(t)\big] - \sum_{i\in \mathcal{B}^\downarrow_k} w_i\,\mathbb{I}\big[d\geq d^\downarrow_i(t)\big]\;. \end{align} \label{eq:delta-eps}\tag{8}\] The first two sums aggregate harmful flips (\(i\in\mathcal{H}^\uparrow_k \cup \mathcal{H}^\downarrow_k\)), which increase the weighted error once their indicators become 1. The last two sums aggregate beneficial flips (\(i\in\mathcal{B}^\uparrow_k \cup \mathcal{B}^\downarrow_k\)) with a minus sign, so they decrease the weighted error when they occur. As a function of \(d\), \(\Delta \varepsilon(k,t,d)\) is a right-continuous step function whose value can change only at the (finite) integer thresholds where upside or downside flips occur: \[\begin{align} \mathcal{T}(t) := &\big\{d^\uparrow_i(t) \mid 1\le i \le n,\;d^\uparrow_i(t)<\infty\big\}\\ &\cup \big\{d^\downarrow_i(t) \mid 1\le i \le n,\;d^\downarrow_i(t)<\infty\big\}\;. \end{align}\] Hence we can restrict our search for the optimal number of copies \(d\) to \(\mathcal{T}(t)\) instead of the infinite set \(\mathbb{N}\).

We choose the optimal number of copies for candidate prompt \(t\) as \[d(t) = \mathop{\mathrm{argmin}}_{d \in \mathcal{T}(t)} \frac{\Delta \varepsilon(k,t,d)}{d}\;, \label{eq:dup-opt}\tag{9}\] where ties are broken in favor of smaller \(d\); see 6.3 for motivation. Using this, the best prompt for class \(k\) is selected as \[t^\star = \mathop{\mathrm{argmin}}_{t\in P_k} \frac{\Delta \varepsilon\big(k,t,d(t)\big)}{d(t)}. \label{eq:best-prompt}\tag{10}\] If the corresponding error change is negative, , \(\Delta \varepsilon\big(k,t^\star,d(t^\star)\big) < 0\), we update the current prompt bank as \(B_k \leftarrow B_k^{(t^\star, d(t^\star))}\) and proceed to the next class in 6.

6.3 Penalty on Larger Duplication↩︎

Prior work on text prompts for VLMs suggests that using a diverse set of prompts for each class improves robustness and generalization [9]–[11]. Because class scores are defined as averages over prompts in \(B_k\), prompts whose similarity \(s(\mathbf{x},t)\) is close to the current mean \(s(\mathbf{x},B_k)\) have low influence and only affect the weighted error when duplicated many times, which in turn causes \(B_k\) to be dominated by repeated copies of such prompts and harms diversity. To encourage diversity in the prompt banks, we penalize larger duplication by normalizing the error change by the duplication factor and using \(\Delta\varepsilon(k,t,d)/d\) as the objective in 9 and 10 .

7 SAMME.R Algorithm↩︎

SAMME.R [23] is a multi-class extension of AdaBoost [22] that uses real-valued, confidence-rated weak classifiers. It can be derived via forward stagewise additive modeling with a multi-class exponential loss under the symmetric \(K\)-class label coding. The full procedure is given in 7. At initialization (a), sample weights are uniformly set to \(1/n\) and each label \(y_i\) is recoded into a \(K\)-dimensional vector \(\tilde{\mathbf{y}}_i\). At boosting round \(m\), (b) a classifier is fit to the training data using the current weights, which outputs \(K\)-class probability estimates under the weighted empirical distribution. The fitting algorithm is typically a decision tree; in our framework we propose the GPC algorithm. (c) Then the sample weights are updated using the log-probabilities and this fit–reweight cycle repeats. In practice, to ensure that the logarithm is well defined, each probability is clipped to \([\epsilon, 1]\) for a small \(\epsilon>0\). After \(M\) rounds, (d) the final prediction is obtained by summing the class scores from all rounds and taking the class with the largest total score.

8 Implementation and Experimental Details↩︎

8.1 Hardware Setup and Model Specification↩︎

8.1.1 Hardware setup.↩︎

All experiments run on a single NVIDIA RTX 3090 (24 GB). For configurations whose GPU memory usage exceeded 24 GB, we used an RTX A6000 (48 GB). All other settings were identical across GPUs.

8.1.2 Model specification.↩︎

We evaluated publicly released CLIP‑style vision-language encoders using each implementation’s default tokenizer and preprocessing. The families and variants were: OpenCLIP [3] (ViT‑B/16, ViT‑L/14, ViT‑H/14, ViT‑g/14; trained on LAION‑2B [45]); DFN‑CLIP [7] (ViT‑B/16, ViT‑L/14 on DFN‑2B; ViT‑H/14 on DFN‑5B); CLIPA‑v2 [4] (ViT‑L/14, ViT‑H/14; trained on DataComp‑1B [46]); SigLIP2 [5] (ViT‑B/16, ViT‑L/14, ViT‑g/16; trained on WebLI [47]); MetaCLIP2 [6] (ViT‑H/14, ViT‑G/14; trained on curated worldwide public‑web data spanning 300+ languages); and EVA‑02‑CLIP [8] (ViT‑B/16 and ViT‑L/14 trained on the Merged‑2B, a mixture of LAION‑2B and COYO‑700M [48]; ViT‑E/14 trained on LAION‑2B).

8.2 Baseline Implementation Details↩︎

8.2.1 PEZ baseline.↩︎

PEZ [21] is a gradient-based hard-prompt optimization method that learns discrete text prompts by projecting continuous prompt embeddings onto the nearest vocabulary tokens at each optimization step. For the baseline of gradient-optimized text prompts, we re-implement PEZ for our image-classification setting, following the official implementation. Concretely, we optimize 16 learnable context tokens that are concatenated with the class name, while keeping the vision-language backbone frozen. We train these prompts using the AdamW optimizer [49] with cross-entropy loss for 200 epochs, using a batch size of 32 and a learning rate of 0.1, closely following the hyperparameter choices reported in the original work.

8.2.2 EFT baseline.↩︎

We include emulated fine-tuning (EFT) [44] as a baseline for transferring adaptation knowledge across models. EFT uses a pre-trained model and its fine-tuned counterpart as source models, and adds the difference between their logits to the target model’s logits so that the target approximately behaves as if it were fine-tuned in the same way. Following the EFT formulation for VLMs in [50], we adapt EFT to our prompt-learning baselines: given logits \(z_{\mathrm{pt-s}}\) and \(z_{\mathrm{ft-s}}\) from a pre-trained and prompt-tuned source VLM and logits \(z_{\mathrm{pt-t}}\) from a pre-trained target VLM, we form EFT logits \(z_{\mathrm{EFT}} = z_{\mathrm{pt-t}} + (z_{\mathrm{ft-s}} - z_{\mathrm{pt-s}})\) for classification. Because this construction requires running three VLMs at inference time (source pre-trained, source prompt-tuned, and target pre-trained), the EFT baseline incurs additional inference cost compared to other text-based transferable baselines.

9 Additional Experiments and Analyses↩︎

9.1 Text Prompts from Consecutive Boosting Rounds↩︎

8 presents the prompts selected by GPC across boosting rounds on a two‑class toy task (beagle vs. basset hound). In Round 1, the weak classifier mostly relies on appearance cues, e.g., “large, floppy ears” for beagle and “low stature” with “front legs slightly splayed apart” for basset hound. This round correctly handles most images but fails on Basset Hound 2, where ears and legs are largely occluded by the lying pose. In Round 2, the newly composed prompts introduce behavioral cues for beagle; “enjoy socializing with humans and other animals, …wagging tails and playful behavior”. This change helps correct the earlier occlusion case; however, Beagle 1 is now misclassified, likely because the image is dark and the dog does not visibly display the “playful” traits suggested by the text. In Round 3, the prompts add further behavioral cues centered on scent‑following. Both classes emphasize similar aspects: the beagle prompt stresses “follow scents enthusiastically …wander off if not properly supervised,” and the basset hound prompt mentions “a tendency to follow scents” and being “distracted during walks.” These cues fix the previous mistakes, yet Beagle 2 is newly misclassified. We cannot attribute this to a single phrase with high confidence; a plausible factor is that the muzzle and leash dampen visible “sniffing/exploratory” cues, leaving the image weakly aligned with either class description and hence easy to confuse. After three additional boosting rounds, the ensemble correctly classifies all examples in 8b.

9.2 Effect of duplication.↩︎

Introducing a learnable weight for each prompt yields too many parameters relative to the few-shot training signal, leading to overfitting and degraded transferability. To avoid this, GPC uses within-round duplication as a parameter-free approximation of continuous weighting. As shown in Table 6, even without duplication, TPB still substantially outperforms the strongest baseline, confirming that the core gains come from the boosting framework itself. Duplication serves as a refinement that further improves performance, with \(53\%\) of selected prompts chosen more than once.

4pt

9.3 Prompt pool robustness.↩︎

We restrict the prompt pool to CuPL+DCLIP and evaluate transfer to ViT-L/H models. Even with this weaker pool, TPB outperforms the baselines, as shown in Table 7, indicating that TPB remains competitive even with a restricted pool, while the full pool provides additional gains.

4pt

9.4 Large-to-small transfer↩︎

We test reverse-direction transfer from OpenAI ViT-L/14 to EVA-02 B/16 in 16-shot setting on 11 datasets. TPB yields a \(+10.17\) pp gain on the source model (\(71.07\!\rightarrow\) \(81.24{\scriptstyle \pm 0.37}\)) and retains a \(+6.98\) pp gain after transfer (\(69.93\!\rightarrow\) \(76.91{\scriptstyle \pm 0.46}\)). This provides initial evidence for TPB’s bidirectional transferability.

9.5 Shot-wise Transfer Performance on Larger Target VLMs↩︎

8 complements 2 by presenting shot-wise transfer accuracy for larger target VLMs (, ViT-E, ViT-G) when text-based baselines and our method are adapted on OpenAI CLIP ViT-B/32. As observed in the results of 2, TPB consistently surpasses the baseline, even when applied to larger VLMs.

4pt

10 Full Experiments Results↩︎

10.1 Full shot-wise scalability.↩︎

2pt

2pt

9 and 10 reports the complete per-dataset shot-scalability results on the source models. The full results confirm the trend observed in the main paper: TPB achieves strong average performance among text-based methods and benefits more clearly from additional shots than prior text-prompting baselines. The improvements are observed across diverse recognition domains, indicating that the shot-scalability of TPB is not driven by a single dataset.

10.2 Full shot-wise and target-wise transfer.↩︎

11 and 12 expands the grouped transfer results in the main paper by reporting shot-wise performance for each target model. This target-wise breakdown confirms that TPB consistently improves over text-prompt baselines such as PEZ and ProAPO, while also revealing a more nuanced comparison with EFT-based prompt-learning baselines. Although CoOp+EFT and PromptSRC+EFT can be competitive, and occasionally outperform TPB at 16 shots on some target models, their behavior is highly target-dependent; in particular, they suffer severe degradation on DFN targets, suggesting that logit-space emulation can be brittle under changes in architecture or training recipe. In contrast, TPB transfers by simply re-embedding natural-language prompt ensembles on the target VLM, yielding stable performance across heterogeneous ViT-L and ViT-H targets. This makes TPB a robust and lightweight transfer option, requiring only the target VLM at inference rather than the multi-model inference used by EFT-based baselines.

3pt

3pt

10.3 Full target-wise table of boosting . augemntation ablation.↩︎

13 provides the full target-model-wise results of the fixed-exposure ablation, where we keep the total exposure budget fixed as \(M \cdot a = 200\) and vary the allocation between the number of boosting rounds \(M\) and the augmentation factor \(a\). Under the same exposure budget, allocating more budget to boosting rounds with moderate augmentation (\(M/a=50/4\)) achieves the best performance on almost all target models, with only one exception where \(M/a=10/20\) is slightly better. This consistent trend across heterogeneous target VLMs supports the role of iterative reweighting and ensemble construction, rather than augmentation alone, in learning transferable prompt ensembles.

4pt

References↩︎