Keywords: Prompt tuning; Vision-language models; Semantic relation; Few-shot learning

1 Introduction↩︎

Fine-grained image recognition (FGIR) plays a pivotal role in scenarios demanding precise differentiation between visually similar subcategories, rich image captioning [1]–[3], image generation [4], [5], food recognition [6]–[8], and food recommendation [9], [10]. The essential challenge stems from the requirement for expert-level annotation precision, which necessitates a nuanced understanding of subtle visual differences, often demanding domain-specific knowledge [11]. The exorbitant cost associated with acquiring such human-expert annotations has emerged as a critical bottleneck, that fundamentally constrains the development of FGIR systems in novel domains.

Vision-language models (VLMs), such as CLIP, offer a promising solution by leveraging large-scale web data for zero-shot learning, thereby mitigating the reliance on extensive manual annotations [12]. CLIP employs contrastive learning to align text and image representations, enabling category classification without task-specific training. While VLMs demonstrate strong generalization in broad categories, this advantage diminishes notably in fine-grained recognition contexts, predominantly stemming from the model’s constrained capacity to discern nuanced inter-class variations.

Prompt tuning adapts task-specific prompts to better leverage the latent knowledge in vision-language models (VLMs), driving strong performance on downstream tasks. It falls into two categories: visual and textual prompt tuning, with the latter optimizing text inputs for improved alignment with visual representations. In this work, we focus exclusively on textual prompt tuning, omitting modifications to the visual encoder. Instead of relying on manually crafted prompts, textual prompt tuning replaces static templates with learnable context vectors [13], [14], which are dynamically optimized using a small set of training samples to enhance image-text alignment, thereby alleviating data scarcity constraints.

However, existing prompt tuning approaches treat category labels as independent, discrete entities, failing to leverage the rich semantic relationships among them. This oversimplification limits the model’s capacity to capture hierarchical dependencies and inter-class correlations, which are crucial for distinguishing visually similar categories. The challenge is particularly pronounced in fine-grained classification, where effective recognition hinges on understanding the intricate semantic structure underlying label relationships.

We address this limitation by introducing Structured-Condensed Prompt Tuning (SCPT), which formulates a structure-aware semantic alignment mechanism within the CoOp-style prompt tuning framework. Instead of relying on discrete class tokens, SCPT embeds Semantic Relation Encoding (SRE) into the prompt space, preserving the global semantic topology among labels through pairwise relational distances, as shown in Figure 1 (c). This enables vision-language models to reason over structured category representations, where each category embedding explicitly encodes its semantic relations to other categories in a shared embedding space, rather than memorizing disconnected labels. We further propose Semantic Condensation loss (ScLoss) to enhance semantic focus by suppressing intra-class noise and amplifying discriminative cues essential for classification. Together, SRE and ScLoss form a unified framework that captures both inter-class structure and intra-class focus, enhancing both few-shot adaptation and base-to-novel generalization. Experimental results show that SCPT consistently outperforms state-of-the-art prompt tuning methods, achieving substantial gains in both few-shot learning and base-to-novel generalization.

In summary, the key contributions of this paper are as follows:

(1) We propose a structure-aware prompt tuning method, Structured-Condensed Prompt Tuning (SCPT), which enhances the semantic structure modeling in vision-language models for FGIR. This approach explicitly captures the inter-class semantic relationships and improves the model’s ability to distinguish subtle differences between visually similar categories.

(2) We present SRE to model the inter-class semantic topology by encoding structured label relationships. This method leverages pairwise relational distances to preserve the global semantic structure, enabling the model to better understand the hierarchical dependencies and correlations between classes.

(3) We design ScLoss to suppress redundant supervision signals and extract discriminative components from the global semantic space. This loss function enhances the model’s focus on task-relevant semantics, improving both few-shot adaptation and generalization to novel classes.

(4) We conduct extensive experiments on 14 diverse FGIR benchmarks. SCPT demonstrates superior performance compared to existing prompt tuning methods. It achieves significant gains in both few-shot learning and base-to-novel generalization tasks, establishing a new state-of-the-art in this domain.

2 Related Works↩︎

2.1 Vision-Language Models (VLMs)↩︎

VLMs such as CLIP [12], ALIGN [15], and BLIP [16] leverage large image-text datasets to achieve superior discriminative power and generalization over traditional single-modality models. By aligning visual and textual information through self-supervised learning, these models excel in zero-shot tasks and transfer their learned representations to downstream applications like image retrieval [17], [18], image segmentation [19], [20], and visual question answering [21]. Although CLIP has shown strong zero-shot performance, fine-tuning for specific tasks can enhance its effectiveness, especially in resource-limited settings. Recent works [13], [14] have explored fine-tuning VLMs for few-shot image recognition, addressing challenges like data scarcity and computational constraints. In this paper, we propose a novel text-prompt tuning method to adapt CLIP for few-shot fine-grained visual recognition, enhancing the model’s flexibility and applicability.

2.2 Prompt Tuning for VLMs↩︎

Prompt tuning seeks to enhance the adaptability of VLMs by introducing a limited number of learnable parameters, thereby allowing the generic features acquired during pre-training to be better aligned with relevant downstream tasks. The fundamental concept is to integrate category information through learnable textual or visual prompts, replacing handcrafted templates such as “a photo of a classname”. These prompts are optimized via backpropagation, while the pre-trained model remains frozen. Existing prompt tuning methodologies often improve model performance by leveraging learnable parameters across various modalities while incorporating additional semantic information. For instance, CoOp[13] replaces the handcrafted templates in CLIP with learnable soft prompts that integrate class names, resulting in substantial advancements in few-shot classification tasks. CoCoOp[14] further enhances the generalization capability of the CoOp model by merging image features with textual soft prompts. Moreover, methods such as MaPle [22], and PromptSRC[23] also introduce learnable visual prompts to bolster model performance. More recently, DGPrompt [24] proposes a dual-guidance prompt generation framework that jointly exploits textual semantic cues and visual structural information to enhance cross-modal alignment in vision-language models. ProDa [25] optimizes the direction of prompt updates while discarding conflicting adjustments to constrain the semantic differences between learnable text features and general text semantics. Conversely, KgCoOp [26] limits the distance between learnable text features and their general semantic counterparts, ensuring that specific semantics remain closely aligned with broader concepts. Building upon KgCoOp, TCP [27] integrates class-related general semantic knowledge extracted from CLIP into deeper layers of the text encoder, thereby enhancing the encoder’s capacity to differentiate between classes. Related to structured semantics, graph-based methods rely on explicit graph construction and additional supervision to model inter-class relationships, whereas SCPT leverages the implicit semantic structure of CLIP’s pretrained embedding space without introducing extra graphs or annotations.

Despite the advancements made by existing prompt tuning approaches in fine-grained image recognition, many still utilize original categories as classification labels, inadequately addressing the semantic limitations inherent in FGIR tasks. Beyond prompt-based adaptation, recent work has explored test-time adaptation for vision-language models. Bayesian Test-Time Adaptation [28] frames adaptation as Bayesian inference to handle distribution shifts at inference time, but does not explicitly model the semantic granularity or structural relationships required for fine-grained recognition. Therefore, when processing fine-grained categories, CLIP may lack sufficient associative semantics learned from its training data, leading to suboptimal generalization performance. In contrast, the proposed method explicitly models semantic structural relationships and enhances FGIR performance by extracting discriminative components from the global semantic space.

3 Method↩︎

The proposed method is grounded in the observation that employing isolated, discrete category labels within prompt templates for prompt tuning underutilizes the rich semantic information inherent in these labels. This drawback is especially pronounced in fine-grained image recognition tasks. To mitigate this issue, we introduce Structured-Condensed Prompt Tuning (SCPT), which enhances the semantic expressiveness of fine-grained label prompts by incorporating Semantic Relation Encoding (SRE) alongside a Semantic Condensation loss (ScLoss), as illustrated in Figure 2. In the following sections, we first provide a brief review of the CoOp-style prompt method, followed by a detailed description of the proposed SCPT framework.

3.1 CoOp-style Prompt Tuning↩︎

Existing CoOp-style prompt tuning algorithms utilize the powerful CLIP as their backbone. CLIP consists of a Transformer-based text encoder and a vision encoder based on either Vision Transformers [29] or Residual Networks [30], where the vision encoder generates visual embeddings from images, and the text encoder transforms text prompts into textual embeddings. During training, CLIP employs contrastive loss to align these embeddings, facilitating effective zero-shot inference. For zero-shot classification, CLIP compares image features with class weights generated from descriptive text, enabling classification without requiring specific category training. Formally, let the image feature extracted by the image encoder be denoted as \(\mathbf{Z}\), and let \(\mathbf{X} = \{ X_i \}_{i=1}^{N}\) represent the class weight vectors generated by the text encoder, where \(N\) is the number of classes. Each class is associated with a handcrafted prompt template, which is first transformed into a vectorized textual token using the word embedding function \(e (.)\). Specifically, for the \(i\)-th class, the textual token is given by:

\[t_i = e (\text{``a photo of [CLASSNAME]''}).\] Subsequently, the text encoder \(\theta\) maps these vectorized textual tokens into class-level embeddings: \[X_i = \theta (t_i).\] Finally, the prediction probability is defined as:

\[p (y = i | x) = \frac{\exp\left (\frac{\mathrm{sim} (X_i, Z)}{\tau}\right)}{\sum_{j=1}^{N} \exp\left (\frac{\mathrm{sim} (X_j, Z)}{\tau}\right)},\] where \(\tau\) is a temperature parameter learned by CLIP and \(\mathrm{sim} (\cdot, \cdot)\) denotes cosine similarity.

To enhance CLIP’s performance on downstream tasks, CoOp replaces the handcrafted prompt template with learnable prompts. By training on a small number of label-relevant images, CoOp improves the discriminability of class-specific textual embeddings. Specifically, CoOp introduces \(m\) learnable context vectors \(V = \{ v_1, v_2, \dots, v_m \}\). The class label embedding \(c_i\) for the \(i\)-th class is concatenated with the context vectors to form the prompt \(V^{*}_i = \{ c_i, v_1, v_2, \dots , v_m \}\). This prompt \(V^{*}_i\) is then fed into the text encoder \(\theta\), yielding the textual class embedding \(Y_i = \theta (V^{*}_i)\). Finally, the textual embeddings for all classes are defined as \(\mathbf{Y} = \{ Y_i^{coop} \}_{i=1}^{N}\).

CoOp optimizes the learnable context vectors \(V\) by minimizing the contrastive loss between the image embedding \(z\) and its corresponding class embedding \(\mathbf{Y}\): \[L_{ce} = -\sum_{\mathbf{z} \in \mathbf{Z}} \log \frac{\exp \left ( \mathrm{sim}\left ( \mathbf{Z}, \mathbf{Y}_{gt} \right) / \tau \right)}{\sum_{i=1}^{t} \exp \left ( \mathrm{sim}\left ( \mathbf{Z}, \mathbf{Y}_i \right) / \tau \right)},\] where \(gt\) is the corresponding label of the image embedding, \(\mathbf{Z}\) is the complete set of embeddings.

To further enhance textual semantics and improve the discriminability of CoOp, previous works [27] have explored refining textual embeddings by reducing their dimensionality and incorporating class-aware tokens into the text encoder. Building on these approaches, SCPT applies this enhancement within the text encoder, facilitating the generation of more discriminative textual representations.

3.2 Structured-Condensed Prompt Tuning↩︎

Unlike existing CoOp-style prompt tuning methods, the proposed SCPT framework enhances the modeling of label semantic structure by jointly leveraging two complementary components. SRE explicitly captures the inter-class semantic topology, replacing isolated category labels with structure-aware embeddings that encode pairwise semantic relations among categories. ScLoss compresses supervision signals by suppressing redundancy and extracting discriminative components from the global semantic space. As illustrated in Figure 2, these two modules work in tandem to improve semantic alignment and fine-grained discriminability. We detail each component in the following sections.

3.2.1 Semantic Relation Encoding↩︎

In this work, we define structured category representations as category embeddings that preserve global inter-class semantic topology, rather than treating each category as an independent token. The CLIP model, with its strong zero-shot generalization capabilities, is employed to extract semantic features for category classification. For fine-grained image recognition tasks with \(N\) categories, we construct category-specific prompts \(c_i\) in the form “a photo of a \(c_i\)", which are passed through CLIP’s text encoder to generate category embeddings, denoted as \(\mathbf{X}\). Each \(X_i\) represents the semantic feature of the \(i\)-th category. To quantify semantic relation between categories, we compute a similarity matrix using cosine similarity: \[S_{ij} = \frac{\mathbf{X}_i \cdot \mathbf{X}_j}{\|\mathbf{X}_i\| \|\mathbf{X}_j\|}, \quad \forall i, j \in \{1, 2, \ldots, N\}, \label{eq1}\tag{1}\] where \(\mathbf{X}_i\) and \(\mathbf{X}_j\) are the feature vectors of the \(i\)-th and \(j\)-th categories. Each row \(S_i\) in the similarity matrix captures the semantic similarity between the \(i\)-th category and all others. This design stems from the research that CLIP’s text embeddings inherently encode taxonomic relationships, and preserving such relations through similarity constraints Eq.(1 ) helps prevent semantic distortion during prompt tuning.

However, due to CLIP’s text prompt length limitations [31], directly using the similarity matrix \(S\) as category labels is not feasible. Inspired by the Johnson–Lindenstrauss (JL) Lemma [32], which states that a set of \(N\) points in a high-dimensional space can be embedded into a lower-dimensional space \(\mathbb{R}^d\) (\(d \geq \mathcal{O} (\epsilon^{-2} \log N)\)) while approximately preserving pairwise distances, we propose a signed random projection method for compact semantic representation. Given a similarity matrix \(S \in \mathbb{R}^{N \times N}\), we generate a binary embedding \(P = \{P_i\}_{i=1}^N \in \{0, 1\}^{N \times d}\) as follows:

\[P = \text{sign}\left (S W^\top\right), \quad\] where \(W \in \mathbb{R}^{d \times N}\) is a random projection matrix whose entries are drawn i.i.d. from \(\mathcal{N} (0, 1)\). Each row of \(W\) defines a hyperplane that randomly partitions the semantic space, and the sign operation encodes whether category embeddings lie on the positive side of these hyperplanes. This binarization compresses semantic relations into a Hamming space where similar categories share overlapping bit patterns. The sign function \(\text{sign} (\cdot)\) applies element-wise binarization, mapping positive values to \(1\) and non-positive values to \(0\). Geometrically, each row vector \(W_i \in \mathbb{R}^{1 \times N}\) defines a randomly oriented hyperplane that partitions the semantic space, and \(P_i \in \{0, 1\}^d\) encodes the relative positioning of categories with respect to \(W_i\). According to JL Lemma, the number of parameter \(d\) is defined as \(d = \lceil \log_2 (N) + d_{free} \rceil\), where \(d_{free}\) introduces additional space to enhance randomness. This ensures the vectors remain independent and non-repetitive after dimensionality reduction.

Similar to the CoOp approach, we first input the binary encoding \(P\) into the word embedding function \(e (\cdot)\) to generate the SRE, denoted as \(R = \{R_i^N\}\), where \(R_i=e (P_i)\). Subsequently, we combine the SRE with a learnable context vector to construct a semantic distance-aware prompt vector: \[V_i = \{R_i, v_1, v_2, \dots, v_m\}.\] This approach efficiently encodes relational semantics while preserving inter-class structure in the lower-dimensional space.

3.2.2 Semantic Condensation loss↩︎

KgCoOp [26] demonstrates that aligning learnable prompts with handcrafted prompt-generated embeddings via an MSE regularizer helps extract discriminative components from the global semantic space, thereby improving classification performance. However, this approach may simultaneously introduce redundant or noisy signals—particularly problematic in fine-grained settings, where class semantics are narrowly scoped and visually diverse. Such coarse supervision can mislead the optimization process and hinder the learning of compact, class-specific representations. Motivated by this hypothesis, we propose a straightforward yet effective strategy: Specifically, let \(\mathbf{X}\in R^{N\times M}\) denote the text embeddings generated from handcrafted prompts and \(\mathbf{Y}\in R^{N\times M}\) represent the embeddings obtained from learnable prompts. Treating \(\mathbf{X}\) as a semantic data matrix and applying singular value decomposition (SVD) to remove noise. By preserving dominant singular values and eliminating smaller ones, we aim to denoise the embeddings. Specifically, we perform SVD on \(\mathbf{X}\):

\[\mathbf{X} = \mathbf{U} \mathbf{\Sigma} \mathbf{V}^\top.\] Here, \(\mathbf{\Sigma} = (\sigma_1, \sigma_2, ..., \sigma_r)\) represents the singular values arranged in descending order, i.e., \(\sigma_1 \geq \sigma_2 \geq ... \geq \sigma_r \geq 0\), where \(r\) denotes the rank of the matrix. We retain only the top \(K\) singular values, yielding:

\[\left\{ \begin{align} \mathbf{U}_{K} &= \mathbf{U}_{[:, :K]} \in \mathbb{R}^{N \times K}, \\ \mathbf{\Sigma}_{K} &= \text{diag} (\mathbf{\Sigma}_{[:K]}) \in \mathbb{R}^{K \times K}, \\ \mathbf{V}_{K}^\top &= \mathbf{V}^\top_{[:K, :]} \in \mathbb{R}^{K \times M}. \end{align} \right.\] The denoised embedding matrix \(\mathbf{X}^{'}\) is then reconstructed as:

\[\mathbf{X}^{'} = \mathbf{U}_{K} \mathbf{\Sigma}_{K} \mathbf{V}_{K}^\top.\] Our goal is to mitigate the model’s tendency to forget general semantics by minimizing the divergence between inter-class and generic semantic knowledge: \[L_{\text{sc}} = \frac{1}{N} \sum_{i=1}^{N} \| Y_i - X_i^{'} \|_2^2,\] where \(\| \cdot \|_2\) is the Euclidean norm, and \(N_c\) is the number of seen classes. Finally, we integrate the standard contrastive loss \(L_{\text{ce}}\) with our Semantic Condensation loss \(L_{\text{sc}}\) to define the overall objective function: \[L = L_{\text{ce}} + \lambda L_{\text{sc}},\] where \(\lambda\) controls the contribution of \(L_{\text{sc}}\) within the total loss.

As illustrated in Figure 3 (a), applying this denoising process improves the MSE performance compared to using the original embeddings. However, the results also indicate that the choice of \(K\) significantly influences the outcome across different datasets. This suggests that SVD-based denoising is highly sensitive to the selection of \(K\), raising an important question: Can a robust mechanism be devised to determine the optimal value of \(K\)?

3.2.3 Optimal Value of \(K\)↩︎

To explore this question, we visualize the distribution of nonzero singular values after performing SVD on semantic embeddings from various fine-grained datasets, as shown in Figure 3. Compared to Figure 3 (a), a consistent pattern emerges: despite differences in matrix rank, the optimal number of retained singular values tends to fall between the mid-range and lower-range singular values. This suggests that smaller singular values predominantly capture noise, with the smallest ones being particularly susceptible to noise-induced distortions.

To further substantiate this observation, We assume that \(\mathbf{X} \in \mathbb{R}^{N \times M}\) is a purely noisy Gaussian random matrix, where each element is independently and identically distributed (i.i.d.) with zero mean and variance \(\sigma^2\). The covariance matrix of \(\mathbf{X}\) can be expressed as: \[\mathbf{S}=\frac{1}{M}XX^T.\] According to the Marchenko-Pastur theorem [33], when \(N, M\rightarrow\infty\), and \(\frac{M}{N}\rightarrow r\in (0, 1]\). The singular values of the sample covariance matrix follow a characteristic distribution given by: \[p (\sigma) = \frac{1}{2\pi r\sigma^2} (\sigma_{\max} - \sigma) (\sigma - \sigma_{\min}), \quad \sigma_{\min} \leq \sigma \leq \sigma_{\max},\] where \(\sigma_{\min}\) and \(\sigma_{\max}\) denote the lower and upper bounds of the singular value spectrum, determined by the data dimensions \(N, M\) and the noise variance \(\sigma^2\). Notably, the density of singular values near \(\sigma_{\min}\) is markedly high, indicating that most singular values in a pure noise matrix reside in this region. Conversely, singular values exceeding \(\sigma_{\max}\) are statistically improbable under a pure noise assumption, implying that they are more likely to correspond to meaningful semantic structures rather than random fluctuations.

Building on this theoretical analysis, we propose an adaptive filtering mechanism that removes singular values below a predefined threshold, effectively reducing noise while preserving as much semantic information as possible. Notably, the optimal value of \(K\) tends to shift slightly to the right of the theoretical optimum. This shift occurs because some smaller singular values may still carry semantic information. To address this, we design a probabilistic selection function \(P (\sigma_i)\) as follows:

\[P (\sigma_i) = \begin{cases} 1, & \text{if } \sigma_i \geq \tau, \\ e^{ (\sigma_i - \tau)^2-N_{\text{below}}}, & \text{otherwise}. \end{cases} ,\] where \(\tau\) represents the threshold, and \(N_{\text{below}}\) denotes the number of singular values below \(\tau\). This formulation ensures that singular values exceeding the threshold are always retained, while those closer to the threshold have a higher probability of being preserved. The term \(e^{-N_{\text{below}}}\) serves as a regularization factor, reflecting the observation that a larger number of small singular values increases the likelihood of noise. Consequently, when more small singular values are present, the probability of selecting any individual one is reduced, reinforcing the suppression of noise-dominated components. Thus, the final selection of \(K\) is determined as the index of the \(i\)-th singular value that is retained based on the probabilistic selection function, formulated as:

\[K = \arg\max_{i} \{ P (\sigma_i) \neq 0 \}.\] This ensures that \(K\) corresponds to the largest index among the selected singular values, striking a balance between noise reduction and semantic preservation.

SCPT introduces a structured prompt tuning strategy that jointly leverages inter-class relational encoding and intra-class semantic compression. SRE encodes inter-class relations via randomized hyperplane projections that preserve CLIP’s semantic topology, while ScLoss distills task-critical semantics through singular value truncation. Algorithm 4 provides a unified view of how SRE and ScLoss are integrated into the SCPT framework. Their coordinated operation, with SRE expanding inter-class discriminability and ScLoss contracting intra-class redundancy, establishes a robust foundation for fine-grained adaptation.

4 Experiments↩︎

To evaluate the effectiveness of the proposed method, we follow standard practice on a series of downstream tasks, including few-shot experiments, base-to-novel generalization, and ablation study.

4.1 Experimental Setup↩︎

Datasets: We employed 14 fine-grained classification datasets including Dog Breed [34], Oxford Pets [35], Oxford Flowers [36], Stanford Cars [37], Web Cars [38], Stanford Dogs [39], [40] , Fruit92 [41] , CUB200 [42], Food101 [43], Food172 [44], Food200 [45], Food500[46], FGVC Aircraft [47] and Veg200 [41].

Implementation Details: For all experiments, we utilize ViT-B/16 as the vision backbone. In the SCPT model, the context length for learnable prompts is fixed at 4, and the context vectors are initialized from a Gaussian distribution (\(\sigma=0.02\)), threshold \(\tau\) is 0.3. The hyperparameter \(\lambda\) is 8.0, same as KgCoOp. The models are trained on the RTX 4090 GPU with 24GB memory using the SGD optimizer. The training process spans 50 epochs with a batch size of 32. The initial learning rate is set to 2e-3 and dynamically adjusted according to the cosine annealing schedule. All comparisons are conducted under a consistent experimental protocol, with method-specific settings applied only when required by the original implementation.

SOTA Methods: We compared SCPT with the zero-shot inference capability of CLIP [12] (ICML 2021) and conducted a comprehensive comparison with recent textual prompt tuning methods. e.g., CoOp [13] (IJCV 2022), CoCoOp [14] (CVPR 2022), KgCoOp [26] (CVPR 2023), TCP [27] (CVPR 2024), ProText [48] (AAAI 2025) and ATPrompt [49] (ICCV 2025).

0.85pt

4.2 Few-Shot Learning Evaluations↩︎

To validate that the SCPT method offers enhanced fine-grained semantic distinction compared to existing CoOp-style prompt templates, we conducted few-shot classification across all 14 fine-grained image datasets using K-shot labeled source images. Evaluations were performed on a standard test domain within the same class space as the training set.

Table 1 reports the performance of SCPT on 14 fine-grained datasets under the 16-shot setting, achieving an average accuracy of 76.70%. Compared with the state-of-the-art method TCP, SCPT consistently outperforms it on all benchmarks except Oxford Pets. This dataset contains the fewest fine-grained categories, which constrains the expressiveness of the induced semantic topology and thus limits the potential gains from semantic-aware tuning. Overall, SCPT achieves an average improvement of more than 1% across all datasets.

4.3 Base-to-Novel Generalization↩︎

2.5pt

To ensure consistency with prior studies such as TCP, we partition each dataset into two disjoint sets: base classes and novel classes, following a zero-shot learning framework where base and novel classes do not overlap. This setup enables us to quantify the model’s generalization capability to fine-grained, out-of-domain categories. For each evaluation stage, SRE is computed from the textual semantic representations of the candidate class names and projected using the same random projection function as in training; the resulting SRE is then combined with the class name and the learnable context vectors to form the final textual prompt.

Table ¿tbl:tab-sota? compares SCPT with CLIP and recent prompt tuning methods under the 16-shot setting. SCPT outperformed existing approaches, showing improvements on both base and novel classes. The SRE-enhanced prompt captured semantic relation, yielding a 1.10% average gain over TCP across 14 datasets. The incorporation of Semantic Condensation loss (ScLoss) mitigated irrelevant semantic interference, resulting in a 71.15% harmonic mean, demonstrating strong generalization to novel classes. In contrast to CoOp methods, which often overfit to base classes, SCPT achieved an 77.84% improvement in harmonic mean performance, a 9.79% increase in novel class generalization, and a 2.46% boost in base class accuracy. These results highlight SCPT’s superior ability to balance generalization and base class performance, achieving the highest overall accuracy across all 14 datasets.

0.85pt

4.4 Ablation Study↩︎

Effects of SRE and ScLoss. The proposed SCPT method integrates two core components: SRE and ScLoss. SRE encodes class names to reflect inter-class relationships and enhance class distinction, while ScLoss suppresses irrelevant semantic information to improve robustness. To ensure a fair and controlled evaluation of the proposed components, we adopt CoOp-style prompts with semantic regularization following the TCP formulation as the baseline setting in the ablation study. This design choice allows us to isolate the individual contributions of SRE and ScLoss under a consistent semantic regularization framework, avoiding performance variations caused by changes in the underlying prompt formulation.

As shown in Table 2, SRE and ScLoss contribute to performance improvement through different mechanisms. SRE enhances class separability by encoding fine-grained inter-class similarity structures, but its reliance on semantic relations makes it more vulnerable to semantic noise. In contrast, ScLoss improves robustness by suppressing irrelevant variations via low-rank filtering, yet lacks the capacity to explicitly capture structural relations among classes. When combined, the two components complement each other—SRE promotes discriminability, while ScLoss mitigates noise sensitivity—resulting in an average performance gain exceeding 1%. These results highlight their strong complementarity and validate the effectiveness of joint optimization.

Effect of \(d_{free}\) in SRE. This study examines the impact of the additional free space \(d_{\text{free}}\) on encoding length and recognition accuracy in a 16-shot setting across 14 datasets, as shown in Figure 5. We evaluate \(d_{\text{free}}\) values from 0 to 8, with results indicating that \(d_{\text{free}} = 3\) yields the best performance. However, the accuracy difference between the highest and lowest values is only 0.36%, suggesting that recognition accuracy is not highly sensitive to variations in \(d_{\text{free}}\). This implies that the dimensionality reduction is not strongly dependent on the reduced space size in the absence of conflicts.

Effect of Prompt length M. In a 4-shot experiment, we investigate the impact of prompt length \(M\) on few-shot performance, as shown in Figure 6. We evaluate prompt lengths of 1, 2, 4, 8, and 16, finding that SRE-encoded prompts are largely insensitive to learnable parameter variations. While a prompt length of 2 yields the highest accuracy, the difference from the lowest result is just 0.16%. Shorter prompt lengths generally outperform longer ones, indicating that SRE encoding inherently captures more class-specific information, which is particularly advantageous in fine-grained recognition tasks.

Effect of Hyperparameter \(\lambda\). We evaluate the sensitivity of the loss weight \(\lambda\) by measuring average accuracy on four fine-grained datasets, namely Fruit92, FGVC-Aircraft, CUB200, and Stanford Dogs, using the ViT-B/32 backbone. The value of \(\lambda\) is varied over \(\{1, 2, 4, 8, 16\}\). As shown in Figure 7, the performance remains stable when \(\lambda\) ranges from 1 to 8, with accuracy variations below 1%. The highest accuracy is achieved at \(\lambda=4\), while \(\lambda=8\) yields a comparable result. For consistency with prior work and to avoid dataset-specific tuning, we use \(\lambda=8\) in all experiments.

Effect of Threshold \(\tau\). The threshold \(\tau\) controls the strength of feature noise filtering. We evaluate the sensitivity to \(\tau\) by varying its value from 0.1 to 0.5 under the same experimental setting, using the ViT-B/32 backbone on Fruit92, FGVC-Aircraft, CUB200, and Stanford Dogs. As shown in Figure 8, the average classification accuracy remains nearly unchanged across the entire range, with variations below 0.3%. Based on this observation, we use a fixed value of \(\tau\) in all experiments.

Sensitivity to Random Projection Initialization. We evaluate the impact of initialization on the projection matrix \(\mathbf{W}\) using five random seeds on the Fruit92 dataset (ViT-B/32). As shown in Table 3, SCPT exhibits remarkable robustness, yielding a mean accuracy of 75.3% \(\pm 0.23\). With a variation of less than 0.6% across all trials, the results conclusively demonstrate that the proposed signed random projection strategy is insensitive to initialization noise, validating its reliability for consistent deployment.

Comparison of Computational Efficiency. Table 4 summarizes the complexity analysis on the large-scale Food-500 dataset. Results indicate that while SCPT incorporates the SRE module, it maintains a lightweight design with 0.79M learnable parameters, representing merely 0.77% of the total model capacity. In terms of computational speed, SCPT achieves a training throughput of 140.1 img/s, which is more than 20\(\times\) faster than CoCoOp and comparable to both CoOp and TCP. During inference, SCPT achieves 228.4 ms per batch, remaining close to TCP in latency while maintaining a similar memory footprint. These results suggest that SCPT maintains a similar parameter scale and computational efficiency to TCP, while providing consistent performance improvements.

6pt

Robustness Across Backbones. While our main experiments are conducted with ViT-B/16, we further evaluate SCPT on two additional visual backbones, ViT-B/32 and ResNet-50, to assess its generality across architectures. Experiments are performed on five representative fine-grained datasets: Fruit92, Veg200, WebCar, FGVC Aircraft, and Dog Breed. As shown in Table 5. SCPT achieves consistent state-of-the-art performance on both ViT-B/32 and ResNet-50, exceeding the strongest prior methods by a clear margin in average accuracy. This confirms that the effectiveness of SCPT generalizes well across both transformer- and CNN-based visual backbones.

4.5 Comparison with Visual Prompt Tuning Methods↩︎

Although SCPT is designed as a purely textual prompt tuning method, we include a comparison with representative recent multi-modal prompt tuning approaches to analyze the relationship between prompt complexity and recognition performance. As shown in Table 6, MaPLe and PromptSRC adopt deep prompt injection across nine transformer layers and leverage both visual and textual modalities, whereas SCPT employs a single-layer shallow prompt and operates exclusively in the textual domain.

Despite this substantially lighter design, SCPT attains an average accuracy of 76.70%, which is comparable to PromptSRC in terms of average accuracy and markedly higher than MaPLe across 14 datasets. These results indicate that strong performance can be achieved with shallow, text-only prompt tuning, suggesting that explicitly modeling semantic structure in textual prompts is a viable alternative to deep or multi-modal prompt designs.

4.6 Visualization↩︎

To investigate the impact of textual prompts on image feature representations, we compute similarity scores between target images and textual prompts using CoOp, KgCoOp, and the proposed SCPT method. Grad-CAM [50] is then applied to backpropagate these similarity scores through the image encoder, producing gradient-based heatmaps that visualize image regions most sensitive to the target similarity scores, as shown in Figure 9.

Textual embeddings act as explicit supervisory signals during backpropagation, guiding gradient responses toward the target class and facilitating more discriminative region localization. Compared with CoOp and KgCoOp, SCPT produces heatmaps that exhibit stronger focus on target regions, clearer foreground-background separation, and finer class-specific details, while suppressing irrelevant visual information.

5 Limitations↩︎

While our proposed framework demonstrates strong effectiveness in text-based prompt learning, several limitations remain. First, our method focuses exclusively on textual prompts and does not incorporate learnable visual prompts. Recent works such as MaPLe [22] and PromptSRC [23] have shown that jointly optimizing textual and visual prompts can further enhance adaptation and generalization in certain scenarios. Extending SCPT to a multimodal prompt learning framework may provide additional representational flexibility, which we leave for future work.

Second, our current design adopts shallow prompt tuning and injects learnable prompts only at a limited number of layers. Although this design choice helps maintain parameter efficiency and training stability, it does not explore deeper prompt mechanisms that interact with intermediate layers of the backbone, as investigated in VPT [51] and DualPrompt [52]. Incorporating deep prompts into the proposed structured and condensed prompt formulation is a promising direction for future research.

Finally, Although SCPT is formulated as a complete prompt tuning model within the CoOp-style framework, extending it to other prompt learning paradigms is an interesting direction for future work.

6 Conclusion↩︎

This work revisits prompt tuning for fine-grained recognition from a structural perspective and argues that treating category prompts as isolated tokens fundamentally limits the ability of vision–language models to discriminate subtle inter-class differences. To address this issue, we propose Structured-Condensed Prompt Tuning (SCPT), which explicitly incorporates global semantic structure into textual prompt learning. By introducing Semantic Relation Encoding (SRE), SCPT captures inter-class semantic topology in a lightweight and parameter-efficient manner, while the Semantic Condensation loss (ScLoss) further refines supervision by suppressing redundant or noisy semantic components in the shared embedding space.

Extensive experiments on 14 fine-grained benchmarks demonstrate that SCPT consistently improves both few-shot adaptation and base-to-novel generalization, indicating that structured semantic modeling is particularly beneficial under limited data and distribution shift. The results suggest that incorporating global semantic relations provides a more effective inductive bias than treating category prompts as independent tokens.

Despite these advantages, SCPT has certain limitations. Its effectiveness depends on the availability of a sufficiently rich category set, as semantic topology becomes less informative when the number of classes is small. In addition, the current framework focuses exclusively on textual prompt learning and does not exploit complementary visual or multimodal prompting mechanisms. Future work will explore these directions and extend SCPT to large-scale and open-vocabulary recognition settings.

7 T-SNE visualizations of visual embeddings↩︎

To investigate the impact of SCPT on feature space distribution, we conduct a comparative analysis with the baseline method CoOp using t-SNE visualizations of visual embeddings extracted from test set samples across three datasets with varying numbers of categories: Fruit92 (92 classes), Oxford Flowers (102 classes), and Stanford Cars (196 classes), as shown in Figure 10.

The Silhouette Coefficient (\(S\)) is employed to quantitatively characterize the clustering structure of the visual embeddings, measuring the relative compactness of intra-class samples with respect to inter-class separation. Higher \(S\) values indicate more coherent and well-separated clusters.

As illustrated in Figure 10, the baseline CoOp exhibits relatively diffuse cluster structures with indistinct boundaries across all evaluated datasets, which is reflected by its lower Silhouette Coefficients. In contrast, SCPT consistently produces more compact intra-class distributions and clearer separation between categories, yielding higher \(S\) values. Furthermore, the performance gap between SCPT and CoOp becomes more pronounced as the number of categories increases, indicating that SCPT is effective in maintaining structured feature representations under large-scale and fine-grained recognition settings.

8 Comparison with Conventional Few-Shot Learning Settings↩︎

Conventional few-shot learning (FSL) methods are typically formulated under the \(C\)-way-\(K\)-shot episodic paradigm, where models are trained on base classes and evaluated on disjoint novel classes via repeated few-class classification tasks. While widely adopted, this protocol imposes restrictive assumptions on task scale and data availability that diverge from real-world recognition settings.

Specifically, episodic FSL evaluates performance on tasks with a small number of classes (e.g., 5-way or 10-way), whereas practical applications often require discriminating among hundreds of categories simultaneously. Moreover, episodic sampling emphasizes local class subsets and assumes abundant query samples, limiting its ability to reflect global inter-class confusion and large-scale recognition behavior. Most critically, its closed-vocabulary evaluation assesses performance only within individual tasks, offering limited insight into open-vocabulary generalization.

To better align with large-scale and open-world scenarios, we adopt two complementary evaluation protocols commonly used in prompt-based and representation-learning methods: standard few-shot classification and base-to-novel generalization. In the few-shot setting, the model is trained with \(k\) labeled samples per class under a global supervised setup and evaluated on a held-out test set, directly measuring scalability under limited supervision. In the base-to-novel setting, training is restricted to base classes, followed by evaluation on both base and novel categories, with generalization quantified by the harmonic mean of their accuracies.

9 Theoretical Analysis of Signed Random Projection↩︎

We provide a theoretical justification for the sign-based binarization by showing that the Hamming distance between binary projections consistently approximates the angular distance between the original semantic embeddings. Since vision–language models such as CLIP rely on cosine similarity, this result confirms that our binarization preserves the essential semantic topology of the embedding space.

9.1 Equivalence between Hamming Distance and Angular Distance↩︎

Lemma 1 (Angular Distance Preservation). Let \(S_i, S_j \in \mathbb{R}^N\) be two non-zero embedding vectors, and let \(W \in \mathbb{R}^{d \times N}\) be a random projection matrix whose rows are independently sampled from a spherically symmetric distribution. Define the binary codes \(P_i = \operatorname{sign}(W S_i)\) and \(P_j = \operatorname{sign}(W S_j)\). The normalized Hamming distance between the two codes is defined as: \[d_H(P_i, P_j) = \frac{1}{d} \sum_{k=1}^d \mathbf{1}[P_{ik} \neq P_{jk}],\] where \(\mathbf{1}[\cdot]\) is the indicator function. This is an unbiased estimator of the normalized angular distance between \(S_i\) and \(S_j\), namely: \[\mathbb{E}[d_H(P_i, P_j)] = \frac{\theta_{ij}}{\pi},\] where \(\theta_{ij} \in [0,\pi]\) denotes the angle between \(S_i\) and \(S_j\).

Proof. Consider a single random projection vector \(w\), corresponding to one row of \(W\). The induced hyperplane \(\{x : w^\top x = 0\}\) separates \(S_i\) and \(S_j\) if and only if \(\operatorname{sign}(w^\top S_i) \neq \operatorname{sign}(w^\top S_j)\). Due to the rotational invariance of the distribution of \(w\), the probability of this event depends solely on the angle \(\theta_{ij}\) between the two vectors. It is a classical result in Locality Sensitive Hashing (LSH) that a random hyperplane separates two vectors with probability \(\theta_{ij}/\pi\). Averaging over \(d\) independent projections yields \(\mathbb{E}[d_H(P_i, P_j)] = \theta_{ij}/\pi\).

9.2 Concentration of the Hamming Distance↩︎

Theorem 1 (Concentration). For any \(\varepsilon > 0\), the empirical Hamming distance concentrates around its expectation as \[\operatorname{Pr}\!\left( \left| d_H(P_i, P_j) - \frac{\theta_{ij}}{\pi} \right| \ge \varepsilon \right) \le 2 \exp(-2 d \varepsilon^2).\]

Proof. Each bit disagreement in the Hamming distance is an independent Bernoulli trial with success probability \(\theta_{ij}/\pi\). The normalized Hamming distance is therefore the empirical mean of \(d\) i.i.d.Bernoulli variables. Applying Hoeffding’s inequality directly yields the stated bound.

9.3 Rationale for Dimension Selection↩︎

The projection dimension \(d\) governs both the stability and the capacity of the Hamming space. While signed random projection is inherently probabilistic, increasing \(d\) reduces the variance of the angular distance estimation (as per Theorem 1) and lowers the probability of code collisions. In practice, for \(N\) semantic categories, we choose \[d = \left\lceil \log_2 N + d_{\mathrm{free}} \right\rceil,\] where \(d_{\mathrm{free}}\) is a small safety margin. As shown in Figure 5, performance saturates once \(d\) is sufficiently large to produce stable Hamming representations, indicating diminishing returns from further increasing the dimensionality.

References↩︎