1 Introduction↩︎

Recent advances in Vision-Language Models (VLMs) have inspired the development of Audio-Language Models (ALMs), which achieve strong performance on zero-shot audio recognition tasks [1]–[4]. In the zero-shot setting, audio features are aligned with textual descriptions of class labels, enabling recognition without task-specific training. This paradigm provides flexibility and generalization to unseen categories while eliminating the need for large labeled datasets. Despite these advantages, zero-shot performance often falls short of fully supervised or few-shot adaptation. The primary limitation lies in the reliance on manually designed text prompts, which may not optimally align with the target dataset. Prompt learning addresses this limitation by replacing hand-crafted templates with learnable context tokens that are optimized using labeled data from base classes [5], [6]. By adapting prompts to the downstream task, prompt learning significantly improves performance over zero-shot baselines.
However, in the context of audio-language models, we observe a critical trade-off: while prompt learning consistently improves accuracy on base classes (seen during few-shot training), it often degrades performance on novel classes (unseen during few-shot training). In several cases, prompt learning performs even worse than the original zero-shot model on novel categories (refer to Figure 1 and Table 1). This reveals a fundamental base-to-novel generalization gap under prompt learning. Optimizing prompts solely using base-class supervision encourages overfitting to seen categories, distorting the semantic alignment learned during large-scale pre-training and weakening zero-shot transferability.

To address this challenge, we propose ZEBRA: Zero-shot Entropy-Regularized Prompt Learning for Base-to-Novel GeneRAlization. ZEBRA is a plug-and-play framework that operates on top of existing prompt learning methods and is designed to preserve zero-shot generalization while still benefiting from supervised adaptation. ZEBRA introduces two complementary mechanisms: First, it fuses zero-shot logits with the prompt-learning logits, effectively anchoring adaptation to the original zero-shot decision space. Second, it introduces a self-entropy regularization term in the loss function to mitigate overconfidence in base classes during few-shot training. By discouraging over-confident predictions, self-entropy prevents excessive specialization to seen categories and promotes smoother decision boundaries, thereby improving generalization to unseen classes. ZEBRA (refer to Figure 2 for an overview) is lightweight, as it introduces no additional learnable parameters to existing prompt-learning baselines. The zero-shot logits are computed once and reused during both few-shot training and inference, eliminating extra forward passes through the text encoder and resulting in negligible computational overhead. Our contributions are as follows:

• We identify and analyze a base-to-novel generalization gap in prompt learning for audio-language models, where improvements on base classes often come at the cost of degraded novel-class performance (see Figure 1 or Table  1).

• We propose ZEBRA (Zero-shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization), a plug-and-play framework that builds upon existing prompt-learning methods without introducing additional learnable parameters and with negligible computational overhead.

• ZEBRA integrates zero-shot logit fusion to anchor adaptation to the pre-trained decision space and employs self-entropy regularization to reduce overconfidence on base classes, enhancing generalization to unseen categories.

• We demonstrate consistent improvements in novel-class generalization across multiple audio classification datasets, significantly reducing the base-to-novel performance gap compared to vanilla baseline prompt learning methods.

2 Related Work↩︎

Inspired by the success of CLIP [7] in the image–language domain, many audio–language models have adopted a similar contrastive learning framework. CLAP [2] and AudioCLIP [3], for example, extend this paradigm to align audio and textual representations, enabling robust audio classification and cross-modal retrieval. Like CLIP, these models are trained to maximize similarity between matched audio–text pairs while minimizing similarity for mismatched pairs.
Few-shot adaptation has attracted significant attention in vision–language models, particularly for adapting CLIP [8]. Among these, prompt learning has emerged as a parameter-efficient alternative to full fine-tuning, enabling adaptation to downstream tasks by introducing a small set of learnable parameters while keeping the pretrained backbone frozen [5], [6]. This strategy substantially reduces computational and storage costs compared to full model updates. Prompt learning has been successfully applied across natural language processing [9], computer vision [8], [10], [11], and more recently, the audio-language domain [12]–[14]. In audio–language models, approaches such as PALM [12] demonstrate that learning only a small set of prompt vectors can effectively adapt frozen pretrained ALM to achieve strong audio classification performance.

3 Methodology↩︎

Zero-Shot Classification in ALM. In CLIP-style audio-language models (ALMs) [7], zero-shot classification is performed by measuring the similarity between the audio representation and a set of class-specific text descriptions. Let \(\boldsymbol{\mathrm{x}}\) denote an input audio waveform, and let \(\boldsymbol{\mathrm{t}}= \{t_1, t_2, \dots, t_c\}\) represent the set of textual class descriptions for \(c\) classes. The prediction logits are computed as:

\[\label{eq:zs95logits} f_{\mathrm{zs}}(\boldsymbol{\mathrm{x}}) = \bigg\{\operatorname{sim}\big(f_{A}(\boldsymbol{\mathrm{x}})~,~f_{T}(t_i)\big)\bigg\}_{i=1}^{c},\tag{1}\]

where \(f_A\) and \(f_T\) denote the audio and text encoders, respectively, and \(\operatorname{sim}(\cdot)\) is a cosine similarity function. The predicted label corresponds to the class with the highest similarity score.
Base-to-Novel Generalization. In the base-to-novel setting, the full label space is defined as \(\mathcal{C} = \{1,2,\ldots,c\}\) and is partitioned into base classes \(\mathcal{C}_B\) and novel classes \(\mathcal{C}_N\), such that \(\mathcal{C}_B \cup \mathcal{C}_N = \mathcal{C}\) and \(\mathcal{C}_B \cap \mathcal{C}_N = \emptyset\). The model is trained using labeled data from \(\mathcal{C}_B\) and is expected to generalize effectively to the unseen (novel) classes in \(\mathcal{C}_N\). In the context of prompt learning, the prompts are optimized using base-class samples from the training set, and evaluation is conducted on both base and novel class samples from the test set to assess generalization performance.
Prompt Learning in ALM. Textual class descriptions play a central role in zero-shot inference for ALMs; however, manually designed prompts often introduce performance variability and sensitivity to wording. Prompt learning mitigates this issue by introducing auxiliary learnable parameters \(\varphi\) into the text encoder \(f_T\). These parameters are optimized in a few-shot setting to adapt the frozen pre-trained model to downstream tasks while reducing reliance on manual prompt engineering [9]. Formally, prompt learning learns \(\varphi\) by minimizing a task-specific objective over a few-shot training dataset \(\mathcal{D}\): \[\label{eq:pl95logits} \underset{\varphi}{\operatorname{minimize}} \sum_{(\mathbf{x},y)\in\mathcal{D}} \mathcal{L}\big(f_{\mathrm{pr}}(\mathbf{x};\varphi), y\big),\tag{2}\] where \((\mathbf{x}, y)\) denotes an audio-label pair from \(\mathcal{D}\), \(f_{\mathrm{pr}}(\mathbf{x};\varphi)\) represents the model’s prediction conditioned on the learnable prompt parameters \(\varphi\), and \(\mathcal{L}(\cdot)\) is the task-specific loss function, typically the cross-entropy loss. Two representative instances of \(f_{\mathrm{pr}}(\mathbf{x};\varphi)\) are COOP [5] and COCOOP [6]. In COOP, a set of shared learnable context tokens is optimized and prepended to each class name before being passed to text encoder. In COCOOP, the context tokens are dynamically generated conditioned on the audio feature vector, enabling instance-specific adaptation. Although these approaches improve performance on base classes, they optimize prompts solely using base-class supervision and do not explicitly preserve the original zero-shot alignment. As a result, the adapted representation space can become biased toward seen categories, leading to degraded performance on novel classes, as shown in Figure 1.
ZEBRA: Zero-Shot Entropy-Regularized Base-to-Novel Generalization. To mitigate the base-to-novel generalization gap, we propose ZEBRA, a plug-and-play framework that operates on top of existing prompt learning methods. ZEBRA preserves zero-shot transferability while benefiting from supervised adaptation through two complementary mechanisms.
(i) Zero-shot Logit Fusion: Instead of relying solely on prompt-learning logits, we fuse them with the original zero-shot logits during both few-shot training and inference: \[\label{eq:zebra95logits} f_{\mathrm{zebra}}(\boldsymbol{\mathrm{x}};\varphi) = \lambda_{\mathrm{zs}}\cdot f_{\mathrm{zs}}(\boldsymbol{\mathrm{x}}) + \lambda_{\mathrm{pr}}\cdot f_{\mathrm{pr}}(\boldsymbol{\mathrm{x}};\varphi),\tag{3}\] where \(f_{\mathrm{zs}} \in \mathbb{R}^{c}\) denotes logits from zero-shot setting, \(f_{\mathrm{pr}} \in \mathbb{R}^{c}\) denotes logits from any prompt-learning method (e.g., COOP or COCOOP) and \(\lambda_{\mathrm{zs}},\lambda_{\mathrm{pr}}\) control the contribution of the zero-shot and prompt learning components, respectively. This fusion anchors adaptation to the pre-trained decision space.
(ii) Self-Entropy Regularization: To prevent overconfidence on base classes, we introduce a self-entropy regularization term: \[\mathcal{L}_{\text{ent}} = - \sum_{i} \mathbf{p}^{i}(\mathbf{x}) \log \mathbf{p}^{i}(\mathbf{x}),\] where \(\mathbf{p}(\mathbf{x})\) is the softmax probability derived from \(f_{\mathrm{zebra}}(\boldsymbol{\mathrm{x}};\varphi) \in \mathbb{R}^{c}\). The final training objective becomes: \[\nonumber \mathcal{L}_{\mathrm{zebra}}\big(\boldsymbol{\mathrm{x}},y;\varphi\big) = \mathcal{L}_{\mathrm{ce}}\big(f_{\mathrm{zebra}},y\big) - \mathcal{L}_{\mathrm{ent}}(\mathrm{softmax}(f_{\mathrm{zebra}})),\] where \(\mathcal{L}_{\mathrm{ce}}(\cdot)\) is the cross-entropy loss. By combining zero-shot logit fusion with entropy regularization, ZEBRA reduces overfitting to base classes while preserving the semantic alignment necessary for robust novel-class generalization. Formally, we optimize following objective during few-shot training on base classes: \[\label{eq:zebra95objective} \underset{ \varphi }{\operatorname{minimize}}~~ \sum_{(\boldsymbol{\mathrm{x}},y)\in\mathcal{D}} \mathcal{L}_{\mathrm{zebra}}\big(\boldsymbol{\mathrm{x}},y;\varphi\big).\tag{4}\] During optimization, we minimize the cross-entropy loss while maximizing the self-entropy term, which constructively counteracts overconfident predictions, thereby discouraging overconfidence and reducing overfitting to the base classes. At inference time, predictions are computed from the fused zero-shot and prompt-learning logits, without applying entropy regularization. The predicted label \(\hat{y}\) is obtained as follows: \[\hat{y} = \underset{ i\in \{1,2,\dots,c\} }{\mathrm{argmax}} ~~~ f_{\mathrm{zebra}}^{i}(\boldsymbol{\mathrm{x}};\varphi),\] where \(f_{\mathrm{zebra}}(\boldsymbol{\mathrm{x}};\varphi) \in \mathbb{R}^{c}\) denotes the fused logit vector over all \(c\) classes, and \(f_{\mathrm{zebra}}^{i}(\boldsymbol{\mathrm{x}};\varphi)\) represents the logit corresponding to \(i_{\mathrm{th}}\) class. It should be noted that few-shot training is performed using classes in \(\mathcal{C}_B\), while evaluation is conducted separately on base classes \(\mathcal{C}_B\) and novel classes \(\mathcal{C}_N\).
It is worth noting that ZEBRA introduces no additional learnable parameters to existing prompt-learning baselines. The zero-shot logits are computed only once and can be reused during both few-shot training and inference, eliminating the need for additional forward passes through the text encoder. As a result, ZEBRA is a lightweight approach that incurs negligible computational overhead. For an overview of the proposed approach, see Figure 2.

4 Experiments and Results↩︎

Models and Datasets. For the CLIP-style audio-language backbone, we adopt Pengi [1], a generative audio-language model consisting of audio and text encoders followed by an LLM decoder. Following the setup of PALM [12], we discard the decoder and utilize only the pretrained audio and text encoders, effectively employing PENGI in a CLIP-like contrastive framework to leverage its strong pretrained representations and zero-shot generalization capability. We evaluate our method on a diverse set of speech and audio processing tasks, including instrument classification, sound event classification, emotion recognition, vocal sound classification, surveillance sound event classification, acoustic scene classification, and music analysis. Instrument classification is assessed using the Beijing Opera [15] and NS-Instruments [16] datasets. For sound event classification, we use ESC-50 and its ESC50-Actions subset [17], as well as UrbanSound8K [18]. Emotion recognition is evaluated on CREMA-D [19] and RAVDESS [20]. Vocal sound classification is conducted using VocalSound [21], while surveillance sound event classification relies on SESA [22]. Acoustic scene classification is performed on TUT2017 [23], and music analysis is evaluated using GT-Music-Genre [24].

Baseline Methods. We consider ZERO-SHOT, COOP [5], and COCOOP [6] as baselines. COOP and COCOOP are prompt learning methods originally proposed for vision-language models that replace handcrafted prompts with learnable context tokens optimized in the text encoder’s input embedding space; COCOOP further introduces instance-conditioned prompts via feedback from the audio encoder, whereas COOP learns context tokens shared across instances. We adopt their audio-language adaptations as implemented in PALM [12], where the vision encoder is replaced with an audio encoder. We exclude PALM itself as a baseline because its class-specific learnable vectors limit base-to-novel generalization, while COOP and COCOOP are class-agnostic and can generalize to unseen classes. Our method builds upon COOP and COCOOP in a plug-and-play manner, seamlessly integrating with their learned prompt representations without introducing additional trainable parameters or requiring further hyperparameter tuning. It operates on top of the existing framework, preserving their training protocols and generalization capabilities, while providing consistent improvements with minimal computational overhead and no modifications to the underlying model architecture.
Implementation Details. Except for ZERO-SHOT, all methods are trained for \(50\) epochs under the standard few-shot setting, using \(16\) randomly sampled training examples per base class. Inference is conducted on the full test set using the predefined base–novel class split. Few-shot training is done with stochastic gradient descent (SGD) at a learning rate of \(0.05\), and performance is measured in terms of accuracy. Each method is evaluated with three different random seeds, and we report the average results. For ZERO-SHOT, we adopt the default prompt template: This is a recording of {CLASS NAME}, for a fair comparison. We run all experiments using NVIDIA RTX A6000 GPU. For the contribution weights in Equation 3 , we empirically set \(\lambda_{\mathrm{zs}} = 0.5\) and \(\lambda_{\mathrm{pr}} = 0.5\). We also scale the \(\mathcal{L}_{\mathrm{ent}}\) by a factor of \(0.05\) during few-shot training.
Results and Discussion. We report a comparison of ZERO-SHOT, prompt-learning methods, and their ZEBRA-enhanced variants in Table 1. As discussed earlier, baseline methods (COOP and COCOOP) substantially improve performance on base classes compared to ZERO-SHOT (e.g., \(+26.2\%\) and \(+28.5\%\), respectively). However, they generalize poorly to novel classes, exhibiting notable drops in accuracy (e.g., \(-7.13\%\) and \(-4.74\%\), respectively) and, in both cases, performing below the ZERO-SHOT novel-class average of \(55.18\%\). On the other hand, ZEBRA enhances their novel-class performance by \(+4.19\%\) and \(+4.31\%\), respectively, bringing the accuracy above the ZERO-SHOT novel-class average of \(55.18\%\) while maintaining competitive performance on base classes. Results demonstrate that, on average, ZEBRA improves novel-class generalization while maintaining base performance, achieving these gains with no additional parameters and negligible computational overhead (see Table 3).
Ablative Analysis. We analyze the impact of the fusion of zero-shot logits and self-entropy loss term on ZEBRA’s performance in Table 2 (using the COOP baseline). The results indicate that most of the improvement comes from incorporating zero-shot logits, while the self-entropy term provides a further, albeit marginal, gain. We also report runtime (training and testing, averaged across all datasets) and Expected Calibration Error (ECE) in Table 3. ZEBRA introduces negligible computational overhead while consistently reducing ECE on average across both base and novel classes, demonstrating improved calibration without sacrificing efficiency.

5 Conclusion↩︎

We introduced ZEBRA, a lightweight, plug-and-play framework for improving base-to-novel generalization in audio–language models. While existing prompt-learning methods substantially boost base-class performance, they often suffer from degraded generalization to novel classes, frequently underperforming the zero-shot baseline. ZEBRA effectively mitigates this trade-off by leveraging zero-shot knowledge and entropy-regularized logits, consistently improving novel-class accuracy while preserving strong base-class performance. ZEBRA adds no learnable parameters and incurs negligible computational overhead, making it a simple yet effective enhancement to existing prompt-learning methods.

6 Generative AI Use Disclosure↩︎

We confirm that an LLM was used solely for writing refinement (grammar, wording, and clarity). All ideas, analyses, and conclusions are the authors’ own.

References↩︎