1 Introduction↩︎

Large-scale Vision-Language Models (VLMs) [1]–[3] have become widely used foundation models for effective transfer learning in various downstream tasks. A representative VLM is CLIP [1], which is trained on 400 million image–text pairs and aligns the two modalities via contrastive pre-training. As a result, CLIP serves as an effective transferable backbone for various downstream tasks, including few-shot classification [4], [5], semantic segmentation [6], [7], and visual question answering [8], [9]. However, adapting VLMs by updating all model parameters requires considerable computational resources and increases the risk of overfitting, especially in few-shot scenarios. Therefore, researchers explore parameter-efficient fine-tuning (PEFT) methods for adapting large VLMs to downstream tasks with a small number of trainable parameters.

Existing few-shot VLM adaptation approaches include prompt learning and adapter-based methods. Prompt learning methods typically introduce learnable prompts to adapt VLMs to target tasks. For instance, CoOp [10], CoCoOp [11], and ATPrompt [12] replace hand-crafted textual templates with learnable continuous context tokens or incorporate attribute-aware words into the text prompts to facilitate task adaptation. Beyond input-level prompt tuning, some methods such as MaPLe [13], PromptSRC [14], and HiCroPL [15] integrate learnable tokens into intermediate layers of pre-trained VLMs, enabling the models to capture task-specific knowledge while maintaining cross-modal alignment. Adapter-based methods aim to design a lightweight module that are inserted into pre-trained VLMs to enable parameter-efficient adaptation. For example, GraphAdapter [16] and Hegraphadapter [17] leverage graph convolutional networks to capture structured cross-modal relationships, while CLIP-Adapter [18] and MMA [19] introduce feature transformation modules that operate on visual and/or textual representations, enhancing visual-text alignment.

In this paper, we first observe that, for few-shot learning tasks, pre-trained VLMs can exhibit a notable capability in filtering candidate categories and thus obtain a high Top‑\(K\) accuracy. Nevertheless, they often struggle with fine-grained discrimination among visual similar categories, resulting in unsatisfactory Top‑1 performance. Figure 1 shows an example: while Top-\(1\) accuracy is low, Top-\(5\) accuracy exceeds \(90\%\). A similar observation has also been obtained in a recent study [20]. This suggests that pre-trained VLMs tend to encode only coarse representations for visual data. Therefore, how to enhance these representations with fine-grained discrimination becomes crucial when adapting VLMs to downstream tasks. However, after reviewing the above studies on VLM fine-tuning adapters, we observe that existing adapter approaches primarily focus on global alignment between visual and textual representations in traditional feature spaces. They generally overlook fine-grained cues among similar categories, thereby limiting their effectiveness in few-shot scenarios.

Based on the above observations, this paper proposes a novel coarse-to-fine VLM fine-tuning approach for few-shot learning by leveraging quantum computation theory [21]–[24], termed Multi-Modal Quantum Adapter (MQAdapter). MQAdapter begins by retrieving the Top-\(K\) textual categories most similar to the input visual image \({I}\) via pre-trained VLM model, which serve as semantic anchors \(\mathcal{P}\) to guide the adaptation process. Then, the subsequent goal of our MQAdapter is to learn a highly discriminative visual representation for image \({I}\) by fully exploiting the modality interactions between \({I}\) and its semantic anchors \(\mathcal{P}\). To achieve this, inspired by the superposition and entanglement mechanism [25] in quantum computation field, we equip MQAdapter with a novel cross-modal quantum learning mechanism to enhance the representation of visual features under the guidance of semantic anchors. Specifically, it first encodes visual and textual features into a unified quantum representation space using a shared variational quantum circuit (VQC). Then, it models higher-order cross-modal interaction effectively by leveraging quantum entanglement and superposition within a high-dimensional Hilbert space [26], resulting in more discriminative representations than those from traditional Euclidean adapters. Finally, the quantum-enhanced visual features are projected back to the original feature dimension and injected into the frozen visual representation to achieve adaptation. MQAdapter is parameter-efficient and can be combined with various existing fine-tuning approaches to achieve fine-grained VLM adaptations.

We note that the quantum learning models have been employed some large-scale foundation model fine-tuning tasks in recent years [27]–[29]. These works generally utilize quantum or quantum-inspired parameterizations to enable compact full-rank or high-rank adaptation for LLMs. Differently, we focus on multi-modal VLM fine-tuning. The key difference of our MQAdapter is to leverage parameterized quantum circuits to achieve fine-grained modality-interaction between visual and textual semantic anchors, which ultimately enhances visual-text fine-grained alignment for VLM adaptation on downstream tasks. To our best knowledge, this is the first work to explore quantum computation for multi-modality adapter.

Overall, the main contributions of this paper are summarized as follows:

• We propose a novel coarse-to-fine VLM fine-tuning approach, termed MQAdapter by leveraging quantum computing model. To the best of our knowledge, this is the first work to explore multi-modal quantum computation for parameter-efficient few-shot VLM adaptation.

• We design a cross-modal quantum learning model to capture high-order interactions between visual and textual modalities in a shared quantum Hilbert space, resulting in fine-grained image-text representation learning.

• MQAdapter follows a plug-and-play design for VLM adaptation. It can be integrated into different VLM fine-tuning frameworks and consistently improves performance while requiring only 0.078M additional trainable parameters.

The remainder of this paper is organized as follows. In Section II, we review the related work. Section III introduces the preliminaries of quantum computing. Section IV presents the proposed MQAdapter. Section V reports experiments and ablation studies. Section VI concludes the paper.

2 Related Work↩︎

2.1 Efficient Transfer Learning for VLMs↩︎

Large-scale Vision-Language Models (VLMs) are pre-trained on large-scale image–text pairs and demonstrate strong generalization abilities across various downstream applications [6], [7], [30]–[32]. VLMs such as CLIP [1], FILIP [2], and SigLIP2 [3] learn a shared embedding space through contrastive learning, enabling effective zero-shot and few-shot transfer. Despite their success, directly adapting these models to downstream tasks remains challenging. To address these challenges, recent works are focused on Parameter-Efficient Transfer Learning (PETL) methods that adapt pretrained VLMs to downstream tasks while maintaining their strong generalization ability. PETL methods can be broadly categorized into prompt learning and adapter-based approaches.

Prompt learning [13], [14], [33], [34] optimizes a limited set of tokens for adapting pretrained VLMs. For example, CoOp [10] introduces learnable context tokens to construct task-specific text prompts, replacing hand-crafted templates. CoCoOp [11] further refines the textual representation by conditioning the prompts on visual features from each input sample. TextRefiner [35] enriches text prompts with local visual concepts extracted from the image branch of VLMs. ATPrompt [12] enriches textual templates with class-specific attribute descriptions, thereby enhancing the transferability of prompt learning. TCP [36] enhances generalization by transforming textual priors into class-aware prompt tokens.

Adapter-based methods [16], [19], [37] introduce lightweight modules into pretrained VLMs to facilitate task-specific adaptation. For example, CLIP-Adapter [18] introduces two-layer MLP adapters in the visual branch to learn task-specific feature representations. Tip-Adapter [38] adopts a training-free strategy by caching few-shot visual embeddings and performing similarity-based inference at test time. However, these methods typically adapt visual and textual modalities independently. To enable cross-modal interaction, MMA [19] uses a shared fully connected layer to establish connection between encoders, allowing gradients to propagate across branches and improving alignment. PVA [39] introduces modality-specific variational adapters attached to align visual and textual representations at the distribution level, enabling robust adaptation under data imbalance.

2.2 Quantum Machine Learning↩︎

Quantum Machine Learning (QML) [21]–[23] introduces a novel learning paradigm that integrates quantum computing with classical machine learning methods. QML approaches employ variational quantum circuits, leveraging quantum entanglement and superposition to model complex dependencies within data. Recently, QML has also been explored for efficient adaptation of Large Language Models (LLMs). For example, Quantum-PEFT [27] leverages parameterized quantum circuits to realize full-rank and parameter-efficient LLM adaptation. QTHA [28] integrates quantum neural networks as an adaptation module to alleviate the limitations of classical low-rank fine-tuning. Similarly, Quantum Parameter Adaptation (QPA) [29] adopts hybrid quantum-classical mappings to generate compact fine-tuning parameters. QuanTA [40] is inspired by quantum circuits and uses a unitary matrix and tensor parameterizations to enable high-rank LLM fine-tuning. These methods aim to overcome the expressiveness limitations of conventional low-rank fine-tuning, enabling compact full-rank or high-rank adaptation for LLMs. Beyond language models, QML has also been investigated in vision and multi-modal learning. For example, Zhang et al. [41] simulate quantum computation by representing hyperspectral cuboids in a quantum-like form with amplitude and phase, thereby modeling spectral–spatial interactions for hyperspectral image classification. Pokharel et al. [25] leverage quantum entanglement for multi-modal fusion and introduce a modality-aware circuit isolation mechanism to ensure stable quantum training under missing modalities. QViLa [42] first combines extracted visual and textual embeddings and then employs a quantum-augmented layer to enhance global multi-modal feature fusion. MEDQUA [43] inserts a quantum adapter into pretrained medical VLM decoders, where selected decoder tokens are routed to a shallow quantum module and all tokens are processed by a LoRA-like classical path. Unlike previous works, our MQAdapter fully leverages multi-modal cues to guide visual feature adaptation for VLMs, where different modalities are mapped into a unified quantum-enhanced representation space to model fine-grained cross-modal interactions and generate adaptation biases for frozen VLM layers.

3 Preliminaries of Quantum Computing↩︎

3.1 Quantum States and Superposition↩︎

The state of a quantum system is represented in a complex Hilbert space, which encodes both the amplitude and phase. Phase information enables quantum interference, allowing different components of a quantum state to combine constructively or destructively [26]. An \(N\)-qubit system is represented in a \(2^N\)-dimensional complex Hilbert space \(\mathbb{C}^{2^N}\). Quantum states are commonly represented using Dirac notation [44]. The state of a single qubit can be written as: \[|\psi\rangle = \alpha |0\rangle + \beta |1\rangle,\] where \(\alpha, \beta\) are the amplitudes and satisfy \(|\alpha|^2 + |\beta|^2 = 1\). The \(|0\rangle\) and \(|1\rangle\) denote the computational basis of Hilbert space. The probabilities of observing the states \(|0\rangle\) or \(|1\rangle\) are given by \(|\alpha|^2\) and \(|\beta|^2\), respectively. In addition, the qubit state can be visualized on the Bloch sphere which is rewritten as: \[|\psi\rangle = \cos(\frac{\theta}{2})|0\rangle + e^{i\varphi}\sin(\frac{\theta}{2})|1\rangle,\] where \(\theta \in [0,\pi]\) and \(\varphi \in [0,2\pi)\). The \(i\) represents the imaginary number and satisfies \(i^2=-1\). Unlike classical bits, which are restricted to binary states, qubits can exist in superpositions of \(|0\rangle\) and \(|1\rangle\), thereby allowing more expressive representations with compact parameterization.

3.2 Parameterized Quantum Circuit↩︎

Quantum gates serve as the fundamental building blocks of quantum computation. Parameterized rotation gates adjust the orientation of individual qubits, and controlled gates establish correlations and entanglement between qubits. A sequence of parameterized quantum gates forms a quantum circuit. Using a classical optimizer [45], the parameters of the quantum circuit are updated to approximate a desired quantum state. Formally, it can be represented as a parameterized unitary transformation \(U(\boldsymbol{\phi})\) acting on an \(n\)-qubit system as: \[|\psi(\boldsymbol{\phi})\rangle = U(\boldsymbol{\phi}) |0\rangle^{\otimes N},\] where \(\boldsymbol{\phi}\) denotes the set of trainable parameters associated with the quantum gates.

3.3 Quantum Measurement↩︎

Quantum measurement aims to extract classical representations from a quantum state [46]. Given an \(n\)-qubit state \(|\psi(\boldsymbol{\phi})\rangle\) output from the quantum circuit, the expectation value associated with an observable \(O_m\) is given by: \[\boldsymbol{q}_m(\boldsymbol{\phi}) = \langle \psi(\boldsymbol{\phi}) | O_m | \psi(\boldsymbol{\phi})\rangle,\] In quantum computation, classical data are first encoded into quantum states. A quantum circuit is then applied to transform these states. Finally, the quantum outputs are decoded into classical representations through measurement.

4 The Proposed MQAdapter↩︎

Following previous studies [12], [18], [47], [48], we build on the pre-trained CLIP model [1]. MQAdapter first retrieves Top-\(K\) candidate categories as semantic anchors using a frozen pre-trained CLIP model. Then, it models high-order cross-modal interaction between visual features and semantic anchors via quantum learning, thereby refining visual representations and enhancing few-shot downstream adaptation. In this section, we introduce these processes in detail.

4.1 Semantic Anchors↩︎

The pre-trained VLM model (e.g., CLIP) consists of a visual encoder and a text encoder. For few-shot learning tasks, given an input visual image \({I}\) and all textual categories \(\{{t}_1, {t}_2\cdots {t}_C\}\), where \(C\) denotes the number of classes, we first obtain the feature descriptors for them via the frozen pre-trained CLIP visual and text encoder, respectively. We compute the similarities between image \({I}\) and all \(C\) text categories using their feature descriptors, and then select Top-\(K\) candidate categories for image \({I}\) with the highest similarity scores. In the following, we denote \[\mathbf{P}=[\boldsymbol{p}_1,\boldsymbol{p}_2\cdots \boldsymbol{p}_K]\] as the features of top \(K\) candidate categories, and refer to them as semantic anchors.

As illustrated in Figure 1, pre-trained CLIP model can obtain a high Top-\(K\) accuracy, exhibiting a notable capability in filtering candidate categories. However, it often struggles with fine-grained discrimination among visual similar categories, resulting in unsatisfactory Top-1 performance. Based on this observation, the aim of our MQAdapter is to obtain a highly discriminative visual representation by fully exploiting the relationships between visual image \(I\) and its semantic anchors \(\mathbf{P}\) via cross-modal quantum learning. In the following, we first introduce the variational quantum circuit (VQC) in our cross-modal quantum learning, and then present the adaptation process of our MQAdapter.

4.2 Variational Quantum Circuit↩︎

We leverage quantum computation theory and design a Variational Quantum Circuit (VQC) function \(f_{\phi}(\cdot)\) that transforms classical input features into quantum-enhanced feature representations. As shown in Fig. 2 (c), the VQC consists of three steps: data encoding, parameterized quantum circuit (PQC), and measurement. Given a feature vector \(\boldsymbol{x}=(x_1,\dots,x_N)\in\mathbb{R}^{N}\), we first encode it into an \(N\)-qubit quantum state via angle encoding [49], [50]: \[\ket{\psi(\boldsymbol{x})}= \bigotimes_{n=1}^{N} \left(\mathrm{R_y}(x_n)\ket{0}_n\right),\] where \(\mathrm{R_y}(\cdot)\) is a single-qubit rotation gate. Subsequently, we construct a parameterized quantum circuit \(U(\boldsymbol{\phi})\) to transform the encoded quantum state \(\ket{\psi(\boldsymbol{x})}\) as follows: \[\begin{align} U(\boldsymbol{\phi}) &= \Big(\bigotimes_{n=1}^{N}\mathrm{R_y}(\boldsymbol{\phi}^{(2)}_{n})\Big)\cdot \Big(\prod_{(n,m)\in\mathcal{E}}\mathrm{CRZ}(\boldsymbol{\phi}_{n,m})\Big) \nonumber\\ &\quad\cdot \Big(\bigotimes_{n=1}^{N}\mathrm{R_y}(\boldsymbol{\phi}^{(1)}_{n})\Big), \\ \ket{\psi'(\boldsymbol{x})}&=U(\boldsymbol{\phi})\ket{\psi(\boldsymbol{x})}, \end{align}\] where \(\mathcal{E}=\left\{(n,n+1)\right\}_{n=1}^{N-1} \cup \left\{(N,1)\right\}\) adopts a ring nearest-neighbor entanglement, and \(\boldsymbol{\phi}^{(1)}\) and \(\boldsymbol{\phi}^{(2)}\) denote the learnable rotation parameters. \(\mathrm{CRZ}\!\left(\boldsymbol{\phi}_{n,m}\right)\) represents a two-qubit controlled-Z rotation gate [24] that introduces entanglement between qubits \(n\) and \(m\). Finally, each qubit is measured via the Pauli-\(Z\) [24] observable to produce a quantum-enhanced feature representation: \[\begin{align} \label{eu95vqc} f_{\phi}(\boldsymbol{x}) &= \big[ \langle \psi'(\boldsymbol{x})|Z_1|\psi'(\boldsymbol{x})\rangle,\dots, \langle \psi'(\boldsymbol{x})|Z_N|\psi'(\boldsymbol{x})\rangle \big], \end{align}\tag{1}\] where \(Z_N\) denotes the Pauli-\(Z\) operator acting on the \(N\)-th qubit.

4.3 Fine-grained Adaptation↩︎

As shown in Fig. 2 (b), given the input image \(I\) feature representation and associated semantic anchors \(\mathbf{P}\), our MQAdapter first projects them to the same dimension and employs a shared VQC (Eq. 1 ) module to map visual features and textual semantic anchors into a unified quantum representation space. This design yields modality-consistent representations, effectively alleviating the visual-text modality gap in subsequent interactions. We then compute attention weights to quantify cross-modal relevance and obtain multi-modal features. These features are subsequently fed into the same shared VQC, where quantum superposition and entanglement operations enable higher-order cross-modal interactions within a high-dimensional Hilbert space.

To be specific, given an input image \({I}\), we obtain the visual representations of each CLIP layer as: \[\begin{align} [\boldsymbol{c}_0,\boldsymbol{v}_0]&=\mathrm{ImageEmbed}({I}),\\ [\boldsymbol{c}_\ell,\boldsymbol{v}_\ell]&=\mathcal{V}([\boldsymbol{c}_{\ell-1},\boldsymbol{v}_{\ell-1}]) \quad \ell = 1, \ldots, L \end{align}\] where \(\mathrm{ImageEmbed}(\cdot)\) splits the input image \({I}\) into fixed-size patches and projects them into feature embeddings and \(\mathcal{V}(\cdot)\) denotes visual encoder. \(\boldsymbol{c}_\ell\in \mathbb{R}^{d_v}\) denotes the class token at the \(\ell\)-th visual layer. Next, we project the visual class token \(\boldsymbol{c}_\ell\) and the text semantic anchors \(\mathbf{P}=[\boldsymbol{p}_1,\cdots, \boldsymbol{p}_K]\) into a \(N\)-dimensional space as: \[\begin{align} \bar{\boldsymbol{c}}_\ell = \mathbf{W}^v_\ell\boldsymbol{c}_\ell ;\; \bar{\mathbf{P}} = \mathbf{W}^t\mathbf{P}, \end{align}\] where \(\mathbf{W}^v_\ell\) and \(\mathbf{W}^t\) denote learnable modality-specific projection matrices and \(N\) denotes the number of qubits. This projection aligns the input dimensions with the number of qubits, since each input dimension is encoded as the rotation angle of one qubit.

The projected visual and textual features are encoded by using a shared VQC module \(f_{\phi}(\cdot)\) (Eq.(8)): \[\tilde{\boldsymbol{c}}_\ell = f_{\phi}(\bar{\boldsymbol{c}}_\ell), \quad \tilde{\mathbf{P}} = [f_{\phi}(\bar{\boldsymbol{p}_1}),\cdots,f_{\phi}(\bar{\boldsymbol{p}_K})].\] By sharing the VQC across modalities, visual and textual features are mapped into a unified quantum-enhanced representation space. Then, we compute the cross-modal attention between the quantum-enhanced visual token and semantic anchors as: \[\begin{align} \boldsymbol{\alpha}_\ell &= \mathrm{softmax}\!\left(\frac{ \tilde{\mathbf{P}}^{\top}\tilde{\boldsymbol{c}}_\ell}{\sqrt{ N}}\right)\in \mathbb{R}^K,\\ \boldsymbol{h}_\ell& = \tilde{\mathbf{P}}\boldsymbol{\alpha}_\ell= \sum_{k=1}^{K} \alpha_{\ell,k}\,\tilde{\boldsymbol{p}}_k, \end{align}\] where \(\boldsymbol{h}_\ell\) aggregates semantic priors from the selected textual anchors and obtains a multi-modal representation. After that, we feed \(\boldsymbol{h}_\ell\) into the shared VQC to enable cross-modal higher-order interaction. Finally, the quantum-enhanced multi-modal feature is projected back to the original visual feature dimension through projection \(\mathbf{W}_\ell\) and injected into the visual representation via a residual connection as: \[\begin{align} \hat{\boldsymbol{c}}_\ell &= \boldsymbol{c}_\ell + \gamma\,\mathbf{W}_\ell f_{\phi}(\boldsymbol{h}_\ell), \end{align}\] where \(\gamma\) is a scaling hyperparameter.

4.4 Fine-tuning for Downstream Tasks↩︎

For the downstream task, as shown in Fig. 2 (a), we integrate MQAdapter into the last several layers of CLIP model to enable task-specific adaptation while maintaining training efficiency. Given an input image \({I}\) and a set of \(C\) category-level textual descriptions \(\{{t}_1, \cdots,{t}_C\}\), we obtain the refined visual representations \(\boldsymbol{z}^v\) and textual representations \(\{\boldsymbol{z}^t_1,\cdots,\boldsymbol{z}^t_C\}\). The model is optimized using a temperature-scaled cross-entropy loss, which is formulated as: \[\mathcal{L}_{\mathrm{cls}} = - \log \frac{ \exp\!\left( \cos\!\left( \boldsymbol{z}^v, \boldsymbol{z}^t_y\right) / \tau \right) }{ \sum_{i=1}^{C} \exp\!\left( \cos\!\left( \boldsymbol{z}^v, \boldsymbol{z}^t_i \right) / \tau \right) },\] where \(y\) denotes the ground-truth label. By modeling cross-modal interactions in the quantum representation space, our framework jointly optimizes the image and text encoders, enhancing adaptation performance for downstream tasks.

5 Experiments↩︎

5.1 Experimental Setup↩︎

5.1.1 Datasets and Tasks↩︎

Following prior works [13], [14], [19], [51], we evaluate our models’ performance on four tasks, including base-to-new generalization, few-shot image classification, cross-dataset transfer, and domain generalization. All experiments are conducted under the 16-shot setting, except for the few-shot learning experiments.

Base-to-New Generalization: In this task, each dataset is equally split into base and new classes. The model is fine-tuned on base classes using a \(16\)-shot setting and is evaluated on both base and new classes. This evaluation assesses the balance between effective transfer learning on base classes and preserving the generalization when adapting to new classes. We perform these experiments on 11 image classification datasets: ImageNet [52], Caltech101 [53], OxfordPets [54], StanfordCars [55], Flowers102 [56], Food101 [57], FGVCAircraft [58], SUN397 [59], DTD [60], EuroSAT [61], and UCF101 [62].

Few-Shot Image Classification: This task is designed to evaluate the model’s adaptation ability under limited-data conditions. Specifically, we train the model with 1, 2, 4, 8, and 16 samples (shots) per class, and report the performance on the full test set. This evaluation is performed on the same 11 image classification datasets as the base-to-new generalization task.

Cross-Dataset Transfer: To assess the model’s ability to generalize to new datasets, we regard ImageNet as the source dataset and others as target datasets. After training with few-shot samples from all 1,000 ImageNet classes, the model is directly applied to the target datasets without updating any parameters.

Domain Generalization: To evaluate robustness to unseen domains, we train the model on ImageNet and evaluate it on four ImageNet variants with different distribution shifts, including ImageNet-V2 [63], ImageNet-Sketch [64], ImageNet-A [65], and ImageNet-R [66].

5.1.2 Implementation details↩︎

To verify the effectiveness of the proposed Multi-modal Quantum Adapter (MQAdapter), we plug MQAdapter into three representative PEFT frameworks, including MaPLe [13], PromptSRC [14], and MMRL++ [51]. All experiments are conducted under a 16-shot setting, except for the few-shot learning task. We adopt the CLIP model with a ViT-B/16 backbone for all methods. For a fair comparison, we follow the training protocols of the corresponding PEFT frameworks, keeping the batch size, learning rate, and number of training epochs consistent with their original implementations. We implement the 8-qubit shared VQC using TorchQuantum. The MQAdapter is inserted into the visual encoder starting from the \(6\)-th transformer layer to the final layer. The adapter scaling parameter is set to \(\gamma=0.001\) for the base-to-new task. For the few-shot image classification, we set \(\gamma=0.001\) for MaPLe, \(\gamma=0.0001\) for PromptSRC and \(\gamma=0.002\) for MMRL++. During the coarse stage, we select the Top-\(5\) most relevant text categories as semantic anchors for guiding fine-grained visual feature representation.

5.2 Comparison Results↩︎

5.2.1 Base-to-New Generalization↩︎

We integrate MQAdapter into three representative fine-tuning methods: MaPLe [13], PromptSRC [14], and MMRL++ [51]. We further compare MQAdapter-enhanced methods with several state-of-the-art VLM adaptation approaches, including CLIP [1], COMMA [34], MMA [19], and recent plug-and-play methods, PromptSRC+DePT [67], TCP [36], PromptSRC+DPC [48], and PromptSRC+ATP [12]. We report accuracy on both base classes (Base) and new classes (New), and use their harmonic mean (H) to evaluate the comprehensive performance between them in Tab. 1. MQAdapter improves the harmonic mean of MaPLe, PromptSRC, and MMRL++ from 78.03%, 79.67%, and 81.54% to 80.31%, 80.73%, and 82.36%, achieving gains of 2.28%, 1.07%, and 0.82%, respectively. For MaPLe, MQAdapter brings significant improvements on several challenging datasets, such as EuroSAT, DTD, UCF101, and FGVCAircraft. For PromptSRC and MMRL++, MQAdapter also achieves consistent gains on all datasets, demonstrating its robustness across different adaptation frameworks. Moreover, MQAdapter also achieves the best average harmonic mean than existing plug-and-play approaches. These results demonstrate that MQAdapter effectively retrieves the Top-\(K\) textual categories most relevant to the input image and uses them as semantic anchors to guide visual feature adaptation. By further exploiting the cross-modal interactions between the visual representation and these semantic anchors via quantum computation, MQAdapter learns more discriminative visual features and improves generalization on both base and new classes.

5.2.2 Few-shot Image Classification↩︎

We further evaluate MQAdapter on few-shot image classification under 1-, 2-, 4-, 8-, and 16-shot settings. MQAdapter is integrated into three representative frameworks, i.e., MaPLe [13], PromptSRC [14], and MMRL++ [51]. The results are reported in Tab. 2. We compare our methods with recent competitive few-shot adaptation methods, including CoOp [10], CoCoOp [11], GraphAdapter [4], MaPLe [13], PromptSRC [14], and MMRL++ [51]. As shown in Tab. 2, MQAdapter consistently improves the performance of MaPLe, PromptSRC, and MMRL++ across different shot settings. Specifically, when combined with MaPLe, MQAdapter improves the average accuracy by 3.75%, 2.91%, 2.07%, 1.34%, and 0.94% under the 1-, 2-, 4-, 8-, and 16-shot settings, respectively. For PromptSRC, the corresponding gains are 1.94%, 1.74%, 0.96%, 0.89%, and 0.72%. For MMRL++, MQAdapter brings improvements of 1.49%, 1.60%, 0.79%, 1.05%, and 0.76%, respectively. MQAdapter achieves the largest gains in low-shot scenarios. For example, MQAdapter improves MaPLe, PromptSRC, and MMRL++ by 3.75%, 1.94%, and 1.49% under the 1-shot setting, and by 2.91%, 1.74%, and 1.60% under the 2-shot setting, respectively. The consistent gains across different baselines, datasets, and shot numbers verify the robustness and effectiveness of MQAdapter for few-shot VLM adaptation.

5.2.3 Cross-Dataset Transfer and Domain Generalization↩︎

As shown in Tab. 3, MQAdapter consistently improves cross-dataset transfer performance when applied to MaPLe, PromptSRC, and MMRL++. The average accuracy increases by 0.62%, 0.45%, and 0.61%, respectively. Tab. 4 further shows that MQAdapter also improves domain generalization for all baselines across different ImageNet variants. These results indicate that MQAdapter enhances adaptation performance while preserving the transferability of pre-trained VLMs.

5.3 Ablation Experiments↩︎

5.3.1 Effectiveness of Cross-modal Quantum Learning mechanism↩︎

To evaluate the effectiveness of the cross-modal quantum learning mechanism, we conduct ablation experiments on the MMRL++ baseline. Specifically, we replace the cross-modal quantum learning mechanism with a conventional cross-attention module [68] to perform fine-stage cross-modal interaction. This variant improves the harmonic mean by 0.23%, indicating that the text semantic anchors can provide useful guidance for visual feature learning. The proposed MQAdapter achieves the harmonic mean of 82.36%, outperforming the baseline by 0.82% and conventional cross-attention by 0.59%. This demonstrates that MQAdapter learns more discriminative visual representations than conventional cross-attention by effectively modeling higher-order cross-modal interactions. Our MQAdapter use a shared VQC to align different modalities and facilitate cross-modal interaction in a unified quantum representation space. To validate the effectiveness of this design, we conduct an ablation study by replacing the shared VQC with separate VQCs. The results show that the shared VQC achieves better performance than the separate VQC, demonstrating the effectiveness of our cross-modal quantum learning mechanism while requiring fewer quantum resources.

5.3.2 Influence of hyper-parameters in MQAdapter↩︎

To evaluate the influence of hyper-parameters in MQAdapter, we perform experiments on the MMRL++ baseline. Fig. 3 (a) presents the average base-to-new generalization performance on 11 datasets under different choices of Top-\(K\) textual candidates. When \(K\) increases from 2 to 5, the harmonic mean improves from 81.89% to 82.36%. This indicates that semantically relevant textual candidates provide effective guidance for visual feature refinement. However, further increasing \(K\) does not bring additional gains. This suggests that more candidates may introduce less relevant categories and weaken the discriminative guidance for visual representation learning. Therefore, we set \(K=5\) in the final model. Fig. 3 (b) shows the influence of the scaling factor \(\gamma\). MQAdapter achieves the best harmonic mean of 82.36% when \(\gamma=0.001\). A smaller \(\gamma\) may limit the model’s ability to adapt to downstream tasks. In contrast, a larger \(\gamma\) increases the reliance on task-specific features, which may reduce transferability to new classes.

5.3.3 Influence of MQAdapter insertion depth↩︎

We further analyze the impact of inserting MQAdapter at different layers of the visual encoder. The experiments are conducted on the MMRL++ baseline. We report average base-to-new generalization performance on 11 datasets. As shown in Fig. 4, we obtain the best performance when inserted from layer 6. When MQAdapter is inserted from shallower layers, the accuracy decreases on both base and new classes. This suggests that early CLIP layers mainly capture general visual information and are less suitable for task-specific adaptation. In contrast, when MQAdapter is inserted from deeper layers, only a few blocks can be adapted, which may be insufficient to progressively enhance visual representations. These results show that inserting MQAdapter at intermediate layers provides a better balance between preserving CLIP’s general knowledge and adapting to downstream tasks.

5.4 Training Stability↩︎

To assess the training stability of our MQAdapter, we integrated it into the MMRL++ baseline and report the mean performance and standard deviation (std) over three random seeds. As shown in Table 6, +MQAdapter achieves lower standard deviation than MMRL++ while maintaining superior performance across the evaluated datasets. This indicates that MQAdapter not only improves adaptation accuracy but also enables a more stable optimization process under different random initializations.

5.5 Visualization Results Analysis↩︎

We visualize the learned visual features with t-SNE on the base and new classes of EuroSAT and OxfordPets datasets. The results are demonstrated in Fig. 5. The CLIP provides coarse category-level separability, but still struggles to distinguish semantically similar classes. This observation is consistent with our motivation. Compared with CLIP, MMRL++ adapts the visual representation to downstream data, resulting in more compact class-wise clusters and improved inter-class separability. By incorporating MQAdapter, the learned features become more discriminative, leading to larger inter-class margins and more compact intra-class distributions. These results indicate that MQAdapter effectively leverages semantically related textual cues to refine visual representations. Moreover, MQAdapter models higher-order cross-modal interactions in a high-dimensional Hilbert space between visual and textual modalities. This enhances the visual encoder’s ability to capture fine-grained semantic features, thereby achieving better generalization on both base and new classes.

5.6 Efficiency Analysis of MQAdapter↩︎

To evaluate the efficiency of our MQAdapter, we report the number of learnable parameters and inference speed compared with other state-of-the-art methods. The results are shown in Tab. 7, where FPS is measured with a batch size of 100 on a single NVIDIA RTX 3090 GPU. Compared with the MMRL++ framework, MQAdapter introduces only \(0.078\)M additional learnable parameters. Specifically, the proposed Variational Quantum Circuit (VQC) module requires only \(0.024\)K parameters, which is substantially fewer than a standard fully connected layer operating on conventional feature space. These results indicate that the proposed MQAdapter maintains competitive inference efficiency while consistently improving adaptation performance.

3pt

6 Conclusion↩︎

In this paper, we introduced MQAdapter, a parameter-efficient coarse-to-fine adaptation framework for few-shot VLM fine-tuning. MQAdapter first retrieves semantically relevant Top-\(K\) textual candidates from the frozen VLM as anchors, and then leverages them to guide fine-grained visual representation refinement. By mapping visual and textual features into a quantum-enhanced representation space, MQAdapter leverages quantum superposition and entanglement to capture higher-order visual-text interactions, enabling effective adaptation of frozen VLMs. Extensive experiments on four tasks demonstrate that MQAdapter consistently improves downstream performance while requiring fewer trainable parameters. Moreover, MQAdapter can be integrated with existing VLM fine-tuning methods, leading to further performance gains.

References↩︎

---

1. Yumiao Zhao, Bo Jiang, Xiao Wang, and Jin Tang are with the Anhui Provincial Key Laboratory of Multimodal Cognitive Computation, School of Computer Science and Technology, Anhui University, Hefei, China. Min Lu is with Origin Quantum Computing Company Limited, Hefei, Anhui 230026, China.

   Corresponding author: Bo Jiang and Xiao Wang↩︎

2. Manuscript received April 19, 2021; revised August 16, 2021.↩︎