1 Introduction↩︎

Contrastive Language-Image Pre-training (CLIP) [1] aligns global image and text embeddings within a unified latent space by adopting a dual-encoder architecture and conducting matching on large-scale, noisy image-text pairs. It has become a foundational vision-language model (VLM) for representation learning, and achieves remarkable success on image-level tasks [2]–[4], such as image classification and image-text retrieval.

However, CLIP exhibits notable limitations in comprehending fine-grained details, such as poor region recognition when using its dense features, due to the pre-training focusing on matching the whole image (via the \([cls]\) token) to a text description. Thus, the model struggles to extract meaningful region-level representations from its dense visual features for grounding textual concepts. This shortcoming limits the performance of CLIP on the downstream tasks that require region-aware ability. For example, in dense prediction tasks, e.g., object detection and segmentation, the CLIP model is usually utilized as a classifier [5], [6] or the teacher in distillation [7], [8] to process cropped object patches to obtain region features. Some works [9]–[11] adopt the frozen CLIP model as the backbone to produce spatial feature maps, but they all choose the CNN-based CLIP, which can preserve more position information than the vision transformer (ViT-based) architecture.

Since the resource demand for pretraining a fine-grained VLM is staggering, such as billions of data samples [12]–[14], a series of recent works have focused on advancing the vision-language alignment of local image regions via fine-tuning of CLIP. Considering the impressive generalization capabilities of CLIP models, particularly the powerful ViT-based variants, the fine-grained fine-tuning of CLIP is carried out to enable applications on dense prediction tasks, such as open-vocabulary object detection and image segmentation. Since no region-text annotations are provided in the image-text pair training data, the previous fine-tuning works enhance the fine-grained alignment via: 1) region-based pseudo-labeling, 2) region-based distillation, 3) mosaicking images as pseudo regions, or the combination thereof. In the first category, [15]–[17] generate region proposals by coarsely cropping patches or adopting RPN [18], and then use CLIP or other powerful VLMs to retrieve region labels from a pool of concepts (, Fig. 1a). However, the range of concepts is limited by the number of pre-defined categories, which could hinder downstream applications like open-vocabulary object detection. The recent work FineCLIP [19] (Fig. 1c) adopts large VLMs like BLIP2 [20] to generate detailed sentences to describe region proposals. However, carefully preparing the region annotations in these methods inevitably costs significant extra time and space, which cannot be neglected when increasing the scale of training data. For the 2nd category, CLIPSelf [21] (Fig. 1b) and FineCLIP [19] (Fig. 1c) use distillation to transfer the global features of the cropped regions (either random or proposals) to extracted dense features – however, to obtain superior distillation performance, preprocessing of the generated region proposals by a well-trained detector is still required. Mosaic augmentation is adopted to make pseudo regions in [22], [23] (, Fig. 1d), which produces finer matching than image-level while still coarser than region alignment. Moreover, most of these methods sacrifice, and some even ignore [17], [21]–[23], the global representation ability of CLIP when training focuses on region knowledge, which causes severe performance damage on image-level tasks.

In this work, we propose SFF-CLIP, a novel self-annotated fine-grained fine-tuning method for CLIP, which aims to boost the region awareness of CLIP while maintaining the global visual-semantic consistency, without requiring extra preparation of region proposals or their annotations. As with the original CLIP pre-training, image-text pairs are taken as inputs for fine-tuning. We propose a self-annotated fine-grained alignment scheme, where we first extract phrases from the input sentence and generate the corresponding region importance on the image based on text-specific heat maps. The heat maps are used to aggregate region features for each phrase, and then the region-phrase feature pairs are aligned by the matching loss, whose weight is based on the degree of phrase matching. To help maintain the whole image representation ability of the original CLIP, the contrastive learning objective for image-text pairs is included in the loss function through weighted momentum term. As a result, our SFF-CLIP locally and globally aligns the visual and semantic features in the same representation space. Through comprehensive experiments, we show that SFF-CLIP significantly improves the fine-grained understanding of CLIP, surpasses previous state-of-the-arts on both the dense prediction benchmarks, including open-vocabulary object detection and segmentation, while also maintaining the global representation of the original model in the image-level task.

We summarize our contributions as follows:

2 Related Work↩︎

2.1 Fine-grained understanding in VLMs↩︎

Although CLIP [1] and subsequent VLMs [24]–[26] exhibit strong representation capabilities and exceptional generalizability, the image-level training, which matches an image as a whole to a text description, has been shown to be deficient in fine-grained understanding and alignment between image regions and text [15], [16], [21], [27]. This shortcoming limits their applications on tasks that require region-aware abilities, , dense prediction tasks.

To mitigate this issue, some works build strong fine-grained VLMs by pre-training with patch-token embedding alignment [14], or region-caption matching [12] utilizing well-trained open-vocabulary detectors for assistance. FG-CLIP [13] constructs a comprehensive dataset with billions of region-specific annotations for region-aware training for CLIP. Due to the heavy cost of using large-scale data, some recent works propose to enhance the fine-grained representation by fine-tuning CLIP with region-based pseudo-labeling or region-based distillation. RegionCLIP [16] adopts RPN [18] object proposals, while PTP [15] coarsely crops patches, and then they both use CLIP as a classifier to obtain region labels from a large pre-defined pool of concepts. DenseVLM [17] uses a powerful VLM to retrieve categories (from a category set) for randomly cropped regions. The recent work FineCLIP [19] adopts large VLMs like BLIP2 [20] to generate detailed sentences to describe region proposals. However, the semantic diversity is limited for those works with pre-defined categories or concepts, and these methods inevitably cost significant extra time and space to preprocess the region annotations, which cannot be neglected when increasing the training data. CLIPSelf [21] and ATAS [22] facilitate the transfer of the global features of regions to dense feature extraction by self-distillation – CLIPSelf obtains the region proposals through the object detector, and ATAS uses mosaic augmentation to construct pseudo regions as in CLIM [23]. The distillation-based methods only focus on the image encoder training and ignore the text encoder. Furthermore, among these methods, [17], [21]–[23] are also ignore the global image-text matching ability of CLIP.

Our paper proposes a self-annotated fine-grained fine-tuning framework for boosting the region-aware ability of CLIP that only uses image-text pairs as input and does not use external models. Our approach circumvents the resource-consuming preparation of the region annotations and the limitation of pre-defined visual categories, while successfully preserving the global representation ability of CLIP.

2.2 Open-vocabulary dense prediction↩︎

Open-vocabulary dense prediction aims to identify visual regions of arbitrary categories as described by the text, and primarily comprises open-vocabulary object detection [7], [28], [29] and image segmentation [6], [30], [31]. Recent open-vocabulary approaches have leveraged the strong representation and generalizability displayed by powerful pre-trained VLMs, which are exploited to identify novel objects. Due to CLIP lacking precise local vision-language alignment, CLIP is usually adopted as a classifier [5], [6] or the teacher in distillation [7], [8] to process cropped region proposals. Several works [9]–[11], [28], [30] adopt frozen CLIP encoders as backbone in the detectors to generate visual features, but most of them select the CNN-based CLIP, which can preserve more position information than the vision transformer (ViT-based) architecture. Although recent studies [16], [17], [19], [21], [23] fine-tuning CLIP for fine-grained understanding, they are constrained by the limitations when generating dense annotations as mentioned above.

2.3 Visual explanation of VLMs↩︎

Visual explanation methods [32]–[36] in computer vision have been developed for visually interpreting a specific prediction of the model by generating a heat map that indicates the spatial feature importance. Recently, works [37]–[40] have proposed visual explanations for the Transformer architecture. In our designed run-time self-annotation scheme for region-phrase pairs, we produce text-specific heat maps using Grad-ECLIP [40] as reference, which is a state-of-the-art visual explanation method for VLMs, such as CLIP. The heat map generation process is high-speed and can be easily plug-in the training of proposed fine-grained alignment.

3 Method↩︎

The aim of SFF-CLIP is to realize region-language alignment while maintaining image-level representation in the fine-tuning of CLIP, but without the requirements of generating dense data annotations, including pre-defined categories, region proposals, or region descriptions. To achieve this, the vision-language alignment at region and image levels are unified in our training method, employing our proposed run-time self-annotated fine-grained alignment and CLIP’s original global contrastive learning. The framework of our proposed SFF-CLIP is shown in Fig. 2, where the model has the same architecture as the original CLIP. We first present brief preliminaries about CLIP (§3.1), then introduce fine-grained alignment (§3.2) and global alignment (§3.3) in detail.

3.1 Preliminaries↩︎

CLIP learns visual and language representations from large-scale web-curated image-text pairs. It consists of an image encoder \(\mathcal{I}\left(\cdot\right)\) and a text encoder \(\mathcal{T}\left(\cdot\right)\), which are jointly trained to extract image and text feature embeddings in a unified representation space. Given image-text pair \((I,T)\), the matching score between their extracted image features \(F_{I}\in \mathbb{R}^{D}\) and text features \(F_{T}\in \mathbb{R}^{D}\) (both row vectors) is: \[\begin{align} S(F_{I}, F_{T}) = \cos(F_{I}, F_{T}) = \tfrac{F_{I}F_{T}^\mathsf{T}}{\left \| F_{I} \right \| \left \| F_{T} \right \| } . \label{eq:cos95similarity} \end{align}\tag{1}\] The model is trained using contrastive learning on the matching scores, regarding the ground-truth image-text pairs as positive samples and other mismatched pairs as negatives. For ViT-based encoders, the image feature is \(F_{I}=\mathcal{LP}(x_{cls})\), where \(\mathcal{LP}\) denotes linear projections, and \(x_{cls}\) is the feature vector from the \([cls]\) token. Thus, except for the class token, all the final layer features of the other tokens (image patch tokens) are not used during contrastive learning of CLIP. Since only the class token feature is explicitly optimized during training, the local patch features exhibit weak representation ability for localized matching of semantic content.

3.2 Self-annotated fine-grained alignment↩︎

To realize fine-grained alignment, we propose a matching loss based on extracting the dense feature map from the image and performing fine-grained feature matching of regions to the corresponding text phrase. We dynamically generate self-annotated region-phrase pairs from the input image-text pairs during the training process, which we denote as run-time self-annotation, instead of off-line preparation with detectors or pre-defined categories. Note that our run-time self-annotation is based on the concepts learned by the original CLIP, which makes it more flexible than using pre-defined detectors, since it is not limited to a pre-defined pool of visual categories.

3.2.1 Image dense feature↩︎

Following [21], [41] we extract a dense feature map of the input image from a ViT-based encoder by slightly modifying the last transformer layer to keep the projection and norm layers, and discard the self-attention. This modification is experimentally shown to be capable of preserving more spatial detailed features in the output token embeddings [41]. Specifically, in the last transformer layer, with the input \(x=(x_{cls}, x_{1},...,x_{h\times w})\) comprising a \([cls]\) embedding and \(h \times w\) spatial token embeddings, the layer’s output is obtained as \(v_{i}=\mathcal{LP}(x_i)\), where \(v_{i}\in\mathbb{R}^C\) represent the value embeddings at spatial location \(i\), with \(C\) as their channel dimension. Then, the \([cls]\) embedding is removed and the final spatial token embeddings are reshaped into an \(h \times w\) dense feature map \(F_{d}\), from which we extract fine-grained representations for specific image regions.

3.2.2 Run-time region-phrase self-annotation↩︎

We propose a run-time scheme to automatically obtain region-phrase pairs from the input image-text pairs. Note that our method does not require any manually annotated or model-generated region proposals or their corresponding label annotations. It is also completely encapsulated – it does not require any external vision models (e.g., other VLMs).

Phrase extraction. For the caption \(T\) in each image-text pair \((I, T)\), we use the Natural Language Toolkit (NLTK) [42] to parse and extract the phrases that contain object concepts, by setting the separation and selection rules as “adjective + noun”. For the example in Fig. 2, the parsing results are “dog”, “black car”, and “traffic lights” with the input text “a dog in a black car waiting for traffic lights”. Then, for these extracted words or phrases \(\lbrace p_{t} \rbrace_{t=1}^n\), where \(n\) is the number of extracted phrases, textual descriptions are generated using prompt-template strategy [1]: “This is a photo of the \(p_{t}\).” These descriptions are sent to the text encoder, resulting in a set of phrase embeddings \(\{F_{p_{t}}\}_t\).

Text-specific heat map. The region embedding \(F_{r_t}\) corresponding to each phrase embedding \(F_{p_t}\) is calculated using a text-specific heat maps based on explainable AI (XAI) methods. These heat maps have been proposed to visually explain a specific prediction of the model by showing the spatial feature importance with a heat map. The local regions with relatively high values are interpreted as being important for generating the current output [33], [36], [40]. Referring to Grad-ECLIP [40], a high-speed and easy plug-in gradient-based heat map method designed for ViT-based VLMs, we calculate the spatial importance heat map corresponding to each extracted phrase. Specifically, the phrase embeddings \(F_{p_{t}}\) are used to calculate the cosine similarities with the image embedding \(F_I\), and each similarity \(\cos(F_I, F_{p_t})\) is regarded as a target to be interpreted. Following [40], the gradient (\(g_c\)) for each similarity w.r.t. the output of the last layer is used as channel weights to aggregate the feature maps \(v\), and a loosened attention map technique is adopted to calculate the spatial weight \(u_{i}=\Phi(\cos(q_{cls}, k_{i}))\), where \(\Phi(\cdot)\) is min-max normalization. Finally, the heat map for a specific phrase \(p_t\) is: \[\begin{align}H_{ti} = \textstyle\sum\nolimits_{c} g_c \cdot u_i \cdot v_i, \end{align}\] where \(\sum_c\) is the channel-sum operator. As shown in Fig. 2, the heat map \(H_t\) can localize the corresponding concept by revealing the important spatial locations for matching with the specific phrase. The heat maps are adopted as weights for aggregating the image dense feature \(F_{d}\), resulting in the region embedding for the phrase: \(F_{r_{t}} = \sum_{hw} H_{t} \cdot F_{d}\), where \(\sum_{hw}\) is the sum operator over spatial coordinates.

To investigate the influence of the XAI method used for fine-grained alignment, we conduct an ablation study with several other high-speed Transformer-applicable visual explanation approaches in the Supplemental, which demonstrates that the Grad-ECLIP is the most effective.

3.2.3 Fine-grained matching↩︎

Given the region-phrase embeddings, the positive pairs are made to be similar, while the negative pairs are separated, via the loss function: \[\begin{align}\nonumber\\ & + \sum_{t' \neq t} S( F_{r_{t}}, F_{p_{t'}})^{2} \log ( 1- S( F_{r_{t}}, F_{p_{t'}}) ) \Big], \label{eq:local95loss} \end{align}\tag{2}\] where \(S\) represents the cosine similarity function, and \(t\), \(t'\) means the \(t\)-th and \(t'\)-th phrase in the same batch.

A phrase-matching weight \(w_t\) is applied on the loss term for each region-phrase pair in order to diminish the potential impact of misaligned pairs, e.g., for extracted nouns that have no visual content in the image. Specifically, we adopt the highest response value on the phrase’s heat map, i.e., \(w_{t}=\max(H_{t})\), to evaluate the degree of matching between the phrase and the image. For example, in Figure 3, “dinner” and “Tuesday” are extracted as nouns in the caption, while they are not visually matched with any of the image contents, and this can be reflected by the values on the heat map. In practice, we set the maximum number of phrases in each sentence to \(N=5\). When the actual number of concepts \(n\) is smaller than \(N\), then blank phrases are added with weight \(w_t=0\).

3.3 Global contrastive learning with momentum↩︎

In standard CLIP training, the input batch of image-text pairs \(\lbrace (I_{b},T_{b}) \rbrace_{b=1}^B\) is passed to the image/text encoders and the model outputs the corresponding global image/text embeddings \(\lbrace (F_{I_{b}}, F_{T_{b}} ) \rbrace_{b=1}^B\). The cosine similarity \(S(F_{I}, F_{T})\) between image embedding \(F_{I}\) and text embedding \(F_{T}\) is calculated as in Eq. 1 , then the constrastive loss is applied to learn the global representations by maximizing the cosine similarities of the corresponding image and text embeddings, while minimizing the cosine similarities of other non-paired ones from both image and text sides, which is defined as: \[\label{eqn:glossterm} \begin{align} L_{b}^{I2T} = -\log\tfrac{\exp\left( S(F_{I_{b}}, F_{T_{b}})/\tau \right) }{\sum_{b'=1}^{B} \exp\left( S(F_{I_{b}}, F_{T_{b'}})/\tau \right) }, \;L_{b}^{T2I} = -\log\tfrac{\exp\left( S(F_{T_{b}}, F_{I_{b}})/\tau \right) }{\sum_{b'=1}^{B} \exp\left( S(F_{T_{b}}, F_{I_{b'}})/\tau \right) }, \end{align}\tag{3}\] where \(\tau\) is the trainable temperature parameter.

To better maintain the image-level representation ability of the pre-trained CLIP while enhancing its fine-grained alignment, we adopt a momentum CLIP model \(\hat{M}\), which is initialized with the original CLIP model and then updated by the current trained model \(M\) with a low momentum rate \(\alpha=0.005\) after each epoch, , \(\hat{M} \leftarrow (1-\alpha)\hat{M} + \alpha M\). The updated model \(\hat{M}\) is used to extract image and text embeddings \(\lbrace (\hat{F}_{I_{b}}, \hat{F}_{T_{b}} ) \rbrace_{b=1}^B\), and weights are computed for each image-text pair, which measures how well they match: \[w_{I_{b}} = \gamma\tfrac{\exp\left( S(\hat{F}_{I_{b}}, \hat{F}_{T_{b}})/\tau \right)}{\sum_{b'=1}^{B} \exp\left( S(\hat{F}_{I_{b}}, \hat{F}_{T_{b'}})/\tau \right)} + (1-\gamma), \;\; w_{T_{b}} = \gamma\tfrac{\exp\left( S(\hat{F}_{T_{b}}, \hat{F}_{I_{b}})/\tau \right)}{\sum_{b'=1}^{B} \exp\left( S(\hat{F}_{T_{b}}, \hat{F}_{I_{b'}})/\tau \right)} + (1-\gamma),\] where parameter \(\gamma\) controls the influence of match, which is set to \(0.4\) by default in the experiments. The contrastive loss then applies the weights to the loss terms in (3 ) to focus on preserving the well-matched image-text pairs, \[L_{contrastive} = \dfrac{1}{2B} \sum_{b=1}^{B} \left(w_{I_{b}}L_{b}^{I2T} + w_{T_{b}}L_{b}^{T2I} \right). \label{eq:global95loss}\tag{4}\] The final loss for training is obtained by by adding the fine-grained matching loss to the contrastive learning loss, \(L=L_{fg} + L_{contrastive}\).

4 Experiments↩︎

We evaluate the proposed SFF-CLIP on both improving of fine-grained representation via zero-shot region classification, down-stream tasks of open-vocabulary detection and segmentation, and on maintaining of image-level representation via image-text retrieval task. We compare with the primary fine-grained fine-tuning methods, including RegionCLIP[16], CLIPSelf[21], FineCLIP[19], DenseVLM[17], and CLIM[17]. Moreover, we also compare with fine-tuning CLIP using only the global contrastive loss, , without the fine-grained matching, which is a baseline for continuation of image-level training of CLIP (denoted as “CLIP-g”). Finally, we carry out the ablation studies.

4.1 Implementation details↩︎

For fairness, we re-trained all compared fine-grained fine-tuning methods, by initializing with the same pre-trained model and fine-tuning with the same training data, using the same input image size, which is the same as the CLIP pre-training, 224x224 for ViT-B/16 and 336x336 for ViT-L/14 by default. Following previous works [19], [21], the experiments are conducted based on the pre-trained models from EVA-CLIP [43]. Since some related works [16], [19], [21] that require region annotations open-sourced their region information files based on MS COCO train2017 set [44], we conduct the fine-tuning on the same images with the captions provided by [19]. We mainly report the comparison results with other works on ViT-B/16, with the comparisons on ViT-L/14 provided in the Supplemental. Two RTX 6000 Ada are used, and for our method, we use batch size 128, learning rate 1e-5, and weight decay of 0.1; for other methods, we use the training parameters provided in their codes.

4.2 Comparisons on fine-grained representation↩︎

To evaluate the dense representation ability, we use the mean accuracy (mAcc) of classifying region boxes and panoptic masks for “things” annotated in the ADE20K panoptic [45] val set and COCO panoptic [44] val2017 set. To extract region-level features, RoI or mask pooling are used to extract the region box or mask embeddings from the image dense feature maps. The classification is then performed by selecting the highest score when matching with the text embeddings of the classes.

The results are shown in Tab. 1. Compared with the CLIP base model, CLIP-g can slightly increase the zero-shot classification performance, due to additional iterations of image-level training. However, SFF-CLIP, which uses both the global contrastive learning and fine-grained matching, obtains significant improvements in accuracy on region classification, for both boxes and masks and for both ViT-B/16 and ViT-L/14 architectures. Compared with other region-aware fine-tuning methods, our proposed SFF-CLIP also achieves outstanding performances. In contrast to other methods that require pre-preparing region proposals [16], [19], [21] and region labels [16], [19], or using pre-defined categories [16], [17], our method can be flexibly applied with just image-text pairs as training data, which is the same data source as the original CLIP pre-training, resulting in better performance. Furthermore, in contrast to the mosaicking-based CLIM [23], our region-phrase pairs from self-annotation provide more effective fine-grained supervision than mosaicking images as pseudo regions, as evidenced by SFF-CLIP outperforming CLIM on all metrics. These superior performances shows the effectiveness of our self-annotated fine-grained alignment approach, which even exceeds these methods that need to carefully prepare region annotations, and achieve comparable performances with the FG-CLIP [13], which is a fine-grained model trained with large-scale region annotated data (1.6B).

Following the previous related works [17], [19], [21], we build open-vocabulary object detectors based on the F-ViT [21] architecture, which is a two-stage detector using a frozen CLIP-based ViT as the backbone. In our experiment, each fine-grained fine-tuned CLIP encoder from the various methods is adopted to initialize the backbone. The OVD models are then trained on the OV-COCO benchmark [46], and we use AdamW optimizer with batch size of 64, learning rate of 1e-4, and weight decay of 0.1. For evaluation, we report box AP (average precision) at IoU (Intersection over Union) of base, novel and all categories as with previous works [9], [10], [16], [21], [27].

The results are presented in Tab. 4. F-ViT is the baseline that initializes the detector backbone with the original pre-trained CLIP. With the ViT-B/16 backbone, adopting just CLIP-g results in similar performance to the baseline CLIP. In contrast, SFF-CLIP significantly improves the OVD results, especially on the novel categories (over \(9\%\)). Since the base categories have explicit annotated bounding boxes and labels during OVD training, the performance on the unseen novel categories better illustrates the fine-grained understanding ability brought by the CLIP encoder. Finally, we conduct experiments with ViT-L/14 and further improve the OVD performance on the novel categories by a large extent. We also provide comparisons with related works using ViT-L/14 in the Supplemental, as well as the OVD results with the LVIS benchmark [47]. Compared with the existing OVD methods, which mostly rely on a ResNet-based encoder or modified ViT encoder and require pre-training on large-scale prepared data with extra region information, our method achieves superior or comparable performance by fine-grained training on the easily-obtained image-text pairs.

We next explore the performance of applying fine-grained fine-tuned models to open-vocabulary semantic segmentation. Following the previous works [17], [19], [21], we adopt the CatSeg [48] as the segmentation architecture, which uses the dense image feature from CLIP ViT as the backbone, with the CLIP models fine-tuned by all compared methods using the same input image resolution 384x384. After replacing the backbone of CatSeg with the original CLIP model, and each region-aware model, the following segmentation experiments are conducted with training on COCO stuff [49], and evaluation on ADE20k [45], PASCAL VOC [50], and PASCAL Context [51] dataset using mean IoU (mIoU), and pixel Accuracy (pACC). As shown in Tab. [tab:app95seg], SFF-CLIP comprehensively improves the performance of CatSeg on various datasets across different evaluation metrics, and surpasses the enhancements provided by RegionCLIP, CLIPSelf, DenseVLM, FineCLIP, and CLIM.

Overall, the significant improvements on both OVD and OVS demonstrate that our method effectively boosts the fine-grained understanding ability of the CLIP model in downstream tasks.

In Fig. 5, we provide an example visualization to qualitatively compare the fine-grained understanding ability of models under different alignment methods. Cosine similarities are calculated between the embeddings on each position of the image’s dense features and the text features for “car” and “tree”, and visualized as similarity maps. Without fine-grained alignment, the similarity maps of vanilla CLIP and CLIP-g exhibit weak localization ability (i.e., region-specific attention), where many irrelevant locations have high similarities with the object text. For SFF-CLIP, the text can be matched with the specific regions more accurately than other methods. In particular many of the spurious matches to the background have been removed compared to other methods, which demonstrates the effectiveness of our proposed method to significantly improve the fine-grained representation ability of CLIP model.

4.3 Evaluation of image-level representation↩︎

The previous works rarely considered how their fine-tuning affects the whole image-text matching, which is the original ability of a vision-language model (VLM). However, preserving the original image-level representation is important in order to support multiple tasks from the same feature backbone model. To explore if there are any negative effects to image-level representation when improving fine-grained representations, we conduct an image-text retrieval evaluation with the Flickr30k [52] validation set, and report the recall accuracy of image-to-text and text-to-image on Top\(@\{1, 5, 10\}\) matching. As seen in Tab. 2, with just realizing image-level contrastive learning, the baseline CLIP-g can keep or improve the performance of the original CLIP by continuing the training on the new image-text pairs data from MS COCO. Our SFF-CLIP successfully preserves the image-level retrieval performance, which is comparable to or even better than CLIP-g. In contrast, other region alignment methods all cause degradation to the image-text retrieval performances compared with the CLIP-g. In particular, CLIPSelf and DenseVLM have largely reduced their abilities of global image-level representation. Finally, the similar results between SFF-CLIP and CLIP-g with ViT-L/14 further demonstrate the effectiveness of our method in maintaining the global representation and stable image-level matching.

4.4 Ablation Studies↩︎

In this section, we conduct the ablation studies showing the influence of various components in our fine-grained fine-tuning method.

To show the function of the phrase-matching weight \(w_{t}\) in the fine-grained matching loss (Eq. 2 ), we conduct an ablation study with and without the weight on zero-shot classification performance, as the Top-1 accuracy presented in Tab. 6. Without \(w_{t}\), the fine-grained alignment model degrades significantly, with results even worse than the original CLIP. This demonstrates that many phrases extracted from the caption are not matched well with the image, and could cause large disturbances to the region-phrase alignment. Therefore, after introducing the weights to represent the degree of matching for each phrase into the loss, the model can better focus on the learning of well-aligned samples during the fine-tuning, and produce significant performance improvements.

We conduct an ablation study to investigate the function of using momentum model to generate image and text matching weights in the contrastive learning (Eq. 4 ). From the text-to-image (T2I) and image-to-text (I2T) results shown in Tab. [tab:abl95momentum], the image-text retrieval evaluation results are well kept when adopting the momentum model, but decrease when using the pure contrastive loss without the weights from the momentum model.

Since the text-specific heat maps can be calculated in batch for one phrase in each sentence, the number of loops in obtaining all heat maps is the same as the maximum number of phrases \(N\). Therefore, we conduct the ablation study with \(N=\{0,1,3,5\}\) to observe the influence on the fine-grained alignment’s effectiveness and training efficiency. From Tab. [tab:abl95phrase95num], the zero-shot classification Top-1 accuracy is similar when \(N=3\) and \(N=5\), which is better than \(N=1\), while much higher than no region-phrase matching (CLIP-g). Thus, \(N={3,5}\) can basically cover the effective concepts in the training data. For the training efficiency, due to the high speed of the adopted heat-map generation method, the influence of \(N\) on the training time cost is not large.

To investigate the impact of data scale on model performance, we randomly sample three trainsets from CC3M [2] with different sizes: 100K, 1M, and 3M samples. We fine-tune the CLIP model with our SSF-CLIP, and compare with CLIP-g, CLIM and CLIPSelf, where CLIPSelf uses image patches instead of preparing region proposals. We report the Top-1 accuracy of zero-shot classification with boxes or masks for fine-grained understanding evaluation and the Top-1 recall rate of T2I and I2T retrieval results for image-level representation evaluation. In Figure 7, we present the performance curves on the four evaluations as the number of samples scales up. In terms of the fine-grained understanding, SSF-CLIP surpasses other methods, and the performance continues to grow as the dataset size increases, which shows promising scalability. For the image-level representation, SSF-CLIP maintains similar retrieval performances as CLIP-g, while in contrast, CLIPSelf and CLIM become worse when the training data increases.

5 Conclusion↩︎

In this paper, we propose SFF-CLIP (Self-annotated Fine-grained Fine-tuning for CLIP), a novel framework designed to advance the fine-grained representation ability while maintaining the global visual-semantic consistency. Our framework only requires image-text pairs as inputs, avoiding the process of generating region proposals and region annotations (eithetasksnually or via detectors) required by previous works. We propose fine-grained matching with the help of a run-time region-phrase annotation and alignment scheme, which extracts concept phrases from the input sentence and dynamically matches them with image dense features with the help of text-specific heat maps. A momentum model is adopted to support the preservation of CLIP’s original image-level representation ability. We validate the fine-grained dense representation of SFF-CLIP on zero-shot classification task, and down-stream dense prediction tasks, producing consistently significant performance improvements to the baseline model, as well as superior results compared to state-of-the-art works. Meanwhile, the performance on image-level tasks is preserved from the original CLIP model. Training with just image-text pairs, SFF-CLIP provides a flexible and effective solution for improving fine-grained dense representations of vision-language models like CLIP, while also eliminating the requirements of extra training data annotations, such as region proposals and region annotations, as well as dependence on pre-defined categories.

Acknowledgments↩︎

This was was supported in part by......

6 Comparisons on ViT-L/14↩︎

We compare with related fine-grained fine-tuning methods, including RegionCLIP [16], CLIPSelf [21], FineCLIP [19], DenseVLM [17], and CLIM [23], on ViT-L/14. For fairness, all compared methods are reproduced with their open-sourced codes to fine-tune the CLIP pre-trained model based on the MS COCO train 2017 set [44] with the same input image size of 336x336.

6.1 Results of zero-shot region classification↩︎

We compare the fine-grained dense representation on ViT-L/14 with related fine-grained fine-tuning methods using the mean accuraccy (mACC) of classifying region boxes and panoptic masks for “things”. The results on ADE20K panoptic [45] val set and COCO panoptic [44] val2017 set are shown in Tab. 3. Compared to other region-aware fine-tuning methods that require pre-preparing extra region information, our proposed SFF-CLIP achieves superior performance with the ViT-L/14 architecture, further demonstrating the effectiveness of our self-annotated fine-grained alignment on advancing the region-aware representation ability of CLIP.

6.2 Results of OVD with OV-COCO benchmark↩︎

We provide the results of OVD performance on MS COCO val set with ViT-L/14 in Tab. 4. SFF-CLIP significantly improves the OVD results on the unseen novel categories (9.1%), while surpassing other fine-grained fine-tuning methods.

6.3 Results of image-text retrieval↩︎

Moreover, the comparison results on the image-text retrieval task shown in Tab. 5 also indicate that our SSF-CLIP can successfully maintain the global representation and stable image-level matching with ViT-L/14. In contrast, other fine-grained fine-tuning methods all cause degradation of image-text retrieval performances, especially the region-based distillation method CLIPSelf.

7 OVD results on OV-LVIS benchmark↩︎

We build open-vocabulary object detectors based on F-ViT architecture using the fine-grained fine-tuned CLIP ViTs as backbones, and train the models for 48 epochs on the OV-LVIS [47] benchmark with the input image size of 224x224 for ViT-B/16. The evaluation results are shown in the Tab. 6 with AP for base categories (AP\(_{c}\), AP\(_{f}\)), rare categories (AP\(_{r}\)), and all categories (AP) as comparison indicators. SSF-CLIP surpasses all other fine-grained fine-tuning methods on all categories, especially on the rare categories, which better illustrates the fine-grained understanding ability brought by the CLIP encoder.

8 Ablation study on different XAI methods↩︎

In Sec. 3.2, we calculate the text-specific heat maps region-phrase self-annotation based on explainable AI (XAI) methods, and specifically adopt Grad-ECLIP [40] as reference. To investigate the influence of the XAI method used for fine-grained alignment, we conduct an ablation study with other applicable visual explanation approaches that satisfy a high-speed and easy plug-in, including Grad-CAM [33] and MaskCLIP [41]. The Vision Transformer self-attention cannot be used since it is not text-specific. The comparison of zero-shot classification performances in Table ¿tbl:tab:abl95xai? shows that with the heat maps from these two methods, the phrase-region alignment also obtains obvious performance improvements compared with just using global loss (CLIP-g), but there is still a gap compared with using Grad-ECLIP, which demonstrates that Grad-ECLIP is the most effective.

9 Ablation study on input image sizes↩︎

To explore the impact of input image size on SSF-CLIP, we conduct the fine-grained alignment training and evaluation with four different image resolutions, including \(224, 336, 480, 512\), on both region-level zero-shot classification and image-level retrieval. The results are shown in Table 7. As the resolution of input image is increased gradually from \(224\) to \(512\), the performance on region-level task is improved due to that more visual details can be provided with larger images. On the other hand, the performance on the image-text retrieval task is declined as the image becomes larger, when more number of patches leads to a increase of the complexity for the \([cls]\) token to integrate the global feature.

10 Visualization↩︎

We present more visualizations of cosine similarity maps between text embeddings and the dense feature maps generated by different fine-grained alignment methods. The comparisons are shown in Figure 8.

11 Baselines↩︎

We use the following publicly available source code:

12 Broader impact↩︎

Our work contribute to introduce SSF-CLIP, which boosts the fine-grained understanding ability of CLIP while maintains the global representation with eliminating the constrains from region annotation preparation. With the growing adoption of transformer as a unified architecture for both vision and language tasks, it is of great significant to enable the great generalization ability of CLIP ViT encoders in both image-level and dense prediction tasks. We are the first to utilize the XAI to produce self-annotated supervision. Eliminating the requirements of cumbersome region data preparation, the method is expected to support larger scale of data and promote improvements of models with further development. To ensure a positive social impact, we conduct experiments using academic open-source datasets that do not involve personal privacy issues.

13 Limitation↩︎

The SSF-CLIP is built upon an effective visual explanation (XAI) method for CLIP, the Grad-ECLIP [40]. The self-annotated region-phrase alignment will be influenced by the performance of heat maps generated by the adopted XAI method, as shown in §8. Therefore, the generalization of our idea that utilizes XAI to boost the model itself can be limited by the development of corresponding XAI technique. On the other side, the further progress made in XAI area may push our approach to have a wider range of applications. Since our work provides a flexible and low-cost way to supplement or replace the manually labeling process, SSF-CLIP has the potential to further enhance other pretrained models, which would be explored in our future work.

References↩︎