1 Introduction↩︎

Place recognition plays a significant role in the autonomous navigation of mobile robots, allowing them to locate the global place within previously mapped environments [1], [2]. Existing approaches can be categorized into visual place recognition (VPR) and LiDAR place recognition (LPR). Though LPR methods [3]–[5] have demonstrated good performance in outdoor environments, 3D LiDAR sensors are comparatively more costly and power-intensive. In contrast, cameras offer the advantages of cost-efficiency and abundant perceptual data, leading to growing research interest in outdoor place recognition [6], [7].

Current VPR methods [8] typically compress the query image into a global descriptor that is then used to determine the current location by comparing it with stored data in the database. However, visual sensors are sensitive to environmental changes, such as variations in lighting, weather conditions, and occlusions, which can negatively impact the robustness of these descriptors, resulting in poor VPR performance. Therefore, the main focus in VPR research is to develop a robust network capable of generating stable descriptors. Unfortunately, such networks tend to be complex, with a high number of parameters, and they often operate at computational speeds lower than the sensor’s frame rate, which limits their practical applicability [9].

To fill this gap, Hui et al.. [10] employed knowledge distillation to enable knowledge transfer from the powerful teacher network to the lightweight student network. However, the performance of the student network fell significantly short of that of the teacher network. Cai et al.. [11] introduced a self-learning approach to improving the student network in acquiring the teacher network’s knowledge. While this approach enhanced the student network’s performance, it did not reduce the size of the student network, making is unsuitable for mobile robots with limited onboard resources.

In this paper, we introduce a novel concept called cross-metric knowledge distillation (KD) and apply it to VPR within our teacher-student network. Our network comprises a teacher and a student model. The teacher model integrates components from ResNet [12], vision transformer [13], and inter-transformer [14], resulting in increased complexity and superior performance compared to state-of-the-art baselines in VPR. In contrast, the student network is designed to be lightweight, retaining only essential components. We transfer place knowledge from the powerful teacher to the lightweight student through our proposed cross-metric distillation. This approach empowers the lightweight student network to match or exceed the teacher’s performance while reducing inference time, enabling fast and accurate VPR in real-world applications, as illustrated in 1. We evaluated our model on two large outdoor datasets, Pittsburgh30k and Pittsburgh250k. The experimental results demonstrate that our model outperforms baseline models in terms of both recognition accuracy and parameter efficiency. Furthermore, we provide multiple ablation studies showing that our proposed distillation method surpasses other distillation techniques.

In summary, the contributions of this paper are threefold:

(i) We introduce TSCM, a novel cross-metric knowledge distillation approach applied to VPR, which enables the student model to even outperform the teacher model.

(ii) Both our devised new teacher and student networks achieve better performance in terms of VPR accuracy than the state-of-the-art baseline methods;

(iii) While achieving superior performance, our method is more lightweight than the baseline, displaying outstanding computational efficiency. Our student model generates one descriptor and finds a matching within a 10 k-image database in less than 2 ms, achieving real-time performance.

2 Related Work↩︎

Localization is a crucial capability for robots, commonly achieved through Simultaneous Localization and Mapping (SLAM). However, over time, inevitable drift adversely impacts localization accuracy. Responding to this challenge, researchers have increasingly focused on place recognition. Place recognition aids in loop-closure detection and re-localization by identifying previously visited locations and mitigating localization drift [3], [15]–[17]. This paper primarily centers on visual place recognition (VPR).

Early work by Milford et al.. [18] introduced SeqSLAM for outdoor VPR. They employed handcrafted features to calculate the most suitable candidate among sequences. Vysotska and Stachniss [19] exploit multi-sequence maps as well as sequential image information to achieve effective and robust VPR. With the development of neural network technology, there has been a surge in the development of learning-based methods. Among these, NetVLAD [20], a Convolutional Neural Network (CNN)-based architecture, stands out as a pivotal advancement. It facilitates end-to-end training for place recognition, thereby achieving considerable success. Subsequently, many NetVLAD-based approaches have emerged [8], [21]–[24].

With the advent of the Transformer [25], an increasing number of Transformer-based methods have come to the forefront. Wang et al.. [26] introduced a hybrid CNN-Transformer feature extraction network, addressing the inherent locality limitation of Convolutional Neural Networks (CNNs) by incorporating the Vision Transformer to model image contextual information and dynamically combining task-related features. Hou et al.. [27] used Transformer-based networks for feature extraction, presenting a hierarchical transformer for VPR.

Knowledge distillation was initially introduced for image classification by Hinton et al.. [28]. This technique utilizes a powerful, well-trained teacher model to guide the training process of a lighter-weight student model. The goal is to simplify the model while preserving the original teacher model’s accuracy. Due to the success of knowledge distillation, many related works have emerged. [29]–[31]. Oki et al.. [32] introduced the concept of metric learning into knowledge distillation, leveraging pairs or triplets of training samples to bring the student model closer to the teacher model, which is closely aligned with the concept of cross-metric knowledge distillation proposed in this paper. In recent years, researchers have also applied knowledge distillation to place recognition. Shen et al.. [33] proposed StructVPR, a novel training architecture for VPR, aiming to enhance structural knowledge within RGB global features, thereby enhancing feature stability in dynamic environments. Cui et al.. [34] introduced a continual contrastive learning method named CCL to address the issue of catastrophic forgetting and enhance the overall robustness of LPR approaches. This method employs knowledge distillation to update the model, enabling it to sustain effective place recognition over extended periods and across diverse environments. Cai et al.. [11] introduced STUN, a self-teaching framework that predicts place and estimates prediction uncertainty simultaneously using knowledge distillation without reducing the size.

In contrast to the abovementioned approaches, we introduce a novel knowledge distillation technique called cross-scale knowledge distillation, which helps us achieve a good balance between performance and model simplicity for VPR.

3 Our Approach↩︎

In this section, we present the detailed network architecture of our teacher-student models, TSCM, and how we apply the devised cross-metric KD to VPR.

3.1 Teacher-Student Model↩︎

The architecture of our designed teacher network is presented in 2. It fuses powerful components from ResNet [12], vision transformer [13], inter-transformer [14] to increase complexity, enabling better learning of place knowledge in complex outdoor environments and generating distinctive place descriptors. It first exploits ResNet [12] to extract high-level and fine-grained features from the input image. While deeper network architectures have the potential to extract more intricate features, they often encounter training complexities. ResNet effectively addresses these challenges through the use of residual connections. Therefore, we employ several ResNet blocks as the backbone network, as illustrated in 2. We exclude its final stage to enable the teacher model to allocate more general features relevant to places, which is verified to be more beneficial than the features of the last layer in our ablation studies. Previous research [11] did not utilize the middle features directly for the final descriptor, but our findings show that using these features improves outcomes. As a result, we preserve the \((1\times1024)\) features from the ResNet branch and incorporate them into the final global descriptor aggregation process. This insight is also verified in 7.

While ResNet has demonstrated good performance, its feature extraction tends to overlook interrelationships among distinct regions within an image. In contrast, Vision Transformer (ViT) [13] extracts both global and local features from the image while simultaneously capturing correlation information among various image regions. It establishes connections among these disparate regions, enhancing its capacity to capture global and contextual information. Therefore, we also employ ViT to generate global features from the input image. After acquiring fine-grained features of \((1\times256)\) from ResNet and global features of the same size from ViT, we combine them to create more representative image features, generating distinctive global descriptors.

To this end, we introduce an inter-transformer encoder to fuse the features extracted by ResNet and ViT as shown in 3. It has two branches, each with a multi-head attention(MHSA) and a feed-forward network(FFN). The outputs of each branch are finally fused by a NetVLAD and MLP module to obtain the initial descriptor of \((1\times512)\). The MHSA of Inter-Transformer in 3 can be mathematically formulated as follows: \[\begin{align} F^{\text{Res}}_{\text{Att}} & = \text{Attention}(Q^{\text{Res}},K^{\text{ViT}},V^{\text{ViT}}) \\ & = \text{softmax}\left(\frac{Q^{\text{Res}}K^{\text{ViT}}}{\sqrt{d_k}}\right)V^{\text{ViT}} \end{align}\] where \(Q^{\text{Res}}\) is substantially the query split of the feature from ResNet. \(K^{\text{ViT}}\),\(V^{\text{ViT}}\)are the key and value splits of the feature from ViT \(F^{\text{ViT}}\). \(\sqrt{d_k}\) represents the dimension of splits. \(F^{\text{Res}}_{\text{Att}}\) denotes the output feature from the MHSA in the ResNet branch. This feature is subsequently fed into the FFN to generate the attention-enhanced feature, facilitated by the following mathematical formulation: \[\begin{align} F^{\text{Res}}_{\text{ITF}} = \text{FFN}(F^{\text{Res}}_{\text{Att}}+F^{\text{Res}})+F^{\text{Res}}_{\text{Att}}+F^{\text{Res}} \end{align}\]

Likewise, we perform the same operations on the ViT branch.: \[\begin{align} F^{\text{ViT}}_{\text{Att}} & = \text{Attention}(Q^{\text{ViT}},K^{\text{Res}},V^{\text{Res}}) \\ & = \text{softmax}\left(\frac{Q^{\text{ViT}}K^{\text{Res}}}{\sqrt{d_k}}\right)V^{\text{Res}} \end{align}\] \[\begin{align} F^{\text{ViT}}_{\text{ITF}} = \text{FFN}(F^{\text{ViT}}_{\text{Att}}+F^{\text{ViT}})+F^{\text{ViT}}_{\text{Att}}+F^{\text{ViT}} \end{align}\]

Features \(F^{\text{Res}}_{\text{ITF}}\) and \(F^{\text{ViT}}_{\text{ITF}}\) are concatenated to form the composite feature \(F_{\text{ITF}}\). Subsequently, this feature \(F_{\text{Inter}}\) is fed into the NetVLAD-MLPs combos [20] to generate a descriptor. The final global descriptor is created by integrating this descriptor with those from other branches. While the teacher network gathers maximum information using multiple complex components, including potentially redundant knowledge, this does not affect deployment, as a lighter student network is deployed into practical applications.

The overview of our student network is depicted in 4. Unlike the complex architecture of the teacher network, the student network has a much simpler structure, as illustrated in 7. The ResNet branch is pivotal in place recognition, so we keep it in the student network. To simplify the student network, we replace both the ViT and the Inter-Transformer with a layer of convolutional layers. Our student network is relatively smaller but still includes multiple branches, allowing it to learn various types of place knowledge from the teacher network through cross-metric knowledge distillation, explained in the next section.

3.2 Cross-Metric Knowledge Distillation↩︎

Knowledge distillation (KD) transfers complex knowledge from a larger teacher model to a smaller student model while keeping the student compact. Existing KD methods for VPR typically use triplet loss [35], as shown in the left figure of 5. \(T(\cdot)\) represents the teacher model, and \(S(\cdot)\) represents the student model. \(a\), \(p\), and \(n\) correspond to the anchor, positive, and negative samples. Solid lines indicate distances to be reduced, while dashed lines indicate distances to be increased. Triplet loss is well-suited for supervising VPR tasks, as it helps find similar places and distinguish different places. During KD, the teacher model remains fixed, and the triplet loss is applied to the student model as follows: \[\begin{align} L_{\text{Tri}} = \max\left\{ d(S(a) - S(p))-d(S(a) - S(n))+m),0 \right\}, \end{align} \label{con:explanation}\tag{1}\] where \(m\) is the margin.

Traditional KD aligns outputs of the student model with those of the teacher, keeping distances between \(T(a)\), \(T(n)\), and \(T(p)\) unchanged, known as soft targets (red lines in 5), and the corresponding loss function is: \[\begin{align} L_{\text{soft}} & = \sum^{\substack{N}}_{\substack{i}} d(S(a_i) - T(a_i))+d(S(p_i) - T(p_i)) \\ &+d(S(n_i) - T(n_i)). \end{align} \label{con:soft}\tag{2}\]

In VPR, our objective is to ensure that positive sample outputs are close to the anchor while negative sample outputs are far from it, referred to as hard targets (illustrated by the green lines in 5). This goal should be consistent whether dealing with teacher samples, student samples, or interactions between them. Traditionally, the triplet loss focuses only on the relationship between the outputs of the student model, overlooking the relationship between the outputs of the student and teacher models (e.g., the relationship between \(S(a)\) and \(T(p))\). One can apply the triplet loss within a single model to achieve the desired hard targets (\(L_{\text{Tri}} = L_{\text{hard}}\)). However, there is limited research on achieving these hard targets between the teacher and student models.

To fill in this gap, we investigated the relationships between samples in both the teacher and student models during the distillation process. We aimed to bring a positive student sample, S(p), closer to the teacher anchor, T(a), while pushing negative student samples, S(n), away from T(a), and vice versa. To capture these relationships, we introduced black lines in addition to the red and green lines, as depicted in the second figure of 5, to represent the distances between samples across teacher and student. This can be formulated as: \[\begin{align} L_{\text{cm-full}} & = \max \left\{ d(S(a) - T(p)) + d(S(p) - T(a)) \right. \\ & \quad \left.- d(S(a) - T(n)) - d(S(n) - T(a)) + m, 0 \right\}. \end{align} \label{eq:loss}\tag{3}\]

Ideally, using 3 as the loss for VPR would maximize information utilization during KD. However, including all sample distances presents a challenge: We aim to separate \(S(a)\) from \(T(n)\) and \(S(n)\) from \(T(a),\) but this may inadvertently cause \(S(a)\) to diverge from \(T(a)\) and \(S(n)\) to diverge from \(T(n),\) contradicting our soft targets (red solid line). To address this, we propose our cross-metric KD, which discards the dashed black lines and maintains the solid black lines as cross-model constraints, formulated as: \[\begin{align} L_{\text{cm}} =\sum^{\substack{N}}_{\substack{i}} d(S(a_i) - T(p_i))+d(S(p_i) - T(a_i)) \end{align} \label{con:cm}\tag{4}\] As \(S(a)\) approaches \(T(p)\) and \(S(p)\) approaches \(T(a)\), the distances between \(S(a)\) and \(T(a)\), \(S(p)\) and \(T(p)\), and \(S(a)\) and \(S(p)\) tend to decrease. As illustrated in the right figure of 5, our design benefits both soft and hard targets, thus increasing the KD performance specifically in VPR tasks.

Taking into account the aforementioned considerations, we define the final loss function for our cross-metric KD for VPR as follows: \[\begin{align} L_{\text{total}} = L_{\text{hard}}+L_{\text{soft}}+L_{\text{cm}}. \end{align} \label{con:KD}\tag{5}\]

4 Experimental Evaluation↩︎

The main focus of this work is a teacher-student network exploiting our devised cross-metric knowledge distillation (KD) for fast and accurate visual place recognition (VPR). We present our experiments to show the capabilities of our method and support our key claims, which are: (i) TSCM introduces the concept of cross-metric KD to VPR, allowing our smaller student model to perform similarly, and sometimes even better, than the larger teacher model; (ii) TSCM attains superior recognition accuracy while maintaining a more lightweight model in comparison to the state-of-the-art baseline methods; (iii) TSCM demonstrates exceptional computational efficiency, compressing images into descriptors in 1.3 ms and finding a matching in under 0.6 ms per query using a 10 k-image database.

4.1 Experimental Setup↩︎

1) Dataset: Our evaluation of TSCM and other methods is conducted on the Pittsburgh dataset [36], a comprehensive image repository sourced from Google Street View. The original dataset encompasses 250 k training samples and 24 k test query samples. For the training procedure, we employ the same partitioning scheme as NetVLAD [20], using a reduced subset of 10 k samples for training and validation while employing Pittsburgh30k with 10 k test data and Pittsburgh250k with 84 k test data for testing. Notably, these images have been captured at various times of the day and span multiple years.

2) Configuration: In TSCM, the Vision Transformer (ViT) is pre-trained at a resolution of 224x224, with a patch size of 16. For the Inter-Transformer, our setup aligns with that of CVTNet [14], featuring the following specifications: an embedding dimension of \(d_{\text{model}}\) = 512, several heads of \(n_{\text{heads}}\)= 4, and an intermediate dimension for the feed-forward layer of \(d_{\text{ff}}\) = 1024. In the case of NetVLAD following the Inter-Transformer, we configure it with an input feature vector dimension of \(\text{feature}\_{\text{size}}\) = 512, an output feature dimension of \(d_{\text{output}}\) = 512, and a total of \(d_K\) = 64 clusters.For other instances of NetVLAD, we adjust the input feature vector dimension to \(\text{feature}\_{\text{size}}\) = 256, with an output feature dimension of \(d_{\text{output}}\) = 256, while keeping the remaining parameters consistent with NetVLAD following the Inter-Transformer. During the network training phase, we adhere to the STUN setup [11], employing a batch size of 8, utilizing the Adam optimizer with an initial learning rate of \(1\times10^{-5}\), which undergoes decay by a factor of 0.99 after each epoch, and applying a weight decay of 0.001. Our training infrastructure is hosted on a cloud server equipped with the following specifications: a single V100-SXM2-32GB (32GB) GPU, complemented by a 12-core Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz. The server environment operates on the PyTorch 1.10.0 image, utilizing Python 3.8 (Ubuntu 20.04) and CUDA 11.3.

3) Evaluation Metrics: Our experiments are designed to assess the recognition performance of our model and the impact of cross-metric knowledge distillation. We employ recall@N, mAP@N, and AP as metrics to measure recognition performance and recall@N as a metric to measure the impact of cross-metric knowledge distillation.

2.5pt

2.5pt

4.2 VPR Performance↩︎

The first experiment aims to assess the VPR performance of our approach. The experimental results support that TSCM achieves superior recognition accuracy compared to the baseline and our proposed KD strategy, allowing the smaller student model to perform similarly, and sometimes even better, than the larger teacher model. We follow the experimental setup the same as STUN [11].

1 presents the recognition performance of various methods on the Pittsburgh30k dataset. In terms of VPR performance, both our student and teacher method outperforms all other baseline methods. Notably, our student model outperforms our large teacher model in recall metrics while performing similarly in mAP and AP metrics. Our student model surpasses other baseline models in all metrics, including larger teacher models of baseline methods.

To validate our method’s generalization ability on a larger database, we compared it with STUN on Pittsburgh250k without retraining or fine-tuning. As depicted in 2, our method consistently achieves significantly better performance compared to the baseline methods. A more detailed comparison is also provided in the top-left figure of 7. The recall and precision curves clearly illustrate that our method outperforms STUN on two datasets, affirming its superior performance. We additionally provide more visualization results in 6. As can be seen, our TSCM consistently identifies true positives in VPR with smaller distances to the query compared to STUN, which produces incorrect matches.

4.5pt

2.2pt

4.3 Ablation Studies and Insights↩︎

In this section, we assess the impact of our proposed cross-metric KD and validate our designs on the architecture and our KD strategies. In 4, we denote \(D_{\text{SaTp}}\) as the distances between student anchors and teacher positive samples, and so on. As can be seen, our cross-metric knowledge distillation method enhances place recognition within TSCM. Moreover, the performance of our proposed \(L_{\text{cm}} = xx\) is notably superior using \(D_{\text{SaTp}}\) and \(D_{\text{SpTa}}\), compared to scenarios where \(D_{\text{SpTa}}\), \(D_{\text{SpTa}}\), \(D_{\text{SnTa}}\), and \(D_{\text{SaTn}}\) are all included, or when either \(D_{\text{SpTa}}\), \(D_{\text{SpTa}}\), \(D_{\text{SnTa}}\) or \(D_{\text{SpTa}}\), \(D_{\text{SpTa}}\), \(D_{\text{SaTn}}\) are added. This substantiates our hypotheses presented in 5.

We conducted ablation experiments to assess the effectiveness of specific architectural components within our teacher network. This included investigating the impact of retaining the ResNet branch, removing the last ResNet stage, and evaluating the contribution of the Inter-Transformer Encoder. The results of these experiments, depicted in 7, revealed the following key findings: i) Retaining the ResNet branch significantly improved model performance; ii) Removing the final ResNet stage did not lead to performance degradation but rather resulted in improvement; iii) The inclusion of the Inter-Transformer Encoder had a positive impact on model performance. In summary, our modified architecture outperforms the original one.

We conducted ablation experiments on our teacher network, examining the effects of specific architectural components. We analyze the impact of retaining the ResNet branch, removing the last ResNet stage, and evaluating the contribution of the Inter-Transformer encoder. As shown in 7, the results reveal the following insights: i) Keeping the ResNet branch improved model performance significantly; ii) Eliminating the final ResNet stage leads to performance improvement; iii) The inclusion of the Inter-Transformer Encoder positively affected model performance. In sum, all of our design choices contribute positively to VPR tasks.

4.4 Computational Efficiency and Runtime↩︎

This section provides a comparative analysis of the number of parameters and FLOPS in our teacher model, student model, and STUN’s student model, as illustrated in 3. The results show that our student model, when compared to both our teacher model and STUN’s student model, has fewer parameters and lower computational requirements, making it a more lightweight option. 3 also presents the runtime outcomes for TSCM, affirming our third claim that TSCM exhibits remarkable computational efficiency, capable of compressing captured images into descriptors about 1.3 ms and matching in less than 0.6 ms per query at a database size of 10 k within the experimental environment.

5 Conclusion↩︎

In this paper, we introduce a novel knowledge distillation technique for visual place recognition called cross-metric knowledge distillation. This method empowers the student network to surpass the teacher network’s performance when integrated into our teacher-student model for VPR. We conducted extensive experiments using various datasets and compared with existing methods to validate our claims. The results demonstrate that our approach achieves superior recognition accuracy while maintaining a more lightweight model than the baseline method. Furthermore, our proposed cross-metric knowledge distillation method outperforms conventional knowledge distillation techniques. Finally, we evaluated the model’s processing speed, confirming its ability to compress captured images into descriptors in 1.3 ms and achieve matching in less than 0.6 ms per query with a database size of 10 k, demonstrating real-time performance.

References↩︎