1 Introduction↩︎

Unmanned aerial vehicles (UAVs), commonly known as drones, have rapidly emerged as cost-effective and versatile platforms for high-resolution, real-time aerial data collection. With the global popularity of drones, massive aerial videos have been generated. Therefore, efficient retrieval of relevant content from these large-scale UAV videos has become a critical research issue [1]–[3]. Drone text-video retrieval (DTVR) links natural language queries to video data. It is vital in domains such as urban management, emergency response, security surveillance, and disaster relief [4].

Although text-video retrieval has achieved substantial progress in natural video domains, such as MSR-VTT [5] and MSVD [6], the UAV video domain remains underexplored. A representative attempt to address this problem is CapERA [7], which extends the ERA dataset [8] with human-annotated captions. However, CapERA has two major limitations. First, its captions are coarse-grained. Videos within the same class often share highly similar descriptions that emphasize generic motion patterns while neglecting contextual factors such as environmental conditions and background objects. Second, the captions lack semantic diversity. Many are generated by paraphrasing or back-translation [7] of a single source description, leading to severe redundancy among multiple captions for the same video. This lack of granularity and diversity weakens text-video correspondence and severely limits retrieval models’ ability to discriminate among visually similar aerial videos. Consequently, existing aerial datasets remain inadequate for supporting accurate and robust UAV text-video retrieval.

To overcome these limitations, we present the Drone Video-Text Match Dataset (DVTMD) with fine-grained and semantically diverse captions. DVTMD contains 2,864 videos and 14,320 captions, with five unique descriptions assigned to each video. Unlike prior datasets that rely on generic motion-oriented labels and captions, our annotations comprehensively cover five complementary and fine-grained dimensions of each video: (i) human figures and their actions, (ii) objects and equipment present in the scene, (iii) background settings and structural elements, (iv) weather and environmental conditions, and (v) overall visual tone and stylistic attributes. By systematically integrating these dimensions, DVTMD provides richer semantic coverage and reduces redundancy across captions. This fine-grained annotation strategy strengthens text-video correspondence and facilitates more precise discrimination among visually similar aerial videos.

Recent advances in large-scale vision-language pretraining, exemplified by CLIP [9], have demonstrated strong image-text alignment capabilities. Building upon this foundation, CLIP4Clip [10] adapts CLIP to the video domain via temporal pooling, while subsequent works [11]–[14] refine alignment by modeling visual-text interactions. However, their effectiveness does not fully transfer to drone text-video retrieval. As shown in Figure 1, UAV videos, captured from aerial viewpoints, cover wide scenes with multiple moving targets and complex backgrounds. Compared with ground-view videos, they involve larger spatial scales, diverse object sizes, and dynamic environments. CLIP-based methods emphasize global cues such as scene layout or dominant motion, but this coarse alignment often fails, as brief captions typically describe only subclips or subregions. To overcome this limitation, we introduce fine-grained alignment by establishing correspondences between video patches and word tokens. This approach is capable of capturing localized details, such as individual objects, vehicles, or human actions. However, it often overlooks higher-level contextual information, including the broader environment, spatial layout, and collective activity within the scene. For UAV scenarios, both global semantics and local details are indispensable.

Motivated by these challenges, we propose a Text-Conditioned Multi-granularity Alignment (TCMA) framework. At the global level, coarse video-sentence alignment captures overall semantics. At the temporal level, sentence-guided frame aggregation emphasizes frames most relevant to the description. At the local level, word-guided patch alignment refines correspondence at the object and region scale. This hierarchical design jointly leverages global context and fine-grained detail, thereby achieving more accurate and robust drone text-video retrieval.

Within the proposed TCMA framework, we further refine local-level alignment through a Word and Patch Selection Module. Captions for UAV videos typically focus on specific objects or actions, while video frames often contain extensive background regions and irrelevant elements. As a result, only a limited portion of the visual content is semantically relevant. To address this imbalance, we select the most informative patches from each frame and the most salient words from each sentence. This targeted selection suppresses background noise and redundant textual tokens, such as common stopwords, thereby preventing attention dilution and reinforcing meaningful word-patch interactions.

Finally, we introduce a Text-Adaptive Dynamic Temperature Mechanism to regulate attention sharpness. Conventional approaches apply a static temperature to control attention distribution, yet different textual descriptions demand varying levels of focus. Action-oriented descriptions require sharper attention to highlight a few critical frames, whereas scene-oriented descriptions demand smoother attention to capture broader contextual cues. To address this, our module dynamically predicts the optimal temperature conditioned on each input text, enabling adaptive attention that flexibly adjusts focus according to description type, thereby enhancing retrieval robustness.

Our main contributions of the work are as follows:

• We construct DVTMD, a drone video-text match dataset comprising 2,864 videos and 14,320 captions, which provides fine-grained and semantically diverse annotations across different dimensions.

• We propose a text-conditioned multi-granularity alignment framework that combines global video-sentence alignment, sentence-guided frame aggregation, and word-guided patch alignment to balance global context with fine-grained detail.

• We design a word and patch selection module to extract the most informative visual patches and textual tokens. Meanwhile, we introduce a text-adaptive dynamic temperature mechanism to adjust attention sharpness according to the query text.

• We conduct comprehensive evaluations on DVTMD and CapERA, and establish the standard benchmark for the drone text-video retrieval task. Extensive experiments demonstrate that our method outperforms existing state-of-the-art approaches.

2 Related Work↩︎

In this section, we review prior work on drone video–text retrieval, covering aerial cross-modal methods in Section 2.1 and general text-video retrieval methods in Section 2.2.

2.1 Cross-Modal Retrieval in Aerial Scenarios↩︎

Remote Sensing Image–Text Retrieval (RSITR) was the first step toward cross-modal understanding in aerial scenarios. It enables the retrieval of remote sensing data across visual and textual modalities. Early approaches relied on CNN backbones and recurrent networks trained from scratch [3], [15]–[18]. Abdullah et al. [15] pioneered RSITR using an average fusion strategy, while Yuan et al. [16] introduced visual self-attention and released the RSITMD benchmark. Subsequent studies improved cross-modal alignment through knowledge-aided learning [18], key-entity attention [3], and language-cycle attention [17]. With the advent of foundation models (FMs) such as CLIP [9] and BLIP [19], research has shifted toward leveraging pretrained vision–language representations for RSITR. Recent efforts in parameter-efficient fine-tuning [20] and large-scale benchmarking [21] further demonstrated the promise of FMs in this domain. Extending beyond satellite imagery, VCSR [1] introduced UAV Image–Text Retrieval by incorporating context-region learning and consistency-based semantic alignment. Recent work by Bashmal et al [7], [22] proposed a dataset capERA and a text-to-event retrieval framework for aerial videos by combining a vision transformer (ViT)–based video branch with a BERT-based text branch, optimized via a bidirectional contrastive loss.

2.2 Text-Video Retrieval↩︎

Text-Video retrieval aims to find the most relevant video given a natural language query, or vice versa, retrieving text descriptions corresponding to a video. This task requires modeling both cross-modal correspondence and temporal dynamics in videos. Existing methods can be broadly categorized into single-stream, dual-stream, and CLIP-based approaches.

Single-stream method like ActionBERT [23] jointly embed video-text pairs within a unified network, enabling early cross-modal fusion. While effective at capturing fine-grained interactions, these approaches often have high computational costs and limited adaptability to downstream tasks. In contrast, dual-stream methods employ separate encoders for video and text, followed by contrastive learning in a joint embedding space. This design improves cross-modal representation learning and training efficiency. Representative advances include MIL-NCE [24], which learns video–text representations from narrated videos without human annotation by introducing a multiple instance learning noise contrastive loss; ClipBERT [25], which reduces training cost via sparse clip sampling in end-to-end modeling; and Frozen in Time [26], which unifies image–text and video–text pretraining with a space–time transformer encoder.

More recently, CLIP-based approaches leverage the pre-trained language–vision model CLIP, which is trained on large-scale text–image data. Methods such as CLIP4Clip [10] adopt CLIP as a backbone for video–text retrieval by directly mean-pooling frame-level features to obtain a compact video representation. Beyond global pooling, X-Pool [11] introduces cross-modal attention to allow texts to attend selectively to their most semantically relevant video frames, thereby avoiding noise from irrelevant visual content. Beyond discriminative matching, generative approaches have been explored. DiffusionRet [27] extends the CLIP paradigm by modeling the joint distribution of video and text, combining generation and discrimination objectives. Cap4Video [28] leverages auxiliary captions generated from videos to enrich cross-modal alignment, incorporating captions as additional input data. To enhance scalability, parameter-efficient adaptation techniques have emerged. MV-Adapter [29] inserts lightweight adapters into CLIP layers to transfer knowledge with minimal trainable parameters, while VoP [1] introduces cooperative prompt tuning for both text and video, achieving strong performance with less than 0.1% trainable parameters. TeachCLIP [30] distills fine-grained cross-modal reasoning from heavy teacher models into a CLIP4Clip-based student, maintaining efficiency while achieving near state-of-the-art accuracy. Fine-grained spatial–temporal modeling has also gained traction. TS2-Net [12] dynamically adjusts token sequences to select informative tokens in both temporal and spatial dimensions. MCQ [31] introduces a pretext task formulated as multiple choice questions on erased details of nouns and verbs to explicitly associate intermediate video and text tokens. UCofia [32] and EERCF [33] adopt coarse-to-fine strategies to refine video–text matching from global to local levels.

3 DVTMD Dataset↩︎

In this section, we introduce the DVTMD dataset. Section 3.1 details the dataset construction process. Section 3.2 presents a comprehensive analysis of DVTMD.

3.1 Dataset Construction↩︎

The construction pipeline of the newly proposed DVTMD is shown in Figure 2. We choose the ERA dataset [8] as our aerial video source. ERA comprises 2,864 video clips, each annotated with one of 25 event categories. Each clip lasts 5 seconds, recorded at 24 frames per second with a spatial resolution of 640\(\times\)​640 pixels. To adapt this resource for multimodal retrieval tasks, we perform the following steps.

Step 1: Frame Sampling. To reduce redundancy while preserving temporal information, six evenly spaced frames are uniformly extracted from each video. Formally, given a video \(V \in \mathbb{R}^{T \times H \times W \times 3}\), frames are sampled at \(F_1 = V\left[\tfrac{T}{6}\right], \; F_2 = V\left[\tfrac{2T}{6}\right], \dots, \; F_6 = V[T]\). This ensures a balanced frame set \(V' \in \mathbb{R}^{6 \times H \times W \times 3}\) well-distributed across the entire video duration for subsequent annotation.

Step 2: Detailed Frame Description Generation. Each sampled frame is annotated using state-of-the-art vision-language models (VLMs). After comparative evaluation of Qwen2.5-VL-7B [34], MiniCPM-V-2.6 [35], and InternVL2.5-8B [36], we adopt Qwen2.5-VL-7B due to its higher consistency and accuracy. Our prompt design is driven by the need for objective, literal, and fine-grained descriptions that can distinguish subtle differences across frames within the same or similar video genres. Specifically, the prompts instruct the model to systematically describe human figures, objects, backgrounds, color tones, and spatial arrangements without inferring unseen events or emotional context.

Step 3: Video-Level Summarization. To condense frame-level descriptions into concise video-level captions, we employ Qwen3 [37] due to its strong performance in structured summarization tasks. The summarization prompt is carefully crafted to guide the model towards factual, dense, and temporally-aware descriptions covering subjects, actions, backgrounds, and distinctive details, while excluding imaginative or rhetorical content. Each video is summarized into five captions of approximately 20 words each.

Step 4: Dataset Formation. Through this process, we establish a one-to-many correspondence between videos and their associated captions. Each of the 2,864 ERA videos is paired with five structured captions, producing 14,320 captions in total. The resulting DVTMD dataset serves as the basis for subsequent retrieval and analysis tasks in our study.

3.2 Dataset Analysis↩︎

Our DVTMD contains 2,864 videos, with 1,473 videos designated for training and 1,391 videos for testing. Each video is labeled with one of 25 event categories. For each video, we collect exactly 5 captions, resulting in a total of 14,320 captions across the entire dataset, of which 7,365 belong to the training set and 6,955 to the test set. The captions are constructed to be dense and descriptive, with an average length of 21.65 words. The vocabulary size of DVTMD amounts to 7,471 unique words, reflecting the diversity of expressions and scene compositions captured by UAVs. To more comprehensively understand the characteristics of the constructed DVTMD, we conduct a graphical analysis of the captions from the perspectives of caption length, category distribution, and lexical coverage.

Figure 3 (a) illustrates the caption length distribution. Most captions fall between 15 and 25 words, peaking at 18–22 words. This reflects the design goal of balancing clarity and descriptive richness. The slight skew toward shorter captions indicates a preference for concise, information-dense descriptions without excessive verbosity.Figure 3 (b) presents the number of captions per category across training and testing splits. Despite minor variations in the "NonEvent" category, which contains more captions in the test set than in the training set, DVTMD maintains a balanced distribution over 25 categories. Figure 3 (c) and Figure 3 (d) illustrate word clouds of train set and test set, respectively, generated from the captions after removing stop words unrelated to video content. These visualizations demonstrate the coverage of various elements, including humans, actions, objects, backgrounds, and environmental conditions.

Finally, to assess the granularity of captions relative to the capERA dataset, we compute the weighted average BLEU and METEOR scores between video-caption pairs and visualize the results in Figure 4. The goal is to ensure the captions are distinct and sufficiently detailed. Ideally, the similarity matrix would form a diagonal line from the top-left to the bottom-right, indicating that each caption corresponds only to its respective video. However, in Figure 4 (a), the similarity matrix for capERA shows significant off-diagonal lines and color blocks, suggesting that captions within the same category are overly similar. This observation underscores that our dataset, in contrast to capERA, has a clearer diagonal structure that better distinguishes videos with subtle variations within the same genre, offering a more nuanced representation of intra-category diversity.

4 Methodology↩︎

In this section, we introduce our proposed framework. Section 4.1 explains preliminary knowledge on how we adapt the CLIP model to the text-video retrieval task. Section 4.2 introduces our TCMA framework, including hierarchical alignment at video, frame, and patch levels. Section 4.3 details the hierarchical alignment loss that integrates contrastive learning with pearson correlation regularization across multiple granularities.

4.1 Expanding Language-Image Pre-training Model↩︎

We adopt CLIP [9] Given a video set \(V=\{v_1, \dots, v_N\}\) and a caption set \(T=\{t_1, \dots, t_N\}\), the objective is to learn a similarity function \(s(v_i, t_j)\) that produces high scores for semantically aligned pairs and low scores for mismatched ones. For each video \(v_i\), a sequence of \(T\) frames is uniformly sampled. Each frame \(f_i^t\) is encoded by the pre-trained CLIP [9] encoder to obtain frame-level embeddings: \[F_i = \{\phi(f_i^1), \phi(f_i^2), \ldots, \phi(f_i^T)\}\] where \(\phi(f_i^t) \in \mathbb{R}^D\). The video-level representation is then obtained via mean pooling: \[\hat{v}_i = \frac{1}{T}\sum_{t=1}^{T} \phi(f_i^t).\]

For text input \(t_j\), the Transformer-based encoder with 12 layers, width 512, and 8 attention heads is applied, and the output of the [EOS] token at the final layer is used as the sentence embedding: \(\theta(t_j) \in \mathbb{R}^D\). The similarity between video and text is then calculated by cosine similarity: \[s(v_i, t_j) = \frac{\theta(t_j)^\top \hat{v}_i}{\|\theta(t_j)\|\|\hat{v}_i\|}.\]

4.2 Text-Conditioned Multi-granularity Aggregation↩︎

To effectively align video and text at different levels of granularity, we design a hierarchical aggregation strategy shown in Figure 5 that progressively enhances video representations with textual guidance. Specifically, we introduce three complementary modules: (1) text-agnostic video pooling to capture holistic video semantics, (2) sentence-guided frame aggregation to adaptively weight frames according to sentence relevance, and (3) word-guided patch aggregation to model fine-grained correspondence between salient words and patches.

4.2.1 Text-Agnostic Video Pooling↩︎

We first obtain a video-level representation that is independent of text. Specifically, we apply mean pooling over all frame features to produce a global video embedding \(\hat{v}_{\text{video}}\): \[\hat{v}_{video} = \frac{1}{T}\sum_{t=1}^{T} \phi(f^t),\] where \(\hat{v}_{\text{video}} \in \mathbb{R}^D\) serves as the aggregated representation of video \(v\).

4.2.2 Sentence-Guided Frame Aggregation↩︎

To model the cross-modal correspondence between a sentence and the video frames, we propose a dynamic sentence-guided Frame Aggregation Module, inspired by EERCF [33]. As shown in Figure 6 (a), this module adaptively weights each video frame based on its semantic relevance to the input sentence, producing a sentence-guided video representation \(\hat{v}^{\text{frame}}\).

Given a sentence feature vector \(\theta(t) \in \mathbb{R}^D\) and a sequence of \(T\) frame features \(F = \{\phi(f^1), \phi(f^2), \dots, \phi(f^T)\}\) with \(\phi(f^i) \in \mathbb{R}^D\), the frame aggregation module computes sentence-to-frame similarity through dot products: \[s_i = \phi(f^i)^\top \theta(t), \quad i = 1, 2, \dots, T.\]

The similarity scores are normalized into attention weights using a softmax function with temperature \(\tau\): \[a_i = \frac{\exp(s_i / \tau)}{\sum_{j=1}^{T} \exp(s_j / \tau)}.\]

The sentence-guided video embedding is obtained by weighted aggregation: \[\hat{v}_{\mathrm{frame}} = \sum_{i=1}^{T} a_i \, \phi(f^i).\]

Here, a smaller \(\tau\) emphasizes a few highly relevant frames, while a larger \(\tau\) distributes attention more evenly across frames. Different sentences may require varying attention sharpness: action-oriented sentences benefit from smaller \(\tau\) for concentrated attention, whereas scene-oriented sentences benefit from larger \(\tau\) for smoother aggregation. To address this, we introduce a lightweight Text-Adaptive Dynamic Temperature Mechanism. The sentence feature \(\theta(t)\) is input to a small network to predict the optimal temperature: \[\tau_{\text{dyn}}(t) = \mathrm{Softplus}(W_\tau \theta(t) + b_\tau) + \epsilon,\] where \(W_\tau \in \mathbb{R}^{1 \times D}\) and \(b_\tau\) are learnable parameters, Softplus ensures positivity, and \(\epsilon=0.001\) prevents numerical instability. The attention weights are then recomputed using \(\tau_{\text{dyn}}(t)\): \[\hat{a}_i = \frac{\exp(s_i / \tau_{\text{dyn}}(\theta(t)))}{\sum_{j=1}^{T} \exp(s_j / \tau_{\text{dyn}}(\theta(t)))}.\]

Finally, the compact formulation of frame-level representation is expressed as: \[\hat{v}_{\mathrm{frame}} = \sum_{i=1}^T \mathrm{Softmax}\!\Bigg( \frac{\phi(f^i)^\top\theta(t)}{\tau_{\mathrm{dyn}}(t)} \Bigg) \; \phi(f^i).\]

This design enables frame aggregation to adaptively focus on the most relevant frames for each sentence and produce the most appropriate frame-level representation, improving sentence-to-video alignment without introducing heavy additional parameters.

4.2.3 Word-Guided Patch Aggregation↩︎

High-level frame features often fail to capture subtle cross-modal correspondence. To achieve fine-grained alignment, we leverage patch-level visual tokens and word-level textual tokens. However, directly aligning all patches with all words is inefficient due to the redundancy of background patches. To address this, we design a Word and Patch Selection Module, as shown in Figure 6 (c) and Figure 6 (d). They select the most salient patches and words for alignment.

Word Selection Module. Given the word embeddings \(T = \{\theta(w_1), \dots, \theta(w_L)\}\) with \(\theta(w_l)\in\mathbb{R}^D\), we first augment each patch with the sentence feature \(\theta(t)\) then compute a saliency score for each word: \[u_l^{\text{word}} = G_w(\mathrm{Concat}(\theta(w_l),\theta(t))), \quad l = 1,\dots,L,\] where \(G_w\) is a convolutional layer. Based on \(u_l^{\text{word}}\), we select the top-\(K_w\) informative words: \[\hat{T} = \mathrm{TopK}(T, \{u_l^{\text{word}}\}_{l=1}^L ) \in \mathbb{R}^{K_w \times D}.\]

Patch Selection Module. Given the patch features of the \(n\)-th frame \(P_n = \{\phi(p_1^n), \dots, \phi(p_M^n)\}\) with \(\phi(p_j^n)\in\mathbb{R}^D\), we first augment each patch with the frame feature \(\phi(f^n)\): \[\tilde{p}_j^n = G_a\big(\mathrm{Concat}(\phi(p_j^n), \phi(f^n))\big),\] then fuse with video-level feature \(v\) to produce a saliency score: \[u_j^n = G_b\big(\mathrm{Concat}(\tilde{p}_j^n, v)\big).\] We select the top-\(K_p\) patches per frame based on \(u_j^n\), yielding \[\hat{P}_n = \mathrm{TopK}\big(P_n, \{u_j^n\}_{j=1}^M \big) \in \mathbb{R}^{K_p \times D}.\] Concatenating across all \(T\) frames gives \[\hat{P} = \{\hat{P}_1, \dots, \hat{P}_T\} \in \mathbb{R}^{T \times K_p \times D},\]

Patch Aggregation Module. As shown in Figure 6 (b), for each selected word \(\hat{w}_l \in \hat{T}\) and selected patch \(\hat{p}_j \in \hat{P}\), we compute the similarity: \[s_{l,j} = \hat{w}_l^\top \hat{p}_j.\] The similarities are normalized with a dynamic temperature \(\tau_{\text{dyn}}(\hat{w}_l)\) (as defined previously): \[a_{l,j} = \frac{\exp(s_{l,j}/\tau_{\text{dyn}}(\hat{w}_l))}{\sum_{k=1}^{L_v} \exp(s_{l,k}/\tau_{\text{dyn}}(\hat{w}_l))}.\] The patch representation guided by word \(\hat{w}_l\) is: \[\hat{v}_{\mathrm{patch}} = \sum_{i=1}^{T}\sum_{l=1}^{L}\sum_{j=1}^{K_p} a_{l,j} \hat{p}_j.\]

4.3 Hierarchical Alignment Loss↩︎

Contrastive learning is widely adopted in multi-modal learning tasks. Similarly, our framework employs a contrastive objective, treating paired video-text samples as positives and all other non-matching pairs in the batch as negatives. Given a batch of \(B\) video-text pairs \(\{(v_i, t_i)\}_{i=1}^B\), we define the contrastive loss in two symmetric directions: \[\begin{align} L_{v2t}(v,t) &= -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(s(v_i, t_i))}{\sum_{j=1}^{B} \exp(s(v_i, t_j))}, \\ L_{t2v}(v,t) &= -\frac{1}{B} \sum_{i=1}^{B} \log \frac{\exp(s(v_i, t_i))}{\sum_{j=1}^{B} \exp(s(v_j, t_i))}, \end{align}\] where \(v\in \mathbb{R}^{B\times D}\) and \(t\in \mathbb{R}^{B\times D}\) are the video and text feature matrices, and \(s(\cdot, \cdot)\) denotes cosine similarity. The \(v2t\) term encourages each video to be most similar to its corresponding text, while \(t2v\) ensures each text is most similar to its corresponding video. By jointly optimizing both directions, the model learns robust cross-modal alignments.

In addition to the contrastive objective, we incorporate a Pearson correlation constraint to regularize feature structures. This term encourages same-channel consistency, where the \(d\)-th channel of video features aligns with the \(d\)-th channel of text features, while reducing cross-channel redundancy. The Pearson correlation coefficient between two vectors \(x\) and \(y\) is defined as \[\rho_p(x,y) = \frac{(x-\bar{x})^\top(y-\bar{y})}{\|x-\bar{x}\|\|y-\bar{y}\|},\] and the corresponding Pearson distance is \[d_p(x,y) = 1 - \rho_p(x,y).\] The Pearson regularization term over a batch of features is then defined as \[\begin{align} L_{Pearson}(v,t) &= \beta \sum_{d=1}^{D} \big\| d_p(v_{:,d}, t_{:,d}) \big\|^2 \notag \\ &\quad + \alpha \sum_{d_1=1}^{D} \sum_{d_2 \ne d_1} \big\| \rho_p(v_{:,d_1}, t_{:,d_2}) \big\|^2, \end{align}\] where \(||\cdot||\) denotes the L2-norm and \(\beta\) and \(\alpha\) control the strength of regularization. The first term enforces same-channel alignment, while the second discourages cross-channel redundancy. Thus, we define a general loss function that combines both contrastive and Pearson terms: \[L(v,t) = L_{v2t}(v,t) + L_{t2v}(v,t) + L_{Pearson}(v,t).\]

Video-Level Alignment. The video-level loss captures global semantics across all frames: \[L_{\text{video}} = L(\hat{v}_{\text{video}}, \phi(t)),\] where \(\hat{v}_{\text{video}}\) is the mean-pooled video representation and \(\phi(t)\) is the text embedding.

Frame-Level Alignment. The frame-level loss models the relevance of individual frames guided by sentence-level features: \[L_{\text{frame}} = L(\hat{v}_{\text{frame}}, \phi(t)),\] where \(\hat{v}_{\text{frame}}\) is obtained from sentence-guided frame aggregation.

Patch-Level Alignment. The patch-level loss captures fine-grained correspondence between salient patches and words: \[L_{\text{patch}} = L(\hat{v}_{\text{patch}}, \phi(t)),\] where \(\hat{v}_{\text{patch}}\) is the output of word-guided patch aggregation.

Overall Hierarchical Objective. Finally, the overall loss combines all levels in a weighted manner: \[\mathcal{L}_{\text{hier}} = \lambda_{\text{video}} \, L_{\text{video}} + \lambda_{\text{frame}} \, L_{\text{frame}} + \lambda_{\text{patch}} \, L_{\text{patch}},\] where \(\lambda_{\text{video}}\), \(\lambda_{\text{frame}}\), and \(\lambda_{\text{patch}}\) are hyperparameters controlling the contribution of each hierarchical level.

5 Experiments↩︎

In this section, we conduct extensive experiments to evaluate the effectiveness of our TCMA model. We introduce our experimental settings in Section 5.1, compare it with state-of-the-art methods in Section 5.2, perform ablation studies to analyze the contribution of each component in Section 5.3, and provide qualitative results in Section 5.4.

5.1 Experimental Settings↩︎

Evaluation Metrics. We evaluate our TCMA model on standard text-video retrieval benchmarks using Recall at rank \(K\) (R@1, R@5, R@10), Mean Rank (MnR), and Median Recall (MdR).

Implementation Details. All experiments are conducted on a single H20-NVLink GPU (96GB) and implemented in PyTorch. We initialize our TCMA model with CLIP [9] pre-trained weights. For the visual encoder, we adopt CLIP’s ViT-B/32 and ViT-B/16 backbones, where the feature dimension \(D\) for both visual and textual embeddings is set to 512. For optimization, we use Adam with a cosine warm-up schedule. The learning rates are set to \(1\times10^{-7}\) for the pre-trained visual and text encoders, and \(1\times10^{-4}\) for the newly added modules. The batch size is fixed at 64.

Regarding the method-specific parameters, in the patch selection module, we retain the top-\(K_p = 3\) most salient patches per frame, and in the word selection module, the top-\(K_w = 8\) most salient words are chosen. We uniformly sample \(T = 12\) frames per video, and the maximum text length is capped at 32 tokens. Unless otherwise specified, the hyperparameters in our loss functions are empirically set as follows: the intra-modal loss weight is fixed at \(\alpha = 0.05\), and the total loss incorporates \(\beta = 0.001\) with a weighting ratio of \(\lambda_{V_{video}} : \lambda_{v_{frame}} : \lambda_{v_{patch}} = 5 : 5 : 1\).

5.2 Comparisons With State-of-the-Art Methods↩︎

We compare TCMA against a range of state-of-the-art text–video retrieval methods, including global-matching approaches [9], [10], fine-grained alignment frameworks [12], generative models [27], parameter-efficient approaches [38], and coarse-to-fine alignment models for natural video understanding [32], [33].

Global-matching methods such as CLIP [9] and Clip4Clip [10] achieve strong performance on generic video benchmarks, but they underperform on UAV datasets where targets are small, backgrounds are uniform, and global video representations lack sufficient discriminative capacity. Fine-grained methods such as TS2-Net [12] leverage region-level cues but fail to capture holistic global context, which is critical for integrating information in UAV scenes. Generative approaches like DiffusionRet [27] incur substantial computational overhead and exhibit unstable retrieval quality, as evidenced by their significantly degraded MnR on DVTMD. Finally, recent coarse-to-fine models, including UCoFiA [32] and EERCF [33], improve cross-modal alignment but are primarily optimized for natural video domains. They struggle to adapt to aerial scenarios, where objects vary greatly in scale, noise is abundant, and inter-class distinctions are subtle.

In contrast, TCMA is purpose-built for UAV video retrieval. Its hierarchical aggregation fuses both global and local information, token selection modules filter out irrelevant noise in cluttered aerial scenes, and the dynamic temperature mechanism adapts to variations in object and scene scale. Together, these designs directly tackle the challenges of small-object resolution, fine-grained motion modeling, and context disambiguation, yielding robust performance in aerial video retrieval.

Tables 1 and 2 show that TCMA achieves consistent improvements over all baselines on both DVTMD and capERA datasets. With the ViT-B/32 backbone, TCMA achieves 45.5% R@1 and 75.3% R@5 in text-to-video retrieval, improving by +1.2% and +0.5% over the previous best method EERCF [33]. Using the stronger ViT-B/16 backbone, the margin further widens as TCMA reaches 48.5% R@1. A similar trend holds in video-to-text retrieval. Our TCMA with ViT-B/16 obtains 45.9% R@1, surpassing EERCF by +4.1%, the largest improvement among all settings. The retrieval on capERA is more challenging due to the weak text-video correspondence as mentioned before. However, TCMA maintains the best performance among all methods, raising R@10 by +1.1% on ViT-B/16 and reducing MnR to 21.2.

5.3 Ablation Studies↩︎

To better understand the contributions of each component in our TCMA framework, we conduct ablation studies on the DVTMD dataset with ViT/32 backbone.

Hierarchical Aggregation. We first investigate the effect of progressively introducing different levels of alignment. Starting from a video-only baseline that uses mean-pooled representations, we add frame-level alignment via sentence-guided aggregation, then further incorporate patch-level alignment guided by word features. The results in Table 3 show that each additional level consistently improves retrieval performance. The full TCMA model, which combines all three levels, achieves the best overall performance.

Word and Patch Selection. We then analyze the role of word and patch selection. Without any filtering, all words are aligned with all patches, which introduces noise from uninformative tokens. Selecting only the most salient patches already improves performance, and word selection brings similar benefits. The results in Table 5 and Table 4 show When both strategies are applied jointly, as in our final design, the model achieves the best retrieval accuracy.

Dynamic Temperature. Finally, we evaluate the impact of the dynamic temperature coefficient \(\tau_{\text{dyn}}\) and record the results in Table 6. Using a fixed temperature produces competitive results, but adapting the temperature dynamically yields consistent improvements across recall metrics.

5.4 Qualitative Results↩︎

We visualize some retrieval examples on our DVTMD set for text-to-video retrieval in Figure 7. In the first case, our model accurately identifies the number of objects (“two vehicles”) and their respective attributes (“red” and “silver”), while also distinguishing “mud” from regular road surfaces. In the last case, the model successfully retrieves a video containing both human figures (“firefighters in dark gear”) and relevant objects (“red fire truck”), highlighting its capability to jointly reason over actions and contextual elements.

6 Conclusion↩︎

In this paper, we propose a drone text-video retrieval framework that jointly captures global semantics and fine-grained details through multi-granularity alignment. We mainly establish the TCMA model, which integrates global video-sentence alignment, sentence-guided frame aggregation, and word-guided patch alignment, and contribute the fine-grained DVTMD dataset. The results show that for UAV videos with large spatial scales, diverse objects, and complex backgrounds, retrieval accuracy can be significantly improved by using Top-K Word and Patch selection to filter irrelevant content and a dynamic temperature module to adapt attention sharpness. The qualitative and quantitative results on DVTMD and CapERA indicate that multi-granularity alignment with adaptive attention may be a promising direction for advancing drone text-video retrieval.

References↩︎