1 Introduction↩︎

Multi-animal tracking (MAT), a representative yet challenging multi-object tracking problem, plays a pivotal role in biology, ecology, and wildlife research, supporting applications such as animal monitoring [1], [2], population estimation [3], and behavior analysis [4]. MAT involves accurately locating all animals within a video stream while consistently maintaining their identities throughout the sequence. Despite remarkable advances in human and vehicle tracking [5]–[8], MAT remains technically demanding, particularly under low visual discriminability regimes, where appearance cues provide limited identity separation and geometric dynamics exhibit stronger variability.

Driven by advances in object detection, the bottleneck in multi-object tracking has primarily shifted from accurate localization to robust and consistent identity association [9], [10]. To address this, two major paradigms have emerged: heuristic-based association and Transformer-driven query-based tracking. Traditional heuristic-based paradigms typically adopt a detector–tracker pipeline, relying on meticulously crafted rules (e.g., Kalman filter [11] and Hungarian algorithm [12]) to model geometric cues for association [13], [14]. Despite notable progress, these methods rely on fixed association rules that are inherently incompatible with gradient-based end-to-end optimization. Consequently, they struggle to adapt to evolving visual and geometric cues, leading to suboptimal performance in complex tracking scenarios [15]. In contrast, query-based methods formulate tracking as query propagation across frames, enabling end-to-end identity modeling via cross-frame attention [16]–[19]. Although effective on human and vehicle benchmarks, these methods are heavily appearance-driven. In scenarios with weak visual discriminability, such reliance renders identity association inherently fragile.

Multi-animal tracking presents precisely such a regime (Fig. 1 (a)). First, animals of the same species often exhibit a highly uniform appearance due to social grouping behavior, resulting in minimal inter-instance visual distinction. Second, animals frequently form high-density groups, where close spatial proximity and frequent overlap significantly increase identity ambiguity. Third, irregular and non-planar movement patterns, especially in aerial or aquatic environments, introduce strong geometric variability across frames. Overall, these factors place MAT in a low-discriminability, high-variability regime, in which both appearance and continuous geometric cues become less reliable for stable identity modeling.

This motivates incorporating geometric cues into end-to-end query-based tracking. However, geometric modeling under attention-based architectures is non-trivial. As observed in [6], camera motion, object jitter, and environmental noise introduce subtle yet persistent fluctuations in geometric signals. Although small in raw analog space, such perturbations may induce disproportionate shifts in cross-frame attention weight distributions, leading to unstable identity matching. Empirically, as we further demonstrate in Sec. 4.3, directly incorporating continuous geometric representations can even degrade association performance relative to appearance-only baselines. This phenomenon suggests that continuous geometric embeddings may destabilize attention-based identity modeling under noisy and appearance-ambiguous conditions.

Inspired by quantization principles in signal processing—where noisy analog signals are mapped to stable discrete states (Fig. 1 (b))—we revisit geometric modeling from a representation perspective. Rather than treating geometry purely as a continuous regression signal, we introduce a discretized state-space representation for geometric dynamics. By mapping fluctuating geometric cues to a finite set of learned codewords, we constrain perturbation-induced embedding variance while preserving structural information essential for stable identity association. This discrete formulation mitigates perturbation amplification and yields more stable attention responses.

Based on this insight, we propose HieDG, a Hierarchical Discrete Geometry-guided framework for multi-animal tracking. HieDG employs a coarse-to-fine residual quantization strategy to discretely encode position, size, and velocity cues through independent two-level codebooks. This hierarchical design captures both coarse geometric structure and fine-grained variations, transforming unstable continuous signals into robust discrete embeddings aligned with the visual feature space. The resulting geometric tokens are fused with tracking queries, enhancing identity consistency under challenging conditions. Our contributions are summarized as follows:

• We propose HieDG, a geometry-guided multi-animal tracking framework that unifies heuristic geometric reasoning with end-to-end query-based identity modeling via discrete geometric representations.

• We design a hierarchical residual quantization strategy that encodes position, scale, and velocity cues into structured discrete tokens, reformulating geometric modeling as a robust representation-learning problem within tracking.

• Extensive experiments on AnimalTrack, BFT, and BuckTales demonstrate state-of-the-art performance with significant improvements in HOTA, AssA, and IDF1. Additional evaluations on generic MOT benchmarks show competitive performance, indicating the general applicability of discretized geometric modeling beyond animal-specific scenarios.

2 Related Work↩︎

Tracking-by-Detection has become the dominant paradigm in modern MOT, where object detection and data association are decoupled. With the rapid progress in object detection, recent efforts have increasingly focused on the association stage, aiming to construct coherent trajectories across frames. Many prior works adopt heuristic post-processing strategies to associate current detections with historical trajectories [13]. Early approaches, such as SORT [20], rely on geometry by computing an IoU matrix between frame-wise detections and applying the Hungarian algorithm for optimal matching. ByteTrack [14] further incorporates low-score detections to recover temporarily occluded targets with dropped confidence, significantly improving identity preservation in occlusion-heavy scenes. OC-SORT [21] emphasizes motion continuity by modeling trajectory consistency and smoothing motion direction, showing notable advantages in long-term occlusions and crossing scenarios. While heuristic-based paradigms perform well under certain conditions through carefully crafted geometric modeling, their fixed association rules often require manual tuning and struggle to generalize across diverse or complex scenarios. Nevertheless, their explicit use of geometric consistency provides a useful inductive bias for identity association, motivating us to integrate geometry into end-to-end trainable frameworks.

Tracking-by-Query has emerged as a powerful paradigm in modern MOT due to its end-to-end trainability and its capability for global optimization. By propagating object queries across frames and performing cross-frame attention, these methods unify detection and association, enabling joint learning of localization and identity assignment via gradient descent. TransTrack [22] first introduces Transformers into MOT by propagating tracking queries through temporal attention. TrackFormer [16] further unifies detection and association within a shared decoder to improve alignment. MOTR [17], [18] models identity propagation by dynamically reusing historical queries, allowing flexible handling of object births and terminations. In contrast, MOTIP [19] formulates MOT as an identity prediction problem trained via classification loss, avoiding explicit matching operations.

Although these approaches achieve strong performance on standard human and vehicle benchmarks, their identity modeling remains largely appearance-driven. Under conditions commonly observed in multi-animal tracking, such as uniform appearance, high density, and irregular motion, appearance embeddings alone may provide limited separability, making association more challenging. Inspired by the geometric reasoning in classical heuristic pipelines, we incorporate explicit geometric cues (e.g., position, scale, and velocity) into the query-based framework to enhance robustness under visually ambiguous and dynamically complex scenarios.

2.1 Motivation and Overview↩︎

3 Method↩︎

In this section, we introduce our proposed MAT framework, HieDG, which integrates robust discrete geometric cues into the query-based paradigm. We begin with the motivation and overview in Sec. 2.1. Then, we detail the geometric vector construction and the hierarchical geometry quantization in Sec. 3.1 and Sec. 3.2, respectively. The geometry-aware animal tracking is described in Sec. 3.3. Finally, we outline the training and inference procedures in Sec. 3.4 and Sec. 3.5.

Motivation. In multi-animal tracking scenarios, uniform appearance, high density, and irregular motion patterns pose significant challenges to both heuristic-based and query-based tracking paradigms. Heuristic approaches, while effective at geometric modeling, lack the capacity for end-to-end global optimization, limiting their adaptability across diverse animal scenes. In contrast, query-based methods benefit from joint optimization but heavily rely on visual features, which are often non-discriminative in animal settings. This motivates a natural and intuitive question: can we combine the strengths of both paradigms by leveraging the global optimization capability of query-based frameworks to model geometric cues directly from video sequences, thereby enhancing identity association?

However, within query-based tracking frameworks, association is typically performed via cross-attention between encoded features. In real-world animal monitoring videos, camera shake, object jitter, and visual noise introduce fluctuations in geometric signals. Although subtle at the raw signal level, these variations can be amplified during embedding, leading the attention mechanism to misinterpret them as genuine geometric discrepancies—ultimately compromising association reliability. In signal processing and circuit analysis, quantization is often employed to convert perturbed analog signals into discrete and deterministic states, enhancing both stability and interpretability. Inspired by this, we seek to improve the discriminability of fluctuating geometric embeddings by discretizing them into structured discrete codeword representations.

Overview. The overall architecture of HieDG is illustrated in Fig. 2. We extract object-level geometric cues (position, scale, and velocity) from Deformable DETR [23] outputs and project them into latent vectors. Each geometric cue is independently quantized using a two-stage residual codebook in a coarse-to-fine manner, producing structured discrete tokens. These tokens constrain perturbation-induced embedding variance while preserving essential geometric information. The resulting discrete geometric embeddings are aligned with the visual feature space and fused with appearance features within the query-based tracking framework to guide robust identity association.

3.1 Geometric Vector Construction↩︎

We adopt Deformable DETR [23] as the end-to-end object detector to process each video sequence \(V=\{\mathbf{X}_t\}_{t=1}^{T}\), where \(\mathbf{X}_t \in \mathbb{R}^{H \times W \times 3}\) denotes the RGB image at frame \(t\). A CNN backbone [24] and a Transformer encoder [25] are employed to extract and enhance image features \(f_t \in \mathbb{R}^{h \times w \times c}\). Subsequently, the DETR decoder transforms detection queries into object-level predictions, including bounding box \(bbox = (c_{x,t}^i,c_{y,t}^i,w_t^i,h_t^i)\) and class index. Deformable DETR directly provides decoded embeddings as object-level features \(f_t^i\), eliminating the need for post-processing steps such as RoI cropping or feature alignment, obtaining dimension consistent representations across frames.

To accommodate video sequences with varying resolutions and aspect ratios, we normalize the bounding box parameters as \((c_{x,t}^i/W,c_{y,t}^i/H,w_t^i/W,h_t^i/H)\), where \(W\) and \(H\) denote the frame width and height, respectively. During training, the velocity of object \(i\) is computed by differencing the normalized center coordinates across consecutive frames, yielding Cartesian velocity components \((v_{x,t}^i, v_{y,t}^i)\). As polar coordinates explicitly decouple two identity-relevant factors: magnitude and direction [26], we convert the velocity representation from Cartesian \((v_{x,t}^i,v_{y,t}^i)\) to polar coordinates\((\rho_t^i,\theta_t^i)\). This formulation decouples motion magnitude and direction, reducing entanglement between velocity components and facilitating more stable representation. As a result, the entire geometric state vector \(g_t^i = (g_{p,t}^i, g_{s,t}^i, g_{v,t}^i)\) for object \(i\) at frame \(t\) is thus defined as: \[g_t^i = \left( \frac{c_{x,t}^i}{W}, \frac{c_{y,t}^i}{H}, \frac{w_t^i}{W}, \frac{h_t^i}{H}, \rho_t^i, \theta_t^i \right),\] where \(g_t^i\) represents the geometric vector of object \(i\) at frame \(t\), including its position, size, and velocity.

3.2 Hierarchical Geometry Quantization↩︎

Given that the constructed geometry vectors exhibit heterogeneous scales and units across dimensions, we perform residual quantization separately on position \(g_{p,t}^i\), scale \(g_{s,t}^i\), and velocity \(g_{v,t}^i\). Each component is projected into a latent space using a multi-layer perceptron, resulting in embeddings \({z_{g\in(p,s,v)}}_t^i\in\mathbb{R}^{D}\).

To better capture the fluctuating geometric cues, we design a hierarchical quantization strategy that discretizes the MLP-projected continuous embeddings. Specifically, for position, size, and velocity components, we apply a two-stage residual codebook to perform quantization.

\[E_{g \in (p, s, v)}^{(\ell)}=\{e_{g,1}^{(\ell)},\ldots,e_{g,K}^{(\ell)}\}\subset\mathbb{R}^{D}, \quad \ell=1,2, \label{eq:codebook-level}\tag{1}\] where \(D\) denotes the embedding dimension and \(K\) represents the number of codewords per stage. We set \(D=64\) and \(K=64\) to balance representational capacity and computational efficiency. A detailed analysis of the hyperparameter design is provided in Sec. 4.4.

\[\left\{ \begin{align} {k_1}_{g,t}^{i} &= \mathop{\mathrm{arg\,min}}_{k_1 \in \{1,\dots,K\}} \big\lVert z_{g,t}^i - e_{g,{k_1}}^{(1)} \big\rVert_2, &\quad q_{g,1} &= e_{g,{k_1}}^{(1)}, \\[2pt] r_{g,t}^i &= z_{g,t}^i - q_{g,1}, \\[2pt] {k_2}_{g,t}^{i} &= \mathop{\mathrm{arg\,min}}_{k_2 \in \{1,\dots,K\}} \big\lVert r_{g,t}^i - e_{g,{k_2}}^{(2)} \big\rVert_2, &\quad q_{g,2} &= e_{g,{k_2}}^{(2)}. \end{align} \right. \label{eq:two-stage-quant}\tag{2}\]

The residual formulation allows the second codebook to capture fine-grained geometric variations that persist after coarse quantization. The final geometric embedding \(\tilde{z}_{g,t}^i\) is obtained by summing the coarse and fine codewords, producing a more detailed and robust representation [27]. The discrete embeddings \(\tilde{z}_{p,t}^i\), \(\tilde{z}_{s,t}^i\), and \(\tilde{z}_{v,t}^i\) are then concatenated into a 3D-dimensional vector and linearly projected to \(\tilde{z}_t^i \in \mathbb{R}^{d}\) to ensure compatibility with the visual features.

3.3 Geometry-Aware Animal Tracking↩︎

In addition to the object-level visual embedding \(f_t^i\) extracted in Sec. 3.1, we follow [19] and incorporate identity information by projecting a one-hot encoded ID index into an embedding \(l^m\) for trajectory \(m\). We further integrate the discretized geometric embedding \(\tilde{z}_t^i\) to jointly model appearance, identity, and geometric states. All embeddings \(f_t^i\), \(l^m\), and \(\tilde{z}_t^i\) are projected to dimension \(d\). At frame \(t\), the historical representation of trajectory \(m\) over time window \([t\!-\!T, t\!-\!1]\) is defined as: \[F_{t-T:t-1}^m = \left\{ \left(f_{t-T}^m, \tilde{z}_{t-T}^m, l^m\right), \dots, \left(f_{t-1}^m, \tilde{z}_{t-1}^m, l^m\right) \right\}. \label{eq:frame95sequence}\tag{3}\]

Each detected object in the current frame is represented as \(o_t^i = (f_t^i, \tilde{z}_t^i, l_{\text{init}}^i)\). A Transformer decoder performs cross-attention between the current object query \(o_t^i\) and historical trajectory features \(F_{t-T:t-1}^m\). The concatenation of visual, geometric, and identity embeddings preserves modality-specific information before attention interaction. By incorporating discretized geometric tokens, the attention mechanism becomes less sensitive to small coordinate perturbations, leading to more stable identity matching across frames. The attention output is combined with \(o_t^i\) and passed through a classification head for identity prediction.

3.4 Training↩︎

Loss Function. Following prior works [16], [22], [28], we formulate trajectory association as an identity classification task that includes an additional unknown category. Beyond the standard detection loss \(\mathcal{L}_{\text{det}}\) from the DETR, we incorporate an ID classification loss \(\mathcal{L}_{\text{id}}\) and a vector quantization loss \(\mathcal{L}_{\text{vq}}\) for training process. These components are integrated into a unified objective via a weighted sum, where each coefficient \(\lambda_i\) modulates the contribution of the corresponding loss term \(\mathcal{L}_i\). The overall training objective is formulated as:

\[\mathcal{L}_{\text{total}} = \lambda_{\text{det}}\mathcal{L}_{\text{det}} + \lambda_{\text{id}}\mathcal{L}_{\text{id}} + \lambda_{\text{vq}}\mathcal{L}_{\text{vq}}, \label{eq:total95loss}\tag{4}\] where \(\mathcal{L}_{\text{det}}\) consists of a classification loss, a Generalized IoU loss, and an L1 regression loss. For the ID classification loss \(\mathcal{L}_{\text{id}}\), we adopt the standard cross-entropy formulation. The vector quantization loss \(\mathcal{L}_{\text{vq}}\) is defined as \(\| \text{sg}[z] - e \|_2^2 + \beta \| z - \text{sg}[e] \|_2^2\), where \(\text{sg}[\cdot]\) denotes the stop-gradient operator. The first term updates the codewords \(e\), while the second term encourages the encoder output \(z\) to commit to its assigned codeword [29], [30]. To enable gradient backpropagation through the non-differentiable nearest-neighbor operation, we adopt the straight-through estimator (STE) [31].

3.5 Inference↩︎

During inference, object detections in the current frame lack explicit temporal correspondences with the previous frame, which prohibits direct velocity estimation via frame-to-frame differencing. For each detection with normalized center \(\mathbf{c}_t^i = (c_{x,t}^i, c_{y,t}^i)\), we estimate its velocity using a lightweight KNN-based regression over historical trajectory states. Specifically, we retrieve the \(k\) nearest historical states \(\mathcal{N}_k\) in spatial coordinate space and compute the estimated velocity as \(\hat{\mathbf{v}}_t^i = \sum_{j \in \mathcal{N}_k} w_{ij} \mathbf{v}_j\), where the normalized weights are defined as \[w_{ij} = \frac{ \exp\!\left(-\|\mathbf{c}_t^i - \mathbf{c}_j\|_2^2 / \sigma_s^2\right) \cdot \exp\!\left(\cos(\Delta\phi_{ij}) / \sigma_d\right) }{ \sum_{l \in \mathcal{N}_k} \exp\!\left(-\|\mathbf{c}_t^i - \mathbf{c}_l\|_2^2 / \sigma_s^2\right) \cdot \exp\!\left(\cos(\Delta\phi_{il}) / \sigma_d\right) }.\] Here \(\mathbf{v}_j\) denotes historical velocity vectors and \(\Delta\phi_{ij}\) denotes the angular difference between the motion direction of the current detection and the historical state \(j\). The spatial kernel encourages proximity consistency, while the directional term favors motion-aligned trajectories.

4 Experiments↩︎

4.1 Datasets and Metrics↩︎

Datasets. To comprehensively evaluate multi-animal tracking, we select a set of challenging and representative benchmarks spanning diverse animal study scenarios. Animaltrack [6] is a multi-class animal tracking dataset featuring complex inter-individual interactions, consisting of 58 sequences collected from wild animals across 10 terrestrial and aquatic categories. BFT [32] is a dataset of 22 bird species with diverse morphological characteristics and irregular motion patterns, composed of 106 videos. BuckTales [33] is a UAV-based animal tracking dataset featuring wild antelopes with highly similar visual appearances. Due to incomplete annotations in the full release, we evaluate on the subset with complete identity labels (12 videos, 680 trajectories). These benchmarks cover a broad spectrum of representative challenges in multi-animal tracking. This diverse setting enables a comprehensive evaluation of HieDG in terms of both performance and robustness across complex animal tracking scenarios.

To further assess the generalization of our study beyond animal-specific scenarios, we conduct experiments on two widely adopted multi-person tracking benchmarks, DanceTrack [34] and SportsMOT [35]. Both datasets exhibit extreme intra-class similarity and complex motion patterns, posing substantial challenges for identity association.

Metrics. We adopt Higher Order Tracking Accuracy (HOTA) [36] to jointly evaluate detection (DetA) and association (AssA) performance. MOTA [37] and IDF1 [38] are also reported to cover complementary tracking aspects. Since our method primarily targets identity association, we place particular attention on HOTA, AssA, and IDF1, while reporting MOTA for completeness.

4.2 Implementation Details↩︎

Given its simplicity and generality, we adopt MOTIP [19] as the baseline for this study. Specifically, we employ a ResNet-based [24] Deformable DETR [23] as the detector in the proposed HieDG framework. To maintain consistency with prior work, we initialize the model using COCO [39] pretrained weights. Furthermore, we set the dimension of each codebook and each geometric component to \(D=64\), while the hidden dimension of the quantized geometry embedding \(\tilde{\mathbf{z}}\) is aligned with the Deformable DETR backbone, i.e., \(C=256\).

Following prior work [18], [19], [49], we apply standard image-level augmentations, including random resizing, horizontal flipping, and color jittering, for an apples-to-apples comparison. To improve robustness to estimation noise at inference, we apply a 0.3 dropout to velocity embeddings during training, which regularizes the model against potential motion uncertainty. All experiments are implemented in PyTorch and conducted on NVIDIA Tesla V100 GPUs. We use 4 GPUs for training on the AnimalTrack, DanceTrack, and SportsMOT datasets, and 2 GPUs for BFT and BuckTales.

We adopt the AdamW optimizer with a weight decay of \(5 \times 10^{-4}\) and an initial learning rate of \(10^{-4}\). A linear warmup is applied for the first epoch to stabilize learning rate adjustment. The loss weight coefficients are set to \(\lambda_{\text{det}} = 1.0\), \(\lambda_{\text{id}} = 2.0\), and \(\lambda_{\text{vq}} = 0.1\).

4.3 Comparisons with State-of-the-art Methods↩︎

We evaluate the proposed HieDG in comparison with representative heuristic-based and query-based trackers on the AnimalTrack [6], BFT [32], BuckTales [33], DanceTrack [34], and SportsMOT [35] benchmarks. The quantitative results are illustrated in Tab. 1, Tab. 2, and Tab. 3. Since association mechanisms are inherently coupled with detection backbones, we report results following each method’s official evaluation protocols and configurations.

AnimalTrack and BFT. We evaluate HieDG on standard multi-animal tracking datasets in AnimalTrack and BFT across terrestrial, underwater, and aerial environments, where individuals frequently exhibit uniform appearance, high density, and irregular movements. Heuristic-based methods (e.g., SORT [20], DeepSORT [43], and OC-SORT [21]) rely on handcrafted motion and association rules. Although OC-SORT performs better at handling abrupt nonlinear motion in BFT for bird tracking, such strategies remain limited under complex multi-animal dynamics. Query-based approaches demonstrate greater adaptability, yet HieDG consistently outperforms existing methods and achieves state-of-the-art association performance, yielding clear gains in HOTA, AssA, and IDF1. A detailed comparison reveals a consistent trend: directly incorporating continuous geometric representation (HieDG\(^{*}\)) degrades association accuracy relative to the baseline, whereas discretizing geometric embedding consistently yields performance gains. This contrast suggests that continuous geometry can be unreliable in crowded and appearance-ambiguous scenarios, where it may introduce substantial noise. By contrast, discretization converts raw geometry into stable structural tokens, thereby enhancing association robustness. These results suggest that discretized geometric modeling provides improved robustness in complex multi-animal tracking scenarios.

l0.50

3.2pt

BuckTales. UAV-based remote sensing, widely used in wildlife monitoring, introduces additional challenges for multi-animal tracking. Aerial viewpoints yield small, blurry targets, while frequent camera shake and abrupt viewpoint shifts destabilize both appearance and spatial relationships, making identity association significantly more difficult. As shown in Tab. 2, ByteTrack achieves competitive performance in this UAV setting, suggesting that motion-driven heuristics can remain effective when appearance cues collapse. Nevertheless, HieDG achieves the best overall association performance. The improvements in HOTA, AssA, and IDF1 indicate stronger identity consistency under severe geometric perturbations. These findings are consistent with the observations on AnimalTrack and BFT, further demonstrating the robustness of hierarchical geometric modeling in extreme aerial tracking scenarios.

DanceTrack and SportsMOT. To evaluate generalization in generic low-discriminability multi-person tracking scenarios, we report results on DanceTrack and SportsMOT in Tab. 3. HieDG achieves superior association performance across both benchmarks. The consistent gains in AssA and IDF1 demonstrate improved identity preservation in crowded and appearance-ambiguous scenes. These results indicate that the proposed geometry-guided modeling strategy generalizes beyond animal-specific tracking and remains effective in broader multi-object tracking settings.

4.4 Ablations↩︎

We conduct ablation studies on the BFT [32] dataset, which exhibits dense interactions, strong appearance similarity, and irregular motion patterns. These characteristics make BFT a suitable testbed for validating the effectiveness of each component in our framework under low-discriminability conditions. For clarity, we denote HieDG* as the variant using continuous geometric embeddings without quantization.

Geometric Component. Heuristic-based methods typically leverage geometric cues, such as position and scale, from detectors to guide tracking. To capture the fast yet coherent motion patterns in animal scenarios, we model velocity based on inter-frame position differences. As shown in Tab. ¿tbl:tab:ablation95geom95components?, each geometric cue independently improves association performance, while combining all three yields the highest performance gain. These results demonstrate that complementary geometric cues provide additive benefits for identity association, and that jointly modeling position, scale, and velocity yields the most stable cross-frame matching.

We visualize the t-SNE [50] of trajectory embeddings for all frames in sequence Fa5001 and An1005, as shown in Fig. 3, where each color and shape denotes a unique ID. The first three columns illustrate embeddings derived from individual geometric components (position, scale, and velocity), whereas the last three columns show appearance features, discretized geometry embeddings, and the fused trajectory representation. While appearance features show moderate separability, the quantized geometric embeddings exhibit tighter intra-ID clustering compared to continuous appearance features. The fused embeddings further improve inter-ID separation, suggesting that discretized geometry stabilizes identity representation across frames.

Geometric Vector Quantization. Discretizing geometric cues into stable representations significantly enhances tracking performance. To evaluate our quantization design, we compare it with fixed-bin quantization and explore different depths of quantization. As shown in Tab. [tbl:tab:ablation95encoding], fixed-bin quantization yields only limited gains, while hierarchical residual quantization consistently improves association performance. Although extending from two to three stages slightly increases HOTA and IDF1, the improvement is marginal relative to the additional 49.1K parameters and 98.4K FLOPs. This disproportionate increase in computational cost does not justify the minor accuracy gain. Therefore, we adopt the two-stage design as a favorable trade-off between performance and efficiency.

To assess the impact of codebook size, we set equal dimensions for the position, scale, and velocity codebooks to ensure a fair comparison. As shown in Tab. ¿tbl:tab:ablation95codebook95dim?, increasing the codebook size to 64 substantially improves association performance with moderate computational overhead. However, further scaling to 128 offers marginal gains while introducing significant additional cost. These results suggest that our two-stage residual quantization sufficiently captures fluctuating geometry in animal scenarios, rendering larger codebooks less beneficial.

Hungarian Algorithm and Velocity Representation. Inspired by [19], although the Hungarian algorithm is widely used for global ID assignment, our association strategy (detailed in Sec. 3.3) is formulated independently of this matching mechanism. To enhance robustness to low-speed jitter and rapid directional changes common in animal scenarios, we model velocity in polar coordinates. The ablation results in Tab. [tbl:tab:ablation95hungarian95polar] show that incorporating Hungarian matching brings negligible improvement, indicating that the ID classification-based formulation already provides effective global assignment within the learned framework. In contrast, modeling velocity in polar coordinates consistently improves AssA and IDF1, suggesting that decoupling magnitude and direction provides a more stable motion representation under irregular animal dynamics.

5 Conclusion↩︎

We propose HieDG, a hierarchical discrete geometry-guided framework for multi-animal tracking. By reformulating continuous geometric cues into structured discrete embeddings through residual quantization, HieDG stabilizes cross-frame identity association within attention-based tracking architectures. Extensive experiments demonstrate consistent improvements across diverse multi-animal benchmarks, with competitive generalization to generic tracking datasets. Overall, these findings suggest that discretized geometric representations can improve the stability of attention-based identity modeling under low-discriminability and noisy motion conditions.

6 Acknowledgement↩︎

This work was supported by the National Key Research and Development Program of China under Grant 2023YFC3208303; the Brain Science and Brain-like Intelligence Technology-National Science and Technology Major Project under Grants 2022ZD0211900 and 2022ZD0211902; the National Natural Science Foundation of China (NSFC) under Grant 82471245 from Hang Zhou, Grant 12301651 from Weifu Li, and Grant 62273347 from Yifan Zhang; the Shenzhen Natural Science Foundation under Grant JCYJ20241202130204005; the Guangdong Basic and Applied Basic Research Foundation under Grant 2025A1515010844; and the Characteristic Innovation Project of Guangdong Higher Education Institutions (Natural Science) under Grant 2024KTSCX027.

7 Overview↩︎

In this supplementary material, we primarily:

1. State additional experimental details, in Sec. 8.

2. Provide the pseudo-algorithm of the hierarchical geometry quantization process, in Sec. 9.

3. Present additional visualization results across different benchmarks, in Appendix 10.

4. Discuss the strengths, limitations, and potential future directions of this work in Sec. 11.

8 Experimental Details↩︎

8.1 Dataset Details↩︎

Due to space constraints in the main paper, only a brief summary of the datasets was provided. Here we supplement more detailed information about the three benchmarks used in our experiments. AnimalTrack [6] contains 402.0K annotated instances across 10 animal categories, including aquatic and terrestrial species such as dolphins, deer, and geese. BFT [32] focuses on aerial bird tracking and includes 85.8K annotations covering 22 bird species across the training, validation, and test splits. In contrast, BuckTales [33] is a UAV-based dataset centered on buck tracking. Although it contains fewer image frames, it includes 853.7K annotated instances, indicating a much higher object density and introducing greater tracking challenges due to increased occlusions and complex interactions. The overall statistics are summarized in Tab. 4.

3pt

8.2 Training↩︎

Building on these dataset characteristics, we next describe key aspects of our training strategy. Since the model is initialized with COCO-pretrained DETR weights, the visual features are more reliable than the geometric and ID embeddings at the start of training. Without additional regularization, the model tends to underuse or even discard the parameters linked to geometry and ID cues. To address this, we adopt a U-shaped curriculum. During the first 5% of training, we use a standard setup so the model can establish basic tracking ability with strong visual guidance. Between 5% and 35%, we apply a 0.35 dropout rate to the visual embeddings in the HieDG framework, which shifts the learning focus toward geometric and ID information. To maintain consistency between training and inference, we reduce the visual embedding dropout to 0.1 to recover appearance details. This schedule follows a simple-to-complex trajectory: learn the basic pattern first, strengthen geometric reasoning next, and finally reintegrate appearance cues, resulting in more stable optimization and stronger generalization.

9 Pseudo Algorithm↩︎

We present the pseudo-algorithm of our hierarchical geometry quantization in 4. In this process, we first normalize the positions and scales of the detected \(bbox\) to handle the varying resolutions and aspect ratios across videos. Given the complex motion patterns common in animal scenes, we further convert the frame-to-frame Cartesian displacements into a polar-coordinate velocity representation. We then employ three separate MLPs to map the normalized position, size, and velocity cues into their respective embeddings.

Moreover, we introduce a two-stage codebook for geometric quantization. We first use a nearest-neighbor search to match each continuous geometric embedding to the closest codeword in the first-stage codebook, producing a coarse discrete embedding. We then compute the residual between the coarse embedding and the original continuous representation, and quantize it using the second-stage codebook to obtain a fine discrete embedding. The final discrete representation is formed by adding the coarse and fine components.

We further summarize the velocity estimation procedure during inference in 5. For each detection in the current frame, we first retrieve the \(k\) nearest historical trajectory states in the spatial coordinate space. The velocity of the detection is then estimated by aggregating the historical velocities of these neighbors through a normalized weighted average. The weights combine a spatial proximity kernel and a directional consistency kernel, encouraging contributions from trajectories that are both spatially close and motion-aligned. The resulting velocity estimate is subsequently converted to the polar-coordinate representation and used to construct the geometric vector for quantization. This lightweight procedure provides a stable motion estimate for detections without explicit temporal correspondences while maintaining compatibility with the geometry quantization module.

10 Visualization Results↩︎

10.1 t-SNE Visualization↩︎

Due to space limitations in the main text, we could not provide comprehensive t-SNE and tracking visualizations in the main paper. Here we present these visual results along with more detailed analyses. Fig. 6 shows the t-SNE embeddings for sequences An3008, An1005, An1002, and An6012 from the BFT dataset. For the geometric features, we observe that the position and velocity embeddings across video sequences generally follow spatial and motion consistency, remaining stable or changing gradually. This behavior provides reliable support for the subsequent association stage. In contrast, the size embeddings exhibit less clear separation, both in inter-class distance and intra-class compactness, suggesting potential directions for future improvement.

At a broader level, visual embeddings in animal-tracking scenarios often show weak structure, with inconsistent intra-ID and inter-ID distances. By comparison, incorporating our discrete geometric embeddings produces fused trajectory embeddings that display much clearer separation than appearance embeddings alone. This indicates that the proposed discrete geometric representation effectively complements visual cues in MAT tasks and contributes to more reliable association.

10.2 Tracking Visualization↩︎

In this section, we also present six consecutive-frame tracking visualizations from the AnimalTrack dataset. Representative scenes, including chicken, deer, and dolphins, are shown in Fig. 7. We observe that tracking performance remains stable and reliable under natural conditions. However, in scenes such as chicken and pig, missed detections occur when animals appear near the image boundary or undergo partial occlusion, which prevents correct association in subsequent frames. In addition, under severe occlusion or significant appearance changes, such as ducks dipping their heads underwater or deer undergoing pronounced posture variation, HieDG may treat the target as a newly appearing individual and assign a new identity.

11 Discussions↩︎

Our HieDG framework provides an end-to-end paradigm that handles both object detection in animal scenes and identity association for tracking. Because appearance embeddings often show limited discrimination in animal scenarios, we integrate geometric embeddings into standard query-based tracking pipelines to strengthen the association stage. To cope with the substantial fluctuations in position, size, and velocity that are common in animal motion, we adopt a hierarchical residual quantization scheme that discretizes these geometric cues. The resulting geometric representation complements the visual embeddings, allowing one modality to support the other when its signal is weak or ambiguous.

At this stage, our geometric embedding design combines position, size, and velocity through linear mappings followed by concatenation. Future work may explore more expressive fusion strategies, including element-wise gated fusion, FiLM-style modulation, attention-based fusion such as cross-attention or self-attention, and graph-based message passing. These approaches could more effectively inject geometric information into end-to-end query-based tracking. In addition, the t-SNE visualizations in Appendix 10.1 show that position and velocity form the main discriminative structure in our discrete geometric embeddings, while scale features remain less distinct. This suggests that more advanced size modeling, such as area–shape decomposition, log-space encoding, or ratio-based representations, could further improve performance.

Although our current study focuses on multi-animal tracking, we plan to extend the framework to broader tracking tasks. The proposed HieDG is not limited to animal scenes: its robust geometric modeling is also suitable for human and vehicle MOT. Since transformer-based tracking-by-query methods benefit from large-scale training data, we expect HieDG to perform even better as more comprehensive datasets become available.

References↩︎