1 Introduction↩︎

Feature tracking lies at the foundation of visual perception. It maintains the temporal consistency of salient and geometrically stable keypoints that support 3D reconstruction, SLAM, and motion estimation. Unlike dense optical flow [1]–[3] and track-any-point tasks [4]–[9] that operate on dense motion fields, or high-level object tracking frameworks [10]–[28], feature tracking focuses on sparse yet reliable trajectories that provide an interpretable geometric structure for long-term scene understanding. However, conventional frame-based pipelines [4], [7], [9], [29]–[36] inevitably degrade under challenging conditions such as motion blur and limited dynamic range, while their fixed frame rate introduces latency. These issues make them unreliable for many real-world tracking tasks, such as real-time robotics and high-speed perception.

A compelling alternative lies in event-based feature tracking, which leverages a fundamentally different sensing paradigm. Event cameras [37]–[43] asynchronously record per-pixel brightness changes with microsecond resolution and an extremely high dynamic range. This sparse data modality enables robust perception even in challenging scenarios characterized by fast motion and difficult lighting conditions [28], [44]–[47]. However, the asynchrony that provides these advantages breaks the dense and synchronous assumptions inherent in traditional vision systems [48], [49]. This necessitates the development of new architectures capable of efficiently and continuously modeling long-range spatiotemporal dependencies.

Existing event-based feature tracking approaches can be divided into two main categories. Event-assisted frameworks combine the temporal precision of events with frame-based appearance cues to enhance robustness under fast motion and challenging illumination, but remain frame-dominant, anchoring temporal reasoning to discrete frame intervals and resampling asynchronous data, which reintroduces latency and limits temporal fidelity. In contrast, event-dominant frameworks operate directly on asynchronous streams, preserving high temporal resolution for continuous, low-latency tracking. However, existing approaches [5], [8], [50]–[56] still face architectural bottlenecks: typical RNNs provide efficiency but limited memory, while Transformers capture global context at quadratic cost, restricting scalability for high-frequency data. As a result, event-based 3D feature tracking remains constrained by a trade-off between modeling capacity and computational efficiency.

To overcome these challenges, we design E-TraMamba, the first Mamba-based paradigm for event-based 3D feature tracking. E-TraMamba redefines feature tracking under a linear state-space formulation, enabling efficient long-range spatiotemporal modeling with linear complexity. It incorporates a lightweight affine-transform predictor for motion stability and fuses multi-scale cues—including spatiotemporal patches, correlation maps, and positional embeddings—into a unified representation for continuous 3D tracking. To address the scarcity of large-scale event data, we further construct EvD-PointOdyssey, a high-fidelity semi-synthetic dataset derived from PointOdyssey [57] with synchronized event streams, depth maps, and accurate 3D trajectories. Our contributions are threefold:

• New paradigm for event-driven tracking. We design the first Mamba-based framework for 3D feature tracking, establishing a new direction for efficient event-driven modeling.

• Efficient long-term modeling for feature tracking. Built upon a structured state-space formulation, E-TraMamba explicitly models long-range spatiotemporal dependencies, achieving scalable and stable 3D tracking at high temporal frequencies.

• The first high-fidelity synthetic dataset for event-based 3D feature tracking. We construct EvD-PointOdyssey, a novel semi-synthetic benchmark featuring physically grounded event synthesis, realistic temporal dynamics, and unified evaluation for rigorous benchmarking.

2 Related Work↩︎

2.1 Event-Assisted Feature Tracking↩︎

To address the shortcomings of frame-based tracking under fast motion or adverse lighting, recent studies have explored frame–event fusion for keypoint detection and point tracking. Event cameras provide microsecond-level temporal cues that complement degraded or low frame-rate images, improving robustness and motion estimation [6], [58]–[60]. Despite their success, these fusion schemes remain constrained by the frame domain: the integration process still depends on discrete frame timestamps, which limits the temporal granularity and prevents full exploitation of the event stream’s asynchronous dynamics. These limitations highlight the need for event-dominant frameworks that operate natively in the event domain, enabling high-speed and long-term tracking.

2.2 Event-Dominant Feature Tracking↩︎

Contrary to fusion-based designs, event-dominant frameworks directly process asynchronous event streams, preserving their intrinsic temporal resolution and enabling continuous tracking under extreme motion or illumination. Early studies focused on handcrafted methods such as ICP alignment, patch warping, and event clustering [52], [53], [55]. More recent learning-based approaches—including DeepEvT [61], Ev-TAP [5], and ETAP [8]—demonstrate notable progress but still rely on synthetic video-to-event data and architectures such as RNNs or Transformers, which struggle to balance long-range modeling with efficiency. In contrast, our work reframes the event-dominant paradigm through a Mamba-based state-space formulation that achieves linear-complexity long-term reasoning, and introduces EvD-PointOdyssey, a high-fidelity semi-synthetic benchmark for unified 2D/3D evaluation.

2.3 Mamba in Tracking Tasks↩︎

Structured state-space models (SSMs) based on Mamba have recently been applied to tracking, but mainly at the object level. In multi-object tracking, Bi-Mamba [62] and related variants [10], [63] model motion trajectories for robust MOT, while single-object trackers [64]–[66] emphasize long-context reasoning and modality fusion [67]–[70]. These models excel at object-level trajectory modeling with dense visual supervision, but have not addressed the fine-grained, geometry-aware domain of feature or point tracking, where sparse observations and asynchronous inputs pose distinct challenges. To our knowledge, this work is the first to extend the Mamba paradigm to event-driven 3D feature tracking.

3 Methods↩︎

3.1 Preliminary↩︎

3.1.0.1 Mamba.

Mamba models sequential data using a state-space formulation, where the hidden state \(\mathbf{z}[t]\) evolves over time under the influence of the input \(\mathbf{x}[t]\). At each time step \(t\), the continuous-time dynamics are defined as: \[\mathbf{z}'[t] = \mathbf{A}\mathbf{z}[t] + \mathbf{B}\mathbf{x}[t], \qquad \mathbf{y}[t] = \mathbf{C}\mathbf{z}[t]. \label{eq:continuous95ssm}\tag{1}\] Here, \(\mathbf{A}\) denotes the system dynamics, while \(\mathbf{B}\) and \(\mathbf{C}\) are the input and output projection matrices. Using the zero-order hold (ZOH) method, the system is discretized as: \[\widetilde{\mathbf{A}} = \exp(\Delta \mathbf{A}), \qquad \widetilde{\mathbf{B}} = \mathbf{A}^{-1}\!\big(\exp(\Delta \mathbf{A}) - \mathbf{I}\big)\mathbf{B}, \label{eq:zoh}\tag{2}\] leading to the discrete update rule: \[\mathbf{z}[t] = \widetilde{\mathbf{A}}\mathbf{z}[t-1] + \widetilde{\mathbf{B}}\mathbf{x}[t], \qquad \mathbf{y}[t] = \mathbf{C}\mathbf{z}[t]. \label{eq:discrete95update}\tag{3}\] Unlike classical state-space models, Mamba makes the parameters \(\mathbf{B}\), \(\mathbf{C}\), and \(\Delta\) input-dependent: \[(\mathbf{B},\, \mathbf{C},\, \Delta) = f_\theta(\mathbf{x}[t]),\] which allows dynamic transition behavior conditioned on the current token. A global convolution operator across time further refines \(\mathbf{y}[t]\), enabling long-range temporal interaction with linear time and memory complexity. For detailed derivations, please refer to [71].

3.2 Model Architecture↩︎

3.2.0.1 Track Token Representation.

Following prior works [4], [7], [8], [61], our framework represents each trajectory as a sequence of track tokens that encode spatio-temporal and appearance information. For a trajectory \(i\), we define \[\mathbf{G}^i = [\mathbf{G}^i_1, \mathbf{G}^i_2, \dots, \mathbf{G}^i_T],\] where each token \(\mathbf{G}^i_t\) aggregates multiple complementary features at the \(t\)-th time step.

Our model supports both stereo and monocular+depth configurations. For stereo 3D tracking, we extract features from both left and right event views. For each tracked point, we compute: (1) local appearance features \(\mathbf{F}^i_t\) from event patches via a CNN encoder, (2) correlation features \(\mathbf{C}^i_t\) that measure temporal consistency between consecutive frames, (3) reference template features \(\mathbf{R}^i_t\) from an affine-aligned grayscale patch, and (4) position encodings \(\gamma_{\text{pos}}(\mathbf{x}^i_t)\) that encode spatial coordinates. The left and right view features are then fused via an MLP: \[\begin{align} \mathbf{G}^{i,\text{L}}_t &= \mathrm{CNN}\big[\Omega^{\text{L}}_i(t)\big] + \mathbf{C}^i_t + \mathbf{R}^i_t + \gamma_{\text{pos}}(\mathbf{x}^i_t), \\ \mathbf{G}^{i,\text{R}}_t &= \mathrm{CNN}\big[\Omega^{\text{R}}_i(t)\big] + \mathbf{C}^i_t + \mathbf{R}^i_t + \gamma_{\text{pos}}(\mathbf{x}^i_t - \mathbf{d}_t), \\ \mathbf{G}^i_t &= \mathrm{MLP}_{\text{fusion}}\big(\mathbf{G}^{i,\text{L}}_t,\, \mathbf{G}^{i,\text{R}}_t\big), \end{align}\] where \(\Omega^{\text{L/R}}_i(t)\) denotes the event patch and \(\mathbf{d}_t\) is the disparity.

For monocular+depth tracking, depth information \(\mathbf{D}^i_t\) is directly concatenated with the event data as an additional input channel before CNN encoding: \[\mathbf{G}^i_t = \mathrm{CNN}\big[\Omega_i(t) \oplus \mathbf{D}^i_t\big] + \mathbf{C}^i_t + \mathbf{R}^i_t + \gamma_{\text{pos}}(\mathbf{x}^i_t),\] where \(\oplus\) denotes channel concatenation.

This unified token representation compactly encodes geometric (position, depth/disparity, correlation) and appearance (local features, reference template) cues. In the following sections, we primarily describe the stereo configuration for clarity, noting that the monocular+depth variant follows an analogous pipeline with depth maps replacing the right-view event stream.

3.2.0.2 Feature Extraction and Sliding Windows.

Let \[\mathcal{E}_{j}=\{(x_{k}, y_{k}, t_{k}, p_{k})\}_{k=1}^{N}\] denote the event stream collected between timestamps \([t_{j-1}, t_{j}]\), where \((x_{k}, y_{k})\) are pixel coordinates, \(t_{k}\) the timestamp, and \(p_{k}\in\{-1,+1\}\) the polarity. We process events using a sliding window strategy with window size \(w\) and stride \(w/2\): \[\mathcal{S}_{t} = \{(x, y, t', p) \in \mathcal{E} \mid t' \in [t-\tfrac{w}{2},\, t]\}.\] The half-window overlap ensures temporal continuity across adjacent windows.

For each tracked point \(\mathbf{x}_{i}(t)\), we extract a square event patch \[\Omega_{i}(t)=\{(u,v)\mid \lVert (u,v)-\mathbf{x}_{i}(t)\rVert_{\infty}\leq r\}.\] Point positions in the second half-window are initialized via cubic spline interpolation from the first half-window: \[\hat{\mathbf{x}}_{i}(t)=\mathrm{Spline}\big(\mathbf{x}_{i}(t-\tfrac{w}{2}),\dots\big).\]

Each event patch is fed into a CNN encoder \(\phi(\cdot)\) followed by global average pooling to produce local features \(\mathbf{F}_{i}(t)\). Correlation features \(\mathbf{C}_{i}(t)\) capture temporal consistency by measuring patch similarity and 2D displacement between consecutive frames. Reference template features \(\mathbf{R}_{i}\) are extracted from a grayscale reference patch with learned affine alignment to handle scale and rotation changes. All feature components are concatenated and fused through an MLP to produce the final token representation \(\mathbf{G}^i_t \in \mathbb{R}^{d}\). Detailed architecture specifications are provided in the supplementary material.

3.2.0.3 Mamba-based Refinement.

Given the extracted track tokens \(\mathbf{T}\!\in\!\mathbb{R}^{B \times T \times N \times d}\) (batch size \(B\), time steps \(T\), tracked points \(N\), feature dimension \(d\)), we apply iterative refinement through Mamba blocks to model long-range spatio-temporal dependencies. Our architecture uses multiple refinement groups, each containing four directional Mamba modules that scan the token sequence along different orders.

The token tensor has two logical axes: the temporal axis (\(\tau\)) tracking each point across time, and the spatial axis (\(\sigma\)) relating different points at the same time step. We construct four scanning directions: \(\mathcal{S}=\{\tau^{+},\tau^{-},\sigma^{+},\sigma^{-}\}\), where \(\tau^+\) and \(\tau^-\) scan temporally (forward/backward), and \(\sigma^+\) and \(\sigma^-\) scan spatially (forward/backward).

For each direction \(s\!\in\!\mathcal{S}\), we reshape the tokens into a 1D sequence via permutation \(\pi_{s}\), process through a Mamba module, and inverse-permute back: \[\mathbf{H}_{s} =\pi_{s}^{-1}\!\bigl(\mathrm{Mamba}_{s}\!\bigl(\pi_{s}(\mathbf{T})\bigr)\bigr).\] The outputs from all four directions are averaged and combined with the input via residual connection. After iterative refinement, lightweight prediction heads decode the final tokens into 2D positions (via softmax-weighted binning), disparity/depth values, and visibility confidence scores.

3.3 Supervision and Loss Function↩︎

A unified multi-scale L1 supervision strategy is formulated to jointly constrain the 2D reprojection, disparity, and reconstructed 3D coordinates of each trajectory during training. Given predicted 2D points \(\hat{\mathbf{x}}_{t}\), disparity \(\hat{d}_t\), and reconstructed 3D coordinates \(\hat{\mathbf{X}}_t\), with ground truths \(\mathbf{x}_t\), \(d_t\), and \(\mathbf{X}_t\), the overall loss for a single refinement step is \[\mathcal{L}_{\text{total}} = \lambda_{\text{xy}} \| \hat{\mathbf{x}}_t - \mathbf{x}_t \|_1 + \lambda_{\text{disp}} \| \hat{d}_t - d_t \|_1 + \lambda_{\text{3D}} \| \hat{\mathbf{X}}_t - \mathbf{X}_t \|_1,\] where \(\lambda_{\text{xy}}, \lambda_{\text{disp}}, \lambda_{\text{3D}}\) are balancing coefficients (set to 1.0). All losses are masked by visibility and averaged across all time steps and points. The total loss aggregates all refinement stages: \[\mathcal{L} = \sum_{k=1}^{K} \gamma^{K-k} \mathcal{L}_{\text{total}}^{(k)},\] where \(\gamma\) is the decay factor (0.8). No adversarial or contrastive objectives are used—our supervision remains purely regression-based.

3.4 Semi-Synthetic Dataset↩︎

We construct a high-fidelity semi-synthetic dataset, EvD-PointOdyssey, by extending the PointOdyssey framework [57], which originally provides dense 3D point tracking sequences with precise geometry and long temporal coverage. While prior variants such as Ev-PointOdyssey [5] simulate event data for dense motion supervision, they remain limited to dense tracking formulations. In contrast, our dataset redefines the paradigm toward sparse, event-based 3D feature tracking, combining high-frame-rate RGB rendering with physics-based event simulation and per-frame depth supervision. We further apply strict temporal filtering and trajectory validation to ensure clean, geometrically consistent annotations, enabling large-scale training and benchmarking of event-based tracking models under realistic dynamics. Given all candidate 3D points \({p_i^t}\), we retain only trajectories satisfying:

• Displacement constraint: maximum inter-frame displacement \(\leq 12\) pixels;

• Visibility constraint: each point remains visible throughout the sequence;

• Depth-quality constraint: valid, finite depth values (no NaN or infinite entries);

• Non-overlap constraint: selected temporal windows must not overlap;

• Point-count constraint: up to 30 valid points per sequence.

This filtering eliminates noisy trajectories and yields stable, geometrically consistent tracks. The resulting dataset provides reliable event–depth–trajectory triplets, forming a solid foundation for benchmarking event-based 3D tracking models.

4 Experiments↩︎

4.1 Dataset and Implementation Details↩︎

4.1.0.1 Dataset.

We evaluate our method on two datasets designed for event-based feature tracking, covering both 3D and 2D tasks. These include one large-scale real-world benchmark and one semi-synthetic dataset introduced in Section 3.4.

1) E-3DTrack [51] is a stereo event-based dataset for 3D feature tracking, providing synchronized left–right event streams with ground-truth 3D feature trajectories across diverse motion patterns. It serves as a benchmark for evaluating long-range tracking stability and temporal consistency under real-world dynamic conditions.

2) EvD-PointOdyssey (ours) is a semi-synthetic dataset derived from the PointOdyssey framework [57], as detailed in Section 3.4. Unlike E-3DTrack, it adopts a monocular+depth configuration, pairing event streams with rendered depth maps to enable single-view 3D reasoning. For fair comparison with existing 2D trackers, evaluation is performed in the 2D projection domain. This design bridges 3D structure-aware tracking and conventional 2D event tracking. Representative examples from EvD-PointOdyssey are shown in Fig. 3.

4.1.0.2 Implementation Details.

Our E-TraMamba model is built upon a Mamba-based architecture tailored for event-based feature tracking. We use a hidden dimension of 256 and a patch size of 31. The architecture consists of two Mamba refinement groups, each containing four directional Mamba blocks that process the sequence along both forward and backward directions across temporal and spatial dimensions. Each Mamba block implements a structured state-space model with a state dimension of 32, a convolutional kernel size of 4, and an expansion factor of 2. Our final model uses a window size of 12 to balance temporal modeling capacity and efficiency.

We apply positional encoding and dropout with a rate of 0.1. A two-stage refinement strategy is adopted, with a refinement decay factor of 0.8 to stabilize the optimization.

Training is conducted on 8 NVIDIA A6000 GPUs with a batch size of 16 for 120 epochs. We use the AdamW optimizer with an initial learning rate of \(1 \times 10^{-4}\), a weight decay of \(5 \times 10^{-4}\), and cosine annealing down to a minimum of \(1 \times 10^{-6}\). Each training run takes approximately 2 days to complete.

4.1.0.3 Evaluation Metrics.

To comprehensively evaluate our proposed E-TraMamba across both 3D and 2D event-based feature tracking tasks, we adopt metrics widely used in prior literature and tailored to the characteristics of each dataset.

For E-3DTrack [51], which targets long-range and temporally consistent 3D tracking, we report the Tracked Feature Ratio (TFR), Feature Age (FA), and Root Mean Squared Error (RMSE) under spatial thresholds of 0.1 m, 0.15 m, and 0.2 m. TFR measures the proportion of trajectory duration during which the predicted 3D points remain within a predefined spatial error bound relative to ground truth, FA reflects the average lifetime of tracked features before loss, and RMSE quantifies the overall 3D deviation, providing a holistic measure of accuracy and stability.

width=

For EvD-PointOdyssey, although our model is capable of predicting full 3D trajectories, most existing event-based trackers produce only 2D projections. To ensure a fair comparison, we therefore evaluate all methods in the 2D image plane following standard event-based feature tracking protocols. We report FA and TFR at pixel thresholds of 2 px, 3 px, and 5 px, together with the RMSE-2D that measures the average reprojection error across frames. TFR and FA respectively capture the success ratio and temporal stability of tracked features within each pixel threshold, while RMSE-2D reflects the average spatial deviation of the projected points. The complete quantitative comparison is presented in Table ¿tbl:tab:combined95results95correct?. Together, these metrics jointly assess the precision, robustness, and cross-domain generalization of our approach in both 3D and 2D tracking settings.

4.2 Quantitative Comparison↩︎

4.2.0.1 E-3DTrack results.

Table ¿tbl:tab:combined95results95correct? presents the quantitative comparison of our method against existing event-based feature tracking and stereo depth estimation baselines on the challenging E-3DTrack dataset [51]. E-TraMamba (ours) consistently achieves the best performance across all metrics. Compared to the second-best approach, E-3DTrack [51], our model reduces the RMSE by 22.6% (from 0.1181 m to 0.0914 m) and improves feature age (FA) by +35.4, +36.7, and +32.3 percentage points at the 0.1 m, 0.15 m, and 0.2 m thresholds, respectively.

These improvements stem from the synergy of two key modules: the spline-based motion estimator, which provides smooth local priors, and the Mamba-based state-space module, which refines long-range temporal dependencies with linear complexity \(O(NT)\). This combination yields stronger temporal coherence and spatial stability under high-speed motion and depth variation, without incurring the quadratic cost of attention-based designs.

4.2.0.2 EvD-PointOdyssey results.

Table ¿tbl:tab:combined95results95correct? further presents a quantitative comparison between our model and two representative event-based tracking frameworks on the EvD-PointOdyssey dataset. All methods are evaluated in the 2D image domain for fair comparison. E-TraMamba (ours) achieves the best performance on nearly all metrics: it performs on par with ETAP [8] at the strictest FA@2px threshold (0.2177 vs.) and outperforms both ETAP and DeepEvT [61] on all other metrics. In particular, our model improves feature age (FA) at the 3 px and 5 px thresholds by +10.4 and +33.1 percentage points over ETAP, and reduces the 2D RMSE from 23.26 px to only 4.16 px—a nearly 5.6\(\times\) improvement. These results highlight that jointly modeling spatio-temporal motion and depth-conditioned event cues enables more stable and geometrically consistent long-term tracking under complex, high-speed event dynamics.

Despite the additional preprocessing and channel fusion required for event–depth inputs, our E-TraMamba architecture achieves inference latency nearly identical to lightweight 2D Transformer-based trackers. It operates at 2.11 ms/frame (\(\approx\)​475 FPS), matching DeepEvT (2.10 ms/frame) and surpassing ETAP (2.86 ms/frame). This efficiency stems from the linear-time state-space modeling of the Mamba backbone, which replaces the quadratic attention mechanism in Transformers with recurrent state updates. As a result, E-TraMamba achieves real-time performance while incorporating richer 3D cues and spatio-temporal consistency—a combination that was previously challenging for Transformer-based designs.

4.3 Qualitative Comparison↩︎

4.3.0.1 E-3DTrack results.

Figure 4 shows the qualitative 3D feature-tracking results of our proposed method compared with the prior event-based 3D tracking framework E-3DTrack [51]. The visualizations correspond to three representative sequences—airplane, toy1, and toy5—from our benchmark set, covering a wide range of motion patterns including large-scale translation, rotation, and depth variation. Ground-truth trajectories are shown in red, those predicted by the comparison baseline in blue, and those from our method in green. From the figure, we observe that our method produces smoother and more stable 3D tracks, maintaining accurate temporal coherence even under fast motion and severe viewpoint changes. In contrast, E-3DTrack suffers from noticeable jitter and drift, especially when the objects undergo complex depth changes or partial occlusion. These results highlight the effectiveness of our motion-aware spatiotemporal modeling and the robustness of our long-range state representation in reconstructing continuous 3D trajectories.

4.3.0.2 EvD-PointOdyssey results.

Figure 5 presents the qualitative 2D feature-tracking comparison on the EvD-PointOdyssey dataset. Each row shows a representative sequence exhibiting challenging motion patterns, including fast rotation, large depth variation, and complex textured regions. Ground-truth trajectories are shown in red, while our method E–TraMamba (green) produces smoother and more stable long-term tracks than ETAP [8] (cyan) and DeepEvT [61] (magenta), particularly under severe motion blur and highly textured surfaces. Star markers (\(\boldsymbol{\star}\)) denote the initial feature locations. These results highlight the effectiveness of our depth-conditioned event representation and the robustness of the Mamba-based long-range modeling in maintaining precise temporal correspondence across complex dynamic scenes.

4.4 Ablation Studies↩︎

To further analyze the effectiveness of individual components, we conduct comprehensive ablation studies summarized in Table 1 and Table 2. We investigate three key factors: (1) sequence modeling strategy (Transformer vs.Mamba), (2) temporal window size, and (3) encoding configurations.

Impact of sequence modeling. As shown in Table 1, when the window size is fixed at 8, the Transformer slightly outperforms Mamba (TFR\(_{0.1}\): 0.5772 vs.; RMSE: 0.0973 vs.), highlighting the strength of attention-based modeling. However, Transformers suffer from quadratic complexity with respect to the window size (\(\mathcal{O}(w^2)\)), while Mamba maintains linear complexity (\(\mathcal{O}(w)\)). As the window increases to 12 and 16, Mamba surpasses Transformer both in accuracy and efficiency, making it more suitable for long-range and real-time event tracking tasks.

4pt

6pt

Effect of temporal window size. Window size plays a critical role in balancing context and stability. Although a longer window provides richer temporal information, our sliding strategy uses a stride of half the window size. Thus, overly large windows (e.g., 16) span excessive motion intervals, increasing temporal ambiguity and reducing point-wise precision. A window size of 12 achieves the best trade-off between temporal context, computational cost, and prediction stability.

Encoding configurations. As shown in Table 2, removing the temporal encoding slightly improves performance (TFR\(_{0.1}\): 0.5950 \(\rightarrow\) 0.6192), suggesting that the sequential scanning order of the Mamba modules already conveys sufficient temporal structure, while additionally removing the positional encoding leads to a pronounced drop (TFR\(_{0.1}\): 0.6192 \(\rightarrow\) 0.5952; RMSE: 0.0911 \(\rightarrow\) 0.0960). This indicates that spatial cues are more critical than explicit temporal ordering in our event-based tracking setup.

Number of Mamba groups. We further analyze the effect of stacking multiple Mamba refinement groups—each consisting of four directional Mamba blocks modeling forward/backward dependencies across spatial and temporal axes. Increasing the number of groups from 2 to 3 brings no further improvement (TFR\(_{0.1}\): 0.5950 \(\rightarrow\) 0.5920; RMSE: 0.0921 \(\rightarrow\) 0.0935) while adding computational cost. Therefore, we adopt two Mamba groups in the final model as a balanced configuration between accuracy and efficiency.

5 Discussion↩︎

Efficiency, scalability, and performance of Mamba. Mamba achieves linear-time temporal modeling by reformulating feature propagation as a recurrent state-space process, avoiding the quadratic complexity of attention-based architectures. This efficiency is crucial for high-frequency event streams, where the number of effective virtual timesteps reflects asynchronous sampling rather than real-world time. Under equivalent latency or FLOPs, Mamba processes longer temporal horizons without additional delay, enabling broader receptive fields and stronger temporal coherence. Consequently, E-TraMamba delivers higher accuracy and stability than Transformer-based trackers, particularly under motion blur and complex textures, while maintaining real-time performance. Experiments on the semi-synthetic EvD-PointOdyssey dataset further demonstrate strong cross-domain robustness to noise, domain shifts, and varying motion dynamics.

Limitations and future directions. Despite these advantages, several challenges remain in transitioning from dense track-any-points setups to fine-grained, feature-level 3D tracking. In particular, trajectory filtering, temporal consistency, and supervision reliability remain difficult under real-world asynchronous conditions. Future work should explore decomposing large-scale tracking into interpretable sub-tasks that explicitly evaluate long-term modeling, temporal alignment, and efficiency under strict resource constraints. Incorporating physical priors or event-generation dynamics into Mamba-based architectures also presents a promising direction for further improvement.

6 Conclusion↩︎

We have presented E-TraMamba, a Mamba-based framework that establishes a new paradigm for efficient event-based 3D feature tracking. By reframing temporal modeling under a linear state-space formulation, E-TraMamba enables scalable, long-range reasoning with linear complexity and achieves stable performance where traditional attention or recurrent structures struggle.

Our work demonstrates that E-TraMamba achieves superior tracking accuracy and temporal consistency under high-speed motion and occlusion, at a lower computational cost than Transformer-based methods. These findings establish structured state-space modeling as a powerful and practical foundation for next-generation event-driven vision systems.

Our contribution extends beyond the model itself. We also present EvD-PointOdyssey, a high-fidelity semi-synthetic benchmark designed for unified 2D and 3D event tracking. Together, the model and dataset advance the frontier of event-based perception, emphasizing efficiency, scalability, and physical realism. Moving forward, we envision this combination as a foundation for real-time applications in autonomous navigation, agile robotics, and neuromorphic sensing—where low latency and long-term spatiotemporal understanding are essential.

7 Architecture Details↩︎

This section provides complete implementation details for the E-TraMamba framework, including precise network architectures, channel dimensions, and hyperparameters. Our model supports both stereo and monocular+depth configurations, which we describe separately below.

7.1 Input Representations↩︎

7.1.0.1 Stereo Configuration.

The input consists of synchronized left and right event streams, each represented as a 10-channel tensor \(E^{\text{L}}_t, E^{\text{R}}_t \in \mathbb{R}^{10\times H\times W}\), where the 10 channels encode event polarity (positive/negative) across 5 temporal bins. A reference grayscale patch \(P^{\text{ref}}_i \in \mathbb{R}^{1\times 31\times 31}\) is extracted from the first frame for each tracked point.

7.1.0.2 Monocular+Depth Configuration.

The input consists of a single event stream \(E_t \in \mathbb{R}^{10\times H\times W}\) paired with a dense depth map \(D_t \in \mathbb{R}^{1\times H\times W}\). Depth values are normalized to \([0,1]\) via: \[D_t^{\text{norm}} = \frac{\text{clip}(D_t, d_{\min}, d_{\max}) - d_{\min}}{d_{\max} - d_{\min}},\] where \(d_{\min}=0.9\) m and \(d_{\max}=50\) m for PointOdyssey. The normalized depth is concatenated with events to form an 11-channel input: \[I_t = [E_t,\, D_t^{\text{norm}}] \in \mathbb{R}^{11\times H\times W}.\]

7.2 Feature Extraction CNN↩︎

For each tracked point, we extract a \(31{\times}31\) patch centered at its predicted 2D location. The patch is processed by a CNN encoder:

Stereo Mode: \[\begin{align} &\text{Conv2d}(10, 64, k=3, p=1) \to \text{BN}(64) \to \text{ReLU} \\ &\to \text{Conv2d}(64, 128, k=3, p=1) \to \text{BN}(128) \to \text{ReLU} \\ &\to \text{Conv2d}(128, 256, k=3, p=1) \to \text{BN}(256) \to \text{ReLU} \\ &\to \text{AdaptiveAvgPool2d}(1) \to F_{\text{local}} \in \mathbb{R}^{256}. \end{align}\]

Mono+Depth Mode: \[\begin{align} &\text{Conv2d}(11, 64, k=3, p=1) \to \text{BN}(64) \to \text{ReLU} \\ &\to \text{Conv2d}(64, 128, k=3, p=1) \to \text{BN}(128) \to \text{ReLU} \\ &\to \text{Conv2d}(128, 256, k=3, p=1) \to \text{BN}(256) \to \text{ReLU} \\ &\to \text{AdaptiveAvgPool2d}(1) \to F_{\text{local}} \in \mathbb{R}^{256}. \end{align}\]

The reference patch encoder shares the same architecture but processes only the grayscale template \(P^{\text{ref}}_i \in \mathbb{R}^{1\times 31\times 31}\): \[\begin{align} &\text{Conv2d}(1, 64, k=3, p=1) \to \text{BN}(64) \to \text{ReLU} \\ &\to \text{Conv2d}(64, 128, k=3, p=1) \to \text{BN}(128) \to \text{ReLU} \\ &\to \text{Conv2d}(128, 256, k=3, p=1) \to \text{BN}(256) \to \text{ReLU} \\ &\to \text{AdaptiveAvgPool2d}(1) \to F_{\text{ref}} \in \mathbb{R}^{256}. \end{align}\]

7.3 Correlation Features↩︎

To capture temporal motion consistency, we compute correlation descriptors between consecutive frames.

7.3.0.1 Stereo Mode.

For each tracked point at time \(t\): \[\begin{align} \Delta x_t,\;\Delta y_t &= (x_t - x_{t-1},\, y_t - y_{t-1}),\\ \rho_t &= \sum_{i,j,c} P_t^{\text{L}}(i,j,c)\,P_{t-1}^{\text{L}}(i,j,c), \end{align}\] where \(P_t^{\text{L}}\) and \(P_{t-1}^{\text{L}}\) are \(31{\times}31{\times}10\) event patches from the left view. The concatenated vector \([\Delta x_t,\Delta y_t,\rho_t] \in \mathbb{R}^{3}\) is processed by an MLP: \[\begin{align} &\text{Linear}(3, 128) \to \text{ReLU} \\ &\to \text{Linear}(128, 256) \to \text{ReLU} \\ &\to F_{\text{corr},t} \in \mathbb{R}^{256}. \end{align}\]

7.3.0.2 Mono+Depth Mode.

We additionally include depth change: \[\Delta d_t = d_t - d_{t-1},\] yielding a 4-dimensional input \([\Delta x_t,\Delta y_t,\Delta d_t,\rho_t]\): \[\begin{align} &\text{Linear}(4, 128) \to \text{ReLU} \\ &\to \text{Linear}(128, 256) \to \text{ReLU} \\ &\to F_{\text{corr},t} \in \mathbb{R}^{256}. \end{align}\]

7.4 Position Encoding↩︎

Normalized 2D coordinates \(\left(\tfrac{x_t}{W},\tfrac{y_t}{H}\right)\) are encoded via: \[\begin{align} &\text{Linear}(2, 128) \to \text{ReLU} \\ &\to \text{Linear}(128, 256) \to F_{\text{pos},t} \in \mathbb{R}^{256}. \end{align}\]

7.5 Feature Fusion↩︎

7.5.0.1 Stereo Mode.

Left and right view features are first extracted independently, then fused via an MLP: \[\begin{align} G^{i,\text{L}}_t &= F^{\text{L}}_{\text{local},t} + F_{\text{corr},t} + F_{\text{ref},t} + F_{\text{pos},t}, \\ G^{i,\text{R}}_t &= F^{\text{R}}_{\text{local},t} + F_{\text{corr},t} + F_{\text{ref},t} + F_{\text{pos},t}, \\ G^i_t &= \text{MLP}_{\text{stereo}}([G^{i,\text{L}}_t, G^{i,\text{R}}_t]), \end{align}\] where \(\text{MLP}_{\text{stereo}}\): \[\begin{align} &\text{Linear}(512, 256) \to \text{ReLU} \\ &\to \text{Linear}(256, 256) \to G^i_t \in \mathbb{R}^{256}. \end{align}\]

7.5.0.2 Mono+Depth Mode.

Features are concatenated and fused: \[\begin{align} F_t^{\text{cat}} &= [F_{\text{local},t}, F_{\text{corr},t}, F_{\text{ref},t}] \in \mathbb{R}^{768}, \\ F_t^{\text{fuse}} &= \text{MLP}_{\text{fusion}}(F_t^{\text{cat}}) + F_{\text{pos},t}, \end{align}\] where \(\text{MLP}_{\text{fusion}}\): \[\begin{align} &\text{Linear}(768, 512) \to \text{ReLU} \\ &\to \text{Linear}(512, 256) \to F_t^{\text{fuse}} \in \mathbb{R}^{256}. \end{align}\]

7.6 Mamba-based Refinement↩︎

The fused tokens \(\mathbf{T}\!\in\!\mathbb{R}^{B \times T \times N \times 256}\) are refined through 2 groups of four-directional Mamba modules.

7.6.0.1 Mamba Block Configuration.

Each Mamba module uses:

• Hidden dimension: \(d = 256\)

• State dimension: \(d_{\text{state}} = 32\)

• Convolution kernel size: \(d_{\text{conv}} = 4\)

• Expansion factor: \(\text{expand} = 2\)

This yields an internal dimension of \(d_{\text{inner}} = d \times \text{expand} = 512\).

7.6.0.2 Four-Directional Scanning.

For each refinement group, tokens are scanned along four directions:

1. Temporal forward (\(\tau^+\)): reshape to \([B \cdot N, T, 256]\), process, reshape back.

2. Spatial forward (\(\sigma^+\)): reshape to \([B \cdot T, N, 256]\), process, reshape back.

3. Temporal backward (\(\tau^-\)): flip temporal axis, process, flip back.

4. Spatial backward (\(\sigma^-\)): flip spatial axis, process, flip back.

Outputs are averaged: \(\mathbf{T}' = \tfrac{1}{4}\sum_{s\in\mathcal{S}} \mathbf{H}_s\), followed by residual connection: \(\mathbf{T} \leftarrow \mathbf{T} + \mathbf{T}'\).

7.7 Prediction Heads↩︎

7.7.0.1 2D Position Heads.

X and Y coordinates are predicted via softmax-weighted binning: \[\begin{align} &\text{Linear}(256, 128) \to \text{ReLU} \\ &\to \text{Linear}(128, 31) \to \text{Softmax} \to \text{weighted sum} \to \Delta x, \end{align}\] where the 31 bins span \(\Delta x \in [-12.5, 12.5]\) pixels. Y prediction follows the same structure.

7.7.0.2 Disparity Head (Stereo).

\[\begin{align} &\text{Linear}(256, 128) \to \text{ReLU} \\ &\to \text{Linear}(128, 193) \to \text{Softmax} \to d \in [0, 192]. \end{align}\]

7.7.0.3 Depth Head (Mono+Depth).

\[\begin{align} &\text{Linear}(256, 128) \to \text{ReLU} \\ &\to \text{Linear}(128, 1) \to \Delta d, \end{align}\] predicting depth increment \(\Delta d\), followed by clipping to \([d_{\min}, d_{\max}]\).

7.7.0.4 Confidence Head.

\[\begin{align} &\text{Linear}(256, 128) \to \text{ReLU} \\ &\to \text{Linear}(128, 1) \to \Delta c, \end{align}\] predicting confidence increment \(\Delta c\) in logit space.

8 Dataset Construction Details↩︎

8.1 Overview↩︎

As an implementation supplement to Sec. 3.4, we describe how long PointOdyssey sequences are segmented into training windows for EvD-PointOdyssey.

Each scene is rendered into high-frame-rate RGB videos (240 FPS), converted to events using a single, fixed contrast threshold \(\theta=0.02\) (calibrated once for the dataset), and paired with dense depth maps. We do not use multi-threshold synthesis, ensuring consistent event generation across the entire dataset. The resulting event–depth–trajectory triplets are then organized into windowed training sequences with controlled motion magnitude.

8.2 Sequence Formation↩︎

From the filtered trajectories in Sec. 3.4, we extract sliding temporal windows of length \(\ell \in \{12,16,24,30\}\) frames. A window is valid if it satisfies:

1. Full visibility: all selected points satisfy \(v_{t,n}=1\) for all \(t\in [s,s+\ell)\),

2. Non-overlapping: frame ranges do not overlap with previously selected windows,

3. Motion constraints: maximum inter-frame displacement \(\Delta_{n}^{\max} \leq 12\) px and mean displacement \(\Delta_{n}^{\text{avg}} \geq 2\) px,

4. Depth validity: all depth values are finite and within \([0.9, 50]\) meters,

5. Point count: \(1 \leq N \leq 30\) valid points per sequence.

Each valid sequence contains:

• Per-frame 10-channel event tensors \(E_t \in \mathbb{R}^{10\times H\times W}\) (5 positive & 5 negative temporal bins),

• Dense depth maps \(D_t \in \mathbb{R}^{1\times H\times W}\),

• 2D pixel trajectories \(\mathbf{X}_{t,n} \in \mathbb{R}^{2}\),

• 3D world trajectories \(\mathbf{P}_{t,n} \in \mathbb{R}^{3}\),

• Visibility masks \(v_{t,n} \in \{0,1\}\),

• Camera intrinsics and poses.

This segmentation strategy supports both short-term (\(12\)–\(16\) frames) and longer-term (\(24\)–\(30\) frames) motion patterns, enabling the model to learn across different temporal scales.

8.3 Filtering Algorithm↩︎

Algorithm [alg:sequence] summarizes the selection procedure used to generate valid training sequences from the PointOdyssey assets.

8.4 Dataset Statistics↩︎

The final EvD-PointOdyssey dataset contains:

• Training: 2,847 sequences from 100 unique scenes

• Testing: 312 sequences from 11 held-out scenes

• Length distribution: 30-frame (42%), 24-frame (28%), 16-frame (18%), 12-frame (12%)

• Average points per sequence: 18.3

• Average displacement: 4.2 pixels/frame

• Average depth range: 1.2–28.5 meters

8.5 Reproducibility Assets↩︎

Complete per-scene sequence lists, start/end indices, and detailed statistics (sequence lengths, point counts, displacement distributions) will be released together with the dataset. These assets enumerate the exact training windows used in our experiments, enabling full reproducibility of our dataset construction pipeline.

References↩︎