1 Introduction↩︎

Motion prediction [1]–[5] aims to predict future motion displacements based on past observations from multimodal sensors, playing a crucial role in path planning and navigation for autonomous driving systems. To mitigate failures when dealing with unknown classes, many recent motion prediction approaches [4], [6], [7] adopt a class-agnostic paradigm to directly regress cell-wise motions. These approaches transform the point clouds into bird’s eye view (BEV) maps and represent motions as 2D displacements along the ground plane, achieving convincing performance. However, it is extremely challenging and costly to obtain motion labels on real data due to the sparse and non-uniform nature of point clouds [8], [9]. In contrast, the synthetic 4D LiDAR sequences can directly provide accurate motion labels. The availability of labeled synthetic motion data thus prompts us to explore synthetic-to-real translation for motion prediction (SRMP), where the motion prediction model learns knowledge from synthetic data and transfers it to unlabeled real-world data.

While unsupervised domain adaptation has achieved notable success in tasks such as semantic segmentation and object detection [6], [10]–[12], its extension to synthetic-to-real motion prediction remains underexplored. Motion prediction is characterized by a significant domain gap between synthetic and real-world data, stemming from discrepancies in motion distributions, occlusion patterns, and object compositions. This gap is particularly pronounced in the synthetic-to-real setting; consequently, a model trained solely on synthetic data suffers a severe performance degradation when directly deployed in the real world, exhibiting a \(150.5\%\) increase in prediction error. A common approach to bridge this synthetic-to-real gap is the teacher-student framework [6], [10], [11], [13]–[16], where a teacher model generates pseudo-labels on real data to supervise a student model. However, a naive application of this framework to motion prediction (Fig. 1 (a)) proves ineffective, further degrading performance with a \(202.7\%\) error increase. We attribute this failure to noisy pseudo-labels generated by the teacher, which impede effective knowledge transfer. Specifically, we observe two dominant failure modes in the pseudo-labels (Fig. 1 (c)): (1) motion jitters, characterized by erratic and non-smooth motion vectors at the individual cell level, and (2) object-level inconsistencies, where cells belonging to the same physical object exhibit divergent or misaligned motion patterns. Prior work [3] suggests that such artifacts are inherent to standard motion regression methods, which typically operate in a cell-wise, context-agnostic manner, rendering them highly vulnerable to domain shifts.

To overcome this, we propose SR-Motion, a novel framework that exploits objectness as a structural prior to mitigate the sensitivity of motion regression under domain shift. Inspired by MotionNet [3], which uses ground-truth object labels to regularize motion consistency within instances, we aim to leverage objectness priors to enhance pseudo label reliability. However, this is challenging in synthetic-to-real motion prediction, where real-world data lacks object annotations and raw motion vectors are too noisy for reliable clustering. To address this, SR-Motion introduces two key components: Objectness-aware motion prediction, in which the model jointly predicts motion vectors and objectness priors, specifically, the relative offset from each point to its object centroid, enabling the network to implicitly capture object-level structure during motion estimation; and Objectness-aided motion enhancement, which employs a dual-path consistency mechanism: only points where both the objectness and motion branches agree on cluster assignment are retained and spatially smoothed. This mutual validation between objectness and motion suppresses jitter and enforces coherent object-level motion, significantly improving the quality of pseudo labels and enabling stable knowledge transfer.

Furthermore, since the exploration into SRMP remains at an early stage, there is a lack of synthetic datasets specifically designed for SRMP. To address this, we develop a practical synthesis pipeline and construct a large-scale synthetic LiDAR dataset named Motion4D. This comprehensive dataset comprises \(1,370\) 4D sequences totaling \(124K\) frames, encompassing a wide variety of motion patterns. The diversity and scale of Motion4D significantly facilitate effective motion knowledge translation for SRMP, providing a robust foundation for both training and evaluating motion prediction models in a synthetic-to-real domain adaptation context. Based on Motion4D, we conduct a series of experiments to evaluate the effectiveness of SR-Motion. The results show that SR-Motion effectively enhances the motion regression branch, facilitating robust knowledge translation and further leading to superior performance on real scenes.

Our contributions are three-fold:

• We introduce the Synthetic-to-Real Motion Prediction (SRMP) task, the first formal study on transferring motion knowledge from synthetic to real 4D LiDAR data. We identify that motion jitters and object-level inconsistencies, caused by naive cell-wise regression under domain shift, are key barriers to robust knowledge transfer.

• We propose SR-Motion, a novel framework that leverages learned objectness priors to mitigate the sensitivity of motion regression under domain shift. The proposed objectness-aware prediction and objectness-aided enhancement modules offer a new paradigm for objectness-aware motion transfer.

• We present Motion4D, the first large-scale synthetic 4D LiDAR dataset tailored for SRMP, featuring diverse and realistic dynamic scenes and accurate motion annotations. Motion4D provides a benchmark for evaluating motion transfer methods and facilitates scalable training.

2 Related Work↩︎

2.0.1 Motion Prediction on LiDAR Point Clouds↩︎

Motion prediction [1]–[4], [17] aims to predict the future displacements of current point clouds based on past observations. To predict the motion efficiently, many recent works represent the 3D point cloud environment with bird’s eye view (BEV) map and represent the motion as 2D displacement along the ground plane. Footbots [18] propose a transformer-based architecture to predict motion trajectories for soccer players on BEV maps. PillarFlow [19] adopts PWC-Net [20] structure for 2D flow estimation to establish correspondences between two BEV maps. MotionNet [3] and BE-STI [4] learn to predict 2D motion in a fully-supervised manner by taking past BEV maps as input. SmartRefine [21] proposes a scenario-adaptive refinement method to enhance motion prediction performance, but it still relies on expensive real-world HD maps and labeled trajectories. Regarding the expensive cost of annotations, WeakMotionNet [7] and SSMP [22] propose to train the motion prediction networks in weakly- and semi-supervised manner, respectively. To be completely independent of annotations, PillarMotion [23] and SelfMotion [24] propose to learn from the data itself, training the model in a self-supervised manner. Observing synthetic data can directly provide accurate motion labels, we propose to explore SRMP to acquire real-world motion knowledge from synthetic data.

2.0.2 Synthetic-to-Real Translation↩︎

Synthetic-to-real translation approaches [13], [25]–[29] aim to train a model on synthetic data and transfer the knowledge into real data, which is widely used to avoid the cost of laborious annotation of real data. To narrow the domain gap, some methods adopt representation learning to learn more transferable features across domains. For example, [30] aligns density map features to improve crowd counting in real scenes, [31] explores audio-visual representations for crowd anomaly detection, and [32] learns general-purpose trajectory representations from synthetic data.

Recently, teacher-student frameworks [6], [33], [34] have become popular, where a teacher model generates pseudo-labels on real data to guide the synthetic-to-real learning of the student model. Most of the teacher-student-based works [10], [11], [13]–[16] focus on utilizing task-specific priors to improve the quality of pseudo labels and further alleviate the task-specific domain shifts. ST3D [10] introduces a quality-aware triplet memory bank to update pseudo labels to guide the knowledge transfer on the target domain. DCA-SRSFE [13] addresses the distortions and misalignments for 3D scene flow estimation. However, SRMP is still unexplored, and there are no appropriate techniques for such a dense regression task in the 4D LiDAR sequence domain. Thus, in this work, we concentrate on addressing domain gaps for SRMP to ensure a robust motion knowledge translation.

2.0.3 Synthetic Dataset Generation↩︎

There are many works in multimedia application that generate large-scale synthetic datasets for improved and robust real-world performance. However, there is a lack of suitable synthetic datasets for SRMP. A similar task with motion prediction is scene flow estimation, where synthetic datasets FT3D [35] and GTA-SF [13] are widely used as benchmarks. FT3D focuses on general object movement. GTA-SF employs city engine GTA-V [36] to simulate traffic and offer city scenes. For trajectory prediction, [32] generates and refines synthetic trajectories on real-world HD maps to construct synthetic datasets. While recent works [37]–[39] have explored deep generative models to synthesize LiDAR data, these methods are prone to producing artifacts and risk model collapse through recursive training [40]. To circumvent these limitations, we adopt a physically-grounded simulation pipeline [13], [41]–[46]. Instead of learning data distributions, our approach populates 3D environments with dynamic CAD assets and renders realistic 4D sequences. This is achieved through a high-fidelity simulator that employs ray-casting algorithms [44], [47] combined with z-buffer occlusion handling. The resulting dataset, Motion4D, features diverse and realistic traffic scenarios, providing a robust foundation for the development and evaluation of SRMP.

3 Methodology↩︎

3.1 Overview↩︎

In this work, we tackle the challenge of synthetic-to-real translation for motion prediction (SRMP), with the goal of leveraging synthetic data to perform accurate predictions on unlabeled real-world datasets. To achieve this objective, we address three pivotal questions: 1) How to effectively translate motion knowledge from synthetic to real environments? 2) How to perform robust motion prediction with synthetic-to-real domain gaps? 3) How to improve the quality and robustness of the translation process.

As illustrated in Fig. 2, our proposed solution, SR-Motion, is designed to address these challenges through a cohesive framework comprising three key components: motion knowledge translation framework in Sec 3.2, objectness-aware motion prediction in Sec 3.3, and objectness-aided motion enhancement in Sec 3.5. These components work together to enable effective and robust knowledge translation from synthetic to real environments. In the subsequent sections, we will elaborate on each component of SR-Motion.

3.2 Motion Knowledge Translation↩︎

In the context of SRMP, a critical problem is the effective translation of motion knowledge from labeled synthetic data to unlabeled real-world data. As illustrated in Fig. 2, SR-Motion adopts a teacher-student framework [6] for foundational knowledge transfer. The key insight of this framework is to train a teacher model \(\Phi^t\) using labeled synthetic data, which then generates high-quality pseudo labels \(\hat{M}\) for real data to guide the learning of a student model \(\Phi^s\). The training procedure is structured as follows:

3.2.1 Teacher Model Initialization↩︎

Initially, the teacher model \(\Phi^t\) is trained on labeled synthetic data to capture motion knowledge effectively. Once adequately trained, the weights of \(\Phi^t\) are copied to initialize the student model \(\Phi^s\), ensuring that both models share identical architectures and start with the same learned parameters.

3.2.2 Pseudo Label Generation↩︎

Then, the teacher model \(\Phi^t\) generates high-quality pseudo labels \(\hat{M}\) for real-world data. These pseudo labels serve as a bridge between the synthetic and real domains, enabling \(\Phi^s\) to learn from real-world scenarios without direct supervision. Besides, to mitigate the risk of noisy labels undermining robust knowledge translation, we enhance \(\Phi^t\) with an objectness-aided motion enhancement module (detailed in Sec. 3.5) to further improve the quality of pseudo label.

3.2.3 Student Model Learning↩︎

Obtained pseudo-labeled real-world data, \(\Phi^s\) is able to jointly learn from losses \(\mathcal{L}_{real}\) and \(\mathcal{L}_{syn}\) on synthetic data and real data, respectively, via stochastic gradient descent. This learning strategy allows \(\Phi^s\) to integrate motion knowledge from synthetic data while adapting to real-world conditions. The definitions of these two losses will be detailed in Sec. 3.3.

To boost the robustness of knowledge translation, data augmentation techniques are applied to the input sequences of \(\Phi^s\). Specifically, we randomly apply the same rotation (\(+90^{\circ}\;or -90^{\circ}\)) and flip (\(x\) or \(y\) direction) to all frames in one sequence, with corresponding transformations applied to the (pseudo) labels to align with the modified inputs. This process ensures that motion prediction models produce consistent results despite various disturbances, thereby enhancing robustness. The implementation of augmentation is detailed in Sec. 3.3.

3.2.4 Teacher Model Update↩︎

Finally, to continuously refine the teacher model, after the \(\tau_{th}\) iteration of student model learning, the teacher model \(\Phi^t\) is updated according to the exponential moving average (EMA) of the student model \(\Phi^s\): \[\label{eq:ema} \Phi^t_{\tau + 1} = \alpha \cdot \Phi^t_{\tau} + (1-\alpha) \cdot \Phi^s_{\tau + 1},\tag{1}\] where \(\alpha\) is a smoothing factor to alleviate the noise in the single model \(\Phi^s_\tau\). This update strategy effectively mitigates the impact of noisy or unstable gradients in the student model, leading to a more stable and robust teacher model over time. By gradually incorporating the learned representations of the student model \(\Phi^s\), especially those adapted to the real-world domain through pseudo-labeling, the teacher \(\Phi^t\) becomes increasingly better adapted to the target distribution. This results in higher-quality and more consistent pseudo-labels, which in turn provide more reliable supervision for the student in subsequent iterations.

Following each update, the knowledge translation process iterates back to the pseudo label generation step, allowing for continuous refinement and learning of both \(\Phi^t\) and \(\Phi^s\), progressively enhancing the performance of motion prediction. In the inference stage, the output of the motion branch of the teacher model is utilized for evaluation.

3.3 Objectness-Aware Motion Prediction↩︎

While the above teacher-student framework has enabled basic motion knowledge translation, the simplistic motion regression method is sensitive to the domain shift, inevitably leading to noisy pseudo motion labels. To address this challenge, we introduce the Objectness-Aware Motion Prediction network (OAMNet), designed to improve the robustness of motion prediction across these domain gaps. As shown in Fig. 3, OAMNet builds upon the state-of-the-art MotionNet [3] and enhances its capability to handle domain shifts by incorporating an objectness-aware branch into the architecture. We will provide a brief overview of the motion prediction pipeline from MotionNet, highlighting the feature extraction module and motion branch, while placing particular emphasis on the functionality of the newly introduced objectness-aware branch. We strongly recommend readers refer to MotionNet [3] for details of feature extraction and motion branches.

3.4 Preliminaries: From Point Clouds to BEV Representation↩︎

This section details the nature of this data and the preprocessing steps used to create a structured input representation suitable for our deep learning architecture.

Our model processes 4D LiDAR data, which is a sequence of 3D LiDAR point clouds. A raw LiDAR point cloud is an unordered and sparse set of 3D points that captures the geometry of the surrounding environment. This format is not directly amenable to standard convolutional architectures. Therefore, we convert the sequence of raw point clouds into a dense, spatio-temporal tensor using a voxel-based, Bird’s-Eye-View (BEV) representation, which is a common and effective approach [3], [24], [48], [49] in autonomous driving perception. We follow the previous work to adopt a transformation process that involves the following key steps:

• Temporal Alignment: Given an input sequence of \(N\) LiDAR frames (the current frame and \(N-1\) past frames), each of the frames is described by its local coordinate system. Thus, we first apply Ego-motion compensation to align all past point clouds to the coordinate system of the current frame. This crucial step corrects for the movement of the ego-vehicle and creates a coherent scene representation over time.

• Voxelization: The 3D space of the synchronized point clouds is quantized into a regular grid of voxels. We use a binary occupancy representation, where a voxel has a value of 1 if it contains one or more LiDAR points and 0 otherwise. The implementation details are presented in Sec. 5.1.

• BEV Pseudo-Image Formation: The 3D voxel grid is then projected onto a 2D BEV plane of size \(H\times W\). To retain crucial vertical information, the height dimension is encoded into the feature channel. Specifically, for each grid cell in the BEV map, we form a binary vector of dimension \(D\). Each element in this vector corresponds to a voxel at a specific height in that vertical column, indicating its occupancy status. This process transforms each LiDAR frame into a 2D pseudo-image where the channel dimension encodes height information.

The final input to our network is a tensor with the shape \(N\times H\times W\times D\), where \(N\) is the number of temporal frames in the sequence, \(H\) and \(W\) are the height and width of the BEV grid, \(D\) is the number of voxel layers along the vertical axis, now serving as the feature dimension. This structured tensor effectively captures the spatial and temporal dynamics of the driving scene in a form that can be further effectively processed.

3.4.1 Motion Prediction↩︎

Obtained the BEV representation, we adopt a Spatio-Temporal Pyramid Network (STPN) [3] as the backbone architecture to process these input sequences. The SPTN hierarchically extracts features from a sequence of BEV maps by using efficient 2D convolutions for spatial information and 3D convolutions for temporal information. It then fuses these multi-scale features to combine both local and global spatio-temporal context for dense prediction tasks, resulting in extracted features \(F\in\mathbb{R}^{N\times H\times W\times C}\) (where the output feature dimension \(C=32\)). The features are then forwarded to the following branches to perform various perception tasks.

As the pivotal branch for motion prediction, the motion branch, a regression head with two-layer 2D convolutions, predicts the motion vectors \(M = \{M^t\}^T_{t=1}\in \mathbb{R}^{T\times H\times W \times 2}\) for each cell in the current BEV map. These motion vectors serve as offset indicators to forecast the future positions of cells in the subsequent \(T\) frames, which can be mathematically expressed as: \[\label{eq:motion} X^t = X + M^t, \quad t=1,2,...,T,\tag{2}\] where \(X\in \mathbb{R}^{H\times W \times 2}\) represents the coordinates of cells in the current BEV map, and \(X^t\in \mathbb{R}^{H\times W \times 2}\) denotes their predicted coordinates in the \(t^{th}\) future frame. The motion branch is supervised by a smooth L1 loss: \[\mathcal{L}_{m}=\sum_{t=1}^Tsmooth_{L1}(M^t, \overline{M}^t) \,,\] where \(\overline{M} = \{\overline{M}^t\}^T_{t=1}\in \mathbb{R}^{T\times H\times W \times 2}\) is the ground truth motion vectors.

Additionally, following previous works [3], [22], two auxiliary branches are responsible for perceiving the categories and moving states of all cells, providing supplementary information for regularizing the motion predictions. Both of the branches are supervised by cross-entropy losses \(\mathcal{L}_{p}\), which can be formualted as: \[\mathcal{L}_p=CELoss(A, \overline{A})+BCELoss(S,\overline{S})\,,\] where the first term is the Categorical Cross-Entropy loss for object classification. \(A\in\mathbb{R}^{H\times W \times 5}\) contains the predicted probabilities for five categories (background, pedestrian, car, bike, and other), while \(\overline{A}\in\mathbb{R}^{H\times W \times 5}\) is the corresponding one-hot ground-truth label. The second term is the Binary Cross-Entropy loss for moving state estimation. \(S\in\mathbb{R}^{H\times W}\) is the predicted probability of an object being in motion, and \(\overline{S}\in\mathbb{R}^{H\times W}\) is the ground-truth label indicating its state (static or moving).

3.4.2 Objectness-Aware Branch↩︎

Despite its effectiveness for single-domain motion prediction, naive cell-wise regression within the motion branch is sensitive to domain shifts between synthetic and real-world scenes, leading to noisy predictions (see Fig. 2). Traditional approaches [3] suggest constraining motions with rigid object labels, but this is non-trivial for SRMP due to the lack of labeled real data. To address this, OAMNet incorporates an objectness-aware branch with two advantages:

• Implicit Integration: Via multi-task learning [50], [51], the branch integrates learnt objectness knowledge with motion knowledge, enhancing prediction robustness.

• Explicit Enhancement: The branch predicts objectness priors, which can be utilized to constrain motions without requiring labels on real data (see Sec 3.5).

Detailed implementation. Specifically, OAMNet adds the objectness-aware branch parallel to the motion branch. The objectness-aware branch is also implemented as a regression head with two-layer 2D convolutions. This design preserves the full encoder and context modules of MotionNet [3] unchanged, ensuring backward compatibility while enriching motion prediction with objectness-aware cues.

The objectness-aware branch takes the spatio-temporal feature \(F\) from the shared spatio-temporal pyramid network as input, which is identical to the motion branch. Inspired by detection and segmentation techniques [52]–[56], this branch predicts cell-wise relative offset vectors \(O\in \mathbb{R}^{H\times W \times 2}\) that align cells with respective ground-truth centroids. To this end, the branch consists of two convolutional layers:

• A \(3\times3\) convolution with \(32\) output channels, followed by BatchNorm and ReLU activation;

• A \(1\times1\) convolution with \(2\) output channels (for \(x\) and \(y\) offsets), without non-linearity.

During training on synthetic data, the ground-truth offset \(\overline{O}\) for each cell at position \(X=(x,y)\) is computed as \(\overline{O}=X_{ctr}-X\), where \(X_{ctr}\in\mathbb{R}^{H\times W \times 2}\) denotes the center coordinate of the bounding box enclosing the object which the cell belongs. Cells not associated with any object (i.e., background) are assigned zero offsets. The loss function combines L1 regression and cosine similarity to optimize length and direction jointly: \[\label{eq:centroid} \mathcal{L}_{o}=L1(O,\overline{O}) + Mean(\frac{O \cdot \overline{O}}{\Vert O \Vert \cdot \Vert \overline{O} \Vert})\tag{3}\] On real-world data, where ground-truth object labels are unavailable, we generate pseudo-offset labels using predictions from the teacher model \(\Phi^t\)(see Sec. 3.2), enabling self-supervised objectness learning.

By learning centroid offsets, the objectness-aware branch enriches OAMNet with objectness knowledge, i.e. enabling each cell to notice which instance it belongs to. This method encourages the motion branch to predict consistent motion for cells within the same instance, resulting in more reliable motion predictions.

3.4.3 Losses and Augmentation↩︎

For the synthetic domain, \(\mathcal{L}_{syn}\) contains supervised losses from all branches: \[\mathcal{L}_{syn}=\mathcal{L}_{m}+\mathcal{L}_{p}+0.01\mathcal{L}_{o}.\] In contrast, for the real domain, \(L_{real}\) only considers losses from motion and offset branches, mitigating impacts from the class imbalance, which can be formulated as: \[\mathcal{L}_{real}=\mathcal{L}_{m}+0.01\mathcal{L}_{o}.\]

In addition, to enhance the robustness of the model to geometric variations and domain gaps, we apply a consistent rotation and flip augmentation strategy on both the BEV input sequences and corresponding labels. The procedure is detailed in Algorithm 4.

3.5 Objectness-Aided Motion Enhancement↩︎

Even with OAMNet implicitly enhanced robustness to domain gaps through multi-task learning, initial pseudo motion labels from the teacher model \(\Phi^t\) still contain unavoidable noise, as illustrated in Fig. 2. To mitigate the impact of this noise on knowledge transfer stability, we propose an Objectness-Aided Motion Enhancement (OAME) module that explicitly constrains these motion pseudo labels with objectness priors. The core principle underpinning OAME is the rigid assumption: cells within a rigid object should share consistent motions and common centroids. As depicted in Fig. 5 (a), the OAME module employs three strategies based on this assumption:

• Centroid-aware clustering: Utilizing learned centroid offsets as cues to group cells into clusters that approximate rigid objects.

• Cluster outlier filtering (COF): Identifying and excluding noisy predictions that remain outliers in the clustering process.

• Spatial consistency smoothing (SCS): Ensuring consistent motion predictions for cells within the same cluster.

Subsequently, we will detail the design of each component.

3.5.1 Centroid-aware clustering↩︎

Given a set of centroid offset vectors \(O\) and their corresponding cell coordinates \(X\), we can calculate the centroid positions \(X_{ctr} = X + O\). This operation effectively shifts each cell to its associated centroid location. If multiple cells have centroids that are very close or even identical, it is reasonable to assume these cells belong to the same rigid object. We then cluster these cells together based on their proximity in the transformed space. For example, as illustrated in Fig. 5 (b), this process results in the scattered cells being grouped into two clusters, where cells within a cluster share identical centroid positions when mapped onto a quantized grid. We formulate the clustering process can be described as follows: \[\mathcal{C} = \mathcal{G}(O, X, F),\] where \(\mathcal{C}\) denotes the set of all clusters, and \(\mathcal{G}\) is a function that takes the centroid offset vectors \(O\), cell coordinates \(X\), and a binary mask \(F\in\mathbb{R}^{H\times W}\) indicating valid moving cells as inputs, and outputs the resulting clusters. For each cell \(p\) within a cluster \(\mathcal{C}_i\), we have: \[[O_p + X_p] = X_{\mathcal{C}_i},\] where \([O_p + X_p]\) represents the integer coordinates obtained after applying the rounding function, and \(X_{\mathcal{C}_i}\) is the shared centroid coordinate among all cells in the cluster \(\mathcal{C}_i\). This method aids in identifying objects that share similar spatial positions by grouping each instance into a cluster.

3.5.2 Cluster outlier filtering (COF)↩︎

Given the rigid assumption, cells within identical objects should maintain coherent clusters. Conversely, noisy cells with erratic motions or offsets cannot form valid clusters and will appear as outliers. Thus, we design COF method with two stages to suppress noisy pseudo labels with prediction jitters.

In the first stage, we add the predicted cell-wise motions \(M^T\) to cells to forecast their positions in the future \(T^{th}\) frame, as described in Eq. 2 . The cells with motion jitters will be shifted to undesirable positions. Then, in the second stage, we add centroid offsets to determine the centroids of cells in future frames. Here, we can utilize the centroid-aware clustering technique to group moving cells into \(\mathcal{C}^{fut}\). Thus, the overall procedure can be described by \(\mathcal{C}^{fut} = \mathcal{G}(M^T + O, X, F)\). If any clusters in \(\mathcal{C}^{fut}\) contain no more than \(\mathcal{T}_{N}\) cells, they will be recognized as outliers, and therefore excluded from translation and labeled as invalid cells in \(F\).

Discussion. If we only activate the latter stage to cluster in the current frame, some noisy cells with wrong offsets, especially the ground cells near objects, may be incorporated into valid clusters. Thus, this one-stage filtering leads to inferior enhancement (see Table 2). In contrast, with the two-stage filter, noisy cells with either inaccurate motions or offsets will form invalid clusters, enabling us to filter out most of the unreliable jitters in pseudo labels.

3.5.3 Spatial consistency smoothing (SCS).↩︎

Without explicit object constraints during motion regression, pseudo labels may exhibit inconsistent object-level motions, hindering stable knowledge transfer. Therefore, we design SCS to ensure the spatial consistency of motion across clusters. Here, we first group the reserved valid cells from COF into clusters in current frames, which can be depicted as \(\mathcal{C}^{cur} = \mathcal{G}(O, X, F)\). Since we have excluded the potential outliers via COF, these clusters are clean enough for consistency smoothing. For every cell \(p\) belonging to \(\mathcal{C}^{cur}_i\), we update its motion pseudo labels to the average motion \(\mu M_{i}\) across \(\mathcal{C}^{cur}_i\), which can be formulated as: \[\label{eq:cluster95mean} \forall p \in \mathcal{C}^{cur}_i\;M_p = \mu M_{i},\;\mu M_{i} = \frac{1}{N_i}\sum_{p\in \mathcal{C}^{cur}_i}{M_p},\tag{4}\] where \(N_i\) denotes the number of cell in \(\mathcal{C}^{cur}_i\). Additionally, recognizing that clusters exhibiting high divergence in motion distribution are unreliable for smoothing and often contain significant noise, we introduce the Coefficient of Variation (CV) as a metric to evaluate the divergence within clusters. The CV of \(\mathcal{C}^{s}_i\) can be defined as: \[\begin{align} \label{eq:cluster95cv} CV_{i} = \frac{\sigma M_{i}}{\vert \mu M_{i} \vert},\;\sigma M_{i} = \sqrt{\frac{1}{N_i}\sum_{p\in \mathcal{C}^{cur}_i}{(M_p-\mu M_{i})^2}}. \end{align}\tag{5}\] Here, \(\sigma M_{i}\) represents the standard deviation of the motion values within cluster \(\mathcal{C}^{s}_i\), while \(\vert\mu M_{i}\vert\) denotes the absolute mean motion value of the same cluster. A higher \(CV_{i}\) indicates greater variability and thus lower reliability of the cluster.

Clusters with a CV exceeding a predefined threshold \(\mathcal{T}_{CV}\) in either the horizontal or vertical direction along the ground plane are considered unreliable and are discarded. This filtering process ensures more stable self-training by eliminating clusters that could otherwise introduce noise and instability into the model. By doing so, we enhance the robustness and accuracy of our motion prediction framework, particularly in scenarios characterized by significant domain shifts.

4 Motion4D Dataset↩︎

We create a large-scale synthetic 4D dataset Motion4D with diverse labeled motions, focusing on addressing the lack of synthetic datasets in SRMP. In this section, we present the motion synthesis pipeline of Motion4D and its key features.

4.1 Motion Synthesis Pipeline↩︎

Motion4D is built upon Waymo Open Dataset [8], [57] and Blender [58]-based LiDAR tracing engine BLAINDER [44]. Specifically, it synthesizes a sequence of realistic driving scenes by placing diverse moving objects onto a static background, which is a single static real-world LiDAR point cloud.

We first conduct scene library and motion library to provide backgrounds and objects, respectively. Scene library The real-world LiDAR point clouds with few foreground objects are suitable to serve as the background for motion placement and can be easily collected. Here, for efficiency and versatility, we select \(685\) discrete static LiDAR frames collected from top center LiDARs in Waymo Perception Dataset [8]. Note that the selected frames only serve as backgrounds without any annotations. These frames are expected to contain broad road areas to provide vast space and freedom for the generation of motion. Besides, we implement Patchwork++ [59] for unsupervised road segmentation to judge valid areas of motion placement. Motion library Then, we establish the motion library by combining CAD assets with diverse trajectories from the training split of Waymo Motion Dataset [57]. The CAD assets are adopted from banks in CARLA [46] and Render People [60]. For realism, we apply category-specific speed restrictions and rule-based invalid motion suppression. Please refer to supplementary materials for details.

Now, with the above libraries, we can generate synthetic data via LiDAR simulation in BLAINDER. As the previous motion prediction approach [3] proves that sweep synchronization affects the model performance greatly, we eliminate the ego-vehicle motions and freeze the backgrounds so that we can obtain naturally synchronized sequences. In the simulation, we first obtain rays in LiDAR-axes from static LiDAR backgrounds. Then, we freely place driving objects onto the valid area in the backgrounds. To improve realism, we also implement collision detection to regenerate invalid motions. Finally, to collect LiDAR sequences, we activate the placed driving objects by moving objects along the respective trajectories and iteratively re-scan the rays based on z-buffer occlusion at a predefined frequency (10 Hz). The multi-iteration ray tracing in BLAINDER ensures realistic simulation. In order to annotate motions, following the rigidity assumption [3], [7], [13], we calculate motion displacements depending on object boxes and trajectories. Further details of LiDAR scanning are included in supplementary materials.

4.2 Dataset Features↩︎

With the above pipeline, Motion4D is constructed as a large-scale 4D synthetic dataset specifically designed for motion prediction, which contains \(1370\) sequences with \(124K\) frames of LiDAR point clouds. Our careful design makes the dataset both realistic and cost-effective. Each sequence in Motion4D provides diverse motions with precise labels based on just a single static LiDAR point cloud. Table 1 shows the comparison between real-world datasets and our synthetic datasets, demonstrating the diversity of motions and objects in Motion4D. As depicted in Fig. 7, Motion4D exhibits a rich variety and balanced distribution of motions. Thus, serving as the synthetic source data, Motion4D lays a strong foundation for synthetic-to-real translation for motion prediction.

A similar task with motion prediction is scene flow estimation, where synthetic datasets FT3D [35] and GTA-SF [13] are widely used as benchmarks. We present a visual comparison of FT3D, GTA-SF, and Motion4D as shown in Fig. 8. FT3D focuses on general object movement, which is unsuitable for autonomous driving scenarios. GTA-SF employs city engine GTA-V [36] to simulate traffic and offer city scenes. However, it is worth noting that both datasets currently offer 3D point cloud pairs designed for scene flow estimation. While valuable for such estimation task, it may be considered insufficient for addressing the scope of motion prediction, which involves forecasting future motion displacement based on a comprehensive 4D sequence input. Besides, as for LiDAR simulation, FT3D and GTA-SF transform z-buffer to point clouds and need special designs to rectify the points. Motion4D considers refraction and reflection of lights and implements multi-iteration ray tracing to calculate realistic light paths.

5 Experiments↩︎

5.1 Implementation Details↩︎

5.1.1 Real-world datasets↩︎

We evaluate SR-Motion on two large-scale real-world autonomous driving datasets, Waymo [8] and nuScenes [9]. For Waymo, we follow WeakMotionNet [7] to use \(798\) scenes for training and \(202\) scenes for validation. Note that we collect data from six sensors for a fair comparison. For nuScenes, we follow MotionNet [3] to use \(500\) scenes for training, \(100\) scenes for validation, and \(250\) scenes for testing. The motion ground truths are obtained from detection and tracking annotations [3], [7].

5.1.2 Training details↩︎

Following the same data preprocessing in previous works [3], [7], we crop the point clouds in the range of \([-32, 32]\times [-32, 32]\times [-1, 4]\) and \([-32, 32]\times [-32, 32]\times [-3, 2]\) meters for Waymo and nuScenes, respectively. The voxel size is set to \((0.25,0.25,0.4)m\). The past four sweeps are synchronized and combined with the current sweep as input. The network is trained on a single RTX 4090 GPU. The batch size is set to \(16\) with \(8\) synthetic data and \(8\) real data. By operating directly in the Bird’s Eye View (BEV) space, our architecture achieves high computational efficiency. During inference, it runs at an average speed of 20 ms per frame on the same RTX 4090, enabling real-time performance. Training typically takes approximately 5 hours.

5.1.3 Evaluation metrics↩︎

Following previous works [3], [7], each grid cell is categorized into one of three groups based on ground truth speeds: static, slow (\(\leq 5m/s\)), and fast (\(\geq 5m/s\)). This classification facilitates a more nuanced evaluation of model performance across different motion dynamics. To assess the accuracy of our predictions, we compute both the mean and median errors. These errors are quantified using the L2 distance between the predicted displacements and the corresponding ground truth displacements over a future time horizon of 1 second. Median values are supplementary, and mean metrics are used for primary comparison.

5.1.4 Comparison methods↩︎

We present the comparison with the following methods. (a) Synthetic-only models (labeled with superscripts \(\dagger\)) trained with only synthetic data; (b) Adaptation baseline models (labeled with superscripts \(\S\)) trained in mean teacher paradigm to realize basic knowledge transfer; (c) General domain adaptation techniques. Specifically, the gradient reversal layer (GRL) [61] reverses the backpropagation of domain classification to render features invariant to domain shifts, and Gaussian uncertainty estimation (GU) [62] predicts motions as Gaussian distributions and utilize the variances as uncertainty to exclude noisy pseudo labels; (d) Fully- [3], weakly- [7], semi- [22] and self-supervised [23] motion prediction approaches. MotionNet [3] trained with labeled real data is employed as Oracle models for clear comparison.

5.2 Main Results↩︎

In Table ¿tbl:table:performance?, we evaluate and compare our SR-Motion on different real datasets: Waymo and nuScenes.

Firstly, we evaluated synthetic-only approaches, specifically “MotionNet\(^\dagger\)” and “OAMNet\(^\dagger\)”. Both methods demonstrated respectable performance on real-world datasets, confirming that our Motion4D dataset effectively simulates realistic traffic scenarios and provides a solid foundation for SRMP. Subsequently, we examined the baseline model “MotionNet\(^\S\)”, which fundamentally transfers knowledge and slightly improves the prediction accuracy for slow cells. However, the presence of noisy pseudo labels inevitably degrades the performance for static and fast-moving cells.

To mitigate this issue, we try to directly incorporate general adaptation techniques such as Gradient Reversal Layer (GRL) [61] and Gaussian Uncertainty (GU) [62] methods to enhance the baseline models. Although these adaptations improve performance, they still yield sub-optimal results. In contrast, SR-Motion introduces innovative components, OAMNet and OAME modules, which effectively bridge domain gaps for SRMP. Specifically, compared to the baseline model, SR-Motion reduces the mean error for the fast group from \(2.702\) to \(1.545\) and closes the performance gap by \(76.3\%\) for the static group on the Waymo dataset.

Furthermore, we compared SR-Motion with Oracle, weakly-, semi-, and self-supervised motion prediction approaches. Our method achieves comparable performance to fully-supervised Oracle models. When compared with weakly- and semi-supervised approaches like WeakMotionNet and SSMP, our approach matches their performance without requiring any target data labels. Moreover, SR-Motion significantly outperforms previous self-supervised methods, PillarMotion and SelfMotion, narrowing the performance gap up to \(86.4\%\) at the static speed level.

Real-to-real Domain Adaptation In Table ¿tbl:table:domain-adaptation?, we present the results of real-to-real domain adaptation for motion prediction. On both Waymo and nuScenes datasets, our approach surpasses the baseline in real-to-real domain adaptation, affirming the effectiveness and robustness of SR-Motion.

Moreover, models trained on our synthetic dataset, Motion4D, achieve performance comparable to their real-to-real counterparts across various adaptation settings, highlighting the strong generalization capability of Motion4D. Notably, in the \({Motion4D}\rightarrow{Waymo}\) setting, our approach significantly outperforms the \({nuScenes}\rightarrow{Waymo}\) setting, achieving a mean error of \(1.545\) versus \(2.327\) for the fast-moving agent group. The marked performance gain underscores the high realism and robust domain adaptability of the Motion4D dataset, facilitating accurate and generalizable motion prediction.

Furthermore, we trained MotionNet on a mixed dataset comprising both the original Waymo data and our synthetic Motion4D data. As shown in Table. ¿tbl:table:aug?, this model outperforms MotionNet trained exclusively on the Waymo dataset by a significant margin of \(9.4\%\). This performance gain demonstrates that Motion4D serves as a valuable complementary data source that enriches the training distribution with diverse and realistic scenarios.

5.3 Ablation Study↩︎

We conduct ablation studies to evaluate the effectiveness of different components in SR-Motion. All the experiments are conducted on Motion4D\(\rightarrow\)Waymo setting and evaluated on Waymo val sets. Mean errors are reported.

5.3.1 Evaluation of Different Components↩︎

SR-Motion comprises several key components: motion knowledge translation (MT), objectness-aware motion prediction (OA); objectness-aided motion enhancement with cluster outlier filtering (COF) and spatial consistency smoothing (SCS). As shown in Table ¿tbl:table:ablation?, we evaluate their contributions to SR-Motion.

Objectness-aware motion prediction (OA). We incorporate an objectness-aware offset branch into MotionNet as the objectness-aware motion prediction network, named OAMNet. We compare OAMNet with MotionNet to evaluate the effectiveness of objectness-aware branch. First, group and are synthetic-only approaches without knowledge translation. OAMNet in group effectively reduces the mean error of static cells by \(40.7\%\) compared with MotionNet in group , which demonstrates that the acquired objectness knowledge can complement the motion regression and suppress jitters of background points. Besides, simply combining MotionNet with motion knowledge translation leads to noisy pseudo labels and performance decline (v.s. ). In contrast, OAMNet shows strong robustness to domain shift and offers stable improvement (v.s. ). This consistent improvement in generalization across domains demonstrates that the objectness prior is not only robust to domain distribution shifts but also effectively enhances motion prediction through implicit integration.

Objectness-aided motion enhancement (OAME). Though combining OAMNet with the motion knowledge translation framework obtains basic knowledge transfer, the noisy pseudo labels still contribute to unstable translation. Thus, we propose OAME with COF and SCS to filter out outliers and improve the consistency of pseudo labels. Cooperating with COF in group , the performance gap is reduced by \(58.3\%\) at the static speed compared with group , which proves that COF effectively filters out noisy jitters in static backgrounds. Based on the constraints of rigid objects from SCS, group achieves the best performance and reduces the mean error at fast speed by \(20.2\%\). However, when applying SCS without COF in group , the spatial clusters contain numerous noisy outliers, disturbing the pseudo label smoothing and leading to inferior results. This also demonstrates the effectiveness of COF for filtering outliers and accurate clustering.

We also visualize the predicted center offsets and the behavior of the OAME module in Fig. 10. Despite some noise in offset predictions under domain shift (e.g., first row), the majority of vectors point consistently toward object centers, validating the transferability of objectness knowledge. Crucially, our OAME module incorporates a dual-path consistency mechanism: only points where both the offset and motion branches agree on cluster assignment are retained. This geometric consensus acts as a built-in filter, i.e. COF module, suppressing false positives that may arise in one modality but are not corroborated by the other. This redundancy enhances robustness during synthetic-to-real transfer. Moreover, the SCS module further improves motion consistency within objects using the cleaned predictions from COF. As further illustrated in Fig. 11, combining offsets and motions, the OAME module filters out the noisy motion pseudo labels and produces high-quality pseudo labels.

5.3.2 Comparison of different strategies in COF↩︎

Cluster outlier filtering is a strict two-stage filter, where a cell with either inaccurate motion or offset will be recognized as an outlier. As shown in Table 2, we loosen the constraints by employing the one-stage filter, where group cells in the current frames. As the cells with inaccurate motion prediction are reserved, the noisy pseudo labels result in inferior performance. Besides, we observe that a large \(\mathcal{T}_{N}\) filters out more noisy jitters on static cells but also excludes some correct predictions of small objects. Thus, we only filter out the clusters with a single cell (\(\mathcal{T}_{N}=1\)) as outliers for robust learning.

5.3.3 Comparison of different strategies in SCS↩︎

After filtering out most of the unreliable cells, we apply spatial consistency smoothing to constrain pseudo labels with rigid spatial clusters. Besides, a selection operation is also included to exclude clusters with CVs larger than \(\mathcal{T}_{CV}\). Setting \(\mathcal{T}_{CV}\) to infinity (\(\infty\)) implies that this operation will not take effect. As shown in Table 3, the motion smoothing in Eqn. 4 guarantees motion consistency within clusters and narrows the performance gap with the Oracle model by \(31.7\%\) (strategy 2 v.s. strategy 1). The selection procedure improves the performance at the slow level (strategy 3 v.s. strategy 1). We also conduct an ablation on thresholds \(\mathcal{T}_{CV}\) (strategy 4-6). The lower \(\mathcal{T}_{CV}\) is, the more noisy pseudo labels are excluded but also the fewer clusters are reserved. As we pay more attention to fast objects in autonomous driving systems, \(\mathcal{T}_{CV}\) is determined as \(10\) in our implementation.

5.3.4 Ablation on smoothing factor↩︎

After every step \(\tau\), the weight of the teacher model \(\Phi^t\) is updated by the EMA weights of the student model \(\Phi^s\) as formulated in Eq.@eq:eq:ema , which can smooth the model noise and improve the performance of \(\Phi^t\). To investigate the influence of the smoothing factor \(\alpha\), we conduct an ablation study with different values, and the results are summarized in Table ¿tbl:table:smoothing95factor?. When \(\alpha=1.0\), the teacher model remains static and does not incorporate any new knowledge from the student. In this case, the model fails to adapt to the real-world domain, preventing effective knowledge transfer from synthetic source data and resulting in significantly degraded performance. On the other hand, setting \(\alpha=0.0\) (i.e., directly using the student as the teacher at every step) leads to highly unstable pseudo-labeling, as the teacher becomes overly sensitive to transient errors or fluctuations in the predictions of the student. This instability severely hinders training convergence and degrades overall accuracy.

These results clearly demonstrate that the EMA mechanism is not merely a smoothing trick, but a crucial component for balancing stability and adaptability. It enables the teacher to gradually absorb useful knowledge from the student while filtering out harmful noise, thus facilitating robust domain adaptation and reliable self-training. Based on this analysis, we set \(\alpha=0.999\) to achieve optimal performance, ensuring both effective learning dynamics and label consistency.

5.3.5 Ablation on Data augmentation↩︎

For robust learning, we introduce augmentation for the input BEV maps of the student model upon the input sequences of the student model. As demonstrated in Table ¿tbl:table:augmentation?, training with augmentation significantly boosts performance, particularly benefiting the prediction accuracy for static cells. By introducing transformations such as rotations and flips, the model learns to maintain its predictive accuracy under different conditions, thereby enhancing its reliability and stability across various scenarios. This improvement underscores the effectiveness of our augmentation strategy in fostering a more robust synthetic-to-real knowledge translation.

6 Conclusion↩︎

In this work, our primary focus lies in addressing the challenges of SRMP. Through our proposed novel SR-Motion, overcoming issues associated with naive regression methods, we successfully transfer motion knowledge to real-world data and obtain a motion prediction model robust to domain shift. Limitation. Following previous arts, our method operates on a BEV map, losing the fine-grained geometry needed to model non-rigid motion. Consequently, it assumes objects are rigid and has limitations when applied to deformable agents.

Acknowledgment↩︎

This research is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021). This research is also supported by the MoE AcRF Tier 2 grant (MOE-T2EP20223-0001).

7 References Section↩︎

8 Biography Section↩︎

References↩︎