1 Introduction↩︎

Trajectory prediction plays an important role in applications such as autonomous driving, robotics, and human behavior analysis [1]–[3]. It involves forecasting the future states of agents based on past observations and interactions with their surroundings. The topic has gained significant attention in recent years, driven by advances in deep-learning methods, increased availability of trajectory datasets [4]–[24], and open-source tools [25]–[28]. However, this progress has also highlighted an important challenge: trajectory datasets differ significantly in terms of agent types, environmental complexity, and sensing modalities. This diversity offers valuable opportunities for comprehensive model development, but it also poses difficulties for transferring models across different domains. A key open question is how well a model trained on one dataset generalizes to another—and whether such transferability can be predicted.

Advances in large foundation models demonstrate that training on diverse datasets can yield highly transferable representations [3], [29]–[31]. Theoretically, this could be applied to trajectory prediction by pooling all available data. However, several factors limit the practicality of this approach. First, real-time systems such as autonomous vehicles operate under strict computational and latency constraints, making it impractical to deploy models at the scale and with the inference costs of current foundation models. Furthermore, while model compression and distillation techniques alleviate these issues [32]–[35], recent work shows that more data are not always better; low-quality or irrelevant data can degrade performance [36]–[39]. These observations motivate prioritizing the most relevant datasets for pretraining, rather than indiscriminately aggregating all available data.

A natural first step toward addressing this challenge is to compare datasets based on their surface-level features [25], [40], [41]. These analyses can highlight differences in statistical properties, including agent densities, speed profiles, and map structures. However, they do not inherently capture emergent behavioral characteristics, such as interaction patterns, social norms, or task intent, which are defining features of multi-agent motion-prediction problems [42].

In this work, transferability is studied through a learned latent space that captures behavioral characteristics across datasets, as illustrated in 1. This space is obtained by jointly training a single embedding model on all datasets under investigation to project scenarios into a representation where distances reflect structural and behavioral patterns in addition to surface-level feature similarities. This learned representation provides a more meaningful basis for assessing dataset transferability by leveraging the capacity of deep models to capture complex interactions. Once trained, the embedding model enables querying at the dataset level, supporting data-driven recommendations of suitable source datasets for pretraining or adaptation. In practice, transferability is estimated using the KL divergence between Gaussian approximations of the latent dataset distributions. It is demonstrated that smaller latent distribution divergences correspond to improved transfer performance.

1.1 Contributions↩︎

The primary contributions of this paper are:

• Dataset-level representation learning for trajectory prediction. A latent embedding framework is introduced, modeling trajectory datasets as distributions in a shared latent space to enable comparison through their distributional geometry.

• A directional probabilistic measure of transferability. A divergence-based transferability metric is proposed to capture asymmetric transfer success, demonstrating that latent distances predict both zero-shot and fine-tuned performance.

• A systematic large-scale study of cross-dataset transfer. A comprehensive evaluation across 24 trajectory datasets and 552 transfer pairs is conducted, revealing underutilized datasets that improve performance on popular motion-prediction benchmarks.

Implementations are available at https://github.com/westny/transferatlas.

2 Related work↩︎

2.1 Trajectory prediction↩︎

Trajectory prediction remains an active area of research as a result of its central role in domains such as autonomous driving and robotics [1]–[3]. Advances in deep learning and increased data availability have made learning-based methods the dominant approach. A wide range of architectures have been explored, including recurrent networks [43]–[46], convolutional encoders [44], [47], [48], Transformer-based models [49]–[52], and GNNs [53]–[59].

A substantial portion of the literature focuses on modeling interactivity, , how agents influence one another and how they are affected by their surrounding environment. This includes pooling-based methods [43]–[45], [48], [60], as well as approaches that explicitly model interactions using GNNs or Transformer-based architectures [52], [53], [55]–[58], [61]. Environmental context is processed using specialized modules and usually incorporated through either rasterized scene encodings [47], [48], [55], [56], [62]–[65] or as vectorized maps [57], [65]–[68]. Further, probabilistic modeling is widely used to capture the inherent uncertainty of future behavior. These models can be broadly categorized as discriminative, which directly predict multi-modal trajectory distributions [44], [45], [58], [63], [65], [69], [70], or generative, which aim to learn the underlying distribution from which future trajectories can be sampled [48], [52], [55], [56], [60], [61], [71]–[78].

2.2 Domain adaptation and generalization↩︎

Improving model robustness across environments often involves either domain adaptation or domain generalization. Domain adaptation aims to reduce the discrepancy between the source and target data distributions, while domain generalization seeks models that perform well on unseen domains without additional tuning. In the field of trajectory prediction, several works explore these themes.

Recent approaches propose architectural adaptations such as ensemble-based methods [79], or specialized modules [80], [81] to improve generalization. Other strategies tackle this problem by learning shared representations across datasets [77], [82], [83], or by designing dataset-agnostic input abstractions that reduce reliance on domain-specific cues [25], [27], [39], [84]–[87].

On the domain adaptation side, recent work proposes techniques such as attention-based adaptation [88], feature alignment [89], and instance-level augmentations [90]. Other approaches adapt pre-trained models to new domains [91] or introduce meta-learning strategies for online or few-shot adaptation [92].

While domain adaptation and generalization are central themes in this work, the focus is not on developing new methods to improve these properties, but rather on understanding whether they can be predicted. Furthermore, evaluations are typically restricted to a limited set of source–target pairs. This leaves a clear gap for research that investigates transferability across a broader, more diverse spectrum of datasets.

2.3 Dataset similarity and transferability metrics↩︎

Quantifying dataset similarity is important for anticipating transfer learning success [93]–[97]. A closely related approach to ours is presented in [96], where methods to estimate transferability between time-series datasets are proposed. A shared autoencoder embeds source and target data into a common latent space, from which metrics such as the L1 distance between dataset centroids are derived and shown to correlate with transfer learning gains. In contrast, this work focuses on trajectory prediction, where spatio-temporal structure, interaction dynamics, and map-based context require specialized modeling [1]–[3]. In addition, probabilistic and non-symmetric measures, such as KL divergence, are considered to capture the inherently directional nature of transferability.

Related work on latent representations of trajectory datasets includes ScenarioNet [26], which utilizes TrafficGen [98] to obtain scenario embeddings that are then projected onto a lower-dimensional manifold. These projections highlight differences between synthetic and real data as well as residual variation across datasets. While primarily used for visualization, such an approach also highlights the potential of using latent space analyses to identify distributional gaps.

3 Problem definition↩︎

The problem of learning the approximate distributions of multiple trajectory datasets and using these distributions to quantify pairwise transferability is considered. Each dataset consists of interactive scenes, where a scene is defined as a set of agents (, vehicles, pedestrians) and their associated features (, position, velocity) that evolves over time. The temporal span of a scene is divided into a historical segment and a future segment. Each time instant in the scene is modeled as an undirected graph \(\mathcal{G}= (\mathcal{V}, \mathcal{E})\), where \(\mathcal{V}\) is the set of agents (nodes) and \(\mathcal{E}\) is the set of edges representing the interactions between said agents. Let \(N = |\mathcal{V}|\) be the number of agents, \(H\) the length of the historical segment, and \(F\) the length of the future segment. The node input features are summarized in a tensor \(\mathbf{X}\in \mathbb{R}^{N \times H \times D}\), and the output features in \(\mathbf{Y}\in \mathbb{R}^{N \times F \times D}\), where \(D=5\), corresponding to 2D position, 2D velocity, and orientation. When available, the map \(\mathbf{M}\) is also incorporated, encoded as a lane graph with associated features, shared across all time instants of the scene. For brevity, the notation is overloaded, letting \(\mathbf{X}\) implicitly denote the pair \((\mathbf{X}, \mathbf{M})\) whenever map information is available.

Next, latent variables \(\mathbf{z}_i\) are introduced for each node \(i\), summarized in a matrix \(\mathbf{Z}\in \mathbb{R}^{N \times L}\). Finally, for a scene \(\mathcal{S}\), the latent scene embedding \(\mathbf{Z}_\mathcal{S}\in \mathbb{R}^{L}\) is defined, and for a dataset \(\mathcal{D}\), the corresponding latent dataset embedding \(\mathbf{Z}_\mathcal{D}\in \mathbb{R}^{L}\), where \(L\) denotes the latent dimension. Unless otherwise stated, \(L=32\) in this work. The objective is two-fold:

1. Learn the latent dataset-level representations \(\{{\mathbf{Z}_\mathcal{D}}_j\}_{j=1}^K\) for datasets
   \(\mathcal{D}_1, \mathcal{D}_2, \ldots, \mathcal{D}_K\) and use them to quantify pairwise transferability based on distributional (dis)similarity.

2. Empirically validate the inferred transferability by training a trajectory prediction model \(\mathbf{X}\mapsto \mathbf{Y}\) on dataset \(\mathcal{D}_i\) and evaluating it on a different dataset \(\mathcal{D}_j\), for multiple pairs \((\mathcal{D}_i, \mathcal{D}_j)\).

4 Latent embedding model↩︎

To construct compact and comparable representations of the datasets under consideration, a deep latent embedding model is employed, designed to learn expressive codes of the individual samples (scenes) comprising each dataset.

To learn representations that capture all aspects of the data under a setting relevant to trajectory prediction, the model is trained using two complementary supervision signals: reconstruction of observed inputs and prediction of future states. The predictive objective reflects the trajectory forecasting setting, where past motion is observed and future motion is unknown, while reconstruction promotes information preservation. Latent regularization is used to encourage informative and stable representations [99]. In particular, a normalization constraint is applied on the latent variables, \(\|\mathbf{z}\|_2^2 = 1\) that restricts them to the surface of a unit sphere, encouraging comparability across samples and datasets while preventing degenerate solutions.

4.1 Model architecture↩︎

The latent embedding model follows an encoder–decoder architecture, illustrated in 2. The encoder maps input features to node-level latent variables, used to compute scene- and dataset-level embeddings. Two decoder heads are employed: one reconstructs input features, and the other predicts future states. The main building blocks of both networks are GGRUs [58].

4.1.1 Graph-gated recurrent unit↩︎

The GGRU enables spatio-temporal interactions to be captured by replacing the linear mappings in the GRU [100] with GNN components. The original GGRU design is simplified by using a minimal mechanism based on the minGRU [101], reducing model complexity with minor performance trade-offs.

At each time step \(k\), a message-passing operation incorporates information from node \(i\) and its neighborhood \(\mathbf{N}(i)\), producing intermediate vectors: \[\left[\boldsymbol{v}_{k}^{(i)}, \boldsymbol{w}_{k}^{(i)} \right] = \text{GNN}\left(\boldsymbol{x}_k^{(i)}, \{\boldsymbol{x}_k^{(j)}\}_{j \in \mathbf{N}(i)}\right),\] where GNN\((\cdot)\) is the graph neural network operator from [102]. These are then used to update the hidden state \(\boldsymbol{h}_k^{(i)}\) using the following gating mechanism: \[\begin{align} \boldsymbol{z}_k^{(i)} &= \sigma(\boldsymbol{v}_{k}^{(i)} + \boldsymbol{b}_z), \\ \tilde{\boldsymbol{h}}_k^{(i)} &= \tanh(\boldsymbol{w}_{k}^{(i)} + \boldsymbol{b}_{h}), \\ \boldsymbol{h}_k^{(i)} &= (1 - \boldsymbol{z}_k^{(i)}) \odot \boldsymbol{h}_{k-1}^{(i)} + \boldsymbol{z}_k^{(i)} \odot \tilde{\boldsymbol{h}}_k^{(i)}, \end{align}\] where \(\boldsymbol{b}_z\) and \(\boldsymbol{b}_h\) are learnable bias terms, \(\odot\) denotes the Hadamard product, and \(\sigma\) is the sigmoid function.

4.2 Training objective↩︎

The model is trained to minimize a combined objective of reconstruction and prediction losses. Node-level latent variables \(\mathbf{z}_i\) are inferred from input features \(\mathbf{X}_i\) using an encoder module \(\mathop{\mathrm{\mathsf{Enc}}}\). To ensure the representation captures structural data properties and relevant forecasting features, the total loss \(\mathcal{L}\) is defined as \[\mathcal{L} = \|\mathop{\mathrm{\mathsf{Rec}}}(\mathbf{Z}) - \mathbf{X}\|^2 + \|\mathop{\mathrm{\mathsf{Pred}}}(\mathbf{Z}) - \mathbf{Y}\|^2\] where \(\mathop{\mathrm{\mathsf{Rec}}}\) and \(\mathop{\mathrm{\mathsf{Pred}}}\) denote the reconstruction and prediction heads, respectively. Note that the decoder heads are solely used during training to shape the latent space and are discarded during inference.

4.3 Aggregating node- and scene-level representations↩︎

Let \(\mathbf{z}_{s,i}\in\mathbb{R}^{L}\) denote the representation of agent \(i\) in scene \(s\), where scene \(s\) contains \(n_s\) agents. The scene-level mean and covariance are computed as \[\boldsymbol{\mu}_s = \frac{1}{n_s} \sum_{i=1}^{n_s}\mathbf{z}_{s,i}, \qquad \boldsymbol{\Sigma}_s = \frac{1}{n_s-1} \sum_{i=1}^{n_s} \left(\mathbf{z}_{s,i}-\boldsymbol{\mu}_s\right) \left(\mathbf{z}_{s,i}-\boldsymbol{\mu}_s\right)^{\mathsf T}.\]

The aggregated dataset-level mean and covariance for a dataset \(\mathcal{D}\) consisting of \(S\) scenes are defined as \[\label{eq:total95covariance} \bar{\boldsymbol{\mu}}_\mathcal{D} = \frac{1}{S} \sum_{s=1}^{S}\boldsymbol{\mu}_s, \qquad \boldsymbol{\Sigma}_{\mathcal{D}} = \underbrace{ \frac{1}{S} \sum_{s=1}^{S} \left(\boldsymbol{\mu}_s-\bar{\boldsymbol{\mu}}_{\mathcal{D}}\right) \left(\boldsymbol{\mu}_s-\bar{\boldsymbol{\mu}}_{\mathcal{D}}\right)^{\mathsf T} }_{\text{Between-scene cov.}} + \underbrace{ \frac{1}{S} \sum_{s=1}^{S}\boldsymbol{\Sigma}_s }_{\text{Within-scene cov.}}.\tag{1}\] The first covariance term captures variation between scene-level means, whereas the second captures the average variation among agents within each scene. Together, these terms provide a characterization of the dataset’s latent structure and enable direct comparisons between datasets using probabilistic measures.

However, the empirical covariance may be ill-conditioned or singular, particularly in high-dimensional latent spaces with limited data. To improve numerical stability, a low-rank approximation with additive jitter regularization is used. Let \(\boldsymbol{\Sigma}_{\mathcal{D}} = \mathbf{U}\Lambda \mathbf{U}^\top\) denote the eigendecomposition of the covariance matrix, where the columns of \(\mathbf{U}\) are the eigenvectors and \(\Lambda\) is a diagonal matrix containing the corresponding eigenvalues in descending order. The regularized covariance is defined as \[\widetilde{\boldsymbol{\Sigma}}_{\mathcal{D}} = \mathbf{U}_r \operatorname{diag} \left( \lambda_1,\ldots,\lambda_r \right) \mathbf{U}_r^\top + \varepsilon \mathbf{I}, \qquad \varepsilon = \alpha \frac{\operatorname{tr}(\boldsymbol{\Sigma}_{\mathcal{D}})}{L},\] where \(\mathbf{U}_r\) contains the eigenvectors associated with the \(r\) largest eigenvalues. The jitter parameter \(\varepsilon\) is scaled according to the average marginal variance, where \(\alpha\) is a hyperparameter set to \(3 \times 10^{-3}\) and \(r\) is set to \(16\), both values chosen based on empirical results (see the supplementary material for more details).

5 Evaluation and results↩︎

This section evaluates the proposed framework. The datasets are introduced in 5.1, followed by implementation details in 5.2. The embedding model is assessed in 5.3 through complementary quantitative and qualitative latent space analyses. 5.4 investigates how well latent distances predict cross-dataset generalization under zero-shot and fine-tuning settings. Finally, an ablation on the latent dimension \(L\) analyzes its influence on transferability prediction accuracy.

5.1 Datasets↩︎

The investigations are conducted on a diverse set of trajectory-prediction datasets spanning pedestrian, vehicle, and mixed-traffic domains, covering both map-based and map-free settings, and collected using static cameras, drones, or instrumented vehicles. An overview is provided in 1, where datasets are grouped by collection method and listed chronologically by release year.

All datasets are converted to a unified format using the Dronalize toolbox [28], including resampling to 10 Hz and adopting a common feature representation. Official train/validation/test splits are used when available; otherwise, the splitting protocol in Dronalize is applied. For cross-dataset evaluation, all models are evaluated using a common 3 s prediction horizon, determined by the shortest fixed horizon among datasets with official test splits (, View-of-Delft). During training, however, models predict up to a maximum horizon of 8 s, with supervision provided where future trajectories are available.

Following common practice [43], [60], the ETH and UCY datasets are organized into the five standard evaluation subsets: eth, hotel, univ, zara1, and zara2. Each subset is treated as a separate dataset throughout the experiments. Accordingly, the evaluation comprises 24 datasets in total.

5pt

5.2 Implementation details↩︎

The embedding model was implemented in PyTorch [103] and PyTorch Geometric [104]. It was trained on a single NVIDIA A100 GPU using the AdamW optimizer [105] with a batch size of \(512\). To mitigate the imbalance in dataset sizes when jointly training on all datasets, mini-batches were constructed using weighted sampling, with each sample \(j \in \mathcal{D}_i\) assigned the weight \[w_j \propto \frac{1}{|\mathcal{D}_i|^\alpha},\] where \(\mathcal{D}_i\) denotes the \(i\)-th dataset. Here, \(\alpha=0\) corresponds to sampling proportional to dataset size and \(\alpha=1\) to uniform sampling across datasets. A value of \(\alpha=0.5\) is used, reducing the dominance of larger datasets while preserving a moderate bias toward them. Training was conducted for \(100\) epochs, with an initial learning rate of \(10^{-3}\), which was decayed to \(10^{-5}\) using a cosine annealing schedule [106]. Teacher forcing was used for both heads during the first 25 epochs, with the probability annealed linearly from 1 to 0. The final models used in all the experiments were selected based on minimum ADE performance on the validation set.

5.3 Embedding evaluation↩︎

In the latent space, each dataset is modeled as a multivariate Gaussian distribution defined by its dataset-level mean and covariance. Let \(\mathcal{I}=\{1,\dots,K\}\) index the datasets \(\mathcal{D}_1,\dots,\mathcal{D}_K\). For \(i,j\in\mathcal{I}\), the directed dissimilarity of \(\mathcal{D}_i\) from \(\mathcal{D}_j\) is defined as the KL divergence between their corresponding Gaussian distributions: \[\label{eq:kl} D_{\text{KL}}(\mathcal{D}_i \| \mathcal{D}_j) = D_{\text{KL}}\!\left( \mathcal{N}\!\left( \bar{\boldsymbol{\mu}}_{\mathcal{D}_i}, \widetilde{\boldsymbol{\Sigma}}_{\mathcal{D}_i} \right) \,\middle\|\, \mathcal{N}\!\left( \bar{\boldsymbol{\mu}}_{\mathcal{D}_j}, \widetilde{\boldsymbol{\Sigma}}_{\mathcal{D}_j} \right) \right), \qquad i,j\in\mathcal{I},\tag{2}\] where \(D_{\text{KL}}(\mathcal{D}_i \| \mathcal{D}_j)\) measures how well the distribution of dataset \(\mathcal{D}_j\) approximates that of dataset \(\mathcal{D}_i\). Since the KL divergence is asymmetric, the resulting dissimilarity captures directional differences in dataset coverage and variability.

3 presents a heatmap of the KL divergence between embedding distributions for each dataset pair, where lower values indicate greater latent similarity. Several interesting patterns emerge. For example, nuScenes appears easier to approximate using other instrumented-vehicle datasets such as Lyft L5, WOMD, Argoverse, and Argoverse 2 than vice versa. For instance, \[\begin{align} D_{\text{KL}}(\text{nuScenes} \| \text{WOMD}) = 358, \quad D_{\text{KL}}(\text{WOMD} \| \text{nuScenes}) = 36602, \end{align}\] indicating that these datasets contain information that transfers to nuScenes more effectively than nuScenes transfers to them. This asymmetry aligns with the empirical findings in both [27] and [86], where models pretrained on WOMD, Argoverse, or Argoverse 2 and evaluated on nuScenes achieve notably stronger performance than the reverse. The t-SNE [107] projection of scene latents in 4 (a) supports this: nuScenes occupies a distinct subregion within the other datasets’ broader clusters. This indicates that the asymmetric relationship observed in the high-dimensional latent space is also reflected in a low-dimensional projection.

Highway datasets are more similar to each other than to urban datasets, though asymmetrically. The t-SNE projection in 4 (b) clearly shows overlap among highD, exiD, and A43, all collected in Germany, likely reflecting regional driving patterns. Furthermore, AD4CHE, I-80, and US-101 cluster together, suggesting the latent space encodes shared behavioral or traffic-flow characteristics beyond geographic proximity.

Another important observation is the large divergence between the pedestrian-focused datasets and the rest. The results indicate that the pedestrian datasets are substantially different from the others, suggesting low potential for direct transferability. Partial exceptions are uniD and View-of-Delft, which are heavily pedestrian-oriented and appear to be closer in the embedding space.

Most interestingly, the embedding space also uncovers relationships between datasets that do not share presumed commonalities, such as acquisition method or agent type. For instance, several drone-acquired datasets, including inD, openDD, SIND, INTERACTION, exhibit low KL divergence with instrumented-vehicle datasets such as nuScenes, Argoverse 2, and WOMD. This suggests that cross-domain transfer is possible even between heterogeneous datasets. This is significant, as trajectory-prediction research often suffers from scarce, domain-specific data [3], indicating untapped potential in leveraging datasets that are comparatively underused in the motion prediction literature.

5.4 Transferability evaluation↩︎

To evaluate the interpretation of closeness in the latent space as a measure of transferability, a series of experiments is conducted in which a trajectory-prediction model is trained on one dataset and evaluated on another. Motivated by its state-of-the-art performance and its architectural dissimilarity to the proposed embedding model, QCNet [65] is adopted as the baseline predictor, following the original training objectives and inference procedure. To ensure comparability across datasets, a lightweight input transformation layer is introduced, and model capacity is reduced to maintain computational feasibility across all datasets (see the supplementary material for more details). Model performance is reported using the minimum average positional displacement error for the best of \(K = 6\) predicted trajectories (minADE\(_6\)) over a 3-second horizon.

5.4.1 Zero-shot transferability↩︎

To evaluate cross-dataset generalization, a model is trained on one dataset and is evaluated, without any adaptation, on all remaining datasets. With \(24\) datasets, this yields \(24 \times 23 = 552\) transfer pairs.

Using the divergence introduced in 2 , it is investigated how well this measure predicts zero-shot transfer performance. To this end, minADE\(_6\) is examined against \(D_{\text{KL}}(\mathcal{D}_e \| \mathcal{D}_t)\), where \(\mathcal{D}_e\) and \(\mathcal{D}_t\) denote evaluation and training datasets, respectively. As shown in 5, pairs with larger divergence exhibit consistently higher minADE\(_6\), indicating poorer transfer.

A Spearman’s rank correlation analysis confirms a strong positive association: increasing divergence aligns with reduced zero-shot performance. Overall, the KL divergence achieves a rank correlation of \(\rho = 0.811\), with a \(95\%\) confidence interval of \((0.782,\,0.840)\).

5.4.2 Transferability under fine-tuning↩︎

Beyond zero-shot performance, it is important to understand how datasets support transfer when used for fine-tuning toward a target domain, as this has direct practical utility. This is studied from two complementary perspectives: whether the latent divergence can guide pretraining source selection, and whether it predicts how much source knowledge is lost during adaptation.

5.4.2.1 Pretraining source selection.

As a practical case study, Argoverse is used as the target dataset. Models pretrained on a set of candidate source datasets are subsequently fine-tuned on Argoverse, and target performance is recorded after adaptation. As shown in 6, sources with lower KL divergence to Argoverse generally yield stronger post-fine-tuning performance on the target domain, indicating that the latent metric can serve as a practical guide for pretraining source selection, even before any task-specific training is conducted.

5.4.2.2 Catastrophic forgetting.

A complementary concern when fine-tuning is catastrophic forgetting [108]: the tendency for adaptation to a new domain to overwrite previously acquired source knowledge. To examine this, models pretrained on a source dataset \(\mathcal{D}_i\) are fine-tuned on a target dataset \(\mathcal{D}_j\), and performance is subsequently evaluated on \(\mathcal{D}_i\). Let \(m_{\text{org}}^{\mathcal{D}_i}\) and \(m_{\text{ft}}^{\mathcal{D}_j}\) denote the original and fine-tuned models, respectively. The performance change is defined as: \[\Delta M(\mathcal{D}_i, \mathcal{D}_j) = M(m_{\text{ft}}^{\mathcal{D}_j}, \mathcal{D}_i) - M(m_{\text{org}}^{\mathcal{D}_i}, \mathcal{D}_i),\] where \(M(\cdot, \cdot)\) denotes minADE\(_6\) evaluated on a specific dataset. Lower \(\Delta M\) indicates less forgetting, and it is expected to correlate with \(D_{\text{KL}}(\mathcal{D}_i \| \mathcal{D}_j)\).

Experiments across five datasets, AD4CHE, nuScenes, rounD, View-of-Delft, and WOMD, produce \(20\) transfer pairs. [fig:forgetting-kl-finetune-fig] indicates a positive trend: larger divergence between datasets generally coincides with stronger forgetting. Spearman’s rank correlation is \(\rho=0.729\), with a \(95\)% confidence interval of \((0.403,\,0.913)\), confirming a strong positive association. Taken together, the two results show that the latent divergence is predictive of fine-tuning behavior from both directions: it anticipates adaptation quality before training, and captures the degree of source knowledge loss afterward.

5.4.3 Motivation for the Gaussian assumption↩︎

To assess the Gaussian modeling assumption, 2 compares the predictive performance of the KL divergence with alternative similarity measures, including the non-parametric MMD [109] computed directly on scene-level embeddings. The KL divergence consistently achieves the highest rank correlation across all evaluated latent dimensionalities in both zero-shot and fine-tuning tasks, empirically justifying the Gaussian approximation.

5.9pt

5.4.4 Ablation on latent dimension size↩︎

Finally, an ablation study examines how the latent dimensionality \(L\) influences the predictive quality of the learned embeddings. Models with \(L \in \{16, 32, 64, 128\}\) are evaluated on both zero-shot transfer and post-fine-tuning degradation tasks. As shown in 2, the two tasks exhibit opposing trends. For zero-shot transfer, the correlation decreases from \(\rho=0.811\) at \(L=32\) to \(\rho=0.736\) at \(L=128\), indicating that the more compact representation better preserves the ordering of cross-dataset transfer performance. In contrast, the correlation for post-fine-tuning degradation increases from \(\rho=0.729\) to \(\rho=0.874\), suggesting that larger latent spaces may better capture differences associated with adaptation and forgetting. However, the wide and overlapping confidence intervals, together with the smaller number of fine-tuning pairs, warrant a more cautious interpretation of this trend.

5.5 Investigation into latent space quality↩︎

To evaluate whether latent dataset embeddings capture complex discrepancies beyond basic statistical heuristics, transfer performance \(T_{i,j}\) from source dataset \(\mathcal{D}_i\) to target \(\mathcal{D}_j\) is modeled using the KL divergence and a set of candidate explicit heuristic metrics \(\gamma^{(\ell)}_{i,j}\) \[\label{eq:linear95model} T_{i,j} \approx a\,D_{\mathrm{KL}}(\mathcal{D}_j \| \mathcal{D}_i) + \sum_{\ell \in \mathcal{S}} b_{\ell}\,\gamma^{(\ell)}_{i,j} + c.\tag{3}\] Here, \(\mathcal{S}\) encompasses five dataset differences, with definitions and correction ratios detailed in 3. To isolate the most informative predictors, this model is optimized using Lasso regularization, applying an \(L_1\) penalty to the coefficients \(a\) and \(b_{\ell}\). Formally, the parameters \(\theta = \{a, \{b_{\ell}\}_{\ell \in \mathcal{S}}, c\}\) are estimated by minimizing \[\min_{\theta}\; \sum_{i,j} \Bigl( T_{i,j} - a\,D_{\mathrm{KL}}(\mathcal{D}_j \| \mathcal{D}_i) - \sum_{\ell \in \mathcal{S}} b_{\ell}\,\gamma^{(\ell)}_{i,j} - c \Bigr)^{2} + \lambda \Bigl( |a| + \sum_{\ell \in \mathcal{S}} |b_{\ell}| \Bigr).\] This formulation enforces sparsity. Coefficients driven to zero correspond to predictors that do not contribute substantially to explaining transfer performance.

4.2pt

The regression results in 7 demonstrate the representational capacity of the learned latent space. In 7 (b), the KL divergence coefficient remains nonzero for much higher values of the regularization strength \(\lambda\) compared to the other coefficients, indicating that the learned representation captures these underlying patterns.

The exception is average agent speed, as its coefficient remains nonzero under stronger regularization. Nevertheless, using the speed factor by itself achieves a lower rank correlation, \(\rho=0.624\), with a \(95\%\) confidence interval of \((0.561,\,0.669)\), compared with \(\rho=0.811\) for the KL divergence, with a \(95\%\) confidence interval of \((0.782,\,0.840)\). This further indicates that the latent representation captures discrepancies that are not adequately described by the considered statistical heuristics.

6 Conclusion↩︎

A latent embedding framework that systematically characterizes trajectory datasets and quantifies their transferability was introduced. The framework reveals structured relationships between datasets and offers guidance on selecting effective pretraining sources. Empirical results across 24 diverse datasets show that distances in the latent space strongly correlate with both zero-shot generalization and fine-tuning outcomes, demonstrating that transferability can be predicted rather than discovered through exhaustive experimentation. The analysis further identifies datasets that are less commonly used in prior work but still hold potential to improve performance on widely used motion-prediction benchmarks. These findings offer practical guidance for dataset selection and model development, contributing toward more robust and generalizable motion prediction systems.

Acknowledgments↩︎

This research was supported by the Strategic Research Area at Linköping-Lund in Information Technology (ELLIIT) and the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. Computations were enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.

7 Latent embedding model↩︎

This section expands on the latent embedding model and provides additional details on its architecture and hyperparameters.

7.1 Network configuration↩︎

As shown in 2, the latent embedding model comprises five main components:

1. GGRU encoder for agent features and interactions

2. GNN-based map encoder for processing the lane graph

3. Feed-forward module that fuses agent and map features into the latent embedding \(\mathbf{Z}\)

4. GGRU decoder head for reconstructing the input

5. GGRU decoder head for trajectory forecasting

All modules use a hidden dimension of \(128\), and all graph operations rely on the GNN operator of [102].

Following a simple embedding layer that includes agent type information, the GGRU encoder processes agent features sequentially for each instance, performing message passing between neighboring agents. The map encoder operates in two stages. First, it performs message passing on the lane-graph to compute node-level map embeddings. The raw map inputs consist of node positions and dataset-defined lane-graph connectivity. Edge features include the map-node type specified by the Dronalize toolbox [28]; these are one-hot encoded, projected through a linear layer, and processed by a three-layer GNN. Second, a directed three-layer GNN propagates information from map nodes to agents. The resulting map features are then passed through a LayerNorm layer [110].

The outputs of the GGRU encoder and the map encoder are concatenated and passed through a two-layer feed-forward network with SiLU activations [111] to produce the latent embedding \(\mathbf{Z}\), followed by normalization. This embedding initializes the hidden state of the two GGRU decoder heads, which autoregressively reconstruct the input and predict future trajectories. At each step, the decoders take their own previous prediction, transformed by a two-layer feed-forward network with SiLU activations, as the next input.

7.2 Graph construction↩︎

Graphs are constructed on a per-sample basis using heuristic rules tailored to capture relevant spatial and semantic structure. Inter-agent graphs are built by connecting each agent to others within a 150 m radius, limited to a maximum of 10 neighbors per node. All edges are undirected, with edge weights computed as the exponential of the negative relative distance. Map-agent graphs are constructed from the lane graph, if available, which is initially represented as a set of polylines with distinctive types. These polylines are discretized into graph nodes by sampling points at approximately 3 m intervals. Agents are then connected to nearby map nodes using a nearest-neighbor criterion, allowing up to 50 directed edges from map nodes to agents. For both graph types, edge dropout [112] is applied during training with a probability of 0.25.

8 Trajectory prediction model↩︎

QCNet [65] serves as the baseline predictor for the transferability experiments. While the original implementation is largely followed, a few modifications were made to adapt it to this work.

12pt

8.1 Network configuration↩︎

The only functional modification concerns the map encoder, which was adapted to match the input format defined by the Dronalize toolbox [28]. In practice, this required adjusting the architecture to accommodate a different set of map-node types and edge definitions. To reduce computational demands and training time across the extensive experimental suite, several hyperparameters of the original QCNet model were scaled down. 4 summarizes the modified settings alongside the original values. These adjustments lower the model size from roughly 7.66 million parameters to 1.35 million, yielding substantial reductions in memory usage and training duration. Initial studies showed that these changes had only minor effects on performance and, where differences appeared, they followed the same trends observed with the full-size model, making the compact configuration sufficient for the purposes of the transferability analyses of this work.

8.2 Training details↩︎

Models are trained for up to \(50\) epochs using AdamW [105]. Batch sizes and learning rates are selected separately for each dataset (see 5). Early stopping is applied based on the validation minADE\(_6\), with a patience of \(5\) epochs. Given the number of datasets and total runs, learning-rate tuning is limited. The reported values represent stable, workable settings rather than the result of an extensive hyperparameter search. As for the latent embedding model, the final QCNet models used in all experiments were selected based on minADE\(_6\) performance on the validation set.

3.2pt

9 Experiments↩︎

This section presents additional experiments that complement the main results. These include extended analyses, alternative evaluation settings, and further qualitative examples that provide additional insight into the behavior of the proposed framework.

9.1 Recommender system↩︎

One objective of the latent embedding model is to provide a principled, data-driven mechanism for identifying suitable source datasets for pre-training or adaptation. When a new dataset becomes available, its compatibility with existing sources can be assessed by computing its embedding and comparing it with the existing embeddings.

To illustrate this process, an unseen-data scenario is simulated by training a new embedding model after removing WOMD, exiD, and uniD from the training set. The resulting KL divergence heatmap from this model is shown in 8. Despite the exclusion, the model positions WOMD near Argoverse, uniD near inD, and exiD near highD—relationships that closely match those in the full-training heatmap (3). Although the similarity to the fully trained model may appear unsurprising, it is precisely this consistency that is meaningful: the embedding space recovers the same structural relationships even without direct exposure to the omitted datasets.

This stability suggests that the learned embedding generalizes well to unseen data and that it can reliably support both transferability assessment and source-dataset recommendation.

9.2 Effect of latent dimensionality on correlation dynamics↩︎

Figure 9 illustrates how the latent dimensionality \(L\) influences the evolution of the learned correlations during training. The \(L=32\) model achieves the highest and most stable Spearman’s \(\rho\), while the \(L=16\) model performs slightly worse. Increasing the latent dimension to \(L=64\) or \(L=128\) does not provide additional benefit and results in lower final correlations. In particular, the \(L=128\) model exhibits a downward trend, which may indicate overfitting. These results suggest that increasing the latent capacity beyond \(L=32\) may amplify noise rather than improving the learned representation.

9.3 Transferability results↩︎

For completeness, the full zero-shot transferability results are provided in 10, showing minADE\(_6\) performance over a 3 s prediction horizon. Several patterns emerge. First, transferring from urban sources to highway targets is noticeably more difficult than the reverse.

Although transferring from urban sources to highway targets is challenging, highway prediction itself is inherently easier once trained on in-domain data. This is reflected in the relatively low displacement errors observed for highway-to-highway transfer (, exiD, highD, A43). In contrast, urban-to-urban transfer proves substantially more difficult, with datasets such as rounD and openDD showing markedly higher errors despite the within-domain setting. This aligns with prior empirical findings [46], which similarly report that highway scenarios are generally easier to predict.

One row that stands out is WOMD, which appears to be one of the most challenging target datasets. Only Argoverse and Lyft achieve reasonably competitive zero-shot performance. On the other hand, WOMD is a strong source dataset, offering solid transfer to many targets. Lyft and Argoverse 2 show similar behavior. As noted in the main text, these datasets are also among the largest in the study, covering broad regions of the latent space and thus providing strong potential for pre-training.

Finally, several datasets transfer surprisingly well to the pedestrian-centric ones. This should, however, be interpreted with caution, as these are predominantly low-speed scenarios where best-of-\(K\) metrics tend to overestimate performance [113]–[115]. Regardless, it is noteworthy that pre-training on highway datasets can serve as an effective starting point for zero-shot or fine-tuned pedestrian prediction.

9.4 Single-source selection for zero-shot transfer↩︎

To assess the practical utility of the proposed KL-based similarity measure, a single-source selection experiment is conducted using the zero-shot transfer matrix in 10. For each target dataset, one source dataset is selected according to each selection rule. The oracle source is defined as the candidate source achieving the lowest minADE\(_6\) on the target dataset.

The quality of a selected source is evaluated using four metrics. First, its rank is determined by ordering all candidate sources according to their actual minADE\(_6\) on the target dataset. Second, the top-3 rate measures the fraction of targets for which the selected source is among the three best-performing candidate sources. Third, the absolute performance gap is computed as the difference between the minADE\(_6\) obtained using the selected source and that obtained using the oracle source. Finally, the relative gap normalizes this difference by the oracle minADE\(_6\). The reported values are averaged across the target datasets.

As shown in 6, the KL-based rule selects one of the three best-performing sources for \(83.3\%\) of the target datasets and achieves a mean selected-source rank of \(3.00\). It also produces the smallest mean absolute and relative minADE\(_6\) gaps, outperforming selection based on mean speed, source-dataset size, and random sampling. These results indicate that the proposed similarity measure can provide a useful criterion for selecting a source dataset when target-domain performance measurements are unavailable.

The experiment is restricted to single-source zero-shot transfer and does not consider multi-source training or target-domain fine-tuning. The effectiveness of the proposed measure in these more general and complex transfer settings therefore remains to be investigated.

5.3pt

9.5 Covariance estimation ablation↩︎

The covariance estimation ablation considers two design choices: the source of the covariance matrix and the estimator applied to it. The covariance source is either the within-scene covariance, the between-scene covariance, or their sum, with the latter defined in 1 . The evaluated estimators range from the raw empirical covariance to regularized and low-rank approximations.

9.5.0.1 Covariance estimators.

Consider the decomposition of the empirical covariance matrix, \[\boldsymbol{\Sigma} = \mathbf{U}\boldsymbol{\Lambda}\mathbf{U}^{\mathsf T},\] where \(\mathbf{U}\) contains the eigenvectors and \(\boldsymbol{\Lambda} =\operatorname{diag}(\lambda_1,\ldots,\lambda_d)\) contains the corresponding eigenvalues. The four covariance estimators are defined as \[\begin{align} {2} &\widehat{\boldsymbol{\Sigma}}_{\mathrm{raw}} \quad &&= \boldsymbol{\Sigma}, \\ &\widehat{\boldsymbol{\Sigma}}_{\mathrm{jitter}} \quad &&= \boldsymbol{\Sigma} + \frac{\mathrm{tr}(\boldsymbol{\Sigma})}{L}\mathbf{I}, \\ &\widehat{\boldsymbol{\Sigma}}_{\mathrm{floor}} \quad &&= \mathbf{U} \operatorname{diag}\!\bigl( \max\{\lambda_1,\alpha\}, \ldots, \max\{\lambda_d,\alpha\} \bigr) \mathbf{U}^{\mathsf T}, \\ &\widehat{\boldsymbol{\Sigma}}_{\mathrm{LR}} \quad &&= \mathbf{U}_r \operatorname{diag}(\lambda_1,\ldots,\lambda_r) \mathbf{U}_r^{\mathsf T} + \alpha \frac{\mathrm{tr}(\boldsymbol{\Sigma})}{L} \mathbf{I}, \end{align}\] where \(r\) denotes the retained rank and \(\alpha\) controls the regularization strength.

Performance was assessed primarily using Spearman’s rank correlation coefficient, while the median KL divergence was used as a secondary diagnostic of distributional discrepancy and sensitivity to covariance regularization. For each latent dimension, the selected configuration was defined as the one with the lowest median KL among those retaining at least 99% of the maximum correlation.

7 compares the maximum-correlation and selected configurations. All optima use a low-rank estimate of the summed covariance source. The selected configurations preserve near-maximal correlation while generally exhibiting substantially lower median and upper-tail KL divergence. At 32 dimensions, the selected configuration retains 99.3% of the maximum correlation while reducing the median and 95th-percentile KL divergences by factors of approximately 875 and 596, respectively.

4pt

11 shows the relationship between correlation and median KL under the selection convention of maximizing correlation while preferring lower KL among near-optimal configurations. Each frontier contains configurations that are nondominated within the same latent dimension; moving along a frontier changes the balance between correlation and KL. The 32-dimensional frontier extends to the highest correlation and contains a strongly regularized solution close to its maximum-correlation endpoint. Specifically, the selected configuration achieves \(\rho=0.811\), compared with the maximum value of 0.817, while reducing the median KL by approximately \(875\times\).

Overall, the ablation reveals a clear interaction between covariance estimation and latent dimension. The best correlation is obtained at 32 dimensions, while both lower and higher-dimensional representations perform worse. This non-monotonic trend suggests that 32 dimensions provide a favorable balance between representational capacity and reliable covariance estimation. Since the selected 32-dimensional configuration preserves 99.3% of the maximum correlation while avoiding the extreme KL values observed for the weakly regularized estimate, it was adopted as the main covariance configuration.

References↩︎