1 Introduction↩︎

Traffic simulation is essential to autonomous vehicle (AV) development, enabling scalable and repeatable evaluation in controlled settings. It supports systematic validation under diverse driving conditions, shortens development cycles, and provides a safe platform for training and assessment. A key requirement is controllability: the ability to deliberately induce targeted behavior patterns that probe specific system responses. Many traffic scenes are non-interactive and trivial, offering limited insight into failure modes. With controllability, simulation enables the construction of counterfactual “what-if” scenarios that expose potential weaknesses and improve diagnostic effectiveness.

Despite its importance, enabling controllable traffic simulation remains challenging. Prior approaches typically fall into two paradigms: inference-time guidance and direct conditioning through retraining. Inference-time methods, such as diffusion via optimization or gradient-based control during sampling [1]–[3], offer flexible guidance. However, such approaches often rely on carefully designed objectives and introduce additional computational overhead at inference. Alternatively, direct conditioning enables models [4]–[7] to internalize control signals during training, often yielding stable and faithful behavior. However, this comes at the cost of large-scale paired data and expensive retraining for each new control modality. Also, most existing evaluations focus on alignment with ground-truth trajectories, leaving the ability to generate controllable counterfactual scenarios across diverse contexts less explored.

In this work, we introduce a data-efficient fine-tuning framework that enables controllable traffic simulation using traffic simulation models pretrained on large-scale data, hereafter referred to as traffic models. We freeze the base model and learn lightweight control adapters that inject conditioning signals into intermediate features, enabling efficient adaptation to new control modalities with limited paired control data. The adapters are trained under closed-loop simulation to enable controllability while preserving long-horizon interaction dynamics. Our framework supports multiple control modalities. Trajectory sketches provide precise spatial guidance, latent behavior embeddings extracted by our BehaviorVAE capture driving patterns learned from data, and natural language offers an intuitive user interface for specifying high-level behaviors. The same adaptation mechanism generalizes across different model paradigms, including diffusion-based and autoregressive traffic models. Finally, we introduce a context-aware retrieval pipeline that transfers learned behaviors into compatible scenes, enabling counterfactual and long-tail scenario generation.

We evaluate on the Waymo Open Sim Agents Challenge (WOSAC) [8] under closed-loop simulation. Across both diffusion-based and autoregressive models, our lightweight adapters reduce control error by up to 83% while improving closed-loop realism by 0.02–0.03 absolute points in the WOSAC Meta score (see Sec. 4.2). Remarkably, these gains are achieved using as little as 0.01%–1% paired fine-tuning data. Despite this data efficiency, our method matches or surpasses fully fine-tuned baselines trained on substantially larger datasets [5].

Our main contributions can be summarized as follows:

• We propose a data-efficient control adaptation framework that adds multi-modal controllability (sketch, latent, and text) to both diffusion and state-of-the-art autoregressive traffic models.

• We demonstrate strong sample efficiency. With identity-initialized FiLM adapters, our method achieves competitive controllability using minimal paired supervision (as few as 500 scenarios, 0.1% of the paired control data) while keeping the pretrained backbone frozen.

• The proposed context-aware condition transfer mechanism enables counterfactual and long-tail scenario generation while preserving closed-loop realism.

2 Related Work↩︎

Controllable Traffic Scenario Generation. Recent work has made significant progress in building data-driven, realistic traffic models that support multi-agent interactions in closed-loop environments [9]–[11]. Recent autoregressive (AR) and reactive traffic models achieve strong closed-loop realism by modeling traffic generation as sequential next-token prediction [12]–[14]. However, these models are primarily designed for unconditional simulation and provide limited mechanisms for controllable scenario synthesis, which is crucial for structured, target-case testing [15], [16].

Several works explore the generation of controllable traffic scenarios to synthesize targeted or safety-critical situations [3], [15], [17]. Existing approaches can be broadly categorized into three mechanisms: (1) Inference-time guidance steers unconditional samples during generation via gradient-based or constraint-based objectives [1], [18]. While flexible, it often introduces significant inference overhead. (2) Direct conditioning trains generative models from scratch to internalize explicit control signals (e.g., goal specifications, scene constraints or language descriptions) [4]–[7]. However, introducing new modalities requires expensive retraining. (3) Retrieval-based control guides generation using similar scenarios retrieved from large datasets, such as RealGen [19]. In contrast, we adapt pretrained traffic models to support new control modalities without retraining the backbone. Our framework further extends retrieval-based guidance by introducing multiple control interfaces, including behavior latents learned via BehaviorVAE, enabling context-aware control within closed-loop simulation.

Control Adaptation in Generative Models. Recent works in image and video synthesis explore adapting pretrained generative models to enable controllable generation. Rather than retraining large conditional models from scratch, these approaches augment pretrained backbones with mechanisms that incorporate new conditioning signals. For example, ControlNet [20] introduces dedicated control branches connected via zero-initialized convolutions that progressively learn to modulate a pretrained diffusion model in response to structural inputs such as edges or poses. Similarly, LoRA [21] adapts pretrained models by injecting low-rank updates into existing weight matrices, enabling new capabilities while preserving the original model parameters. These works demonstrate that powerful generative models can be extended to new control modalities by leveraging pretrained generative priors.

Despite their success in visual generation, such control adaptation strategies remain largely unexplored in traffic modeling, where preventing compounding errors requires closed-loop training. We introduce a control adaptation framework optimized under closed-loop simulation to enable controllability while preserving the behavior of pretrained traffic models.

3 Method↩︎

3.1 Problem Formulation↩︎

Let \(\pi_{\textrm{ref}}\) denote a model pre-trained on large-scale, unconditional driving data that generates future trajectories in a closed-loop manner given a scene context \(c\). Our goal is to extend this model to conditional generation, \(\pi_{\theta}(\mathbf{S}_{t:t+T} \mid c, r)\), where \(r\) denotes a control signal (e.g., trajectory sketches, latent behavior codes, or language descriptions) and to enable controllable multi-agent generation in a data-efficient manner. The notation \(\mathbf{S}_{t:t+T}\) denotes the joint multi-agent future states over a prediction horizon. The generator should control designated agents according to \(r\) while preserving realistic interactions of the pre-trained model \(\pi_{\textrm{ref}}\). For each control modality \(m\), we assume access to a small control-annotated dataset \(\mathcal{D}_m \subset \mathcal{D}\) and aim to adapt the pretrained traffic models with limited supervision.

3.2 Control Adaptation via Feature Modulation↩︎

Given a raw control input \(r_m\) from modality \(m\), we first map it to a unified \(d\)-dimensional conditioning embedding \(z \in \mathbb{R}^d\) via a modality-specific encoder (where \(d=256\) in our implementation). For each agent \(i\), only targeted agents receive their corresponding \(z_i\), while non-targeted agents are assigned \(z_i = \mathbf{0}\).

To enable controllable generation while preserving the pretrained traffic model prior, we inject the conditioning embedding \(z_i\) into intermediate representations of the frozen backbone through lightweight feature modulation. We adopt Feature-wise Linear Modulation (FiLM) [22], which conditions intermediate activations via an affine transformation. For an intermediate feature map \(\mathbf{h}^{(l)} \in \mathbb{R}^{D}\) at the \(l\)-th decoder layer (where \(D\) is the hidden dimension of the backbone), FiLM applies: \[\mathrm{FiLM}(\mathbf{h}^{(l)}; z_i) = \boldsymbol{\gamma}^{(l)}(z_i) \odot \mathbf{h}^{(l)} + \boldsymbol{\beta}^{(l)}(z_i), \label{eq:film95multiply}\tag{1}\] where \(\boldsymbol{\gamma}^{(l)}, \boldsymbol{\beta}^{(l)} \in \mathbb{R}^{D}\) are layer-specific modulation functions (implemented as lightweight MLPs) predicted from \(z_i\).

A key design principle of our framework is extensibility: pretrained traffic models should be incrementally augmented with new controllability without retraining or destabilizing the original model. To this end, we initialize the modulation layers near identity (\(\boldsymbol{\gamma}^{(l)} \approx \mathbf{1}\), \(\boldsymbol{\beta}^{(l)} \approx \mathbf{0}\)), ensuring that the adapted model initially approximates the base model. This identity-preserving initialization is particularly important in closed-loop traffic simulation, where small perturbations can compound over long horizons and disrupt multi-agent interactions. Conceptually related to zero-initialized conditioning strategies such as ControlNet [20], this design enables controllability to be learned gradually through lightweight additional parameters while maintaining the backbone’s learned dynamics.

3.3 Modality-Specific Control Representations↩︎

3.3.1 Sketch and Language Conditioning.↩︎

Trajectory sketches provide coarse spatial guidance in the form of future waypoints expressed in the agent’s local frame. We encode the waypoint sequence using a lightweight temporal encoder and aggregate it into a fixed-dimensional embedding \(z_i\), capturing the intended motion trend while allowing reactive interactions.

Following [6], we utilize a DistilBERT [23] model adapted via Low-Rank Adaptation (LoRA) [21] for natural language commands (e.g., “turn left at the intersection”). While language instructions are inherently abstract, we establish spatial and kinematic grounding by jointly optimizing the LoRA parameters with the downstream closed-loop control objective. Specifically, given a text prompt \(p_i\), we extract the language features \(\mathbf{H}_i = \mathrm{BERT}(p_i)\). We pool the [CLS] token embedding and apply a learned linear projection to match the shared \(d\)-dimensional control space.

3.3.2 Context-Aware Behavior Latent.↩︎

Beyond explicit sketches or language commands, we introduce a context-aware behavior latent that captures high-level driving patterns directly from data, without manual annotations. This compact control space abstracts interaction styles such as yielding or merging, enabling scalable and data-efficient controllability (see 4.3).

To learn this representation, we employ a Conditional VAE (CVAE) [24], referred to as BehaviorVAE, which is trained separately from the control adapters. We adopt a CVAE instead of deterministic autoencoder-based approaches [19], [25] because KL regularization encourages a smooth latent space where similar behaviors cluster together. This representation enables strong sample efficiency, as shown in 4.3. Given a traffic scene, the encoder jointly processes all agents’ future motion sequences together with scene context in a single forward pass. By modeling relative geometry and multi-agent dependencies, it infers a latent variable \(z_i\) for each agent that captures structured interaction behaviors in a context-aware manner (see 2).

Formally, the encoder produces a Gaussian latent distribution \(q_\phi(z_i \mid \boldsymbol{\xi}, c_{\mathrm{env}})\), where \(\boldsymbol{\xi}\) denotes the set of agent trajectories in the scene. Because latent inference is performed jointly at the scene level, the resulting \(z_i\) encodes interaction-aware behavior patterns rather than independent per-agent dynamics. The model is trained using a conditional ELBO objective: \[\mathcal{L}_{\mathrm{VAE}} = \mathbb{E}_{q}[\log p_\psi(\boldsymbol{\xi}_i \mid z_i)] - \beta D_{\mathrm{KL}}(q_\phi(z_i \mid \boldsymbol{\xi}, c_{\mathrm{env}}) \parallel p(z)),\] with KL annealing for stable optimization.

At inference time, sampling \(z_i\) from the prior enables counterfactual behavior generation, and the latent is projected into the shared conditioning space for downstream control.

3.4 Backbone Integration and Closed-Loop Fine-Tuning↩︎

To enable controllable traffic simulation, the proposed adapters must be integrated into existing generative traffic models while preserving their learned generative priors. A key challenge is that the model operates in a closed-loop setting, where small prediction errors can accumulate over time and destabilize multi-agent interactions. We integrate control adapters into the generative cores of both diffusion [2] and autoregressive traffic models [12] while keeping backbone parameters frozen. In both cases, control is injected at the decoder level to directly influence trajectory generation, systematically modulating the representations at every transformer decoder layer output across both the diffusion and autoregressive backbones. Only the control encoders and FiLM layers are updated during closed-loop fine-tuning [6], [13]. By leveraging FiLM-based feature modulation at the decoder level, the same control adaptation mechanism can be applied to different generative backbones without modifying their core architectures.

The key idea of our closed-loop fine-tuning is to bias sampled trajectories toward those aligned with the control target while maintaining stable multi-agent interactions. The exact procedure depends on the generative backbone.

3.4.0.1 Diffusion: Receding-Horizon Control.

Following a “plan–select–execute” paradigm [6], we generate \(M=8\) candidate rollouts at each replanning step and select the candidate closest to the ground-truth trajectory: \[m^* = \operatorname*{argmin}_{m} \mathcal{L}_{\mathrm{match}}(\hat{\mathbf{S}}^{(m)}, \mathbf{S}^{gt}),\] where \(\hat{\mathbf{S}}^{(m)}\) denotes the \(m\)-th predicted joint trajectory and \(\mathbf{S}^{gt}\) is the ground truth. The adapter parameters are updated using the selected rollout to compute the training loss during closed-loop fine-tuning.

3.4.0.2 Autoregressive: Closed-Loop Fine-Tuning.

For the autoregressive (AR) model, trajectories are generated sequentially under control conditioning \(z_i\). We optimize a cross-entropy objective under closed-loop unrolling: \[\mathcal{L}_{\mathrm{AR}} = - \frac{1}{NT} \sum_{i,t} \log \pi_\theta(s_{i,t}^{gt} \mid \hat{\mathbf{S}}_{\le t}, z_i),\] where \(\hat{\mathbf{S}}_{\le t}\) denotes the dynamically generated joint state history up to time \(t\), where \(N\) is the number of agents and \(T\) is the prediction horizon. To reinforce controllability, we bias sampling toward trajectories [13] aligned with the conditioning signal, enabling the adapter to recover from its own prediction errors.

3.5 Context-Aware Scenario Transfer↩︎

In traffic simulation, it is important to explore alternative futures or synthesize rare interactions that deviate from recorded data while remaining realistic. However, directly forcing arbitrary control signals often leads to implausible behaviors in the current traffic context. To address this, we introduce a context-aware scenario transfer pipeline that retrieves compatible agents from similar scenarios and transfers their control signals into the target traffic scene, enabling realistic variations within existing environments.

Specifically, we use context embedding similarity to retrieve compatible agents across scenarios. Given a target agent, we search the dataset for agents whose encoded scene context embeddings are similar (see 3). The retrieved agents serve as sources of plausible behaviors that can be transferred to the target scene.

Using this scenario transfer mechanism, we support two types of controllable scenario generation: (i) Counterfactual Rollouts, where we retrieve context-matched agents representing different driving intentions and transfer their behavior signals (e.g., sketch or latent trajectory) to the target scene to generate alternative futures, and (ii) Long-Tail Synthesis, where we identify rare behaviors using SMART likelihood and transfer their latent behavior codes to context-matched agents in new scenes to synthesize rare interactions.

4 Experiments↩︎

In this section, we demonstrate that our framework enables effective multi-modal controllability for both autoregressive and diffusion traffic models using only small amounts of paired control data, and further show its practical utility for counterfactual traffic simulation.

4.1 Experimental Setup↩︎

Datasets and Benchmark. We evaluate our framework on the Waymo Open Motion Dataset (WOMD) [26], following the standard WOSAC [8] protocol for closed-loop traffic simulation. BehaviorVAE is trained separately on 10,000 WOMD training scenarios.

Backbone Models. To demonstrate the architecture-agnostic nature of our framework, we evaluate two state-of-the-art generative traffic models as unconditional backbones: VBD [2], a diffusion-based model, and SMART [12], an autoregressive model.

Metrics. Following established practices in controllable generation [5], [6], we evaluate along two axes: Realism, measured by the WOSAC Meta score (\(\uparrow\)), evaluates distributional realism using likelihood-based metrics across multiple aspects of driving behavior. Controllability, measured by the minimum Average Displacement Error (mADE \(\downarrow\)) and relative mADE Gain (\(\uparrow\)), which quantify alignment with the intended control signals. For counterfactual scenario synthesis (Sec. 4.4), we additionally report driving quality and safety metrics, including collision and off-road rates, as well as an adapted closed-loop PDMScore [27].

Control Signal Generation. To quantitatively evaluate controllability, we derive control signals from held-out ground truth (GT) trajectories: (i) Sketch: We downsample the GT future trajectories to obtain sparse waypoints as the sketch input; (ii) Latent: We process the GT trajectories through our pre-trained BehaviorVAE encoder to extract latent codes \(\mathbf{z}\); and (iii) Text: We utilize natural language descriptions of each agent’s motion from the ProSim-Instruct-520k dataset [5]. We explicitly note that derived controls are used only for standardized quantitative evaluation (e.g., mADE), not as a deployment assumption. Practical non-oracle controllability is evaluated via retrieval-based sketch/latent/text transfer in Sec. 4.4.

4.2 Controllable Simulation Evaluation↩︎

1 reports controllable simulation results across both diffusion-based (VBD) and autoregressive (SMART) traffic models. To establish robust backbones for controllable simulation, we first convert the open-loop VBD and SMART models into closed-loop traffic models via additional training, denoted as VBD-CL and SMART-tiny-CLSFT [13]. All control adapters in this comparison are trained using only 1% of the paired control data. Despite this limited supervision, our adapters consistently reduce mADE and improve the overall WOSAC Meta score relative to the corresponding unconditional backbones across all three control modalities. Spatially grounded signals provide the strongest gains: Sketch and Latent reduce mADE by approximately 65% on VBD and by up to 83% on SMART. Text conditioning also improves controllability and overall realism, although its control gain is smaller because natural-language instructions provide weaker spatial grounding than trajectory sketches or behavior latents. To isolate the effect of the proposed adaptation mechanism from backbone strength, we apply our FiLM adapters to a retrained unconditional ProSim backbone. As shown in 2, using only 1% of the paired control data achieves sketch controllability comparable to the original ProSim model trained with 100% paired control data. For text conditioning, FiLM adaptation provides substantially stronger controllability despite using 100\(\times\) less paired supervision. These same-backbone results show that the observed data efficiency comes from the proposed adaptation mechanism rather than solely from differences in backbone strength.¹

4pt

4.3 Sample Efficiency and Data Scaling↩︎

In this section, we focus on evaluating the sample efficiency of our proposed adaptation framework. Adapters are trained on logarithmically spaced subsets of the Waymo paired control dataset (50 to 50,000 scenarios; 0.01% \(\sim\) 10% data). Figure 4 illustrates how controllability (mADE \(\downarrow\)) and distributional realism (Meta Score \(\uparrow\)) evolve as the amount of training data increases. For reference, we additionally report the performance obtained by fine-tuning the base model with LoRA on the full \(100\%\) dataset using the same control adapters.

4.3.0.1 Data Efficiency Across Modalities.

Our adapters demonstrate strong sample efficiency across control modalities. Even with extremely small amounts of paired control data, they are able to learn effective controllable behaviors while preserving the generative prior of the frozen base model. Among the evaluated modalities, Latent control is the most data-efficient: with only 0.01% of the training data, it already achieves performance close to the reference model trained with the full dataset. The Sketch modality also performs strongly under limited supervision, approaching the reference performance with merely 0.1% of the data. In contrast, the Text modality exhibits a delayed scaling curve, requiring \({\sim}1\%\) data to outperform the baseline. Because natural language provides abstract guidance and relies on noisy auto-generated captions (e.g., ProSim-Instruct-520k), text control primarily serves as high-level intent biasing rather than precise trajectory supervision.

4.3.0.2 Realism Preservation and Saturation.

Importantly, extremely low-data regimes do not compromise simulation realism. While Meta scores for Sketch and Text dip slightly at \(0.01\%\), they rapidly recover and match the base model by \(0.1\%\) data, while Latent control improves realism from the start. Performance across all modalities saturates around \(10\%\) data, where Sketch, Latent and Text adapters achieve mADE comparable to the reference performance obtained by LoRA fine-tuning on the full \(100\%\) dataset (see 4). This suggests that effective controllable simulation can be achieved without large-scale paired annotations.

4.4 Counterfactual and Long-Tail Scenario Generation↩︎

Having established the sample efficiency of our control adaptation framework, we next evaluate its practical utility for scenario generation using the context-aware scenario transfer pipeline described in 3.5. Specifically, we generate counterfactual and long-tail scenarios within existing traffic scenes. For these experiments, we use logits-level classifier-free guidance during decoding; see Section E.1 of the supplementary material.

4pt

(i) Counterfactual Rollouts: We select 250 query agents and retrieve context-matched agents representing common driving intentions (turning, lane changing, accelerating, decelerating, cruising, stopping). For each query agent, we select up to four distinct intentions with the highest context-match scores and transfer the corresponding behavior annotations (sketch or latent) to the query agent in the original scene to generate counterfactual rollouts.

Evaluation Metrics. For counterfactual generation, we evaluate three aspects: (i) controllability, i.e., how well the transferred behavior controls the target agent, (ii) behavioral diversity, and (iii) driving realism. Controllability is measured using Control ADE, which computes the displacement error between the generated trajectory and the transferred control behavior (e.g., sketch or latent trajectory). We also report the maneuver Success Rate, defined as the fraction of rollouts that successfully execute the same high-level maneuver in each scenario. To evaluate diversity, Coverage measures the spatial distributional spread of generated trajectories, computed as the number of grid cells whose kernel density estimate (KDE) exceeds a predefined threshold [28]. Driving realism is evaluated using Collision, Offroad, and the closed-loop PDMScore [27]. We do not report WOSAC metrics in this setting, as they measure similarity to ground-truth trajectories, which is not appropriate for counterfactual generation where trajectories intentionally deviate from the recorded future.

Table 3 shows that our control adapters enable effective counterfactual generation across modalities. Sketch and latent controls achieve strong controllability, reflected by low Control ADE and high maneuver success rates ( 70%). At the same time, driving realism is preserved, with all modalities maintaining stable safety metrics and achieving PDM scores comparable to or higher than the unconditional baseline. Finally, the generated trajectories exhibit greater diversity: conditioning on different high-level behaviors increases trajectory coverage compared to the unconditional baseline, indicating that our framework can explore multiple plausible futures within the same scene (see Sec. A of the supplementary material for qualitative results).

(ii) Long-Tail Synthesis: Beyond generating alternative futures within a scene, we also evaluate whether our framework can synthesize rare behaviors in new contexts. We use the SMART model’s likelihood from [13] to identify the top \(0.1\%\) hardest-to-predict scenarios in the dataset. From these scenarios, we curate eight long-tail behavior categories, including plaza turning, meandering, threading stopped cars, multi-lane weaving, unsignalized crossing, fork-merge turning, tight merging, and aggressive U-turns, as shown in 5.

Similar to the counterfactual setup, for each behavior, we retrieve the top 30 context-matched candidate agents and transfer the corresponding behavior latent \(\mathbf{z}_{\text{target}}\) to the selected agent while leaving surrounding agents unconditioned. As a baseline, we compare against randomly selecting agents within the scene to receive the transferred behavior. Since long-tail behaviors intentionally deviate from nominal driving, we additionally report Traj ADE to quantify the extent to which the generated trajectory diverges from the unconditional simulation.

Table 4 highlights the importance of context-aware transfer. Random context transfer often places the target behavior in incompatible environments, leading to significantly higher collision rates (\(0.3175\)) and degraded driving performance. In contrast, Context Match better aligns the injected behavior with compatible traffic contexts, reducing collisions (\(0.1574\)) and improving the PDMScore.

3pt

Nevertheless, transferring long-tail behaviors across scenarios remains inherently challenging, as these maneuvers (e.g., tight merges) are rare and highly constrained. We observe that many collision cases result from the trade-off between enforcing the conditioned long-tail behavior and preserving safe interactions with surrounding agents. Finally, the large trajectory deviation (Traj ADE \(=11.30\)m) indicates that the model successfully overrides the nominal unconditional behavior to execute the targeted long-tail interactions.

4.5 Ablation Studies↩︎

We conduct additional ablations to validate our core design choices regarding adapter modulation design and conditioning strategies. Unless otherwise specified, models are trained on a 5,000-scenario subset (\(\sim\)​1% of WOMD) utilizing the autoregressive backbone to ensure rapid iteration while preserving statistical significance.

4.5.1 Adapter Modulation Design.↩︎

We study the architectural design of the control adapter using the Sketch modality. Figure 6 compares our proposed FiLM with Identity Initialization against two baselines: FiLM with random initialization and additive modulation. The additive baseline injects control signals via bias addition, i.e., \(\mathbf{h}^{(l)}_{\text{add}} = \mathbf{h}^{(l)} + \boldsymbol{\beta}^{(l)}(z_i)\).

As shown in 6, identity initialization is critical. Randomly initialized adapters significantly disrupt the pretrained model early in training, causing a severe drop in realism, whereas identity-initialized FiLM preserves the base model’s behavior from the start. Moreover, FiLM is substantially more efficient: while additive modulation fails to converge after 50k steps (\(\sim\)​9 hours on a single RTX 4090), our FiLM adapter reaches better mADE in under 12k steps (\(\sim\)​2.2 hours). These results suggest that identity-initialized multiplicative modulation provides a stronger inductive bias for control-adapter fine-tuning.

5pt

4.5.2 Control Conditioning Strategy.↩︎

We compare two conditioning strategies: (i) Static, which encodes the control signal once and injects it with a time embedding, and (ii) Dynamic, which re-encodes the control signal at each timestep.

As shown in Table 5, dynamic conditioning achieves lower mADE, while static conditioning preserves competitive realism and avoids strict timestep alignment. This flexibility is beneficial in closed-loop simulation, where agents must adapt to surrounding interactions.

5 Limitations and Future Work↩︎

Despite these gains, our method still faces limitations in handling infeasible or ambiguous controls. When a user-specified instruction conflicts with the surrounding traffic context, the model may prioritize control satisfaction over realistic and safe interactions. Future work could address this through RL fine-tuning or guidance-based inference to better balance control fidelity and realism. In addition, our experiments focus primarily on controlling a single agent, and future work will extend the framework toward coordinated multi-agent control.

6 Conclusion↩︎

We present a data-efficient control adaptation framework that enables controllable traffic simulation across multiple conditioning modalities, including sketch, latent behavior codes, and natural language. Built on top of pretrained generative traffic models, our lightweight adapters can be applied to both autoregressive and diffusion-based architectures without modifying the backbone models. Experiments show that our approach achieves strong controllability while preserving simulation realism using only 1% of the paired control data. Beyond standard control evaluations, our framework enables counterfactual scenario generation and long-tail behavior synthesis, producing diverse yet realistic traffic interactions.

7 Acknowledgements↩︎

This work was supported by the National Science and Technology Council under Grants 113-2628-E-A49-022 and 114-2628-E-A49-007, the H&J Global Chair, the Ministry of Education, and the Yushan Fellow Program Administrative Support Grant. W.-J. Chang was supported by the National Science Foundation Graduate Research Fellowship Program under Grant DGE-2146752. The views expressed herein are those of the authors and do not necessarily reflect those of the National Science Foundation. Y.-H. Chen thanks the HCIS Lab at NYCU and the MSC Lab at UC Berkeley for their support.

Compared to prior work, our approach enables data-efficient finetuning to introduce multiple control knobs for generating counterfactual behaviors in closed-loop simulation. Section [sec:app95qualitative] provides qualitative results, Section 9 presents additional baseline comparisons and ablation studies, Section 10 provides discussion and limitations, Section 11 details the experimental setup and metrics, and Section 12 covers guidance strategies, architecture and implementation details.

8 Qualitative Results↩︎

To better illustrate the dynamic closed-loop performance of our framework, we invite reviewers to view the supplementary videos, which showcase complex counterfactual and long-tail simulation scenarios, visualizations shown in Fig. 7.

9 Additional Experimental Results↩︎

9.1 ProSim Baseline and Counterfactual Setup↩︎

9.1.0.1 Evaluation Setup

We additionally report counterfactual generation results compared with ProSim [5]. Evaluation is conducted on the subset of validation scenarios that match ProSim’s processed data. Metrics are computed exclusively on the target controlled agents across all generated rollouts. The ProSim Unconditional (Uncond) setting serves as a baseline representing the model’s unguided prior, where all condition inputs are fully masked.

9.1.0.2 Results and Comparison

As detailed in Table 6, ProSim suffers a performance trade-off when explicit controls (Sketch or Text) are introduced: while it attempts to follow the instructions, its overall simulation quality degrades (e.g., Text PDMScore drops to 70.25) and off-road rates increase. In contrast, our framework (using the SMART [12] backbone trained on a 10% subset of WOMD) demonstrates transfer fidelity and generative diversity. Our model achieves lower Control ADE across all modalities, translating to higher Success Rates (\(\sim\)​87% vs. 67-79%). Crucially, our approach enforces control conditions without compromising base realism, maintaining high PDMScores (\(\sim\)​77-78) and lower off-road rates (\(\sim\)​0.040). Furthermore, our method generates substantially broader Spatial Coverage (up to 148.70 vs. ProSim’s 105.60), proving it can execute specific prompts while exploring a richer diversity of plausible futures.

9.2 Target-Only vs. Full-GT Control Conditioning↩︎

We further examine the control-conditioning scope used in Sec. 4.2 of the main paper. Here, target agents refer to the agents designated by the WOSAC protocol for evaluation in each WOMD scenario. Under the Full-GT setting, control signals are provided to all ground-truth-valid agents. Under the Target-Only setting, control signals are provided only to the WOSAC-designated target agents, while all other agents receive zero conditioning. Both settings are evaluated using the same metrics and evaluation protocol. As shown in Table 7, restricting control conditioning to the designated target agents leads to a reduction in control precision, particularly for Sketch control, while the realism metrics remain largely comparable. These results indicate that our method remains effective under target-only conditioning, although conditioning additional surrounding agents can further improve control accuracy and interaction consistency.

9.3 Ablation on BehaviorVAE Posterior and Latent Space Analysis↩︎

To validate the BehaviorVAE’s posterior formulation using the VBD [2] backbone, we investigate the impact of concatenating scene context embeddings with encoded trajectories before the CVAE posterior head (\(\mu, \log \sigma^2\)).

As shown in Table 8, the Without Context baseline achieves a slightly higher Meta metric (0.7361 vs. 0.7335) by defaulting to conservative, generic kinematics that easily satisfy rigid rule-based safety metrics. However, explicitly conditioning on scene context (With Context) improves spatial fidelity, reducing the Minimum Average Displacement Error (mADE) to 1.0228 by adapting generative predictions to specific road topologies. This quantitative trade-off is visually corroborated by our t-SNE [29] projections (Figure 8). Relying strictly on raw kinematics (Without Context) collapses latents into rigidly separated clusters. In contrast, concatenating scene context introduces crucial intra-cluster variance while maintaining clear semantic manifolds (e.g., straight, stop, turns). This context-induced diversity prevents CVAE mode collapse, providing the generative flexibility necessary for accurate behavior transfer across novel scenes.

9.4 Ablation on Language Instruction Formats↩︎

To determine the optimal language conditioning strategy, we evaluate three different formats of descriptions derived from the ProSim-Instruct-520k dataset:

• Text Only: Utilizing the original auto-generated scene descriptions from ProSim-Instruct-520k [5] (e.g., "Make vehicle 1, vehicle 2... and the ego vehicle remain parked for the entire simulation").

• Tag Only: Employing simple, template-based sentences filled with discrete kinematic motion tags (e.g., "The target vehicle is turning left and accelerating").

• Text + Tag: A concatenation of both high-level scene descriptions and low-level motion tags.

As shown in Table 9, the Tag Only format achieves the lowest Minimum Average Displacement Error (mADE) of 1.4011 and the highest mADE reduction (15.60%). This is expected, as explicit motion tags provide direct kinematic "shortcuts" that make trajectory matching easier for the model. However, we ultimately adopt the Text + Tag strategy for our framework. While it yields slightly higher mADE compared to the tag-only approach, it maintains the highest Map realism score (0.9024) by retaining crucial high-level scene context. This fosters a more diverse and comprehensive language-guided controllability rather than merely overfitting to kinematic templates. Furthermore, this observation highlights that future efforts in curating precise, manually annotated, and diverse language descriptions will be pivotal in unlocking the full potential of instruction-following in traffic simulation [6].

10 Discussion and Limitations↩︎

While our modulation framework demonstrates effective controllable generation with data efficiency, challenges remain in handling out-of-distribution behaviors and instruction conflicts. In particular, retrieved counterfactual scenarios can be imbalanced: simple behaviors are common, while complex interactions such as overtaking or merging are harder to identify and label due to their rarity in the dataset. Furthermore, we notice that the model may prioritize control instructions over reactive safety, leading to failures such as sudden brake collisions (e.g., ignoring trailing vehicles) or at-fault collisions (e.g., forcing maneuvers in dense traffic), as illustrated in Fig. 9. Future work can utilize reinforcement learning fine-tuning to mitigate these safety-related failures by better balancing instruction adherence with environment-aware reactions. Finally, our current experiments focus on transferring behaviors of a single controlled agent; future work will explore conditioning on multiple agents simultaneously to enable richer multi-agent interaction modeling.

11 Experimental Setup and Evaluation Metrics↩︎

11.1 Datasets and Oracle Validation Setup↩︎

In our evaluation for controllable simulation (Sec 4.2 and 4.3), we feed ground-truth (GT) condition inputs (e.g., sketches, latent codes, or text instructions) directly to the target agents within the scene, ensuring a direct measure of the model’s trajectory synthesis performance. Conversely, for counterfactual and long-tail generation (Sec 4.4), we condition target agents on behavioral inputs transferred from context-matched agents to evaluate practical, non-oracle scenario synthesis.

11.2 WOSAC Challenge Metrics Formulation↩︎

The Waymo Open Sim Agent Challenge (WOSAC) evaluates simulation quality by computing negative log-likelihood (NLL) scores across nine statistical features, categorized into kinematics, agent interactions, and map adherence. The challenge requires simulating up to 128 agents for 8 seconds, generating 32 joint future samples per scenario.

For a given scenario \(i\), target agent \(a\), at timestep \(t\), and statistical feature \(F_j\), the likelihood of the ground-truth trajectory is given by: \[\text{NLL}(i, a, t, j) = -\log p_{i,j,a}(F_j(x^*(i, a, t)))\] where \(p_{i,j,a}(\cdot)\) represents the empirical histogram distribution constructed from the simulated samples. A lower NLL indicates better alignment with real-world behavior.

The per-scenario metric is obtained by aggregating NLL values over valid timesteps: \[m(a, i, j) = \exp \left( -\frac{1}{N(i, a)} \sum_{t} v(i, a, t) \cdot \text{NLL}(i, a, t, j) \right)\] where \(N(i, a) = \sum_{t} v(i, a, t)\) denotes the number of valid timesteps for agent \(a\). The scenario-level metric is computed by averaging over all target agents \(A_{\text{target}}\): \[m(i, j) = \frac{1}{|A_{\text{target}}|} \sum_{a \in A_{\text{target}}} m(a, i, j)\] Finally, the composite WOSAC Meta metric \(M\) is a weighted average over all \(F\) statistical features across \(N\) scenarios: \[M = \frac{1}{N} \sum_{i=1}^{N} \sum_{j=1}^{F} w_j m(i, j), \quad \text{s.t. } \sum_{j=1}^{F} w_j = 1\]

11.2.1 Component Metric Definitions↩︎

• Kinematic Metrics:

  • Linear Speed: Magnitude of velocity, \(\|v\| = \| \frac{x_{t+1}-x_t}{\Delta t} \|_2\).

  • Linear Acceleration: Magnitude of acceleration, \(\| \frac{v_{t+1}-v_t}{\Delta t} \|_2\).

  • Angular Speed: Rate of heading change, \(\omega = \frac{d(\theta_{t+1}, \theta_t)}{\Delta t}\).

  • Angular Acceleration: Rate of change of angular speed, \(\frac{d(\omega_{t+1}, \omega_t)}{\Delta t}\).

• Interaction Metrics:

  • Distance to Nearest Object: Signed distance calculated via the GJK algorithm.

  • Collisions: Boolean flag triggered when signed distance \(< 0\).

  • Time-to-Collision (TTC): Estimated time until collision under constant velocity.

• Map Metrics:

  • Distance to Road Edge: Signed distance to the nearest map boundary.

  • Road Departure: Boolean indicator of whether the agent exits the road boundary.

• Displacement Metrics:

  • Average Displacement Error (ADE): Mean Euclidean distance between the predicted and ground-truth 3D trajectories. For agent \(i\) in rollout \(r\) over valid timesteps \(T_i\), it is computed as \[\mathrm{ADE}_{r,i} = \frac{1}{|T_i|} \sum_{t\in T_i} \left\| \mathbf{p}^{(r)}_{i,t} - \mathbf{p}^{\mathrm{log}}_{i,t} \right\|_2 .\] Averaging across all \(R\) rollouts and \(N\) agents gives \[\mathrm{ADE} = \frac{1}{RN} \sum_{r=1}^{R} \sum_{i=1}^{N} \mathrm{ADE}_{r,i}.\]

  • Minimum Average Displacement Error (mADE): The minimum displacement error among all generated rollouts. It first averages \(\mathrm{ADE}_{r,i}\) over all \(N\) evaluated agents within each rollout and then selects the minimum across \(R\) rollouts: \[\mathrm{mADE} = \min_{r\in\{1,\dots,R\}} \frac{1}{N} \sum_{i=1}^{N} \mathrm{ADE}_{r,i}.\]

For further implementation details, refer to the WOSAC Challenge documentation [8].

11.3 PDMScore Formulation↩︎

We adapt PDMScore [27] to evaluate controlled closed-loop rollouts on the Waymo dataset. Compared with the original ego-centric formulation, our adaptation (i) evaluates the controlled target agent, (ii) measures progress along the queried condition trajectory when available and along the associated route otherwise, and (iii) sets the traffic-light compliance term to one because traffic-light states are not evaluated. The original score structure, component weights, and remaining metric definitions are retained and implemented using WOMD map and agent geometry.

The final score \(S\) is the product of multiplicative compliance terms (\(M\)) and weighted performance metrics (\(W\)): \[\begin{align} S &= 100 \, M W, \\ M &= NC \cdot DAC \cdot DDC \cdot TLC, \\ W &= \frac{5EP + 5TTC + 2LK + 2HC}{14}. \end{align}\] We set \(TLC=1\) because traffic-light states are not evaluated in our setting. The remaining metric components are defined as follows:

• No At-Fault Collision (\(NC\)): A binary multiplier indicating whether the target agent avoids at-fault collisions. Collisions are detected using oriented bounding-box overlap, and at-fault status is classified from the relative positions and motions of the interacting agents.

• Drivable-Area Compliance (\(DAC\)): A binary multiplier indicating whether the target agent remains within the drivable region. We associate the rollout with nearby lane and drivable-area polylines and evaluate the distances of both the bounding-box center and corners. Brief off-road deviations are tolerated according to the thresholds specified in our implementation.

• Driving-Direction Compliance (\(DDC\)): A route-compliance multiplier that penalizes sustained motion opposite to the direction of the associated lane.

• Ego Progress (\(EP\)): Following the original PDMScore terminology, this metric measures the positive projected displacement of the target agent along the queried condition trajectory when available, or along the associated route otherwise. The resulting progress score is normalized and clipped to \([0,1]\).

• Time-to-Collision Proxy (\(TTC\)): A binary short-horizon safety score obtained by extrapolating the target agent under a constant-velocity assumption and checking for potential at-fault collisions within the TTC horizon.

• Lane Keeping (\(LK\)): A binary route-adherence score based on the smoothed lateral distance to the associated lane. We set \(LK=0\) when this distance continuously exceeds the lane-keeping threshold for the configured duration.

• History Comfort (\(HC\)): A binary comfort score based on bounds on longitudinal and lateral acceleration, jerk, yaw rate, and yaw acceleration. Temporal smoothing and quantile-based aggregation reduce sensitivity to isolated noise in generated rollouts.

11.4 Additional Evaluation Metrics↩︎

Beyond WOSAC and PDMScore, we report the following metrics in Sec. 4.4 of the main text to provide a comprehensive analysis of trajectory fidelity and generative diversity. For each scenario, metrics are computed exclusively on the target controlled agents across \(K=32\) generated rollouts.

• Trajectory ADE (TrajADE): Calculates the mean \(L_2\) distance between the predicted trajectories and the unconditioned model rollout trajectories over the overlapping valid frames \(T_{\text{overlap}}\). Averaged over the \(K\) rollouts, it measures the overall spatial deviation: \[\text{TrajADE} = \frac{1}{K} \sum_{k=1}^{K} \left( \frac{1}{|T_{\text{overlap}}|} \sum_{t \in T_{\text{overlap}}} \|x_{k,t}^{\text{pred}} - x_t^{\text{ref}} \|_2 \right)\]

• Control ADE: Measures the geometric fidelity between the synthesized trajectory and the queried condition trajectory after applying a local coordinate alignment (translation and rotation). This alignment removes ego-motion bias to strictly evaluate control adherence. Averaged over the \(K\) rollouts: \[\text{Control ADE} = \frac{1}{K} \sum_{k=1}^{K} \left( \frac{1}{|T_{\text{overlap}}|} \sum_{t \in T_{\text{overlap}}} \|x_{k,t}^{q, \text{local}} - x_{k,t}^{\text{pred, local}} \|_2 \right)\]

• Success Rate (%): The average ratio of valid rollouts that successfully meet specific safety and goal completion criteria, averaged across all \(N_{\text{cases}}\) scenarios: \[\text{Success} = \frac{1}{N_{\text{cases}}} \sum_{i=1}^{N_{\text{cases}}} \left( \frac{n_{\text{success}, i}}{n_{\text{valid}, i}} \right) \times 100\%\]

• Coverage: Quantifies the exploration breadth of the generative rollouts. Following [28], we discretize the map into a set of grid cells \(\mathcal{G}\), where each cell \(g\) has a center \(c_g\) and an area \(\Delta A\). Let \(d_g \in \{0, 1\}\) denote the drivable space mask (\(1\) for drivable). We apply Kernel Density Estimation (KDE) over all rollouts and timesteps to obtain the spatial density \(\hat{\rho}(x)\). A cell is defined as "covered" if its density exceeds a threshold \(\tau\), denoted by the indicator function: \[z_g = \mathbf{1}[\hat{\rho}(c_g) > \tau]\] From this, we define the Coverage Ratio (\(C_{\text{all}}\)) as follows: \[C_{\text{all}} = \frac{1}{|\mathcal{G}|} \sum_{g \in \mathcal{G}} z_g\]

12 Implementation Details↩︎

12.1 Inference and Decoding Strategy↩︎

To explicitly balance control fidelity and natural driving priors, our synthetic scenario experiments utilize the SMART [12] model trained on a 10% subset of the full training data. During autoregressive decoding, we employ a logits-level Classifier-Free Guidance (CFG) [30] mechanism. At each decoding step, we extract unconditional logits (\(L_{\text{base}}\)) from the base model and conditional logits (\(L_{\text{control}}\)) from the control model. The final extrapolated logits are then computed as: \[L_{\text{final}} = L_{\text{base}} + \omega \times (L_{\text{control}} - L_{\text{base}})\] This mixed \(L_{\text{final}}\) is subsequently converted to a probability distribution via Softmax, followed by standard Top-K and Top-P sampling strategies to predict the next trajectory token. To optimize the trade-off between instruction adherence and scene realism, we empirically configure the guidance scale \(\omega\) based on the control modality: \(\omega=0.5\) for Sketch, \(\omega=0.7\) for Latent, and \(\omega=1.0\) for Text conditioning.

12.2 LoRA Base-Model Finetuning on the Full Training Set↩︎

As a reference performance in Sec. 4.3 (Fig. 5), we additionally consider a setting where the control branch is trained jointly with LoRA adapters inserted into the base decoder using the full training dataset. This setting increases both adaptation capacity and data scale.

Concretely, we keep the pretrained SMART backbone frozen at the full-parameter level but inject LoRA adapters into the base decoder’s attention projections with a rank of 16, a scaling factor of 0.4, and a dropout of 0.05. The control branch remains active and uses the same FiLM-based conditioning interface as in the standard control experiments. In the sketch-conditioned example run, the control encoder utilizes a 16-step control horizon, 256-dimensional control tokens, 2 temporal transformer layers, 4 attention heads, and FiLM modulation at decoder blocks 0–5 with a FiLM hidden dimension of 256.

This setting is trained on the full Waymo training set (486,995 scenarios) using a batch size of 4, a learning rate of \(5\times10^{-5}\), 16-bit mixed-precision training, and gradient clipping at 0.5 for up to 100 epochs.

12.3 ProSim Unconditional Baseline Implementation↩︎

To obtain the true unconditional ProSim baseline, which has not been exposed to any paired control data during training, we mask all prompt inputs and retrain the model from scratch following the original ProSim training procedure [5]. We use the original optimizer and training hyperparameters and select the checkpoint after the validation loss has converged. The retrained model achieves an mADE of 2.641, comparable to 2.679 from the official ProSim checkpoint, providing a consistent unconditional baseline for evaluating our FiLM adaptation.

12.4 BehaviorVAE Training Details↩︎

Our BehaviorVAE was trained on 10,000 Waymo scenarios (approximately 2% of the full training set), processing up to 64 agents per scene and predicting 41 future time steps. The model employs a 6-layer context encoder inherited from VBD [2], which is jointly optimized with the VAE. Each agent’s future trajectory is transformed into its local coordinate frame, represented as \((\Delta x, \Delta y, \sin \Delta \theta, \cos \Delta \theta)\), and processed by a 2-layer GRU (hidden size 512) followed by a single relative-geometry-aware self-attention layer to model inter-agent interactions. This extracts a 512-dimensional agent-specific latent variable without relying on a global scene latent. The posterior is parameterized by concatenating trajectory and context features, while the conditional prior is predicted solely from context features. The decoder, a 6-layer MLP, reconstructs trajectory deltas by conditioning on the latent code and a learned time embedding. The training objective minimizes a masked MSE reconstruction loss alongside a KL regularization term (\(\mathcal{L} = \mathcal{L}_{\mathrm{recon}} + \beta \mathcal{L}_{\mathrm{KL}}\)). Using latent sampling after a 1,000-step warm-up, the KL coefficient (\(\beta\)) is linearly annealed to 0.001 over the first 20,000 steps. The model is optimized using AdamW [31] (learning rate \(3\times10^{-4}\), batch size 4, gradient clipping 1.0) in FP32 precision on two RTX 4080 GPUs for 200 epochs. At convergence, the training loss reached 0.0269 and the validation loss was 0.0389, establishing a diverse and structured behavior prior.

12.5 Control Representation Encoders↩︎

We adopt the same control-conditioning interface for both the SMART and VBD backbones, where each control modality is first encoded into a standardized 256-dimensional per-agent control token and injected into the frozen generative decoder through FiLM layers. For text control, both models utilize a frozen DistilBERT [23] (distilbert-base-uncased) backbone to process prompts (truncated to 384 tokens) from the ProSim-Instruct-520k corpus. We insert LoRA [21] adapters (rank 16, scaling factor 0.4, dropout 0.05) into the attention projections of 6 adapted layers, and the resulting text features are linearly projected into the shared control space. For sketch control, each agent’s relative \((x,y,\theta)\) trajectory over a 16-step control horizon is processed by a 2-layer temporal transformer (with 4 attention heads and a feed-forward dimension of 256) and pooled into the control token. Finally, for latent control, we retain the identical FiLM interface but replace the sequence encoders with a 3-layer MLP that maps the 512-dimensional BehaviorVAE latent codes directly into the 256-dimensional conditioning space. Across all settings, the backbone generative decoder remains completely frozen; only the modality-specific control branches and FiLM layers are optimized.

12.6 FiLM Parameters and Structure↩︎

The FiLM [22] module in both the SMART and VBD backbones utilizes direct feature-wise affine modulation: \[\mathrm{FiLM}(x,c) = \gamma(c) \odot x + \beta(c)\] where \(x\) denotes the decoder hidden feature and \(c\) denotes the control token. In both models, the FiLM MLP takes a 256-dimensional conditioning feature as input and predicts channel-wise modulation parameters for the decoder hidden state.

For the SMART backbone, FiLM is attached to decoder blocks 0–5, with a FiLM hidden dimension of 256. For VBD, FiLM is attached to denoiser blocks 0–1, with a FiLM hidden dimension of 512 in the referenced text-conditioned run. Crucially, in both implementations, FiLM is initialized close to the identity transformation. The final FiLM layer is initialized with very small weights, the scale branch is biased toward \(\gamma \approx 1\), and the shift branch is biased toward \(\beta \approx 0\) (with a minor random perturbation added to avoid exact identity at initialization). This design stabilizes control fine-tuning by ensuring the pretrained decoder behavior is perfectly preserved at the beginning of training, allowing the control branch to gradually learn non-trivial modulations.

12.7 Optimization Hyperparameters↩︎

For control fine-tuning, both models are optimized using AdamW under 16-bit mixed-precision training. Unless otherwise noted, the base decoder remains frozen, and only the control modules are updated. For the SMART backbone, the learning rate is set to \(5\times10^{-5}\) with no warm-up, a minimum learning-rate ratio of 0.05, gradient clipping at 0.5, and synchronized batch normalization. For VBD control training, we use a learning rate of \(1\times10^{-4}\) for the Control branch, with a weight decay of 0.01 and gradient clipping via value clamping at 5.0.

Both SMART and VBD are trained under receding-horizon closed-loop control. In VBD, replanning is performed every 11 steps, utilizing a prediction horizon of 40 steps and 5 denoising steps per planning update; additionally, during closed-loop training, we generate 8 candidate trajectories and select the one closest to the ground truth. In SMART, control is similarly applied in a receding-horizon manner, processing 11 historical steps and 80 future steps in the decoder, with control signals injected over a 16-step control horizon (window-aligned latent indexing is used for latent control).

12.8 Context Matching Heuristic Filter↩︎

To retrieve compatible donor scenarios, we use a non-leaky context-matching heuristic based only on observed history. For each query agent, candidate agents are represented by a SMART history embedding and five historical descriptors: (i) Dynamic, including speed, acceleration, yaw rate, curvature, and displacement; (ii) Trajectory, the recent ego-aligned trajectory; (iii) Map Context, the local map composition; (iv) Map kNN, nearby map geometry and types; and (v) Neighbor Context & kNN, surrounding traffic layout, relative motion, and agent types.

Retrieval proceeds in three stages. First, candidates are restricted to the same agent type and the query scenario is excluded. Second, a coarse pool is formed by combining top candidates ranked by encoder-feature similarity and trajectory distance. Third, candidates are re-ranked using \[\begin{align} s ={} & w_{\mathrm{feat}} s_{\mathrm{feat}} + w_{\mathrm{traj}} d_{\mathrm{traj}} + w_{\mathrm{map\_knn}} d_{\mathrm{map\_knn}} + w_{\mathrm{nbr\_knn}} d_{\mathrm{nbr\_knn}} \\ & - w_{\mathrm{dyn}} d_{\mathrm{dyn}} - w_{\mathrm{map}} d_{\mathrm{map}} - w_{\mathrm{nbr}} d_{\mathrm{nbr}}, \end{align}\] which rewards feature similarity and penalizes descriptor distances. We use geometry-aware weights of \(w_{\mathrm{feat}}=0.18\), \(w_{\mathrm{traj}}=0.40\), \(w_{\mathrm{map\_knn}}=0.22\), \(w_{\mathrm{nbr\_knn}}=0.20\), \(w_{\mathrm{dyn}}=0.08\), \(w_{\mathrm{map}}=0.04\), and \(w_{\mathrm{nbr}}=0.03\). We further require \(d_{\mathrm{traj}} \le 0.08\), \(d_{\mathrm{map\_knn}} \le 1.4\), and \(d_{\mathrm{nbr\_knn}} \le 1.3\). If no candidate passes these thresholds, a score-based fallback retrieves the minimum required number of matches.

12.9 Qualitative Examples of Language Prompts↩︎

We show five prompts from scenario b6975ecc8d04afde to illustrate our Text + Tag format, which combines scene descriptions with motion tags.

References↩︎

---

1. Original ProSim uses the official checkpoint; FiLM uses a retrained backbone (details in Supp. E3); gains are computed relative to each model’s unconditional baseline.↩︎