1 Introduction↩︎

Due to the significant expenses and risks associated with conducting large-scale real-world tests [1], autonomous vehicle (AV) developers rely heavily on comprehensive testing in simulation to ensure the development of reliable systems [2]. To maximize the effectiveness of simulators, it is crucial for them to offer realism that is aligned with human knowledge. Accordingly, realistic traffic models are essential to ensure that insights gained from simulation testing apply seamlessly to real-world scenarios [3], [4]. However, developing traffic models that balance realism and diversity remains an ongoing challenge. To tackle this challenge, we leverage recent advancements in alignment research for traffic modeling; moreover, such alignment is model-agnostic and can improve various existing traffic models. Consequently, our primary research objective is to develop a framework based on reinforcement learning with human feedback (RLHF) to widely improve the realism of existing traffic models.

In order to align human preferences for generating realistic traffic simulations, two significant challenges need to be addressed: (1) the limited expressiveness of existing methods in capturing human preferences for realism in traffic simulations, and (2) unifying diverse traffic simulation models. The first challenge relates to traditional approaches relying on predefined rules or statistical models that fail to encompass the wide range of human preferences and subjective perceptions of realistic traffic scenarios. The second challenge revolves around the need to unify diverse traffic simulation models. The field of traffic simulation encompasses a wide range of models, each with their own strengths and limitations, underlying assumptions, data requirements, and simulation algorithms, making it challenging to incorporate human preferences seamlessly.

To address these challenges, we introduce TrafficRLHF, a three-stage framework utilizing human feedback to enhance traffic models for realistic scenario generation. The framework tackles key challenges in the field of realistic traffic simulation, leveraging recent advancements in reinforcement learning with human feedback (RLHF) and a state-of-the-art autoregressive backbone model.

To tackle the first challenge—capturing human preferences on realism—we propose using human feedback as a tool for alignment. RLHF has recently demonstrated its effectiveness in conjunction with large language models. Its impressive sample efficiency makes it a prime candidate for realism alignment in traffic models. Moreover, through the labeling of a modest quantity of data with human preferences, one can establish a reward model suitable for refining a wide range of traffic models. Addressing the second challenge—the need to unify diverse traffic simulation models—requires a method that can be applied across diverse model architectures. This is achieved using our RLHF-based method, which only requires a universal interface for reward model input. To fulfill this requirement, we employ a state-of-the-art autoregressive backbone model, CTG [5], with a preferred roll-out length to optimize the simulation of the near future. Experiments on the real-world nuScenes autonomous driving dataset [6] confirm TrafficRLHF’s ability to generate realistic trajectories that align with human preferences. As one result, TrafficRLHF is able to reduce unrealistic collisions or off-road driving by up to 80%.

Contributions. The main contributions of our work are threefold: (1) We introduce the first dataset designed for realism alignment in traffic modeling, (2) we formulate a reward model that quantifies realism in alignment with human preferences, and (3) we propose a model-agnostic framework, leveraging RLHF, that effectively enhances the realism of a broad range of existing traffic models.

2 Related Work↩︎

Traffic simulation. Traffic simulation techniques can be broadly categorized into rule-based and learning-based approaches. Rule-based methods, employing analytical models such as cellular automata and intelligent driver models [7], tend to set fixed routes for vehicles and separate longitudinal from lateral agent movements. This inflexibility restricts their capacity to mimic the dynamic variety inherent in real-world driving behaviors.

On the other hand, learning-based approaches utilize deep generative models trained on trajectory datasets to mimic real-world driving behaviors [3], [4], [8]–[10]. However, they often lack the flexibility for users to dictate customized attributes of generated traffic behaviors during inference, thereby falling short in capturing the full spectrum of realistic traffic scenarios.

Additionally, certain researches emphasize on generating adversarial or safety-critical scenarios, crafting trajectories that provoke autonomous vehicle misbehavior [11]–[14]. Models like STRIVE [15] and CTG [5], and TrafficGen [16] have made strides in this direction. But, these models often struggle to generate traffic scenarios that align with human perceptions of realism.

Despite the advancements in traffic simulation methodologies, a significant gap persists in the field: the ability to produce traffic scenarios that closely align with human perceptions of realism, an essential component for effective AV testing. Therefore, integrating human preferences seamlessly into traffic simulation models remains an unfulfilled challenge and a key focus of our research.

Reinforcement Learning from Human Feedback. The investigation into Reinforcement Learning from Human Feedback (RLHF) spans at least a decade, with significant contributions from numerous researchers [17]–[21]. RLHF is a component of a more comprehensive paradigm known as human-in-the-loop learning process [22]. In scenarios where reward engineering for RL proves challenging or costly, human feedback data emerges as a crucial asset.

Recent advances in the field have seen the use of human feedback for fine-tuning LLMs [21], [23], [24]. A notable example is InstructGPT [25], which leverages both human demonstrations and preferences to achieve significant improvements over the GPT-3 baseline in terms of human preferences from annotators. This demonstrates the scalability and potential of RLHF as a method for tuning large models using human feedback.

Our proposed RLHF approach offers unique advantages for generating realistic traffic scenarios with human feedback, particularly in terms of realism that lack of explicit formulations. This approach minimizes data collection and feedback time, avoiding the need for costly labeled data. The preference for videos requires minimal human effort, thus enabling the production of a large volume of labeled data.

3 TrafficRLHF↩︎

In this section, we formally specify the traffic simulation problem and delineate the three core components of TrafficRLHF. As shown in Figure 1, these components include data collection, involving the accumulation of human preferences for realism using a blend of genuine traffic scenes and those generated from traffic models; reward model training, which entails training a model capable of evaluating the realism of a traffic scenario based on human preferences; and fine-tuning, wherein we harmonize the model with the reward model trained in the previous stage. Ultimately, this methodology enables a wide array of traffic models to generate more authentic traffic scenarios.

3.1 Traffic Simulation Formulation↩︎

We formulate the problem of traffic simulation similar to [5]. For a scenario with \(M\) vehicles, their state (comprised of \(x, y\) position, speed, and yaw) at timestep \(t\) is represented as \(s_{t}=[s_{t}^1~...s_{t}^M]\) where \(s_{t}^i = (x_{t}^i,y_{t}^i, v_{t}^i, \theta_{t}^i)\), and their action (i.e., control) as \(a_{t}=[a_{t}^1...~a_{t}^M]\), where \(a_{t}^i=(\dot{v}{t},\dot{\theta}{t})\) (acceleration and yaw rate). We designate \(\mathbf{c}=(I,s_{t-T_\text{hist}:t+1})\) as decision-relevant context, comprising local semantic maps for all agents \(I={I^1,...,I^M}\), as well as their current and \(T_{hist}\) preceding states \(s_{t-T_\text{hist}:t+1}={s_{t-T_\text{hist}}, \dots, s_{t}}\). The state \(s_{t+1}\) at time \(t+1\) is derived from a transition function \(f\) that computes \(s_{t+1}=f(s_{t}, a_{t})\) given the previous state \(s_{t}\) and control \(a_{t}\) following unicycle dynamics.

3.2 Human Feedback Collection↩︎

Our proposed framework TrafficRLHF aligns human preferences with generated traffic scenarios under identical initial conditions. It requires only a moderate amount of video data collection and minimal human feedback. Since all trajectories are collected in simulation using existing traffic models, a considerable amount of data can be generated and enhanced iteratively. Furthermore, any human with driving experience can serve as a labeler after a brief familiarization with the user interface, eliminating the need for specialized expertise.

To garner human feedback, pairs of traffic scenarios \((S_1, S_2)\) are generated from the same context \(\mathbf{c}\). Each scenario \(S_i = [s_{t-T_\text{hist}:t+T_\text{fut}}]_i, i=1,2\) represents stacked state sequences of length \(T\text{fut}\). Human annotators are shown videos of these paired traffic scenarios, with maps and vehicles appropriately labeled. The annotators are then tasked with specifying which scenario appears more realistic.

As current traffic models often yield less realistic data, we increase the total amount of generated traffic scenarios to boost the likelihood of generating at least one realistic scenario. Moreover, since the generated scenarios are usually covering multi-modal future and not always comparable, we select the most realistic example instead of a pairwise comparison. Concretely, we generate \(N\) traffic scenarios \(S_1,\cdots,S_N\) and request human annotators to select the most realistic one from the set. This process results in \(N-1\) scenario pairs with preference relationships \((S_i\succ S_j)\), where \(S_i\) is the scenario chosen by the human annotator and \(S_j\) is any other scenario. In cases where the traffic model fails to generate realistic scenarios, annotators are provided the option to declare that none of the presented scenarios appear realistic. If this option is selected, we will use the ground truth scenario \(S_\text{gt}\) as the preferred scenario, denoted as \((S_\text{gt} \succ S_j)\).

3.3 Reward Modeling↩︎

The training methodology for the reward model \(r_{\text{HF}}(\cdot;\phi)\) is inspired by Abramson et al. [26]. Two traffic scenarios, generated under the same initial conditions, are compared and the reward model is trained with the following loss: \[l_\text{RM} = \mathbb{E}_{S_1, S_2 \sim \mathcal{D}}[-\log\sigma(r_\text{HF}(S_1;\phi) - r_\text{HF}(S_2;\phi))],S_1 \succ S_2. \label{eq:rm95loss}\tag{1}\] Here, \(r_\text{HF}(S_i;\phi)\) represents the reward model. The training dataset \(\mathcal{D}\) is compiled using human preferences over traffic scenarios, generated with the existing traffic models (without any improvements). And \(S_1, S_2\sim \mathcal{D}\) is a pair of scenarios sampled from the human labeled dataset. Note that \(r_\text{HF}\) can be either an iterative model or an encoder-decoder model taking a fixed-length sequence. To maximize TrafficRLHF’s ability to adapt a variety of traffic models during the final fine-tuning stage, we choose an iterative model that can take as input an arbitrary sequence length. To calculate the reward, we compute the average of the reward over a sequence \(S_i\).

3.4 Traffic Model Fine-tuning↩︎

Similar to the fine-tuning method proposed by Abramson et al. [26], our objective is: \[\tilde{r} = r + \alpha \cdot r_\text{HF}(S),\] where \(r\) is the original loss for training the provided traffic model and \(\alpha\) is a scaling term. This mixed objective is used to avoid overfitting the reward score and degrading task performance. In RLHF for training large language models (LLMs) (e.g., ChatGPT [27]), usually the majority of the model parameters are frozen for the computational benefits. However, existing traffic models are usually much smaller than LLMs. In our experiments, we will show that fine-tuning the entire motion prediction model can lead to higher performance improvements.

4 Experiments↩︎

We conduct experiments to validate that (1) TrafficRLHF can generate more realistic traffic behaviors, and (2) the reward model is generalizable and can be reused for a diverse range of traffic models.

4.1 Experimental Setup↩︎

Datasets. The nuScenes dataset [6] is a vast compilation of real-world driving data, totaling 5.5 hours of meticulously logged vehicle trajectories from two unique urban settings. It captures a myriad of driving scenarios, including instances of high traffic density. In our study, we use mainly the training split of the nuScenes dataset for data collection, reward model training and traffic model fine-tuning. Evaluation of the traffic models’ performance is carried out on a randomly sampled subset of 100 scenes from the validation split.

Metrics. Following [4], [5], we evaluate stability (i.e., avoiding collisions and off-road driving), realism of generated trajectories by each model. We evaluate stability by reporting the failure rate (fail), measured as the average percentage of agents encountering a critical failure (i.e., a collision or road departure) in a scene. To assess realism, we compare data statistics between generated traffic simulations and ground truth trajectories from the dataset by calculating the Wasserstein distance between their normalized histograms of driving profiles. We measure realism using realism deviation (real) which is the average of realism values for three properties: longitudinal acceleration magnitude, lateral acceleration magnitude, and jerk. Also, since the reward model naturally captured the human preference and we measure the human preference score using the reward cost score (reward cost).

Traffic Model Benchmarks. Recent advancements in the field of traffic generation include three particularly noteworthy models. Firstly, the Conditional Traffic Generation model (CTG) is a model predicated upon the principles of conditional diffusion [5]. Secondly, the Bi-Level Imitation Learning System (BITS) is a model that utilises a two-tiered imitation learning strategy to approach traffic prediction tasks [4]. Lastly, TrafficGen is an autoregressive generative model that leverages an encoder-decoder architecture, enabling the diverse initialization of traffic scenarios [16]. We also need the initialized model parameters for the fine-tuning (3.4). Given that both CTG and BITS were initially trained on the nuScenes dataset [6], we elected to use their original parameters. As for TrafficGen, we managed to reproduce its performance on the same dataset using the original implementation provided by the authors.

4.2 Experimental Details↩︎

Traffic Generation and Data Collection. In the generation of traffic scenarios, we utilize the state-of-the-art Conditional Traffic Generation (CTG) model. CTG model is built with a encoder for traffic context and a diffuser model for traffic generation. As diffuser presents the concept of guidance, which allows for the sampling of trajectories from an unconditional diffusion model to achieve a specific pre-set objective, CTG use such guidance for controllable generation. For the data collection, we use CTG to generate traffic scenarios, operating under the guidance of a “no collision” principle, a feature that is evident in their outcomes. From the nuScenes dataset, we selected a subset of 500 scenes from the training split [6]. Each scene was processed through the CTG model to generate five unique scenarios. To ascertain human preferences, we presented these five scenarios to labelers, tasking them with identifying the most realistic scenario or indicating if all scenarios lacked realism. Consequently, we constructed our human preference dataset by pairing either the most realistic or the ground truth sample with another sample. This curated data was then employed in training the reward model, with the loss calculated according to Equation  1 .

Reward Model and Training. In our experiments, the Reward Model (RM) is parameterized using the CTG encoder and fully-connected neural networks for the output layer. This selection of inputs for the RM could be seen as a universal module applicable to any traffic scenario. Given the status of CTG as a state-of-the-art model, we assume it offers superior expressiveness across diverse traffic scenes. Furthermore, the simplicity of the CTG encoder facilitates a straightforward training schema, thereby reducing overhead when compared to other models that incorporate more complex components, such as RL policy networks [4], [9]. Nevertheless, it is essential to note that the choice of the reward model is not restricted to our current selection and has the potential for expansion and improvement in the future.

4.3 Reward Model Validation↩︎

Before proceeding to fine-tune the traffic model, we first assess the Reward Model (RM) using the human preference data we collected. This evaluation involves comparing the reward scores with human preferences. For the evaluation, we partition the collected human preferences into a training subset and a validation subset, comprising 400 and 100 units respectively. As demonstrated in Figure 2, the accuracy of the RM swiftly converges with an increase in sample size. However, the variance of the accuracy tends to broaden with further increments. This increasing variance potentially indicates overfitting during the training process [28], since NuScenes dataset is relatively small in size ( 850 scenes). To optimize performance, we opt to utilize the RM trained with 200 samples in the fine-tuning process.

4.4 Evaluation of Traffic Model↩︎

CTG+RM. Having evaluated the learned RM, we apply it to fine-tune the traffic model, as outlined in 3.4. As depicted in Figure 3, fine-tuning with the RM significantly enhances realism by decreasing the failure rates associated with collisions and driving off the road. For quantitative results, given that CTG offers a robust feature for controllable traffic scenario generation via guidance, we evaluate the model with varying guidance, following the original authors’ setup. As demonstrated in Table 1, fine-tuning enables the generated traffic scenarios to decrease failures (i.e., collisions and driving off the road). We observe that the realism metrics deteriorate when guidance is related to the goal positions/speeds, possibly because the guidance during the diffusion process in CTG overpowers the loss provided by the RM penalty losses.

Generality of Learned RM. To highlight the versatility of our trained Reward Model (RM) in generalizing and enhancing a variety of models, we apply fine-tuning to two recent traffic models—BITS [4] and TrafficGen [16]—using the learned RM. Given the complex designs of BITS and TrafficGen, we restrict fine-tuning to the motion forecasting components, while other model parameters remain static. For a balanced comparison, we also preserve the initialization for TrafficGen.

As depicted in Figure 4, fine-tuning with RM decreases the frequency of collisions and off-road conditions in both models. We also present quantitative evaluations in Table 2. When fine-tuned with the reward model—which was trained on traffic scenarios generated from CTG—there were measurable improvements across all three metrics for both BITS and TrafficGen. Although the degree of enhancement is not as substantial as that seen with CTG, it underscores the broad applicability of our proposed method.

4.5 Ablation Study↩︎

In our research, we delve further into the performance capabilities of our proposed method by executing an ablation study involving the fine-tuning of the CTG model.

Fine-tuning components. We experiment with fine-tuning specific components of the model and present the results in Table 3. First, we tweak the encoder portion of the CTG model, incorporating the reward model (RM) in the process. This step leads to substantial enhancements across all performance measures, suggesting the pivotal role of human feedback, captured via the reward model, in refining the model’s ability to generate realistic traffic scenarios. For our next variant, we focus on the decoder portion of the CTG model, integrating the reward model into it. Interestingly, this setup outperforms the previous variant, reinforcing the idea that the diffusing model (decoder) might play a more integral role in the generation of more realistic traffic scenarios. Lastly, we carry out an all-encompassing fine-tuning, involving both the encoder and decoder of the CTG model, while also adding the reward model. This configuration performs the best among all, underscoring the importance of simultaneous fine-tuning of all model components alongside the incorporation of human feedback via the reward model. All things considered, while full fine-tuning yields the most promising results, the decoder (diffusing model) appears to be more responsive to modifications, likely owing to its inherent role in CTG’s mechanism for generating realistic traffic scenarios [5].

Human feedback. We experiment with training reward model using a diverse range of human feedback on traffic scenarios generated by different models. To do so, we repeated the process of collecting human feedback on generated traffic scenarios on both TrafficGen and BITS. The rest of the process, including reward model training and traffic model fine-tuning, remain the same. As the result in Table 4, we observe a slight improvement of realism for the fine-tuned model.

Reward model. We also experiment with training reward model on different architectures. We use TrafficGen encoder as the backbone and different human feedback to train a reward model and fine-tune the other traffic models. We didn’t use the BITS model due to its complicated architecture for enabling bi-level optimization. As the result shown in Table 4, we observe a more significant improvements on realism compared to using human feedback on different models. Based on the findings above that fine-tuning decoder being more effective, one hypothesis is that the latent representation for the traffic model and reward model should be similar and RLHF does not significantly help with learning better representations [27].

5 Conclusion↩︎

Summary. In conclusion, this research has demonstrated the value of reinforcement learning with human preference (RLHF) in developing more realistic traffic models for simulation testing in autonomous vehicle development. Our method addresses the identified challenges of capturing nuanced human preferences and unifying diverse traffic simulation models. By using human feedback for alignment, we have harnessed the data efficiency of RLHF and used an autoregressive backbone model to provide a generalizable interface for the reward model input. Our TrafficRLHF model, tested on the nuScenes dataset, has proven capable of generating trajectories closely aligned with human preferences. Our contributions have not only provided the first dataset of realism alignment for traffic modeling but also offered a versatile RLHF-based framework that enhances the realism of a wide range of existing traffic models. This lays a robust foundation for future work in the field and has the potential to significantly enhance the utility of traffic simulations for autonomous vehicle development.

Limitations and Future Work. Currently, TrafficRLHF does not incorporate the more advanced fine-tuning strategies for diffusion models recently proposed [29]. Furthermore, with BITS and TrafficGen, we have not yet fully harnessed the potential of TrafficRLHF by collecting and labeling generated traffic scenarios for these models. Training a reward model based on a specific traffic model and implementing a bespoke fine-tuning strategy could enhance traffic simulation results. Also, the iterative improvement of the traffic model through the use of updated human preference datasets and reward models might further elevate performance [27]. However, due to the intensive data collection required, we have not yet explored these avenues, although they present exciting opportunities for future research.

6 Algorithm of Training in Details↩︎

CTG. Primarily, we adopt the training and sampling procedures outlined in [5], with detailed algorithms for training presented subsequently. For the reward model, we utilize an encoder from CTG and supplement it with a fully-connected neural network that comprises hidden units of varying dimensions (512-512-512-128-32) and a Tanh activation function for all layers, including the output layer. We set \(\alpha=0.1\) during the fine-tuning process. Furthermore, we consistently utilize the sampling methods from [5] for all different guidance applications.

BITS. Predominantly, we follow the training and sampling procedures outlined in [4]. For the fine-tuning process, we only adjust the spatial goal network, while the parameters for the goal-conditional policy remain unaltered. Though the goal-conditional policy network determines the actions for each agent within the traffic simulation, employing the reward model for policy network fine-tuning necessitates a different strategy, potentially resulting in an uneven comparison. Thus, we opt to fine-tune solely the spatial goal network, which also plays a critical role in guiding the traffic simulation. We set \(\alpha = 0.5\) for the fine-tuning process.

TrafficGen. Largely, we adhere to the training and sampling procedures described in [16]. Notably, TrafficGen also employs a traffic initialization network. For a fair comparison, we maintain a fixed initialization during the fine-tuning process. We set \(\alpha = 0.3\) for the fine-tuning process.

All the fine-tuning process employs an amended PPO from ColossalAI [30] as our fine-tuning strategy with 70 epochs.

Traffic Simulation. Following Zhong et al. [5] in CTG, to perform closed-loop traffic simulation of a scene, our model is applied for all agents in a standard control loop. At each step, the model generates a guided trajectory and executes the initial few actions before re-planning at a set frequency. In all our experiments (4), each scene is rolled out for 10 seconds, starting from a ground truth driving log, and re-planned at a frequency of 2 Hz.

References↩︎