1 INTRODUCTION↩︎

Artificial intelligence is commonly seen as the key enabler for fully autonomous driving. Sensing and Mapping, Perception, and (Path) Planning are often seen as the building blocks of any non-end-to-end autonomous system. The rise of deep learning has led to an unprecedented progress in Mapping and Perception. However, path planning has a hybrid nature - it tends to be model-driven with some sub-components learned using deep learning. This is primarily due to the severely complex interaction of different agents (static and dynamic) and prior knowledge (map and traffic information). Dearth of data which includes various corner cases further limits completely data-driven based planning.

Prediction is a crucial part of autonomous driving, serving as a ‘lego block’ for tasks like Path Planning, Adaptive Cruise Control, Side Collision Warning, etc. In this work, we address the problem of predicting lane changes of vehicles. This is of paramount importance, as around 18% of all accidents happen during lane change maneuvers [1], and lane changes are often executed in high-velocity situations, e.g. on highways. A precise prediction thus decreases risk and enables safer driving. This safety gain stemming from a sensitive prediction is one side of the coin. On the other hand, though, false predictions have to be avoided as they have a negative influence on driver comfort. Each false prediction results in unnecessary braking or acceleration.

For predicting lane changes, several “classical" models, like Support Vector Machines (SVMs) [2] or Random Forests [3], have been proposed, with only one recurrent neural net having been published recently [4]. These classical methods, though theoretically sound, see maneuver prediction as function estimation. Though the weights on different features can give us a hint as to what the function considers important, understanding these models when prior knowledge is also given as input has lacked clarity in analysis. The question we ponder over is: does/can a system see what a human looks at?, e.g. when one approaches a highway entry ramp the probability of a lane change for vehicles on the ramp is higher, and the human driver slows down with this prior knowledge (see 1).

To answer the above intriguing question, we:
(a) propose the first recurrent neural network making use of an attention mechanism over different features and time steps. This model is designed to understand complex situations and also explain its decisions. Like humans it can shift its focus towards certain important aspects of the current scene.

(b) introduce metrics which indirectly reflect driver’s comfort, and thus allow a meaningful quantification of prediction quality. (c) provide the first comprehensive evaluation of several models aimed at the same task on the same benchmark, and analyze critical corner cases and visually interpret them.

We use the publicly available NGSIM [5] dataset as well as proprietary fleet data (2) to demonstrate encouraging results w.r.t. state-of-the-art methods.

2 RELATED WORK↩︎

Lane change prediction, being a fundamental building block for any autonomous driving task, is a hot topic in research and has been investigated for several years [6]–[10]. Picking the most informative features according to a criterion and then using “classical" methods, like SVMs or Random Forests [2], [3], [11], [12] contributed to the core of research in lane change prediction.

Schlechtriemen et al. [13] analyzed the expressive power of a multitude of features and came to the conclusion that lateral distance to the lane’s centerline, lateral velocity, and relative velocity to the preceding car are the most discriminative features. They introduced two models, a Naive Bayesian approach, and a Hidden Markov Model on top of the Naive Bayesian model, with the vanilla Naive Bayesian approach performing better. In another work Schlechtriemen et al. [3] tackled the problem of predicting trajectories, where they consider lane change prediction as a helping subtask. To achieve better generalization, they fed all the available features to a random forest.

Woo et al. [2] proposed a hand-crafted energy field to model the surroundings of a car for prediction with a custom SVM model. Weidl et al. [14] introduced Dynamic Bayesian Networks for maneuver prediction with input features from different sensors and safety distances to the surrounding vehicles.

A main drawback of the above approaches is the improper handling of the temporal aspect of features. A simple concatenation of features across time loses expressibility in the temporal domain, mainly due to a high degree of correlation in the features. Patel et al. [4] introduced a Structural Recurrent Neural Network for this problem. Three Long Short-Term Memory (LSTM) cells handle the driving and neighbouring lanes, with inputs being the features of the surrounding vehicles in the corresponding lanes as well as features of the target.

Zeisler et al. [15] followed a different scheme by using raw video data instead of high-level features. Lane changes are predicted using optical flow of observed vehicles. General intention prediction is a close relative of maneuver prediction. Jain et al. [16] demonstrated impressive results on predicting driver intentions. The key contribution was to fuse two LSTM cells handling complementary feature spaces.

Attention mechanisms were first introduced in vision and translation tasks with outstanding performance [17]–[19]. The key idea is to guide the model towards certain points of the input, such as important image regions for visual tasks, and particularly relevant words in translation. We integrate a temporal attention mechanism into our model which cherry-picks relevant features across a sequence.

3 PROBLEM DEFINITION↩︎

Our goal is to predict lane change maneuvers of cars surrounding the ego car. Let \(F_t\) be a snapshot of the scene at timestep \(t\) containing \(N\) vehicles. A prediction algorithm assigns a maneuver label \(\{\text{left}:L, ~\text{follow}:F, ~\text{right}:R\}\) to each of the \(N\) vehicles present in \(F_t\). Predicting \(L\) or \(R\) expresses the algorithm’s belief that a vehicle has started a lane change maneuver to the respective side. Predicting \(F\), conversely, implies that a vehicle keeps its current lane.

To obtain a prediction, we use the following features for each of the \(N\) cars (considered as target vehicle) in \(F_t\):

• Target vehicle features: \(G^Z = (m, v_{lat}, v_{long}, a_{lat}, h)\). \(m\): target’s lateral distance to its lane’s center line, \(v_{lat}\): lateral velocity, \(v_{long}\): longitudinal velocity, \(a_{lat}\): lateral acceleration, and \(h\): heading angle. These features are computed in Frenet coordinates. ⁴

• Dynamic environment features, i.e., features of cars surrounding the target: \(G^E = (dt_X\) for \(X \in \{\text{PV}, ~\text{RV}, ~\text{PLV}_L, ~\text{PLV}_R, ~\text{PFV}_L, ~\text{PFV}_R\})\), in accordance with the definition of Nie et al. [20] (see 3). Here \(dt_X\) denotes the temporal distance between the target and car \(X\), i.e., the distance divided by the velocity of the trailing car.

• Static environment features: static features describe the environment type, e.g. map-based features. In the NGSIM dataset an on-/off-ramp is present, which is integrated as \(G^M = (d_{\text{on}}\), \(d_{\text{off}}\), \(\text{lane})\). \(d_{\text{on}}\), \(d_{\text{off}}\) denote the distance to the nearest on-/ off-ramp respectively. \(\text{lane}\) is the one hot encoding of the lane identifier.

4 MODEL↩︎

We propose two kinds of recurrent networks for maneuver prediction, (a) consisting of multiple LSTM cells, and (b) an attention layer on top of that network. We train the models in a sequence-to-sequence fashion, i.e., at every timestep \(t\) an output \(y_t \in \{L, F, R\}\) is generated. The input features (\(G^Z\), \(G^E\) and \(G^M\)) used for our proposed approaches are described in Section 3.

4.1 Long Short-Term Memory Network↩︎

Our basic LSTM [21] network is inspired by the work of Jain et al. [16]. We use three different LSTM cells (\(LSTM_Z\), \(LSTM_E\), \(LSTM_M\)), to process the feature groups (\(G^Z\), \(G^E\), \(G^M\)) respectively. This decoupling into separate LSTMs ensures that the intra-group correlation is high but the inter-group correlation is low. We use the following shorthand notation for an LSTM cell: \[(\mathbf{h}_t^X, \mathbf{\tilde{c}}_t^X) = \texttt{LSTM}(\mathbf{X}_t, \mathbf{h}_{t-1}^X, \mathbf{\tilde{c}}_{t-1}^X)\\\] where \(\mathbf{X} \in \{G^Z, G^E, G^M\}\) is the input, \(\mathbf{h}\) denotes the hidden state and \(\mathbf{\tilde{c}}\) the memory unit.

The full network can be seen in 4. Mathematically, the fusion of these \(3\) LSTMs can be formulated as: \[\label{eq:lstm} \begin{align} \mathbf{u}_t & = \mathbf{W}_F[\texttt{concat}(\mathbf{W}_X \mathbf{h}_t^X + \mathbf{b}_X)] + \mathbf{b}_F\\ \mathbf{o}_t & = \texttt{tanh}(\mathbf{W}_{u}\cdot\mathbf{u}_t + \mathbf{b}_{u}) \\ \mathbf{y}^t & = \texttt{softmax}(\mathbf{W}_o\cdot\mathbf{o}_t + \mathbf{b}_o) \end{align}\tag{1}\] where \(\mathbf{W}\)’s are the weight matrices, \(\mathbf{b}\)’s are bias vectors, \(\mathbf{u}\) is the fusion layer, and \(\mathbf{y}\) is the output layer.

4.2 Attention Network↩︎

The idea behind an attention mechanism is to model selective focus, i.e., on certain parts of the input. It mainly consists of a function processing a key (K) and query (Q) to obtain a context vector, which is the accumulation of multiple keys, weighted by their importance w.r.t. the query. We employ two kinds of attention mechanisms, (a) attention over previous time steps, i.e., self-attention [22], and (b) attention over different feature groups. As opposed to traditional attention approaches, the features we use lie in different spaces and have different modalities. We do not accumulate them, but only change their magnitude in accordance with the weighting, and then accumulate these feature vectors over the time steps; see 5 for an intuitive visualization.

We again partition the features into categories, but with a finer granularity than in 3, viz. \(H^Z = G^Z\), \(H^S = [dt_{\text{PV}}, dt_{\text{RV}}]\), \(H^L = [dt_{\text{PLV}_L}, dt_{\text{PFV}_L}]\), \(H^R = [dt_{\text{PLV}_R}, dt_{\text{PFV}_R}]\) and \(H^M=G^M\). The attention function \(\Psi \colon \mathbb{R}^d \to \mathbb{R}\) is given by: \[\begin{align} \Psi(\mathbf{W}, \mathbf{v}, \texttt{Q}, \texttt{K}) & = \mathbf{v}^T \texttt{tanh}(\mathbf{W}[\texttt{Q; K}]) \end{align}\] For time step \(t\) in all calls of \(\Psi\), layer \(\mathbf{u}_t\) serves as query. Let \(T = \{t\!-\!l, \ldots, t\}\) be the time steps used for self-attention; we have used \(l=20\) in our experiments. For each \(i \in T\) the feature categories are embedded into a higher-dimensional space, and the importances of each feature category, \(\mathbf{\beta}^X_i\), as well as each time step as a whole, \(\mathbf{\gamma}_i\), are determined. Let \(C = \{H^Z, H^L, H^S, H^R, H^M\}\): \[\begin{align} \mathbf{E}_i^X & = \mathbf{W}_{EX} \mathbf{X}_i + \mathbf{b}_{EX}\\ \mathbf{\beta}^X_i & = \Psi(\mathbf{W}_{FX}, \mathbf{v}_{FX}, \mathbf{u}_t, \mathbf{E}_i^X)\\ \mathbf{\gamma}_i & = \Psi(\mathbf{W}_{Time}, \mathbf{v}_{Time}, \mathbf{u}_t, \texttt{concat}([\mathbf{E}_i^{X}])\\ \mathbf{\beta}_i &= \texttt{softmax}([\texttt{concat}(\mathbf{\beta}^{X}_i)]) \end{align}\] where \(X \in C\), \(i \in T\). Eventually, the feature categories are scaled by \(\mathbf{\beta}^X_i\) and the weighted sum is calculated over all time steps. The resulting context vector is appended to the fusion layer and the computation follows 1 . \[\begin{align} \mathbf{\gamma}_t &= \texttt{softmax}([\texttt{concat}( \mathbf{\gamma}_i)]) \\ \mathbf{c}_t & = \sum_{i \in T}\mathbf{\gamma}_i \texttt{concat}([\mathbf{\beta}_i \mathbf{X}_i ])\\ \mathbf{u}_t &= [\mathbf{u}_t; \mathbf{c}_t] \\ \end{align}\]

4.2.1 Visualization of Attention↩︎

Apart from improved performance, another large benefit of attention is its interpretability. Traditionally, simply the magnitude of the attention weights, which are used in the calculation of the weighted mean, is shown [18]. Here though, due to the different scales and dimensions of the feature categories, this does not necessarily lead to expected results. Instead, we calculate the derivative of the predicted class by the attention weights \(\mathbf{\beta}_i^X\) and \(\mathbf{\gamma}_i\), summing over all time steps. This derivative denotes the contribution of category \(X\) to the resulting prediction, even providing the information whether this contribution is positive or negative.

4.3 Training Scheme↩︎

As proposed in [16], we employ an exponentially growing loss to encourage early predictions. The used Softmax loss is weighted with \(\alpha\!\cdot\!w_t\!\cdot\!\exp(-T)\), where at time \(t\) a lane change is imminent in the next \(T\) seconds.⁵ We choose \(\alpha\) s.t. the average value of \(\alpha\!\cdot\!\exp(-T)\) over all frames of each lane change maneuver equals \(1\). For a given maneuver at time \(t\), \(w_t\) is inversely proportional to that maneuver’s global size in training data.

As noted by Schlechtriemen et al. [13], simple scenarios cover a majority of lane changes, and a relatively good prediction can already be achieved by using a small subset of features from \(G^Z\). To tackle this imbalance and induce a meaningful gradient flow for the attention in all cases, we introduce a dropout layer in between layer \(\mathbf{u}\) and \(\mathbf{o}\), i.e. \[\texttt{Dropout} = [\mathbf{W}_{Drop, Fusion};\texttt{ }\mathbf{W}_{Drop, c}]\cdot[\mathbf{u}_t;\texttt{ }\mathbf{c}_t] + \mathbf{b}_{drop}\] With a probability \(p = 0.33\), \(\mathbf{W}_{Drop, Fusion}\) and \(\mathbf{W}_{Drop, c}\) are set to 0 independently, forcing the model to rely solely on its recurrent architecture or attention.

5 Datasets and Evaluation↩︎

5.1 Datasets↩︎

NGSIM: The Next Generation Simulation (NGSIM) [5] project consists of four publicly available traffic data sets. We use the US Highway 101 dataset (US-101) and Interstate 80 Freeway dataset (I-80). Data is captured from a bird’s-eye view of the highway with a static camera, and vehicle-related high-level features are extracted from it. The datasets contain measurements at \(10Hz\). After removing noisy trajectories, \(3184\) lane changes are observed.

Fleet data: The fleet data comes from the fused perception model of in-production cars. This data is captured at \(25Hz\) w.r.t. to a moving ego car equipped with several camera and radar sensors to give a complete \(360^\circ\) view. 830 lane changes are recorded.

5.2 Metrics↩︎

A wide variety of metrics is used to measure the performance of lane change prediction algorithms. Predominantly they are inspired by information retrieval and are computed treating each timestep independently of the other.

• Accuracy: percentage of timesteps correctly classified.

Jain et al. [16] introduced a first version of the following maneuver-based metrics:

• Precision: percentage of true predictions w.r.t. total number of maneuver predictions.

• Recall: percentage of true predictions w.r.t. total number of maneuvers.

• Time to Maneuver (TTM): the interval between the time of prediction and the actual start of the maneuver in the ground truth.

For evaluation, we combine Precision and Recall into the F1 score and refer to the metrics introduced by Jain et. al [16] by their associated group’s name Brain4Cars (B4C).

The ground truth labels are event-wise continuous (6). The information retrieval metrics, however, do not reflect this event-wise nature or what the driver experiences in the car. The car’s controller usually reacts to the first prediction event (7). If the prediction is discontinuous then this causes discomfort to the driver (stop-and-go function). In addition, the prediction event should be as early as possible w.r.t. the ground truth, and the earlier the prediction the higher is the comfort. In order to reflect such comfort-related behaviour we propose the following event-wise metrics:

• Delay: delay (measured in seconds) in prediction w.r.t. the ground truth label. If prediction is perfectly aligned with the ground truth then delay is \(0\).

• Overlap: for a given ground truth event the percentage of overlap for the earliest maneuver predicted. The higher the overlap, the smoother is the controller’s reaction.

• Frequency: number of times a maneuver event is predicted per ground truth event. For the ‘follow’ event this indicates the false positive rate (FPR).

• Miss: number of lane changes completely missed. The higher the number of misses, the higher is the discomfort, as the driver has to intervene.

5.3 Labeling↩︎

The perception of the precise moment when a lane change starts differs from person to person, see 8. Therefore, manually labeling lane changes in fleet data gives us a hint at the intention. However, automatic labeling is useful in the case of NGSIM due to a similar time span of lane changes. Thus, like [13] we have used a \(3\)-second criterion, before the target’s lane assignment changes, to label a lane change. Though human labeling is precise and error-free, it is time-consuming and expensive. Intelligent automatic labeling can be slightly imprecise, but on the other hand, is quicker and might prove to be better for deep models, which could pick up on fine cues imperceptible to humans to achieve a better performance.

6 RESULTS↩︎

We denote our two proposed recurrent methods in [sec:LSTM] as LSTM-E (extended LSTM) and LSTM-A (extended LSTM with attention). For both a hidden size of 128 is used. We implement state-of-the-art baselines to demonstrate better performance of our proposed methods.

6.1 Baseline-Methods↩︎

Frame-based: Features from a single timestep are used.

• Random Forest (RF) [3]: The concatenated features (\(G^Z, G^E, G^M\)) serve as input.

• Naive Bayes (NB) [13]: The features \(m, v_{lat}\) and relative velocity to preceding car are used.

Sequence-based:

• Structural RNN (SRNN) [4]: The SRNN consists of three different LSTM cells which cover the target, left, and right lane respectively. To each LSTM cell the features \(Q\) of three vehicles are given, viz. those of the two neighbors of the target car (\(\text{PV}\) - \(\text{RV}\) / \(\text{PLV}_L\) - \(\text{PFV}_L\) / \(\text{PLV}_R\) - \(\text{PFV}_R\)) and the target car itself. \(Q\) consists of absolute world coordinates, lateral and longitudinal velocity, heading angle, and number of lanes to the left and right. The output of the three LSTM cells is passed on to another LSTM cell, which eventually outputs the prediction.
  

• Vanilla LSTM (LSTM): Vanilla LSTM consisting of a single cell with the concatenated features \((G^Z, G^E, G^M)\).
  

6.2 Quantitative Results↩︎

¿tbl:tab:multicol? shows the results of all tested methods w.r.t. all metrics on the NGSIM and fleet dataset. As can be seen, due to the diversity of the evaluation metrics, some methods excel or fail in different categories. Sequence-based methods easily outperform frame-based methods since the latter carry no information regarding the sequence history. Among sequence-based methods, our three recurrent models, LSTM, LSTM-E, and LSTM-A come out on top (refer to the ‘Total Rank’ column in the table).

On the NGSIM dataset, the LSTM network with attention is the best-performing method. It has the lowest delay while predicting lane changes, a lower false positive rate during ‘follow’, and a good continuous prediction indicated by ‘Overlap’. On our fleet data LSTM-A finishes second. This is mainly due to the sparsity of the dynamic environment features \(G^E\) in fleet data. Thus, the prediction falls back to the target features \(G^Z\), as these are the most discriminative features, and the performance is similar to vanilla LSTM.

6.3 Qualitative Results↩︎

As can be seen from ¿tbl:tab:multicol?, the performance of some methods is relatively similar. Analyzing and interpreting a few critical corner cases will help in assessing the performance and give us clarity about the advantage of an attention mechanism. These critical corner cases are not present in the data. They were created by translating around the existing trajectories w.r.t. their position in the scene, and thus remain realistic. For the fleet data we do not have the static data recording, but instead a moving ego car (drawn in red) from which the measurements of the scene are obtained.

Two types of visualizations are used: (a) a snapshot visualization of a single frame, and (b) a visualization of the temporal development of a scene. The first consists of a single image, showing the ground truth and prediction of a single algorithm for that frame, as well as the attention visualization for the five feature categories. For better readability the categories \(H^Z\), \(H^S\), \(H^L\), \(H^R\) and \(H^M\) are denoted by Target, Same, Left, Right, and Street. (b) is the concatenation of several frames spanning a certain amount of time, along with the prediction of different algorithms.

9 and 10 show the influence of attention on the network’s decision making, highlighting its correct and intuitive contribution.

11 shows the temporal development of two scenes while plotting the output of three algorithms - RF, LSTM-E, and LSTM-A. Overall a superior performance of the recurrent models, especially LSTM-A, can be observed.

7 CONCLUSIONS↩︎

We have proposed an LSTM network with an attention mechanism for lane change prediction, which performs better than existing methods w.r.t. to different evaluation schemes. This is the first work applying such a model to this field, which tackles both prediction quality and understandability. We have also proposed new event-wise metrics catering to driver’s comfort. Results on a public dataset as well as fleet data clearly indicate a high level of comfort, in terms of earliness in prediction, false positive, and miss rate, with our proposed methods for the driver. Moreover, with visual analysis of critical cases we have demonstrated the effectiveness of using attention. In the future, analyzing fleet data with complex scenes using our attention mechanism can shine light on circumventing critical cases for fully autonomous driving. Such understandable mechanisms are helpful in diagnosing and minimizing accidents. This can eventually lead to improved path planning algorithms.

References↩︎

---

1. * indicates equal contribution↩︎

2. \(^1\)Oliver Scheel, Naveen Shankar Nagaraja and Loren Schwarz are with BMW Group, 80788 München, Germany, {oliver.scheel, naveen-shankar.nagaraja, loren.schwarz}@bmw.de↩︎

3. \(^2\)Federico Tombari and Nassir Navab are with the Faculty of Computer Science, Technische Universität München, 85748 Garching bei München, Germany, {tombari, navab}@in.tum.de↩︎

4. Coordinate axis is along the target object’s lane center line.↩︎

5. Exponential weighting of the loss function is not done for the fleet data, as the human labels are error-free.↩︎