Sample-efficient Transfer Reinforcement Learning via Adaptive Reward Shaping and Policy-Ratio Reweighting Strategy
June 25, 2026
Abstract
Transfer learning improves policy learning efficiency by reusing knowledge from source tasks, offering a practical paradigm for safe and efficient autonomous highway lane-changing decision-making. However, existing methods often suffer from transfer mismatch caused by distribution shifts between source and target domains, resulting in training instability and performance degradation. Moreover, target-domain adaptation still relies on exploratory interactions, making it difficult to ensure training safety in safety-critical lane-changing scenarios. To address these issues, this study proposes a safe transfer reinforcement learning framework for autonomous highway lane changing. First, an adaptive teacher intervention mechanism based on instantaneous safety cost is developed to reduce unsafe exploration and gradually decay intervention intensity, while the return bounds under the mixed-behavior policy are theoretically analyzed. The intervention process also generates dual-source samples for joint training. Second, a teacher-guided safe transfer learning mechanism incorporates the teacher policy’s action-evaluation information into student learning as a reward-shaping term to improve training safety and efficiency, while gradually weakening teacher guidance as policy safety improves. Third, a teacher-guided optimization weighting mechanism reweights sample contributions during policy optimization via a likelihood-ratio-based factor, thereby improving transfer stability. Experimental results under different traffic densities, together with an additional evaluation on the real-world NGSIM dataset, show that the proposed method achieves improvements of more than 52.2% in safety and 5.0% in efficiency over baseline methods. These results demonstrate the effectiveness and robustness of the proposed method in enabling safety-aware transfer for autonomous highway lane changing across diverse traffic scenarios.
Autonomous driving, safe transfer reinforcement learning, adaptive reward shaping, policy-ratio reweighting.
1 Introduction↩︎
Safe and efficient lane-changing decision-making is critical for autonomous driving systems and intelligent transportation systems envisioned for the 15-Minute City, and reinforcement learning (RL) has shown great potential in autonomous lane-changing tasks [1]. However, learning a lane-changing policy from scratch typically requires extensive environment interactions and exploration, resulting in high training costs and considerable safety risks [2]. To address these issues, transfer reinforcement learning leverages prior knowledge from source tasks to reduce the training burden and improve convergence efficiency [3], [4]. Nevertheless, existing transfer reinforcement learning methods still suffer from several limitations. First, distribution discrepancies between source and target domains often lead to knowledge mismatch, which may further result in poor data utilization, low sample efficiency, training instability, negative transfer, and degraded convergence performance [5]. Second, although transfer mechanisms can improve learning efficiency, target-domain training still generally requires exploration, making it difficult to ensure training safety and limiting the deployment of such methods in safety-critical scenarios [6]. Therefore, existing transfer reinforcement learning methods for autonomous driving still find it challenging to simultaneously achieve training safety and policy learning efficiency.
To address the aforementioned challenges, existing transfer reinforcement learning methods can be broadly classified into three categories: safety-oriented transfer methods [7]–[9], efficiency-oriented transfer methods [10], [11], and domain adaptation-based transfer methods [12], [13]. Safety-oriented transfer methods focus on improving training safety during transfer, typically by reducing hazardous exploration in target-domain training through mechanisms such as teacher guidance [7], robust control [8], and safety constraints [9]. However, unsafe behaviors may still arise during early adaptation to a new environment. Efficiency-oriented transfer methods focus on improving transfer efficiency by reusing prior knowledge from source tasks to reduce the training burden and accelerate policy convergence [10], [11]. Nevertheless, these methods usually lack explicit treatment of target-domain training safety, as well as the distribution shift and training instability caused by source-target discrepancies. Domain adaptation-based transfer methods focus on modeling source-target discrepancies, typically through domain transfer [12], correspondence learning, or representation transfer to improve cross-domain transfer performance [13]. However, a unified and effective treatment of the distribution shift, training instability, and negative transfer caused by source-target discrepancies is still lacking. Therefore, it remains imperative to improve the convergence efficiency and stability of policy learning while ensuring safety, so as to enable fast, stable, and safe reinforcement learning-based decision-making.
To address the above issues, this paper first employs the policy learned from the source task to provide safety guidance for the target task, as illustrated in Fig. 1. On this basis, a teacher-guided safe transfer reinforcement learning framework for autonomous highway lane changing is further developed, as shown in Fig. 2. The proposed framework integrates adaptive teacher intervention, teacher-guided policy learning, and teacher-guided optimization within a unified transfer learning paradigm. Through this design, the method aims to reduce exploration risk, improve the effectiveness of teacher guidance, and enhance transfer stability during target-domain learning.
The main contributions are summarized as follows:
We propose an adaptive teacher intervention mechanism based on instantaneous safety cost, with an adaptive threshold to achieve intervention decay, while the upper and lower bounds of the return under the mixed-behavior policy are theoretically analyzed. In addition, the intervention process naturally generates two types of samples, which can be jointly exploited for dual-source data training according to a mixing ratio determined by the intervention rate.
We propose a teacher-guided safe transfer learning mechanism, in which the action-evaluation information of the teacher policy is incorporated into the student policy learning process as a reward-shaping term. The guiding effect under safety constraints is modulated by the safety-related weight, thereby enhancing the student policy’s safety awareness and learning efficiency.
We further develop a teacher-guided optimization weighting mechanism. By constructing a likelihood-ratio-based weighting factor, the update contribution of samples in policy optimization is reweighted, thereby improving the stability of the transfer process and enhancing the efficiency of teacher knowledge utilization.
Comparative experiments conducted under different traffic densities, together with an additional experiment based on the real-world NGSIM dataset, demonstrate that the proposed method is both effective and robust, and achieves a favorable balance between traffic efficiency and safety.
The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 describes the problem setting and formulation. Sections 4, 5, and 6 present the proposed methodology and the theoretical analysis. Sections 7 and 8 provide the experimental setup and results. Section 9 concludes the whole paper.
2 Related Work↩︎
2.1 Transfer Reinforcement Learning↩︎
Existing studies on transfer reinforcement learning have improved policy adaptation mainly by reusing demonstrations, pretrained policies, or online teacher assistance. Demonstration-based methods accelerate early exploration by incorporating expert trajectories into policy optimization [14], [15], while policy transfer and distillation methods reuse pretrained behavioral or representational knowledge across related tasks [16], [17]. In parallel, teacher-guided approaches further improve learning efficiency through action advising or selective intervention during online training [18], [19]. Such methods are particularly appealing in reinforcement learning because they introduce external guidance without requiring the student to fully rely on offline supervision or direct policy cloning. This online assistance provides a more flexible transfer mechanism than static demonstrations or fixed pretrained policies. However, most existing teacher-assisted methods still provide guidance primarily at the action level. Although such mechanisms can locally reduce unsafe exploration [20], they often fail to transfer teacher-side evaluation signals into the student’s value learning. As a result, the student may follow corrected actions without fully internalizing their long-term safety implications. Moreover, most guidance mechanisms are fixed, heuristic, or only weakly coupled with the student’s learning status [21], making them less suitable for safety-critical tasks that require adaptive guidance.
Different from current methods, we propose a teacher-guided safe transfer mechanism that incorporates teacher-side action evaluation into student policy learning via reward shaping and adaptively modulates the guidance effect with a safety-related weight under safety constraints, thereby improving safety awareness and learning efficiency.
2.2 Safe Transfer Reinforcement Learning↩︎
Safety assurance in teacher-student reinforcement learning is commonly achieved in two different ways. Some studies enforce safety at the execution level by correcting or replacing unsafe student actions before execution; for instance, action shielding methods [22] and their extensions to partially observable settings [23] explicitly prevent actions that may violate safety requirements, while model-based recovery schemes further improve the flexibility of such protection through recovery control or adaptive replanning [24]. Instead of directly correcting unsafe student actions at execution time, another line of research improves training safety by incorporating the teacher into the learning process, either through direct guidance on student decisions or through broader protection of the training process. Some methods provide selective teacher advice only when necessary [25], while others further combine such advice with imitation learning to help the student acquire safer behaviors from teacher-supported decisions [26]. However, existing studies usually use fixed or heuristic intervention schemes, without adapting intervention intensity to the student’s instantaneous safety condition or competence. Moreover, they rarely exploit the student-generated and teacher-guided samples induced by the intervention as complementary data sources for joint training.
Unlike existing approaches, we develop an adaptive teacher intervention framework driven by instantaneous safety cost, in which teacher involvement is progressively reduced through an adaptive threshold, and the resulting mixed-behavior policy is further analyzed theoretically. In addition, the resulting student-generated and teacher-guided samples are jointly utilized for dual-source training under an intervention-dependent mixing strategy.
2.3 Policy Optimization with Heterogeneous Training Data↩︎
Reinforcement learning is typically trained from the agent’s own online interactions, which often suffer from inefficient exploration and unstable policy improvement. To address this issue, early studies augment training with external experience, such as demonstrations or successful trajectories. Some studies exploit demonstrations from related or multiple tasks to improve learning efficiency [27], [28], whereas others incorporate expert state sequences or successful trajectories to guide learning [29], [30]. Building on this idea, subsequent studies began to exploit heterogeneous experience during training, such as combining offline data with online interaction [31], balancing prior data with newly collected samples in offline-to-online reinforcement learning [32], and jointly utilizing demonstrations, successful episodes, and teacher-related data [33]. This setting becomes particularly relevant in teacher-student reinforcement learning, where training experience can also be generated online through teacher intervention. However, these studies usually do not explicitly distinguish student-generated samples from teacher-intervened samples. Consequently, the unique roles of mixed-behavior samples in policy optimization remain insufficiently explored, and their weighting in policy optimization is rarely calibrated according to their relevance to the current policy.
In contrast to conventional methods, we develop a teacher-guided optimization weighting mechanism, where a likelihood-ratio-based factor calibrates the weighting of mixed-behavior samples in policy optimization, thereby improving transfer stability and enhancing the efficiency of teacher knowledge utilization.
3 Problem Description↩︎
This section provides the necessary background on constrained Markov decision processes (CMDPs) and formulates the transfer learning problem considered in this work.
3.1 Constrained Markov Decision Processes↩︎
We consider tasks formulated by constrained Markov decision processes (CMDPs) [34]. A CMDP can be described by a tuple \(\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, \mathcal{P}, r, c, d, \gamma \rangle\), where \(\mathcal{S}\) denotes the state space and \(\mathcal{A}\) denotes the action space. \(\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]\) specifies the transition probability from state \(s\) to the next state \(s'\) after taking action \(a\), with \(s\in\mathcal{S}\) and \(a\in\mathcal{A}\). The function \(r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\) denotes the reward function, and \(c:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}\) denotes the cost function. \(d\in\mathbb{R}_{+}\) and \(\gamma\in[0,1)\) represent the cost limit and the discount factor, respectively. In a safe RL problem, an agent interacts with the environment over time and generates a trajectory \(\langle (s_0,a_0,r_0,c_0,s_0'), (s_1,a_1,r_1,c_1,s_1'), \ldots \rangle\). Starting from the initial state \(s_0\), at each time step \(t\), the agent is in a state \(s_t \in \mathcal{S}\) and takes an action \(a_t \in \mathcal{A}\). It then receives the corresponding reward \(r_t = r(s_t,a_t)\) and cost \(c_t = c(s_t,a_t)\), and transitions to the next state \(s_t' \sim \mathcal{P}(\cdot \mid s_t,a_t)\). This process restarts with \(s_{t+1}=s_t'\) and repeats until a termination condition is met, after which a new trajectory is initiated. The objective is to learn an optimal policy (\(\pi^*\)) that maximizes the expected discounted return while ensuring that the expected discounted cost return remains below \(d\): \[\begin{align} \pi^*= \; &\arg\max_{\pi\in\Pi}\; \underset{(s_t,a_t)\sim\rho_\pi}{\mathbb{E}} \left[ \sum_{t=0}^{\infty} \gamma^t r_t \right], \\ &\text{s.t.} \quad \; \quad \;\;\underset{(s_t,a_t)\sim\rho_\pi}{\mathbb{E}} \left[ \sum_{t=0}^{\infty} \gamma^t c_t \right] \leq d, \end{align}\] where \(\rho_\pi\) is the trajectory distribution induced by the policy \(\pi\), \(\Pi\) is the set of all feasible policies.
3.2 Problem Formulation↩︎
We formalize our problem setting using the transfer learning framework for RL. Given a source task and a target task, transfer learning aims to accelerate the learning process on the target task by leveraging knowledge acquired from the source task [35]. The source task \(\mathcal{M}^\diamond\) is expected to provide useful knowledge \(\mathcal{K} = \{\pi^\diamond \}\) to the agent learning the target task \(\mathcal{M}^\circ\), thereby enabling the agent to learn the target policy more efficiently on \(\mathcal{M}^\circ\).
3.2.1 State Space↩︎
A state describes the environment configuration with which the agent interacts. The state of each vehicle \(i\) is defined as \(s_i=(p_i, \Delta x_i, \Delta y_i, \Delta v_i^{x}, \Delta v_i^{y})\), where \(p_i\) is a presence indicator specifying whether vehicle \(i\) is observable; \((\Delta x_i, \Delta y_i)\) denote the longitudinal and lateral positions of the surrounding vehicle \(i\) relative to the ego vehicle, and \((\Delta v_i^{x}, \Delta v_i^{y})\) represent the relative longitudinal and lateral velocities between the surrounding vehicle \(i\) and the ego vehicle, respectively. The overall state representation is given by \(S=[s_1, s_2, \ldots, s_i, \ldots, s_M]\), where \(M\) is the maximum number of observed vehicles.
3.2.2 Action Space↩︎
The agent selects actions from the action space \(\mathcal{A}\), which consists of high-level control commands. The action space \(\mathcal{A}\) is defined as: \[\mathcal{A} = \left[ A^{\leftarrow}, A^{\rightarrow}, A^{\uparrow}, A^{\downarrow}, A^{\oslash} \right],\] where \(A^{\leftarrow}\), \(A^{\rightarrow}\), \(A^{\uparrow}\), \(A^{\downarrow}\), and \(A^{\oslash}\) correspond to turning left, turning right, accelerating, decelerating, and cruising, respectively.
3.2.3 Reward Function↩︎
The source and target tasks adopt different reward functions. Given the stringent safety requirements of safety-critical applications for transfer, we construct a source task that is free of extrinsic task rewards to train the teacher agent, with the primary objective of acquiring safety-relevant knowledge. During training, no external task return is used; instead, to improve exploration efficiency, an intrinsic auxiliary reward is introduced to encourage active exploration and improve state-space coverage. \[r^\diamond = \left\| \mathbf{p}_{t}-\mathbf{p}_{t-1} \right\|_2 ,\] where \(\mathbf{p}_t = (x_t, y_t)\) denotes the position of the ego vehicle at time step \(t\).
In contrast, the target task follows a standard task-driven extrinsic reward mechanism, aiming to simultaneously promote safety, traffic efficiency, and task completion. In the highway lane-change scenario considered in this paper, the ego vehicle must complete a lane change toward the target lane and maintain stable driving within multi-lane traffic, while avoiding collisions, maintaining appropriate driving speed, and reaching the target lane, thereby enabling safe and efficient longitudinal progression and lateral decision-making. Accordingly, the reward received by the student agent at each time step is defined as a weighted sum of several components, including: a safety term that penalizes collision events, an efficiency term that encourages maintaining an appropriate driving speed, and a goal term that guides the policy toward the target lane. Specifically, the instantaneous reward received is defined as: \[r^\circ = w_s r_s + w_e r_e + w_g r_g,\] where \(r_s\), \(r_e\), and \(r_g\) denote the safe reward, efficiency reward, and goal reward, respectively. \(w_s\), \(w_e\), and \(w_g\) are the corresponding weighting coefficients.
A safe reward is imposed such that: \[r_s = \begin{cases} -1, & \text{if a collision occurs}, \\ 0, & \text{otherwise}, \end{cases}\] which discourages unsafe behaviors that may cause collisions.
To promote driving efficiency, the speed reward is defined as: \[r_e = \min \bigl(\frac{ v_{t} - v_{\min}}{ v_{\max} - v_{\min}},1 \bigr),\] where \(v_{t}\) denotes the speed of the ego vehicle at time step \(t\).
To facilitate task completion in the highway lane-change scenario, a goal term is further introduced as: \[r_g = 1 - \frac{\left|\ell_t-\ell_g\right|}{\mathcal{L}},\] where \(\ell_t\) and \(\ell_g\) denote the indices of the ego vehicle’s current lane and the goal lane, respectively, and \(\mathcal{L}\) is the total number of lanes on the current road. This normalized shaping reward increases as the ego vehicle approaches the target lane, thereby encouraging timely lane-change completion.
3.2.4 Cost Function↩︎
In addition to the reward design, we design a safety-related cost function to explicitly penalize situations with insufficient longitudinal safety margins. Specifically, we consider the time headway (THW) between the ego vehicle and the closest leading vehicle in the current lane. Let \(d_\text{head}(t) \ge 0\) denote the longitudinal distance between the ego vehicle and the leading vehicle at time step \(t\). The THW is defined as: \[\mathrm{THW}_t = \frac{d_\text{head}(t)}{\max(v_t,\varepsilon)},\] where \(\varepsilon>0\) is a small constant to avoid numerical issues.
Subsequently, a THW-based cost term is incorporated to penalize cases where the time headway falls below a predefined safety threshold \(T_{\mathrm{safe}}\). The instantaneous cost is defined as: \[c_{\mathrm{thw}}(t)=\max (0,1-\dfrac{\mathrm{THW}_t}{T_{\mathrm{safe}}}).\] This cost encourages the agent to maintain an adequate following distance, thereby reducing the risk of rear-end collisions and the possibility of longitudinal collision.
4 Transfer Learning Strategy↩︎
This section introduces the core components of the proposed method, including the adaptive action intervention mechanism and the teacher-guided safe policy learning scheme. These components establish the key principles for safe interaction and knowledge transfer, which will later be integrated into a unified training framework.
4.1 Safe Exploration via Adaptive Action Intervention↩︎
After completing the training of the teacher model, we train a new policy, referred to as the student policy, dedicated to the target task. To enhance safety during training and improve the student’s performance, we incorporate the teacher policy into the training loop, allowing the teacher policy \(\pi^t\) and the student policy \(\pi^s\) to work together and form the mixed behavior policy \(\pi^{\mathrm{mix}}\). The term “teacher” only indicates that this policy is introduced to facilitate student training, and the optimality of the teacher model is not considered in this paper.
4.1.1 Action Intervention Function↩︎
During student training, the intervention function determines whether the action at the current time step is generated by the teacher policy network \(\pi^t\) or the student policy network \(\pi^s\). The student policy is then updated using the data collected under such interventions. One intuitive design principle for the intervention function is to trigger intervention when the student’s behavior deviates from that of the teacher. In this work, we design a cost-based intervention function based on the instantaneous cost, as defined below: \[a_t = \begin{cases} a_t^t, & \text{if } c_{t-1} > \tau, \\ a_t^s, & \text{otherwise,} \end{cases}\] where \(\tau\) is the intervention threshold, which controls when the teacher policy intervenes. \(a^t_t\) and \(a^s_t\) denote teacher action and student action at the time step \(t\), respectively. The student policy keeps sampling with \(\pi^{\mathrm{mix}}=\pi^{s}\) at the start of a trajectory. Once the first event \(c_{t-1}>\tau\) is encountered, we set \(\pi^{\mathrm{mix}}=\pi^{t}\) until the end of the episode. Consequently, the teacher policy serves to improve the safety of data collection. Based on this intervention function, the mixed behavior policy can be expressed in the following form: \[\pi^{\text{mix}}(\cdot | s) = \mathcal{T}(s) \pi^t(\cdot | s) + (1 - \mathcal{T}(s)) \pi^s(\cdot | s), \label{General32representation32of32mixed32behavior32policy}\tag{1}\] where \(\mathcal{T}(s)=1\) indicates that the teacher intervenes in action selection, whereas \(\mathcal{T}(s)=0\) indicates otherwise. The mixed behavior policy defined in 1 provides an intuitive mechanism for safe exploration by allowing the teacher to intervene whenever necessary. A theoretical analysis of its effectiveness is presented in Section 6.
4.1.2 Intervention Decay↩︎
Teacher intervention can prevent unsafe behaviors and accelerate the student’s learning process, substantially improving the student’s performance in the early stages of training. However, excessive reliance on the teacher’s guidance may hinder the student’s final performance. To address this issue, we introduce an intervention decay mechanism that gradually reduces the teacher’s intervention, thereby reducing the student’s dependence on the teacher. During training, this mechanism adaptively updates the intervention threshold \(\tau\) based on recent safety performance. Specifically, the violation rate is defined as the proportion of time steps within an epoch for which the instantaneous cost is greater than zero. Let \(r^*\) denote the desired target violation rate, and \(\hat{r}(e)\) denote the average violation rate at epoch \(e\) computed using a sliding window over the most recent \(W\) epochs. The threshold update rule is given by: \[\tau(e+1)=\mathrm{clip}\Big(\tau(e)+\kappa\big(r^{*}-\hat{r}(e)\big),\,0,\,\tau_{\max}\Big), \label{eq:tau95update}\tag{2}\] where \(\kappa > 0\) is the update step size, \(\tau_{\max}\) is the maximum allowable threshold, and \(\mathrm{clip}(\cdot)\) ensures \(\tau\) remains within a feasible range. This update rule admits an intuitive interpretation. If \(\hat{r}(e)>r^{*}\), i.e., the current policy is riskier than desired, then \(r^{*}-\hat{r}(e)<0\) and \(\tau(e+1)<\tau(e)\), which lowers the intervention threshold and leads to more frequent teacher interventions. Conversely, if \(\hat{r}(e)<r^{*}\), then \(\tau(e+1)>\tau(e)\), allowing the student to act more often and thereby progressively decaying teacher intervention.
4.2 Safety-Guided Policy Learning↩︎
To accelerate learning under safety constraints without making the student overly dependent on the teacher, we incorporate teacher-guidance information into an entropy-regularized actor–critic framework from two aspects, namely value-target construction and optimization weighting, as illustrated in Fig. 3. Specifically, the teacher’s action-evaluation signal is introduced as a reward-shaping term in the value-target update, yielding a target with an adaptive safety weight. In addition, teacher-intervened transitions are assigned a likelihood-ratio-based optimization weight according to the relative likelihood of the executed action under the current student and teacher policies. In this way, teacher guidance improves both learning safety and transfer stability during iterative optimization.
4.2.1 Teacher-induced Reward Shaping↩︎
For any transition \(( s, a, r, c, s')\), in addition to the extrinsic environment reward \(r\), we compute an auxiliary signal from the teacher policy to quantify how well the action \(a\) aligns with the teacher’s behavior: \[\text{bonus} = \log \pi^{t}(a \mid s).\] This term encourages the student to produce teacher-consistent actions more frequently during early exploration, thereby improving sampling safety and enhancing learning efficiency. Moreover, to adaptively adjust the influence of the teacher signal according to the current safety status, we scale this auxiliary signal by a safety weight \(\beta\), which inherently serves as a Lagrange multiplier associated with safety cost constraints. This weight varies dynamically with the degree to which the current policy satisfies the safety constraint, strengthening teacher guidance when the policy is unsafe and gradually attenuating it as safety improves, thereby achieving an adaptive trade‑off between safety guidance and task performance.
4.2.2 Value Target Construction↩︎
Within the maximum-entropy actor–critic framework, the update of the action-value function is typically based on a target value obtained by marginalizing over the next-state action distribution, thereby incorporating entropy regularization into value estimation. Specifically, given the next state \(s'\), the policy \(\pi_\theta(\cdot\mid s')\) outputs the action probabilities \(p_\theta(\cdot\mid s')\) and the corresponding log-likelihoods \(\log p_\theta(\cdot\mid s')\). The expected next-state value is then computed as: \[V_{\text{next}}(s') = \sum_{a'} p_\theta(a'\mid s') \Big( \min_{i=1,2} Q_i(s',a') -\alpha \log p_\theta(a'\mid s') \Big),\] where \(\alpha\) is the entropy regularization coefficient, and \(Q_i^{\text{targ}}\) denotes the target action-value function parameterized by a slowly updated target network to stabilize learning.
To incorporate teacher-guidance information into policy learning, we treat the teacher’s evaluation signal \(\text{bonus}\) for the current action as an auxiliary bonus reward and add it to the extrinsic environment reward \(r\), yielding an augmented target: \[Q_{\text{target}}(s,a) = r + \beta \cdot \text{bonus} + \gamma V_{\text{next}}(s'), \label{eq-q-target}\tag{3}\] where \(\gamma\) is the discount factor. Compared with the standard maximum-entropy update, this work introduces an additional safety gain term induced by the teacher signal. By unifying teacher priors, safety constraints, and maximum-entropy value estimation within a single update framework, the learning objective of the student critic is no longer merely to achieve optimal expected return in the environmental sense, but is instead transformed into a risk-sensitive value optimality criterion that incorporates safety-preference-aware corrections.
4.2.3 Optimization Weight Construction↩︎
Besides incorporating the teacher’s evaluation signal into the value target, we further adjust the contribution of teacher-intervened transitions during optimization by introducing a likelihood-ratio-based weighting factor. Specifically, for a teacher-intervened action \(a\), we define: \[\rho = \exp(\log \pi^s(a|s)-\log \pi^t(a|s)),\] where \(\pi^s\) and \(\pi^t\) denote the student and teacher policies, respectively. For student-generated transitions, we simply set \(\rho=1\). This coefficient is used to reweight the optimization losses of both the actor and the critic: when the student policy can adequately explain the teacher’s actions, the corresponding samples are assigned larger update weights, thereby facilitating effective transfer of safety knowledge to the student policy; conversely, when a significant mismatch exists between the student and teacher policies, the weights are clipped and suppressed to prevent unstable updates caused by safety priors in distributionally inconsistent regions. Accordingly, the critic network losses can be written in the weighted form: \[\mathcal{L}_{Q_i} = \frac{1}{2} \, \mathbb{E} \left[ \rho \big( Q_i(s, a) - Q_{\text{target}}(s,a) \big)^2 \right], \quad i \in \{1, 2\}, \label{eq-critic-loss}\tag{4}\] where \(\mathcal{L}_{Q_i}\) is the critic loss. The same weighting strategy is also applied in the policy optimization step.
5 Teacher-Guided Training Framework↩︎
Based on the transfer learning strategy described above, this section further presents the overall teacher-guided training framework of the proposed method. We first introduce the dual-source data training scheme, which jointly utilizes student-generated and teacher-guided samples for policy optimization. Then, the complete algorithm architecture is summarized to show how safe exploration, teacher guidance, and constrained SAC updates are integrated into a unified training process.
5.1 Model Training with Dual-source Data↩︎
The intervention-driven interaction naturally induces two types of transition samples with distinct origins. Let each transition be denoted by \[x = (s, a, r, c, s').\] According to whether teacher intervention is triggered, the collected transitions can be divided into a student-generated replay buffer and a teacher-generated replay buffer, which are defined as follows: \[\mathcal{D}^s = \left\{ x \mid \mathcal{T}(s) = 0 \right\}, \qquad \mathcal{D}^t = \left\{ x \mid \mathcal{T}(s) = 1 \right\}, \label{eq:dual95buffers}\tag{5}\] where \(\mathcal{T}(s) \in \{0,1\}\) is the intervention indicator, with \(\mathcal{T}(s)=1\) indicating teacher intervention and \(\mathcal{T}(s)=0\) indicating student interaction.
Accordingly, the replay data used for training form a dual-source replay structure composed of \(\mathcal{D}^s\) and \(\mathcal{D}^t\), rather than a single-source experience distribution. Samples in \(\mathcal{D}^t\) are mainly collected in safety-critical states and provide safety-aware guidance, whereas samples in \(\mathcal{D}^s\) are generated during autonomous interaction and preserve task-driven exploration in nominal operating states. Therefore, the two replay sources play complementary roles in policy learning. To jointly utilize them, the mini-batch for each update is constructed as \[\mathcal{B}=\mathcal{B}^t \cup \mathcal{B}^s, \quad |\mathcal{B}^t| = \eta |\mathcal{B}|, \quad |\mathcal{B}^s| = (1 - \eta) |\mathcal{B}|, \label{eq:batch95sampling}\tag{6}\] where \(\mathcal{B}^{t} \subset \mathcal{D}^{t}\) and \(\mathcal{B}^s \subset \mathcal{D}^s\) denote the teacher-generated and student-generated subsets contained in the current mini-batch, respectively, and \(\eta\) is a mixing hyperparameter determined by the teacher intervention rate during training.
Under this dual-source replay setting, the SAC actor is optimized on the mixed mini-batch as \[J_{\pi}(\theta) = \mathbb{E}_{s \sim \mathcal{B},\, a \sim \pi_{\theta}(\cdot|s)} \left[ \alpha \log \pi_{\theta}(a|s) - \min_{i=1,2} Q_i(s,a) \right], \label{eq:actor95mixed}\tag{7}\] where \(\pi_{\theta}\) is the student policy, \(\alpha\) is the temperature coefficient, and \(Q_1\) and \(Q_2\) are the twin critics. Since \(\mathcal{B}\) contains both teacher-generated and student-generated samples, the actor update is performed on replay states drawn from both intervention-triggered and autonomous interaction regions.
5.2 Algorithm Architecture↩︎
The overall training procedure of the proposed method is summarized in Algorithm 4. At a high level, the proposed framework integrates teacher-guided safe interaction and constrained policy optimization into a unified SAC-based training procedure. During interaction, the student acts autonomously as long as the current behavior remains safe; once the instantaneous safety cost exceeds the intervention threshold, the teacher takes over control for the remainder of the current trajectory to prevent unsafe exploration and provide corrective guidance (Lines 3–15). For each executed action, teacher guidance is incorporated in two complementary ways. First, the teacher’s action evaluation is transformed into a reward-shaping bonus and injected into the value target, encouraging the student to favor actions that are more consistent with the teacher’s safe behavior. The influence of this bonus is modulated by the adaptive cost penalty coefficient \(\beta\), which scales the strength of teacher guidance according to the current safety status of the student policy. Second, for teacher-intervened transitions, a likelihood-ratio-based coefficient \(\rho\) is computed from the relative likelihood between the student and teacher action probabilities and is used to reweight the optimization losses (Lines 6–10 and 27–30). This design strengthens the influence of teacher-guided samples that are more compatible with the current student policy while suppressing highly mismatched ones. To exploit heterogeneous experience more effectively, two replay buffers are maintained to separately store student-generated and teacher-intervened transitions (Lines 16–20). When the update condition is met, mini-batches are sampled from both buffers with a mixing ratio \(\eta\) and merged for joint optimization. The reward critics and cost critics are then updated with their corresponding targets, followed by the updates of the policy, entropy coefficient, penalty coefficient, and target networks (Lines 25–31). In addition, the intervention threshold \(\tau\) can be adaptively adjusted according to the epoch-level safety violation rate (Lines 33–34), allowing stronger teacher protection during unsafe stages and gradually reducing teacher reliance as training becomes safer. Through this architecture, the proposed method achieves a balance among safe exploration, effective teacher-to-student knowledge transfer, and stable policy improvement.
6 Theoretical Analysis↩︎
This section provides a theoretical analysis of the proposed teacher-policy intervention mechanism. We establish its effectiveness by showing that teacher interventions can raise the student’s attainable performance upper bound while preserving a guaranteed lower bound.
Lemma 1. Consider a MDP with discount factor \(\gamma \in (0,1)\) and bounded reward \(r(s,a) \in [0, R_{max}]\). For any stochastic policy \(\pi\), define the discounted state distribution of \(\pi\) as: \[d_{\pi}(s) = (1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}\Pr\!\left(s_t=s\,\middle|\,\pi,d_0\right),\] where \(d_0\) is the initial-state distribution and \(\Pr(s_t=s\mid\pi,d_0)\) denotes the state visitation probability at time step \(t\) under policy \(\pi\).
Then the following bound holds [36]: \[\begin{align} \left| J(\pi) - J(\pi') \right| \leq \frac{R_{\max}}{(1 - \gamma)^2} \, \mathbb{E}_{s \sim d_{\pi}} \left\| \pi(\cdot \mid s) - \pi'(\cdot \mid s) \right\|_1. \end{align}\]
Proof. According to the Performance Difference Lemma [37], for any two policies \(\pi\) and \(\pi'\), the difference in their discounted returns can be expressed as follows: \[J(\pi) - J(\pi') = \frac{1}{1 - \gamma} \, \mathbb{E}_{s \sim d_{\pi}} \left[ \mathbb{E}_{a \sim \pi(\cdot|s)} A_{\pi'}(s,a) \right], \label{lamme195p1}\tag{8}\] where \(A_{\pi'}(s,a) \triangleq Q_{\pi'}(s,a) - V_{\pi'}(s)\) denotes the advantage function with respect to \(\pi'\).
For a fixed state \(s\), \[\mathbb{E}_{a \sim \pi(\cdot|s)} A_{\pi'}(s,a) = \sum_a \pi(a|s) A_{\pi'}(s,a).\]
From the definition of the advantage function, we have: \[\mathbb{E}_{a \sim \pi'(\cdot|s)} A_{\pi'}(s,a) = 0. \label{lamme195p3}\tag{9}\]
Therefore, \[\begin{align} &\mathbb{E}_{a \sim \pi(\cdot\mid s)} A_{\pi'}(s,a) -\mathbb{E}_{a \sim \pi'(\cdot\mid s)} A_{\pi'}(s,a)\\ =& \sum_a \bigl(\pi(a\mid s)-\pi'(a\mid s)\bigr)\,A_{\pi'}(s,a), \end{align}\] and using 9 we obtain:
\[\mathbb{E}_{a \sim \pi(\cdot|s)} A_{\pi'}(s,a) = \sum_a \bigl(\pi(a|s) - \pi'(a|s)\bigr) A_{\pi'}(s,a). \label{lamme195p5}\tag{10}\]
Applying Hölder’s inequality [38] to 10 yields \[\left| \mathbb{E}_{a \sim \pi(\cdot|s)} A_{\pi'}(s,a) \right| \leq \|\pi(\cdot|s) - \pi'(\cdot|s)\|_1 \cdot \|A_{\pi'}(s, \cdot)\|_\infty. \label{lamme195p6}\tag{11}\]
Because \(0 \leq r(s,a) \leq R_{\max}\), for any policy \(\pi'\) we have: \[Q_{\pi'}(s,a) \leq \sum_{t=0}^{\infty} \gamma^t R_{\max} = \frac{R_{\max}}{1-\gamma}, \quad V_{\pi'}(s) \leq \frac{R_{\max}}{1-\gamma}.\]
Hence, \[|A_{\pi'}(s,a)| = |Q_{\pi'}(s,a) - V_{\pi'}(s)| \leq \frac{R_{\max}}{1-\gamma}, \label{lamme195p8}\tag{12}\] which implies \(\|A_{\pi'}(s, \cdot)\|_\infty \leq \frac{R_{\max}}{1-\gamma}\).
Substituting 11 and 12 into 8 gives \[\begin{align} |J(\pi) - J(\pi')| &\leq \frac{1}{1-\gamma} \, \mathbb{E}_{s \sim d_{\pi}} \left[ \|\pi(\cdot|s) - \pi'(\cdot|s)\|_1 \cdot \frac{R_{\max}}{1-\gamma} \right] \\ &= \frac{R_{\max}}{(1-\gamma)^2} \, \mathbb{E}_{s \sim d_{\pi}} \|\pi(\cdot|s) - \pi'(\cdot|s)\|_1, \end{align}\] which is exactly Lemma 1. ◻
Assumption 1 (Policy Smoothness in Finite Discrete Action Space). Consider a finite discrete action space \(\mathcal{A}\) with \(|\mathcal{A}| < \infty\). The student policy \(\pi^s\) is assumed to be uniformly lower bounded, i.e., there exists a constant \(\delta > 0\) such that for all states \(s \in \mathcal{S}\) and all actions \(a \in \mathcal{A}\),
\[\pi^s(a \mid s) \ge \delta.\]
Theorem 1. With the action intervention function incorporated, the lower and upper bounds of the return of the behavior policy \(J(\pi^{\mathrm{mix}})\) are respectively expressed as:
\[\begin{align} &J(\pi^t) - \frac{\sqrt{2}(1-\omega)R_{\max}\sqrt{-\log \delta}}{(1-\gamma)^2} \\ \leq &J(\pi^{\mathrm{mix}}) \\ \leq &J(\pi^t) + \frac{\sqrt{2}(1-\omega)R_{\max}\sqrt{-\log \delta}}{(1-\gamma)^2}, \end{align}\] where \(\omega = \frac{\mathbb{E}_{s \sim d_{\pi^{\mathrm{mix}}}} \left\| \mathcal{T}(s) \left[ \pi^t(\cdot|s) - \pi^s(\cdot|s) \right] \right\|_1}{\mathbb{E}_{s \sim d_{\pi^{\mathrm{mix}}}} \left\| \pi^t(\cdot|s) - \pi^s(\cdot|s) \right\|_1}\) is the weighted expected intervention rate determined by the intervention function.
Proof. Using the mixed behavior policy as defined in 1 , \[\label{theorem195p1} \begin{align} &\mathbb{E}_{s \sim d_{\text{mix}}} \left\| \pi^{\text{mix}}(\cdot|s) - \pi^t(\cdot|s) \right\|_1 \\ &= \mathbb{E}_{s \sim d_{\text{mix}}} \left\| \mathcal{T}(s) \pi^t(\cdot|s) + \bigl(1 - \mathcal{T}(s)\bigr) \pi^s(\cdot|s) - \pi^t(\cdot|s) \right\|_1 \\ &= (1 - \omega) \mathbb{E}_{s \sim d_{\text{mix}}} \left\| \pi^s(\cdot|s) - \pi^t(\cdot|s) \right\|_1 . \end{align}\tag{13}\]
Applying Pinsker’s inequality [39], we have: \[\left\| \pi^s(\cdot|s) - \pi^t(\cdot|s) \right\|_1 \leq \sqrt{2 D_{\text{KL}}\bigl(\pi^t(\cdot|s) \parallel \pi^s(\cdot|s)\bigr)}. \label{theorem195p2}\tag{14}\]
For any fixed state \(s\), the Kullback-Leibler (KL) divergence between the teacher policy and the student policy is defined as: \[D_{\text{KL}}\bigl(\pi^t(\cdot|s) \parallel \pi^s(\cdot|s)\bigr) = \sum_a \pi^t(a|s) \log \frac{\pi^t(a|s)}{\pi^s(a|s)}.\]
Since \(\pi^s(a|s) \ge \delta\), it follows that \(\frac{1}{\pi^s(a|s)} \leq \frac{1}{\delta}\). Therefore:
\[\begin{align} &D_{\text{KL}}\bigl(\pi^t(\cdot|s) \parallel \pi^s(\cdot|s)\bigr) \\ &\leq \sum_a \pi^t(a\mid s) \log \frac{\pi^t(a\mid s)}{\delta} \\ &= \sum_a \pi^t(a\mid s) \log \pi^t(a\mid s) - \log \delta \sum_a \pi^t(a\mid s). \end{align}\]
Using \(\sum_a \pi^t(a|s) = 1\) and the definition of entropy \(H\bigl(\pi^t(\cdot|s)\bigr) = -\sum_a \pi^t(a|s) \log \pi^t(a|s) \ge 0\), we obtain:
\[D_{\text{KL}}\bigl(\pi^t(\cdot|s) \parallel \pi^s(\cdot|s)\bigr) \leq -H\bigl(\pi^t(\cdot|s)\bigr) - \log \delta \leq -\log \delta. \label{theorem195p3}\tag{15}\]
By combining (13 ), (14 ), (15 ) with Lemma 1, the following result can be obtained: \[\begin{align} &\left| J(\pi^{\text{mix}}) - J(\pi^t) \right| \\ &\leq \frac{R_{\max}}{(1 - \gamma)^2} \mathbb{E}_{s \sim d_{\text{mix}}} \left\| \pi^{\text{mix}}(\cdot|s) - \pi^t(\cdot|s) \right\|_1 \\ &= \frac{(1 - \omega) R_{\max}}{(1 - \gamma)^2} \mathbb{E}_{s \sim d_{\text{mix}}} \left\| \pi^s(\cdot|s) - \pi^t(\cdot|s) \right\|_1 \\ &\leq \frac{\sqrt{2}(1 - \omega) R_{\max}}{(1 - \gamma)^2} \mathbb{E}_{s \sim d_{\text{mix}}} \sqrt{D_{\text{KL}}\bigl(\pi^t(\cdot|s) \parallel \pi^s(\cdot|s)\bigr)} \\ &\leq \frac{\sqrt{2}(1 - \omega) R_{\max} \sqrt{-\log \delta}}{(1 - \gamma)^2}. \end{align}\]
Therefore, we obtain: \[\begin{align} &J(\pi^t) - \frac{\sqrt{2}(1-\omega)R_{\max}\sqrt{-\log \delta}}{(1-\gamma)^2} \\ \leq &J(\pi^{\mathrm{mix}}) \\ \leq &J(\pi^t) + \frac{\sqrt{2}(1-\omega)R_{\max}\sqrt{-\log \delta}}{(1-\gamma)^2}. \end{align}\] ◻
Remark 1. We present a teacher intervention mechanism whose effect can be theoretically characterized by Theorem 1. The result shows that teacher intervention can not only improve the attainable performance level of the student policy during learning, but also maintain stable performance guarantees. Meanwhile, the effectiveness of intervention is jointly influenced by the intensity of teacher guidance, the regularity of the student policy, and the characteristics of long-term returns.
7 Experimental Setup↩︎
7.1 Simulation Environment↩︎
The proposed method is evaluated in a highway lane-changing scenario. In this setting, the ego vehicle makes lane-changing decisions under the dynamic interactions and influences of surrounding vehicles, with the objective of safely completing a lane change toward the rightmost target lane and maintaining stable driving. At the beginning of each episode, the ego vehicle and surrounding vehicles are randomly initialized on the road. The simulation parameters are set as follows: the lane width is 5 m; the vehicle length and width are 5 m and 2 m, respectively; the initial vehicle speed is randomly sampled from 23 to 25 m/s; the maximum speed is set to 30 m/s; and the desired speed is specified within 20–30 m/s. If the ego vehicle makes an improper lane-change decision during driving that results in an excessively close distance to other vehicles, a collision, or departure from the road boundary, the episode is deemed a failure, and the environment is immediately reset to start the next episode.
7.2 Implementation Details↩︎
The simulation environment is developed based on a modified version of the open-source gym-highway-env simulator [40]. Vehicle motion is propagated using a kinematic bicycle model [41]. For HDVs, longitudinal dynamics and lateral lane-changing behaviors are governed by the Intelligent Driver Model (IDM) [42] and the MOBIL model [43], respectively. The simulation frequency is set to 10 Hz, while the decision-making frequency is set to 2 Hz. To ensure reproducibility and statistical robustness, all algorithms are trained by interacting with the environment for \(5 \times 10^5\) steps using three different random seeds. The discount factor and learning rate are set to \(0.99\) and \(1 \times 10^{-4}\), respectively. Parameter gradients are updated every 100 steps, and each update samples a mini-batch of 256 transitions from the replay buffer. Further details are shown in Table 1. All experiments are conducted on a personal PC equipped with an Intel Core i5-10210U CPU and 16 GB RAM 6.
| Parameter | Description | Value |
|---|---|---|
| \(T_\text{safe}\) | Safe threshold for time headway | 1.2 s |
| \(\eta_{\pi}\) | Actor network learning rate | \(1\times 10^\mathrm{-4}\) |
| \(\eta_{Q}\) | Reward critic network learning rate | \(1\times 10^\mathrm{-4}\) |
| \(\eta_{C}\) | Cost critic network learning rate | \(1\times 10^\mathrm{-4}\) |
| \(d\) | Cost limit | 0.05 |
| \(\gamma\) | Discount factor | 0.99 |
| \(\mathcal{B}\) | Mini-batch size | 256 |
| \(n_e\) | Total steps per epoch | 4000 |
| \(\tau_\text{init}\) | Initial intervention threshold | 0.0 |
| \(\kappa\) | Intervention threshold update step size | 0.5 |
We compare it against several state-of-the-art methods: SAC [44], PPO-Lag [45], and TS2C [36]. Considering safety and traffic efficiency, the following evaluation metrics are adopted: (i) Average Reward \((\uparrow)\): the average reward obtained by the ego vehicle per episode; (ii) Average Cost \((\downarrow)\): the average cost of the ego vehicle for each episode; (iii) Crash Ratio \((\downarrow)\): the ratio of the ego vehicle that collides with surrounding vehicles, represented as a decimal value between 0 and 1; (iv) Average Velocity \((\uparrow)\): the average speed maintained by the vehicle during driving.
8 Experiment and Result Analysis↩︎
8.1 Convergence Performance↩︎
Fig. 5 presents the average reward and average cost curves, and the crash-ratio boxplots for all compared methods during training. As shown by the average reward curves Fig. 5 (a), the proposed method exhibits rapid performance improvement in the early stage of training and quickly reaches a high reward level with relatively few environment interactions. In contrast, although SAC gradually attains an average reward comparable to, or even slightly higher than, that of the proposed method in the later stage of training, it shows substantial fluctuations during the early training phase. PPO-Lag also achieves a rapid increase in return at the beginning of training, but its final reward remains clearly lower than that of the proposed method. Notably, the reward curve of TS2C becomes nearly stable at an early stage and shows almost no further improvement throughout training. This is due to TS2C is able to form a relatively fixed behavioral pattern quickly, yet its subsequent policy optimization capability is weak, making it more prone to getting trapped in a local optimum. As a result, its reward level is markedly lower than that of the proposed method and the other baselines.
From the perspective of safety, the average cost curves as shown in Fig. 5 (b) indicate that the proposed method rapidly reduces the cost to a low level in the early training stage and maintains only small fluctuations throughout the entire training process, demonstrating strong safety and training stability. By contrast, the average cost of SAC remains consistently high and fluctuates considerably. Although PPO-Lag yields a lower average cost than SAC overall, it is still significantly higher than that of the proposed method, suggesting that traditional Lagrangian-constrained methods remain limited in their ability to control risk in dynamic traffic environments. TS2C maintains a relatively low average cost, but its unsatisfactory reward level indicates that this low cost mainly results from an overly conservative policy. Its safety improvement is therefore achieved at the expense of task efficiency, rather than through genuinely efficient and safe decision-making.
The crash-ratio boxplots in Fig. 5 (c) further support these observations. The proposed method exhibits a concentrated crash-ratio distribution, with both the median and interquartile range remaining at relatively low levels, indicating not only superior average safety but also better robustness and consistency across different evaluation episodes. In contrast, SAC and PPO-Lag show more dispersed crash-ratio distributions and contain more high-crash outliers, implying that their safety performance is less stable across different scenarios. Although TS2C has a relatively low median crash ratio, it contains more outliers than the proposed method. Overall, our method not only ensures safety but also improves training efficiency and stability in autonomous highway lane-changing tasks.
![Figure 5: Comparative study. (a) Average reward, (b) Average cost, (c) Crash ratio. Training curves of average reward and average cost versus training steps for different methods in the target-task environment, together with a box plot of the crash ratio. The compared methods include the proposed method, SAC [44], PPO-Lag [45], and TS2C [36]. The results show that the proposed method achieves a superior balance between safety and efficiency, with faster and more stable convergence, lower costs, and a reduced crash ratio while maintaining competitive task returns.](/article/2606.26527/pictures/Comparative_experiment.png "fig:")
| Method | Average Reward | Average Cost | Crash Ratio | Average Velocity (m/s) | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2-4 (lr)5-7 (lr)8-10 (lr)11-13 | Low | Medium | High | Low | Medium | High | Low | Medium | High | Low | Medium | High |
| Ours | 48.01 | 47.34 | 46.03 | 0.09 | 0.11 | 0.16 | 0.01 | 0.00 | 0.01 | 21.97 | 21.80 | 21.30 |
| SAC [44] | 45.34 | 43.57 | 43.16 | 1.92 | 1.69 | 1.84 | 0.06 | 0.08 | 0.09 | 21.37 | 20.76 | 20.46 |
| PPO-Lag [45] | 42.99 | 42.54 | 42.16 | 0.64 | 0.68 | 0.85 | 0.05 | 0.05 | 0.06 | 20.38 | 20.31 | 20.18 |
| TS2C [36] | 27.04 | 27.00 | 27.00 | 0.16 | 0.23 | 0.25 | 0.00 | 0.00 | 0.01 | 6.79 | 6.65 | 6.42 |
8.1.1 Comparative Visualization↩︎
8.2 Performance Evaluation↩︎
8.2.1 Traffic Safety↩︎
For traffic safety, the proposed method demonstrates strong safety performance across all three traffic density scenarios. In the medium-density setting, it achieves an average cost of 0.11, representing reductions of approximately 93.5% and 83.8% compared to SAC (1.69) and PPO-Lag (0.68), respectively. Moreover, its collision rate is 0.00, outperforming SAC (0.08) and PPO-Lag (0.05). In both low- and high-density scenarios, the proposed method maintains similarly stable safety performance. In contrast, SAC consistently exhibits higher cost and collision risk across all densities. Although PPO-Lag improves safety relative to SAC, it remains inferior overall. While TS2C achieves a collision rate comparable to the proposed method, its safety performance largely stems from an overly conservative control strategy. Overall, the proposed method effectively satisfies safety constraints under varying traffic densities, demonstrating strong safety robustness.
8.2.2 Traffic Efficiency↩︎
In terms of traffic efficiency, the proposed method maintains high task performance while ensuring safety. As shown in Table 2, in the medium-density scenario, it attains an average speed of 21.80 m/s, exceeding SAC (20.76 m/s) and PPO-Lag (20.31 m/s) by 5.0% and 7.3%, respectively, indicating its ability to sustain high traffic efficiency under safety constraints. In contrast, TS2C achieves an average speed of only 6.65 m/s, reflecting excessive conservatism and failing to meet the fundamental requirements of lane-changing tasks in terms of efficiency and task completion quality. In both low- and high-density scenarios, the proposed method exhibits stable performance. Although SAC also achieves relatively high speeds, its significantly higher safety cost and collision rate indicate that its efficiency is obtained at the expense of safety. PPO-Lag performs poorly compared to the proposed method, while TS2C consistently exhibits conservative behavior. In summary, the proposed method achieves a favorable balance between safety and efficiency across different traffic densities.
To provide a more intuitive assessment of the capability of the proposed method, we visualize the execution process of the highway lane-changing task. Fig. 6 visualizes the comparative results of the proposed method and the baseline methods. As shown in Fig. 6 (a), the proposed method is able to successfully complete the lane-changing task at an early stage of training and maintain stable driving thereafter. By contrast, SAC, shown in Fig. 6 (b), adopts a more aggressive decision-making strategy. In an effort to maintain a high speed, it performs consecutive cross-lane maneuvers without sufficiently accounting for the dynamic gaps and constraints of vehicles in adjacent lanes, which ultimately leads to a collision. Although PPO-Lag (Fig. 6 (c)) also completes the lane-changing task successfully, its policy is relatively conservative, causing the vehicle to gradually decelerate during subsequent driving and maintain an increasingly large distance from the preceding vehicle. This phenomenon is even more pronounced in TS2C (Fig. 6 (d)), where the policy exhibits stronger conservatism and significantly reduces the driving efficiency of the vehicle.
8.3 Real-World Data Validation↩︎
We further validate the proposed method using the US Highway 101 (US-101) dataset from the Next Generation Simulation (NGSIM) open data, with the corresponding scenario illustrated in Fig. 7, and compare its performance with the baseline method PPO-Lag. From the US-101 dataset, we extract information about surrounding vehicles, which then travel in the simulation environment with their initial positions and velocities provided by the dataset.
| Method | ||||
| Reward | ||||
| Cost | ||||
| Ratio | ||||
| (m/s) | ||||
| Ours | 37.46 | 0.74 | 0.00 | 29.90 |
| PPO-Lag | 20.97 | 0.73 | 0.35 | 10.55 |
As shown in Table 3, from the perspective of traffic safety, the proposed method yields an average cost of 0.74 on the US-101 dataset, which is comparable to the 0.73 achieved by PPO-Lag, while attaining a crash ratio of 0, substantially outperforming PPO-Lag’s 0.35. In addition, the proposed method achieves an average reward of 37.46, significantly higher than PPO-Lag’s 20.97, and increases the average velocity from 10.55 m/s to 29.9 m/s, indicating that it can maintain higher traffic efficiency under realistic traffic interactions. These results demonstrate that, in real-world traffic scenarios based on the NGSIM dataset, the proposed method achieves superior traffic efficiency while maintaining a high level of safety, thereby further validating its effectiveness and robustness in realistic driving environments.
8.4 Ablation Study↩︎
>p0.75cm>p0.75cm>p0.75cm> p0.9cm>p0.6cm>p0.9cm> p0.9cm>p0.6cm>p0.9cm>
p0.7cm>p0.6cm>p0.7cm> p0.7cm>p0.6cm>p0.7cm & & & & & &
(lr)4-6 (lr)7-9 (lr)10-12 (lr)13-15 & & & & & high & & & High & & & High & & & High
- & - & - & & & 36.23 & & &0.76 & & &0.34 & & &21.41
- & - & ✔ & & & 38.87 & & & 0.57 & & & 0.24 & & &19.63
- & ✔ & ✔ & & & 45.82 & & & 0.38 & & & 0.03 & & & 19.77
✔ & ✔ & ✔ & & & 46.20 & & & 0.14 & & & 0.01 & & & 21.44
To systematically evaluate the contribution of each module, we conduct an ablation study with four configurations: (i) the complete method, which includes safety-guided (SG), intervention decay (ID), and dual-source data training (DS); (ii) w/o SG, which removes the safety-guided policy learning module, including both the teacher-induced reward shaping and the optimization weight construction; (iii) w/o SG and ID, which further disables the intervention decay mechanism by fixing the intervention threshold; and (iv) w/o SG, ID and DS, which serves as the baseline without any proposed modules. Fig. 8 presents the training curves of average reward and average cost, along with the box plots of crash ratio. After training, all algorithms are evaluated using three random seeds with 100 test episodes per seed, and the averaged results are summarized in Table ¿tbl:tab-ablation?.
8.4.1 Impact of the Safety-Guided Mechanism↩︎
To evaluate the contribution of the safety-guided mechanism, we compare the complete method with the variant in which SG is removed. As shown in Fig. 8, both methods can achieve the high reward levels, indicating that even without SG, the student policy can still benefit from the high-quality samples generated by teacher intervention. However, the complete method consistently maintains a lower average cost throughout the entire training process, demonstrating that SG plays a significant role in improving training safety. The evaluation results in Table ¿tbl:tab-ablation? further confirm this observation. In the medium-density scenario, for instance, the full method outperforms the variant without SG by increasing the average reward from 46.99 to 47.34, while simultaneously reducing the average cost from 0.28 to 0.11 (a 60.7% reduction) and lowering the collision rate from 0.01 to 0.00. Meanwhile, the average speed increases from 19.27 m/s to 21.80 m/s. These results indicate that, by incorporating the teacher’s behavioral evaluation into the student’s learning process, the SG mechanism enhances the quality of knowledge transfer, leading to more stable transfer performance and a more effective balance between safety and reward.
8.4.2 Impact of Intervention Decay↩︎
To analyze the role of intervention decay, we further compare the variant without SG with the variant in which both SG and ID are removed. As shown in Fig. 8, once the intervention decay mechanism is disabled, the model exhibits low reward and high cost during the early stage of training. Although its performance subsequently approaches the convergence region rapidly, both reward and safety deteriorate noticeably in the later stage of training, accompanied by larger fluctuations and an overall degradation in performance. The evaluation results in Table ¿tbl:tab-ablation? show the same trend. After removing the ID module, the average cost increases by up to 91.3% across low-, medium-, and high-density scenarios, while the collision rates rise significantly to 0.13, 0.21, and 0.24, respectively. These findings suggest that the intervention decay mechanism plays a crucial role in balancing early-stage safety protection and late-stage policy autonomy. Without adaptive decay, teacher intervention cannot be relaxed in a timely manner as the student policy becomes safer, hindering a smooth transition from teacher-assisted behavior to autonomous decision-making. As a consequence, both reward performance and safety outcomes are substantially compromised.
8.4.3 Impact of Dual-Source Data Training↩︎
Finally, to assess the effectiveness of dual-source data training, we compare the variant without SG and ID with the variant in which SG, ID, and DS are all removed. As can be observed from the training curves in Fig. 8, introducing DS generally leads to better learning performance, with lower cost and a more stable optimization process. The crash-ratio boxplot also indicates that the collision frequency is significantly reduced when DS is enabled. Table ¿tbl:tab-ablation? further confirms this gain. The baseline method (w/o SG, ID, and DS) exhibits a collision rate as high as 0.39. Although it achieves relatively high speed, this is largely a superficial effect resulting from aggressive decision-making rather than effective optimization under safety constraints. These results demonstrate that DS enhances both the diversity and safety of replay data by jointly leveraging student-generated samples and teacher-intervened samples, thereby mitigating unsafe behavior patterns commonly observed in single-source learning. Consequently, even in the absence of SG and ID, DS remains an effective mechanism for improving training stability and reducing collision risk.
In summary, the ablation results verify that the three modules contribute in different yet complementary ways. SG improves the efficiency of teacher knowledge transfer and achieves a better balance between safety and performance; ID enables a smooth transition from teacher intervention to autonomous control and is crucial for stable learning; and DS enhances safety robustness by enriching the training data distribution. Their integration yields the best overall performance, thereby validating the effectiveness of the proposed teacher-guided safe transfer reinforcement learning framework.
8.5 Sensitivity Analysis↩︎
To evaluate the impact of the initial intervention threshold on algorithm performance, a sensitivity analysis is conducted by varying its initial value, with \(\tau_\text{init}\in \{0.0,0.25,0.4\}\), and training each configuration independently. The corresponding training curves of average reward and average cost are illustrated in Fig. 9. From the overall results, all configurations converge to similar performance levels, indicating that the proposed method exhibits strong robustness to the choice of the threshold. However, in the early stage of training, the learning dynamics are significantly influenced by different threshold settings. When the threshold is small (e.g., \(\tau_\text{init}=0.0\)), more frequent teacher intervention enables the policy to achieve higher initial rewards while maintaining lower costs, reflecting stronger safety guidance and higher-quality early decisions. In contrast, larger thresholds (e.g., \(\tau_\text{init}=0.25\) and \(\tau_\text{init}=0.4\)) allow for greater autonomous exploration. Although this expands the policy search space, it results in slower reward improvement and larger cost fluctuations in the early stage, indicating increased uncertainty and risk exposure during exploration. Despite these differences, all configurations eventually stabilize and achieve comparable performance, further demonstrating the stability and adaptability of the proposed method under varying intervention intensities.
9 Conclusion↩︎
In this paper, we propose a sample-efficient transfer reinforcement learning framework via adaptive reward shaping and policy-ratio reweighting strategy for autonomous highway lane changing. The proposed method combines adaptive teacher intervention, teacher-guided value-target construction, and optimization weighting of teacher-intervened samples within a unified constrained reinforcement learning framework, thereby improving exploration safety, transfer stability, and the utilization of teacher knowledge during training. These components are tightly coupled within a unified constrained reinforcement learning framework, enabling safer exploration in the early stage and more stable knowledge transfer throughout training. In addition, the effect of the mixed-behavior policy was further analyzed through theoretical return bounds. Experimental results show that the proposed method achieves a favorable balance between traffic efficiency and safety and outperforms existing baseline methods in autonomous highway lane-changing tasks. In future work, we will investigate fusing the teacher and student actions in a more flexible manner and explore the meta-transfer reinforcement learning paradigm for safe and few-shot learning.
References↩︎
This work was supported by the National NSF of China (52302493) and the National K&D Program of China (2023YFB2504700). (Corresponding Author: Yang Li)↩︎
Wenjie Huang, Yang Li, Jingjia Teng, Mingwei Jin, Kai Song, Yougang Bian are with the State Key Laboratory of Advanced Design and Manufacturing Technology for Vehicle, College of Mechanical and Vehicle Engineering, Hunan University, Changsha 410082, China (e-mail: huangwenjie@hnu.edu.cn; lyxc56@gmail.com; tengjingjia@foxmail.com; tipsy13103562826@163.com; song_kaivip@163.com; byg10@foxmail.com).↩︎
Yongfu Li is with the Key Laboratory of Intelligent Air-Ground Cooperative Control for Universities in Chongqing, School of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, China (e-mail: liyongfu@ieee.org).↩︎
Qisong Yang is with the Xi’an Institute of High-Tech, Xi’an 710025, China. (e-mail: qisong.yang.93@outlook.com).↩︎
Helai Huang is with the School of Traffic and Transportation Engineering, Central South University, Changsha, Hunan 410083, China. (email: huanghelai@csu.edu.cn)↩︎