1 Introduction↩︎

The empirical findings provide compelling evidence that the proposed NAS-DQN framework achieves a substantial and consistent advantage over both fixed-architecture baselines and random exploration methods across all major performance metrics. Whereas fixed architectures inherently suffer from structural misalignment with task-specific demands—manifesting as underfitting in overly simplistic networks and overfitting in excessively complex ones—NAS-DQN adaptively identifies the optimal network capacity through performance-driven architectural selection. More importantly, NAS-DQN demonstrates pronounced superiority in terms of final performance, sample efficiency, convergence rate, and policy stability relative to its Random-NAS counterpart. The Random-NAS baseline, which maintains identical dynamic update mechanisms but employs uniform architecture sampling, serves as a rigorous control condition isolating the effect of learned selection. Its consistently inferior outcomes confirm that mere architectural variability does not suffice; rather, the intelligent selection mechanism embedded within NAS-DQN constitutes the decisive factor driving performance gains.

Furthermore, the computational efficiency of the proposed method amplifies its practical significance. In contrast to conventional offline RL-NAS approaches that demand extensive computational resources—often amounting to hundreds of GPU-days—NAS-DQN integrates seamlessly within the standard deep reinforcement learning training pipeline, introducing negligible computational overhead while delivering substantial performance improvements. Collectively, these results challenge the prevailing assumption that advanced architecture search is either prohibitively expensive or marginally beneficial in resource-constrained environments. Instead, they establish that online, adaptive architecture optimization should be regarded not as an auxiliary tuning stage but as an intrinsic and indispensable component of the reinforcement learning process itself. Deep Reinforcement Learning (DRL) has achieved remarkable success in solving complex sequential decision-making problems, from game playing to robotics. At the heart of many DRL algorithms, such as the Deep Q-Network (DQN) [1], lies a deep neural network used to approximate a value or policy function. The architecture of this network—its depth, width, and activation functions—is a critical factor that dictates the agent’s ability to learn effective representations and, consequently, its overall performance.

However, the standard practice involves selecting a network architecture a priori and keeping it fixed throughout training. This choice is typically made through heuristics and costly hyperparameter sweeps. To address this, the field of Neural Architecture Search (NAS) has emerged to automate network design. Reinforcement Learning-based NAS (RL-NAS) has become a particularly powerful paradigm, framing architecture design as a sequential decision-making process where an RL controller learns to generate high-performing architectures [2].

Early RL-NAS methods demonstrated state-of-the-art results but were often associated with prohibitive computational costs. This challenge spurred the development of more efficient frameworks. Researchers have introduced techniques like weight sharing, one-shot training, and prioritized experience replay to reduce search time and hardware requirements significantly [3], [4]. Other approaches have focused on improving search efficiency by reformulating NAS as a multi-armed bandit problem [5] or by developing meta-reinforcement learning agents that can generalize architecture design to unseen tasks [6], [7]. The field has also seen a rise in hybrid methods that combine RL with differentiable NAS [8], [9] or evolutionary algorithms [10] to leverage the strengths of multiple search strategies.

While the majority of this research has focused on finding optimal architectures for supervised learning tasks like image classification, the application of NAS to improve the design of RL agents themselves is a crucial and promising research direction [4], [11]. Our work addresses this specific challenge. We propose NAS-DQN, an agent that integrates a lightweight NAS controller directly into the DRL training loop. Instead of committing to a single, static architecture, NAS-DQN periodically evaluates its performance and uses this feedback to intelligently select a new, potentially better, network architecture on the fly, all within a single training run.

This paper investigates the efficacy of online architecture adaptation in DRL. Our contributions are to: (1) implement a complete DRL system from scratch to serve as a controlled testbed; (2) propose the NAS-DQN agent, which integrates a performance-guided controller for dynamic architecture adaptation; and (3) conduct a comprehensive experimental evaluation comparing our learned approach against fixed-architecture and random-search baselines.

The goal of this study is to determine whether an intelligent, learned search strategy provides tangible benefits over simpler fixed or random approaches within a resource-constrained training period. Our results challenge this assumption, providing valuable insights into the trade-offs between search complexity and agent performance in online DRL.

2 Methodology↩︎

To conduct a rigorous and controllable evaluation of architectural adaptation in reinforcement learning, we developed a bespoke simulation framework from first principles. This approach grants us complete transparency and control over the training loop, gradient calculations, and network dynamic. This section formally defines the problem setting, the physical dynamics of the environment, and the mathematical structure of the reward signal.

2.1 The Inverted Pendulum Environment↩︎

We utilize a custom implementation of the classic Inverted Pendulum control task. The problem is modeled as a finite-horizon Markov Decision Process (MDP), defined by the tuple \((\mathcal{S}, \mathcal{A}, P, R, \gamma, T)\).

2.1.1 System Dynamics↩︎

The physical system consists of a pendulum of mass \(m=1.0\) and length \(l=1.0\) under gravity \(g=9.8\). The continuous dynamics are governed by the following second-order differential equation: \[\ddot{\theta} = \frac{g}{l}\sin(\theta) + \frac{\tau}{ml^2}\] where \(\theta\) represents the angle from the vertical, and \(\tau\) is the applied torque.

For simulation, we employ semi-implicit Euler integration with a time step \(\Delta t = 0.02\)s. The discrete time evolution of angular velocity \(\dot{\theta}\) and angle \(\theta\) from step \(t\) to \(t+1\) is given by: \[\begin{align} \dot{\theta}_{t+1} &= \text{clip}\left(\dot{\theta}_t + \left( \frac{g}{l}\sin(\theta_t) + \frac{\tau_t}{ml^2} \right) \cdot \Delta t, \;[-\omega_{\max}, \omega_{\max}]\right) \\ \theta_{t+1} &= \theta_t + \dot{\theta}_{t+1} \cdot \Delta t \end{align}\] where \(\omega_{\max} = 8.0\) rad/s is a saturation limit applied to the angular velocity to maintain numerical stability. Following the update, \(\theta_{t+1}\) is wrapped to the interval \((-\pi, \pi]\).

2.1.2 State and Action Spaces↩︎

To avoid discontinuities in the observation space at \(\theta = \pm\pi\), the raw physical state \((\theta, \dot{\theta})\) is transformed into a 3-dimensional observation vector \(s_t \in \mathcal{S} \subset \mathbb{R}^3\): \[s_t = \begin{bmatrix} \cos(\theta_t) \\ \sin(\theta_t) \\ \dot{\theta}_t \end{bmatrix}\] The agent operates within a discrete action space \(\mathcal{A} = \{0, 1, 2\}\). These discrete actions map to continuous torque values \(\tau_t\) applied to the system as follows: \[\tau_t = (a_t - 1) \cdot \tau_{\max}, \quad \text{for } a_t \in \mathcal{A}\] where \(\tau_{\max} = 2.0\) is the maximum available torque. This results in the action set \(\{-\tau_{\max}, 0, +\tau_{\max}\}\).

2.1.3 Reward Function and Objective↩︎

The objective of the agent is to maximize the expected cumulative discounted reward, \(J = \mathbb{E}[\sum_{t=0}^{T} \gamma^t R_t]\). The immediate reward function \(R(s_t, a_t)\) is designed to encourage balancing the pendulum upright while minimizing energy expenditure: \[R_t = \cos(\theta_t) - \lambda \tau_t^2\] Here, the first term is maximized when the pendulum is perfectly upright (\(\theta=0\)), and \(\lambda = 0.001\) is a coefficient penalizing large control actions. An episode terminates at fixed horizon \(T=200\) steps.

2.2 Q-Network Implementation↩︎

To approximate the optimal action-value function \(Q^*(s, a)\), we employ a flexible, feedforward neural network, denoted as \(Q_{\mathbf{w}}: \mathcal{S} \to \mathbb{R}^{|\mathcal{A}|}\). This network is parameterized by a set of weights and biases \(\mathbf{w} = \{W^{(l)}, b^{(l)}\}_{l=1}^{L+1}\), where \(L\) is the number of hidden layers. The network’s architecture is defined by a configuration \(c = (L, U, \sigma)\), specifying the number of layers, units per layer, and the activation function.

2.2.1 Forward Propagation↩︎

For a given state vector \(s \in \mathcal{S}\), the network computes the Q-values for all actions. Let \(h^{(0)} = s\) be the input. The activation of each subsequent hidden layer \(l \in \{1, \dots, L\}\) is computed recursively: \[h^{(l)} = \sigma\left( W^{(l)T} h^{(l-1)} + b^{(l)} \right)\] where \(W^{(l)} \in \mathbb{R}^{n_{l-1} \times n_l}\) and \(b^{(l)} \in \mathbb{R}^{n_l}\) are the weight matrix and bias vector for layer \(l\), and \(\sigma\) is the element-wise activation function. The final output layer is linear, producing the vector of Q-values: \[Q(s; \mathbf{w}) = h^{(L+1)} = W^{(L+1)T} h^{(L)} + b^{(L+1)}\]

2.2.2 Parameter Initialization↩︎

To ensure effective training, network parameters are initialized carefully. Biases are initialized to zero, \(b^{(l)} = \mathbf{0}\). Weights are drawn from a zero-mean normal distribution, \(\mathcal{N}(0, \sigma_w^2)\), where the variance is chosen to maintain signal propagation, reflecting the specific activation function used: \[\sigma_w^2 = \begin{cases} 2 / n_{l-1} & \text{for ReLU-based activations (initialization)} \\ 2 / (n_{l-1} + n_l) & \text{for tanh (Xavier/Glorot initialization)} \end{cases}\] where \(n_{l-1}\) and \(n_l\) are the fan-in and fan-out of layer \(l\), respectively.

2.2.3 Network Training via Backpropagation↩︎

The network is trained by minimizing the Huber loss over a minibatch of \(N\) transitions sampled from the replay buffer. The loss is given by: \[J(\mathbf{w}) = \frac{1}{N} \sum_{i=1}^{N} \mathcal{L}_{\delta}\left(y_i - Q(s_i, a_i; \mathbf{w})\right)\] where \(y_i\) is the target value from the Double DQN update rule and \(\mathcal{L}_{\delta}\) is the Huber loss with \(\delta=1.0\). The gradient of this loss with respect to the network’s output for the chosen action \(a_i\) is: \[\frac{\partial \mathcal{L}_{\delta}}{\partial Q(s_i, a_i)} = -\text{clip}(y_i - Q(s_i, a_i; \mathbf{w}), -1, 1)\] This initial error is propagated backward through the network layers. The error signal \(\epsilon^{(l)}\) for a layer \(l\) is calculated from the error of the subsequent layer \(\epsilon^{(l+1)}\): \[\epsilon^{(l)} = \left( W^{(l+1)} \epsilon^{(l+1)} \right) \odot \sigma'(z^{(l)})\] where \(z^{(l)}\) is the pre-activation value at layer \(l\) and \(\odot\) denotes the element-wise product. The gradients of the loss with respect to the parameters of layer \(l\) are then computed as: \[\begin{align} \nabla_{W^{(l)}} J(\mathbf{w}) &= h^{(l-1)} (\epsilon^{(l)})^T \\ \nabla_{b^{(l)}} J(\mathbf{w}) &= \epsilon^{(l)} \end{align}\] To stabilize training, we apply gradient clipping. The parameter updates are performed using gradient descent with learning rate \(\alpha\): \[\begin{align} W^{(l)} &\leftarrow W^{(l)} - \alpha \cdot \text{clip}(\nabla_{W^{(l)}} J(\mathbf{w}), -C, C) \\ b^{(l)} &\leftarrow b^{(l)} - \alpha \cdot \text{clip}(\nabla_{b^{(l)}} J(\mathbf{w}), -C, C) \end{align}\] where the clipping threshold \(C\) is set to 1.0.

2.3 The NAS-DQN Agent↩︎

The NAS-DQN agent integrates the process of learning a policy with the process of searching for an optimal network architecture. It is composed of a standard DRL foundation and a super-level NAS controller that dynamically reconfigures the network.

2.3.1 DQN Foundation and Policy↩︎

The agent’s behavior is governed by an \(\epsilon\)-greedy policy, \(\pi(s)\), with respect to its current Q-network, \(Q_{\mathbf{w}}\). At each step, the action \(a_t\) is selected as: \[a_t = \begin{cases} \text{a random action from } \mathcal{A} & \text{with probability } \epsilon \\ \arg\max_{a \in \mathcal{A}} Q(s_t, a; \mathbf{w}) & \text{with probability } 1-\epsilon \end{cases}\] The agent learns by minimizing the loss on transitions \((s_i, a_i, r_i, s_{i+1})\) sampled from a replay buffer \(\mathcal{D}\). It employs the Double DQN algorithm [12] to mitigate overestimation bias. The target value \(y_i\) for a non-terminal state \(s_i\) is calculated using a separate target network parameterized by \(\mathbf{w}^-\): \[y_i = r_i + \gamma Q_{\mathbf{w}^-}(s_{i+1}, \arg\max_{a'} Q_{\mathbf{w}}(s_{i+1}, a'))\] where \(\gamma\) is the discount factor. For terminal states, \(y_i = r_i\). The target network parameters are periodically updated with the main network parameters: \(\mathbf{w}^- \leftarrow \mathbf{w}\).

2.3.2 NAS Controller↩︎

The NASController manages the architecture search. The search space \(\mathcal{C}\) is the Cartesian product of available layers, units, and activation functions: \(\mathcal{C} = \mathcal{L} \times \mathcal{U} \times \Sigma\), where \(\mathcal{L}=\{2,3,4\}\), \(\mathcal{U}=\{32,64,128\}\), and \(\Sigma=\{\text{ReLU, tanh, Leaky ReLU}\}\).

At each architecture update step \(k\), the controller selects a new configuration \(c_{k+1}\) from \(\mathcal{C}\). The selection strategy depends on a decaying exploration probability, \(\epsilon_k\). With probability \(\epsilon_k\), the selection is uniform at random. With probability \(1-\epsilon_k\), selection is guided by past performance. The controller maintains a buffer of the top-performing architectures and their scores, \(\mathcal{C}_{best}\). To stabilize the sampling distribution, the raw performance scores \(S(c)\) are first normalized using their z-score within the buffer: \[\hat{S}(c) = \frac{S(c) - \mu_{\mathcal{C}_{best}}}{\sigma_{\mathcal{C}_{best}} + \varepsilon}\] where \(\mu\) and \(\sigma\) are the mean and standard deviation of scores in \(\mathcal{C}_{best}\), and \(\varepsilon\) is a small constant for numerical stability. The probability of selecting a specific architecture \(c\) is then given by a softmax function over these normalized scores: \[P(c) = \frac{\exp(\hat{S}(c) / \tau)}{\sum_{c' \in \mathcal{C}_{best}} \exp(\hat{S}(c') / \tau)}\] where \(\tau=1.5\) is a temperature parameter that controls the greediness of the selection. After each update, the exploration rate is decayed: \(\epsilon_{k+1} = \max(\epsilon_{min}, \epsilon_k \cdot \rho_{\epsilon})\), with \(\rho_{\epsilon}=0.95\).

2.3.3 The NAS-DQN Training Algorithm↩︎

The complete training process for an agent employing online architecture search (either NAS-DQN or the Random-NAS baseline) is detailed in Algorithm 1. The algorithm highlights the two-level structure of the learning process: the inner loop consists of standard DQN updates on a fixed architecture, while the outer loop periodically triggers an architecture evaluation and transition phase.

2.4 Experimental Protocol and Baselines↩︎

To rigorously evaluate the efficacy of online architecture adaptation, we conduct a comparative study. Each experiment is run for a total of 2000 episodes over 3 independent trials initiated with different random seeds to ensure the robustness of our findings. We compare the performance of our proposed NAS-DQN agent against four distinct baseline strategies, which are detailed below.

2.4.1 Baseline Strategies↩︎

We evaluate against two categories of baselines: fixed-architecture agents, which represent the standard DRL paradigm, and a random-search agent, which serves as a critical ablation study.

1. Fixed-Architecture DQN Agents: Three separate DQN agents are trained with static, pre-defined architectures for the entire duration. These networks, visualized in Figure 2, vary in depth and width to represent a range of model capacities:

• \(c_{\text{small}} = (L=2, U=32, \sigma=\text{ReLU})\)

• \(c_{\text{medium}} = (L=3, U=64, \sigma=\text{ReLU})\)

• \(c_{\text{large}} = (L=4, U=128, \sigma=\text{ReLU})\)

2. Random-NAS DQN Agent: This agent serves as a crucial control to isolate the benefit of intelligent search. It follows the exact same dynamic update protocol as NAS-DQN, including the \(M=200\) episode interval, model transition, and memory pruning. However, its controller samples a new architecture \(c_{k+1}\) from a uniform distribution over the entire search space, \(c_{k+1} \sim \mathcal{U}(\mathcal{C})\), at every update step, irrespective of past performance. This baseline allows us to distinguish the effect of a learned, performance-guided search from the effect of simply introducing architectural diversity during training.

Agent performance is assessed across multiple dimensions:

• Final Performance: The mean and standard deviation of the average episodic return over the final 100 episodes.

• Sample Efficiency: The number of episodes required for the 50-episode rolling average return to first exceed a threshold of 150.

• Peak Performance: The maximum single-episode return achieved during the entire training run.

• Stability: The standard deviation of the episodic return over the final 100 episodes, where lower values indicate a more stable policy.

3 Results and Discussion↩︎

The comprehensive experimental results are summarized in Figure 3, which compares the five agents across various performance dimensions. The plots consistently demonstrate the superiority of the NAS-DQN agent’s dynamic architecture search strategy over both fixed-architecture and random-search baselines.

3.1 Learning Curves and Final Performance↩︎

The learning curve plot (top-left) clearly shows that the NAS-DQN agent learns significantly faster and more consistently than the other agents. It rapidly achieves a high average reward and maintains it, showing less variance across seeds as indicated by the tight error band. The final performance box plot (top-middle) reinforces this observation. NAS-DQN achieves the highest median reward over the last 100 episodes and exhibits a much tighter distribution of outcomes compared to all other methods. In contrast, the "Fixed Large" agent struggles to learn effectively, suggesting its high complexity is a hindrance on this task, likely due to the difficulty of optimizing a larger parameter space with the available data. The "Fixed Medium" agent is the best-performing static baseline, but it is still outperformed by NAS-DQN.

3.2 Sample Efficiency and Stability↩︎

NAS-DQN demonstrates superior sample efficiency, as shown in the "Episodes to Convergence" plot (top-right). It reaches the performance threshold of a 150 average reward in significantly fewer episodes than any other agent. This suggests that by adapting its architecture, the agent can find a suitable level of complexity early in training to facilitate rapid learning. Furthermore, the stability plot (bottom-left) reveals that NAS-DQN achieves the lowest standard deviation in rewards during the final 100 episodes. This indicates that the policy learned by the agent is not only high-performing but also more reliable and less volatile than those learned by the baseline agents.

3.3 Computational Cost↩︎

As expected, the fixed-architecture agents are slightly faster to train because they do not incur the minor overhead of re-initializing network models. However, the "Computational Cost" plot (bottom-left in the first column) shows that NAS-DQN’s training time is comparable to the "Fixed Large" baseline and the Random NAS agent. Crucially, the substantial gains in final performance, sample efficiency, and stability far outweigh this negligible increase in computational cost, making it a highly efficient trade-off.

3.4 Complexity vs. Performance Analysis↩︎

This work challenges the prevailing assumption in deep reinforcement learning that network architectures must be predetermined and fixed throughout training. Our results show that online, performance-guided architecture adaptation delivers consistent improvements in final performance, sample efficiency, convergence speed, and policy stability—without additional computational cost.

Three key conclusions emerge. (1) Integrating learned architecture search within the DRL loop (NAS-DQN) outperforms both fixed and random baselines, achieving an optimal performance–cost trade-off. (2) The pronounced gap between NAS-DQN and Random-NAS confirms that directed selection, not mere structural diversity, drives performance gains. (3) NAS-DQN’s focused exploration of high-reward architectures indicates an emergent meta-learning behavior, as agents adapt their computational structure to task demands.

These findings redefine architecture in DRL as a dynamic, co-evolving property rather than a static hyperparameter. NAS-DQN’s lightweight integration and scalability make adaptive architecture optimization both practical and broadly applicable. Overall, architecture optimization should be viewed as part of learning itself—a continuous process through which agents refine their own computational structure to enhance intelligence and adaptability.

4 Conclusion↩︎

This work challenges the conventional assumption in deep reinforcement learning that network architectures must be fixed and selected offline. Our results demonstrate that online, performance-guided architecture adaptation yields consistent and significant improvements in final performance, sample efficiency, convergence speed, and policy stability—without increasing computational cost. Three key findings emerge. First, integrating learned architecture search into the DRL training loop (NAS-DQN) clearly outperforms both fixed-architecture and random exploration baselines, establishing an exceptional performance–cost balance. Second, the clear gap between NAS-DQN and Random-NAS isolates the true benefit of learned selection, showing that directed architectural choice—not mere diversity—is the critical driver of success. Third, NAS-DQN’s adaptive focus on high-reward architectures indicates an emergent meta-learning capability, as agents self-organize their computational structure to match task demands. These insights redefine how architecture should be treated in DRL: not as a static hyperparameter, but as a dynamic, co-evolving element of the learning process. NAS-DQN’s efficient integration and scalability make online architecture optimization practical across diverse environments and algorithms.

In summary, architecture optimization should be viewed as part of learning itself—a continual adaptation process enabling agents to refine their own computational structures and, ultimately, their capacity for intelligent behavior.

References↩︎