1 Introduction↩︎

The success of deep learning [1] has been driven not only by algorithmic advances but also by innovations in network architecture. In supervised learning, this observation has motivated extensive research on neural architecture search (NAS) [2], [3], producing architectures that are competitive with or better than hand-designed alternatives across vision and language benchmarks. In reinforcement learning (RL) [4], [5], however, automated architecture design has received less attention. Within actor-critic methods, such as proximal policy optimization (PPO) [6], the networks used to instantiate the policy and value functions are typically treated as fixed implementation choices inherited from prior work. This convention may overlook an important design factor: actor-critic architectures can affect learning stability and final performance, motivating automated approaches to architecture discovery for actor-critic systems.

Automated architecture discovery for actor-critic methods faces two main challenges. First, each candidate architecture must be trained before it can be evaluated, making full-budget comparison across many candidates computationally expensive. Second, actor-critic design is open-ended: policy heads, value heads, shared trunks, normalization layers, activation functions, and other modules can be composed in many ways, without a canonical fixed-topology search space. We propose EVOM, an agentic meta-evolution framework that jointly addresses these challenges. A low-fidelity inner evaluation loop provides computationally tractable fitness estimates. At the same time, an LLM-based design agent generates and refines architectures as executable programs, reducing reliance on a manually predefined search space. As illustrated in Fig. 1, EVOM operates as a bi-level optimization: the outer loop evolves a population of architecture programs, and the inner loop evaluates each candidate by training its weights with PPO under a reduced budget.

EVOM also differs from recent LLM-guided policy search methods such as MLES[7], which synthesize executable controller programs that map observations directly to actions. In contrast, EVOM generates trainable actor-critic architectures: the generated programs specify network structure, while PPO learns the weights through environment interaction. This positions the LLM as a reusable design operator within a meta-evolution loop, rather than as a replacement for RL. Our contributions are as follows:

• We identify two challenges in automated actor-critic architecture design, namely evaluation cost and the open-ended design space, and formulate a bi-level meta-evolution framework that addresses them jointly.

• We instantiate architecture search with a low-fidelity PPO evaluator, which provides a practical fitness signal for comparing candidate architectures during evolution.

• We introduce an LLM design agent as a program-level architecture operator for initialization, mutation, and crossover within an evolutionary loop.

• Experiments on Ant-v4 and HalfCheetah-v4 show that EVOM improves over the manually designed baseline, an LLM-guided random search, and an MLES-style programmatic policy search baseline, with ablations indicating the contributions of evolutionary inheritance and LLM-guided design.

2 Related Work↩︎

EVOM is related to three areas: NAS [2] and neuroevolution [8], LLM-assisted automated algorithm design [9]–[11], and actor-critic reinforcement learning [4], [12]. Prior work has studied how to automate architecture design, evolve executable structures, and use LLMs to generate algorithms or programs. In contrast, EVOM focuses on LLM-guided evolution of actor-critic architectures, where the generated programs define network structure and PPO remains responsible for policy learning.

NAS and neuroevolution. NAS automates network design through controller policies, evolutionary search, and differentiable relaxation [2], [13], [14]. Evolutionary computation and genetic programming provide a broader framework for reproduction, evaluation, and selection over executable structures [15], [16]. Neuroevolution applies related principles to policies, weights, and topologies [17]–[20]. EVOM follows this evolutionary view, but applies LLM-guided program generation to the design of trainable actor-critic architectures.

LLM-assisted automated algorithm design. LLMs can generate programs, use feedback, and support iterative design loops [21]–[28]. Recent studies use LLMs for automated algorithm design and program search, including mathematical discovery, heuristic design, self-refinement, black-box optimization, and reward design [10], [29]–[34]. Closely related to our setting, MLES evolves interpretable programmatic controllers with multimodal feedback [7]. In contrast, EVOM uses the LLM design agent to generate trainable actor-critic architectures, leaving policy and value parameters to be learned by PPO through environment interaction.

Actor-critic reinforcement learning. Actor-critic methods, including PPO [6], learn separate policy and value functions and are widely used for continuous-control tasks. In practice, these functions are instantiated by neural networks whose architecture can influence optimization stability, exploration behavior, and advantage estimation. We use PPO from Stable-Baselines3 [35] on MuJoCo tasks [36], [37] as the inner learner for evaluating candidate architectures. Because EVOM outputs explicit programs, the resulting architectures can be inspected, reproduced, and reused rather than treated as opaque policies.

3 Problem Formulation↩︎

We consider episodic continuous-control environments with observations \(o_t \in \mathbb{R}^d\), actions \(a_t \in \mathbb{R}^m\), rewards \(r_t\), and horizon \(H\). Let \(p \in \mathcal{A}\) denote an actor-critic architecture program from an open-ended architecture space \(\mathcal{A}\). Rather than assuming a fixed topology or a closed set of hand-specified choices, \(\mathcal{A}\) is defined by an executable interface and validity constraints. Given \(p\), the actor and critic are instantiated as \[\mu_\theta(o) = \pi_\theta(o; p), \qquad v_\phi(o) = V_\phi(o; p),\] where \(\mu_\theta(o)\) is the continuous-action mean, \(v_\phi(o)\) is a scalar value estimate, and \((\theta,\phi)\) are trainable parameters. Thus, architecture search concerns the outer choice of \(p\), while PPO performs the inner optimization of \((\theta,\phi)\) for an architecture.

Given an architecture \(p\) and a training seed \(s\), let \(U_B(p,s)\) denote PPO training under budget \(B\). The result of \(U_B(p,s)\) is a trained actor-critic policy whose deterministic evaluation return is denoted by \[J(U_B(p,s)) = \mathbb{E}\left[\sum_{t=0}^{H-1} r_t \mid U_B(p,s)\right].\]

Let \(\mathcal{A}_{\mathrm{valid}} \subseteq \mathcal{A}\) denote the subset of programs that compile, satisfy the required actor-critic interface, produce tensors with the expected shapes, and return finite outputs. The full-budget architecture search objective is \[\max_{p \in \mathcal{A}_{\mathrm{valid}}} \; \mathbb{E}_{s}\left[J(U_{B_{\mathrm{full}}}(p,s))\right],\] where \(B_{\mathrm{full}}\) denotes the final training budget. Directly optimizing this objective is expensive because each architecture evaluation requires an RL training run. During search, we therefore use a lower training budget \(B_{\mathrm{low}} \ll B_{\mathrm{full}}\) to obtain a proxy fitness signal. For an architecture \(p\) trained with seed \(s\), the low-fidelity estimate is \[\hat{J}_{\mathrm{low}}(p,s) = \frac{1}{K}\sum_{k=1}^{K} J_k\left(U_{B_{\mathrm{low}}}(p,s)\right),\] where \(J_k\) denotes the return of the \(k\)-th deterministic evaluation episode. This formulation separates architecture selection from parameter learning: the outer problem searches over architecture programs, while PPO learns the policy and value parameters for each architecture.

4 EVOM↩︎

4.1 Overview↩︎

Fig. 1 summarizes the EVOM workflow. Given the problem setup, the LLM design agent first performs population initialization to produce an initial population \(P^{(0)}=\{p_1^{(0)},\ldots,p_N^{(0)}\}\), where \(N\) is the population size. The evolution loop then runs for \(G\) generations, repeating reproduction, evaluation, ranking, and selection. In reproduction, mutation (edit architectures) and crossover (fuse architectures) generate an offspring population \(P'=\{p'_1,\ldots,p'_N\}\). In evaluation, the inner PPO evaluator (low-fidelity) trains each offspring under the low-budget setting and estimates its return using \(K\) deterministic evaluation episodes. Ranking then orders the union of the current elite population and the offspring population according to the evaluation signal. Selection keeps the top \(N\) candidates as the next elite population \(P^{(t+1)}\). After evolution, final evaluation retrains selected elites with full PPO training, and the output stage returns a deployable elite architecture. Algorithm 2 describes the workflow. A key design choice is that the LLM design agent does not observe gradients, trained parameter values, or privileged simulator states. It receives the Environment Description, Fixed State-Action information, Output Requirements, and selected architecture designs, and returns executable actor-critic architecture programs. The LLM proposes architecture programs for the outer loop, while PPO training and environment interaction remain in the inner loop.

4.2 LLM-Guided Evolution Loop↩︎

The evolution process begins with population initialization. The LLM design agent receives the task description, input and output dimensions, program interface, and design constraints, and returns the initial population \(P^{(0)}\). Each architecture \(p_i^{(0)}\) is evaluated immediately, so the initial elite population is selected according to empirical feedback rather than model preference alone. For each generation \(t\), the outer loop constructs a prompt context containing the Environment Description, Fixed State-Action information, Output Requirements, selected architecture designs, and an evolution directive. The evolution directive asks the agent to perform either mutation on one program or crossover on two programs. The LLM design agent then emits the offspring population \(P'\). Each offspring \(p'_i\) in \(P'\) is compiled, trained with PPO under \(B_{\mathrm{low}}\), evaluated, and then ranked together with the current elite population for selection.

The loop is agentic in a restricted sense: it maintains state, takes design actions, and observes empirical feedback. The state is the current elite population together with historical evaluation logs, the actions are program-generation steps for mutation or crossover, and the observation is the evaluation signal returned by the inner PPO evaluator. Because each prompt is conditioned on the current elite population, EVOM can reuse design patterns across generations rather than drawing independent LLM samples at each step.

The prompt state is intentionally compact. For mutation, the prompt records Current Architecture with its label and network design; for crossover, it records Parent 1 and Parent 2 with their labels and network designs. This gives the LLM design agent enough context to compare design patterns without exposing raw trajectories, gradients, or training internals. This compact state representation also supports controlled ablations: removing crossover, removing mutation, or replacing the LLM backbone changes the design operator while keeping the inner PPO evaluator and selection rule fixed.

4.3 Open-Ended Architecture Representation↩︎

Each \(p_i^{(t)}\) is represented as an actor-critic architecture program, which allows the search to evolve architectures beyond a fixed topology or a closed set of layer choices. The program representation remains bounded by a common interface, making open-ended evolution compatible with systematic evaluation. Each program contains a generation label, a structured design description, and an executable Python program. The description records layer types, widths, activations, normalization choices, residual structure, initialization, and actor-critic sharing or asymmetry. The executable program is the authoritative object used for evaluation.

The required interface follows the actor-critic architecture output in Fig. 1. Each \(p_i^{(t)}\) must define PolicyNet(obs_length, act_length), which returns an action mean of shape \((n_b,m)\), and ValueNet(obs_length), which returns a value tensor of shape \((n_b,1)\), where \(n_b\) is the batch size. Depth, width, activation functions, normalization, residual blocks, dropout, initialization, and actor-critic asymmetry remain open to search. Each program is written to an isolated file, imported by a worker, instantiated with task-specific dimensions, and connected to a custom Stable-Baselines3 actor-critic policy wrapper. Programs that fail to compile, produce non-finite outputs, violate tensor-shape constraints, or crash during training are assigned a penalty fitness of \(-1000\). This penalty safeguards PPO evaluation and makes invalid architectures unlikely to survive elitist selection or be used as parents in later generations. This validation step is important because executable-program search is less restricted than a hand-built grammar, and the empirical evaluation loop requires well-defined candidates.

4.4 LLM-Guided Design Operators↩︎

The outer loop uses an LLM as a reusable design agent rather than using it for one-shot architecture generation. The agent receives elite programs, an evolution directive, and an environment description, and returns a structured description and executable program for each offspring architecture \(p'_i\). Population initialization produces the initial set of \(N\) programs. Mutation edits one parent program through semantic changes to components such as activations, capacity, normalization, and residual blocks. Crossover combines two parent programs into a single offspring program, rather than mechanically splicing program text. The default schedule uses mixed reproduction, with \(N/2\) offspring programs generated by mutation and \(N/2\) generated by crossover.

4.5 PPO Evaluation and Final Retraining↩︎

For each candidate architecture, the PolicyNet and ValueNet are trained with PPO under the low-budget setting \(B_{\mathrm{low}}\). The evaluation signal is the mean return over \(K\) deterministic evaluation episodes. After evolution, the best architecture is retrained from scratch under the full-budget setting \(B_{\mathrm{full}}\). Low-fidelity evaluation guides the evolutionary search, while full-budget PPO training determines the reported performance.

5 Experiments↩︎

5.1 Experimental Setup and Details↩︎

We evaluate EVOM on two MuJoCo continuous-control tasks: Ant-v4, with 27-dimensional observations and 8-dimensional continuous torque actions, and HalfCheetah-v4, with 17-dimensional observations and 6-dimensional continuous torque actions. Both tasks use VecNormalize.

For EVOM, the evolutionary search uses population size \(N=16\) and \(G=20\) generations. For each candidate architecture, the low-budget setting \(B_{\mathrm{low}}\) trains PPO for 100k timesteps and estimates the evaluation signal using \(K=3\) deterministic evaluation episodes. The 100k-step budget is used as a search-time proxy to indicate whether an actor-critic architecture can be optimized by PPO, while keeping population-level evaluation feasible on consumer GPUs. The full-budget setting \(B_{\mathrm{full}}\) trains PPO for 5M timesteps, matching the manually designed baseline budget, and we report final results over three independent runs. The code of EVOM can be accessed at https://github.com/xiaofangxd/EVOM.

As shown in Fig. 3, EVOM uses three prompt templates for initialization, mutation, and crossover. Each prompt provides the environment description, fixed state-action information, and output requirements, including network-design JSON and executable PolicyNet/ValueNet program templates. Initialization creates a new actor-critic architecture from the environment description. Mutation supplies the current architecture label and design, then asks the agent to modify either or both networks while producing a different design. Crossover supplies two parent labels and designs, and asks the agent to fuse architectural features into a coherent new actor-critic program. Each output includes a structured architecture description and a runnable Python program.

5.2 Baselines and Ablations↩︎

Manually designed baseline (Manual PPO) uses Stable-Baselines3 PPO with a fixed MLP actor-critic architecture [35]. Random search uses the same LLM generation, PPO evaluation, logging, program interface, and final retraining as EVOM, but samples new architectures without mutation, crossover, and selection. MLES [7] evolves executable controller programs, testing direct programmatic control rather than trainable actor-critic architecture design. For MLES, we use the hyperparameters reported in the original paper. For EVOM, we conduct ablations on the reproduction operator, population size, and LLM backbone. These ablations compare mixed reproduction with mutation-only and crossover-only variants, evaluate \(N \in \{8,16,32\}\), and test DeepSeek V4, Claude Opus 4.7, and Qwen3.6 Plus.

5.3 Evaluation Metrics and Visualization↩︎

For learning curves, we report the historical-best reward. PPO-based methods use logged evaluation returns, while MLES uses controller-level evaluation logs aligned to the same 5M-step horizontal axis. The architecture diagrams in Figs. 4 and 5 compare the manually designed architecture, the best initial-generation actor-critic architecture, and the final selected actor-critic architecture, highlighting design patterns such as residual blocks, normalization placement, actor-critic asymmetry, and output-scale handling.

6 Experimental Results↩︎

6.1 Full-Budget Performance↩︎

Table 1 reports full-budget performance after final retraining. On Ant-v4, EVOM achieves a higher mean return, suggesting that the low-budget evaluation signal can guide the evolution loop toward effective actor-critic architectures. On HalfCheetah-v4, EVOM also obtains the highest mean return. Random search is a stronger baseline on HalfCheetah-v4 than on Ant-v4, but its large standard deviation indicates that independent LLM sampling can occasionally find useful architectures while remaining unstable across full-budget retraining runs.

6.2 Convergence and Architecture Analysis↩︎

Figs. 4 and 5 summarize evolution-loop convergence, PPO learning curves, and discovered architectures. Ant-v4 shows a clear improvement pattern: both the historical-best reward and the elite-population mean increase over generations, and the final selected actor-critic architecture includes residual and normalization components that are absent from the manually designed baseline. On HalfCheetah-v4, the random-search learning curve reaches high historical-best rewards but has much larger final-evaluation variance, whereas EVOM gives a higher and more reliable full-budget mean. This gap between historical-best curves and final retraining results shows why the tables report final full-budget evaluation rather than search-time or best-checkpoint rewards.

6.3 Operator and Population Ablations↩︎

Table 2 reports ablations on the reproduction operator and population size. Mixed reproduction gives the best mean return on both tasks. The crossover-only variant remains competitive on Ant-v4 and improves over mutation-only on HalfCheetah-v4, but both single-operator variants underperform mixed EVOM. This suggests that local editing and design-pattern recombination are complementary. Increasing the population size does not lead to monotonic gains, likely because larger candidate pools also increase reliance on noisy low-budget evaluation signals.

2pt

6.4 LLM Backbone Comparison↩︎

Table 3 compares different LLM backbones. Among the tested backbones, Claude Opus 4.7 yields the highest mean return on Ant-v4, whereas DeepSeek V4 yields the highest mean return on HalfCheetah-v4. The strong but task-dependent results of Claude Opus 4.7 and Qwen3.6 Plus indicate that the LLM backbone changes the distribution of generated architectures, while the PPO evaluator and selection loop remain necessary to identify which designs survive full-budget training.

6.5 Architecture-Level Analysis↩︎

The selected elite architectures differ from the manual PPO architecture in several recurring design choices. The Ant-v4 elite uses residual blocks, normalization, SiLU activations, dropout, and output-scale handling, while the HalfCheetah-v4 elite uses a deeper LayerNorm-SiLU MLP with a larger hidden width than the manually designed 64-by-64 MLP. These design patterns are relevant to PPO because the policy network affects action means and exploration behavior, while the value network affects advantage estimation and policy-update quality.

The best initial-generation architectures serve as controls for one-shot LLM design. In both tasks, the final selected architectures differ from these initial designs, especially in residual organization and normalization placement. This suggests that EVOM does more than select the best initial sample, as the evolution loop preserves and recombines design patterns across generations. Program-level search also provides a practical benefit: each \(p_i^{(t)}\) is an executable actor-critic architecture program, so the final design can be inspected, reproduced, and reused. This contrasts with using the LLM only as a natural-language advisor.

7 Discussion and Limitations↩︎

EVOM uses low-budget PPO evaluation to make population-level search feasible. The resulting proxy can rank many candidate architectures efficiently, but it may not reliably predict full-budget performance. In particular, 100k-step returns may favor architectures that learn quickly but plateau early, or underestimate architectures that require longer training. We therefore treat evolution curves as search diagnostics and report final results after retraining selected elites under the 5M-step full-budget setting.

The benefit of architecture evolution is consistent across the two evaluated tasks. On Ant-v4, EVOM substantially improves over both the manually designed baseline and random search, suggesting that evolutionary refinement helps discover architectures that are not easily obtained by independent sampling. On HalfCheetah-v4, EVOM also outperforms the compared baselines. These results suggest that meta-evolution can improve architecture discovery beyond one-shot or independent LLM-generated candidates.

The LLM design agent may generate programs that are invalid, unstable under PPO, or unnecessarily expensive to train. We assign penalty fitness to failed programs, but future work should reduce such wasted evaluations through program repair or adaptive evaluation budgets.

The current experiments are limited to two MuJoCo locomotion tasks. A broader evaluation is needed to assess generality, especially on tasks with different observation structures, reward densities, contact dynamics, or action spaces. Future work should also study the transfer of discovered architecture patterns across related tasks and coordination with PPO hyperparameter tuning.

8 Conclusion↩︎

We presented EVOM, an agentic meta-evolution framework for automated actor-critic architecture search. EVOM uses an open-ended program representation, LLM-guided design operators, and low-fidelity PPO evaluation to refine candidate architectures across generations. Experiments on Ant-v4 and HalfCheetah-v4 show that EVOM discovers actor-critic architectures that outperform the manually designed PPO architecture, MLES-style direct policy search, and LLM-guided random search under full-budget evaluation. These results suggest that LLMs can serve as architectural design operators within evolution loops, while policy and value parameters are learned via RL.

References↩︎