1 Introduction↩︎

Retrosynthetic analysis, the process of identifying synthetic routes to a target molecule from commercially available starting materials, is central to chemical research and drug development [1]. Modern computational retrosynthesis decomposes this problem into two components [2]–[5]: a single-step retrosynthesis model that predicts candidate reactions for a given molecule, and a multi-step planner that orchestrates the overall search by iteratively calling the predictor. As illustrated in 1 (a), given a target molecule, the planner iteratively selects a frontier molecule, queries the single-step model for candidate reactions, expands the search tree, and filters invalid candidates until a valid route is found. The search space grows combinatorially with route depth, making multi-step retrosynthesis an inherently long-horizon planning task: decisions made early constrain downstream options, and a suboptimal choice can lead the search into regions where no feasible route exists.

For the single-step component, template-based models [6] classify over a library of reaction templates, while template-free approaches [7]–[10] generate product-to-reactant mappings directly. For multi-step planning, research has advanced along two main themes: improving the single-step model through self-play on successful trajectories [11], and training value functions to better guide the search process [12]–[16]. Reinforcement Learning (RL) has also been applied to learn expansion policies and value functions from simulated planning experience [17], [18]. Despite significant progress, these methods rely on value functions or heuristics that are trained independently as proxy objectives rather than directly optimized for route completion. Recently, Large Language Models (LLMs) have been applied as retrosynthetic planners. [19] use the LLM as a template-free route generator within an evolutionary pipeline augmented with retrieval. Retro-R1 [20] trains an LLM agent via RL to interactively construct routes, but its two-action interface provides limited flexibility to explore alternative decompositions.

In the broader literature, LLMs have shown increasing reasoning and planning capabilities, from linear reasoning chains [21] to environment-grounded action generation [22] and tree search over reasoning states [23]. LATS [24] takes a further step by embedding Monte Carlo tree search, which balances exploration and exploitation by building a search tree of possible actions, over LLM agent trajectories with environment feedback. This demonstrates that LLMs benefit from operating within an explicit search structure with environmental grounding, rather than relying solely on internal reasoning. Our work brings this insight to retrosynthesis: rather than using the LLM as a value estimator within a fixed search algorithm, or as an unstructured sequential planner, we give the LLM direct control over a structured search process through tool calls. Unlike traditional value functions that score molecules in isolation, an LLM agent can condition decisions on the full search context, enabling adaptive, state-dependent planning.

We introduce RetroAgent, an LLM agent for retrosynthesis planning built on a harness that couples the agent with a structured-memory environment, as shown in 1 (b). The environment maintains an explicit search tree over an AND-OR graph, where OR-nodes represent molecules that can be decomposed in multiple ways and AND-nodes represent reactions that require all reactants to be solved. It automatically propagates node statuses and computes the search frontier, while every reaction is grounded in a validated template set, ensuring that the search state is chemically valid. The agent interacts with this environment through tool calls, choosing between memory tools that advance the search and chemistry tools that provide heuristic guidance. This cleanly separates deterministic state management, such as status propagation and frontier computation, from strategic decisions that require reasoning, such as which molecule to expand and which reaction to apply. The environment thus guarantees a correct search state, while the agent focuses on learning effective search strategies.

Unlike previous approaches that either expand all candidates [12], [13] or commit to exactly one reaction [20], our harness lets the agent learn its own search strategy, including which molecule to prioritize, which and how many reactions to apply, all based on the current search context. We train this agent end-to-end with GSPO [25] on retrosynthesis targets from the USPTO reaction database [12], using a verifiable, outcome-based reward. The reward further incorporates shaping terms that steer the learned behavior: for example, penalizing route depth encourages shorter routes, while rewarding expansion breadth encourages diverse decomposition pathways.

We evaluate RetroAgent on the in-distribution USPTO-190 [12] and the out-of-distribution ChEMBL-1000 benchmarks [13]. On USPTO-190, our agent achieves the highest pass@1 success rate among all methods with a smaller model than Retro-R1 and without relying on model probabilities to rerank predictions, while remaining competitive at larger budgets. The advantage is most pronounced on ChEMBL-1000, where RetroAgent surpasses all state-of-the-art methods on both pass@1 success rate and success rate at larger budgets. This strong out-of-distribution performance suggests that the harness enables generalizable planning rather than dataset-specific memorization.

2 Method↩︎

2.1 Problem Formulation↩︎

Given a target molecule \(m_0\) and a set of commercially available building blocks \(\mathcal{B}\), the goal of multi-step retrosynthesis planning is to find a synthesis route that decomposes \(m_0\) into molecules in \(\mathcal{B}\) through a sequence of valid reactions. Each reaction applies a retrosynthesis template from a template library \(\mathcal{T}\) to decompose a product molecule into precursors; a single-step retrosynthesis model \(f\) proposes ranked candidate templates for any given molecule. A valid route forms a directed acyclic graph from \(m_0\) to building blocks, where every non-building-block molecule has at least one valid decomposition and the target is never consumed as a reactant.

The solution space of multi-step retrosynthesis can be naturally represented as an AND-OR graph \(\mathcal{G} = (\mathcal{V}_M \cup \mathcal{V}_R, \mathcal{E})\): molecule nodes \(\mathcal{V}_M\) have OR semantics, meaning a molecule is solved if any child reaction is solved, and reaction nodes \(\mathcal{V}_R\) have AND semantics, meaning a reaction is solved only if all child molecules are solved. Starting from the target molecule \(m_0\) as the root, the search proceeds by iteratively expanding frontier molecules with candidate reactions from the single-step model, adding reaction nodes and edges \(\mathcal{E}\) connecting molecules to their child reactions and reactions to their child molecules.

2.2 Environment and Tool Interface↩︎

We design an environment where the agent drives the search by calling tools that query and modify an AND-OR graph. The environment maintains the graph as a structured memory that persists across the agent’s interaction turns: each node carries a status \(\sigma \in \{\texttt{Open}, \texttt{Solved}\}\), where a molecule starts as Open and becomes Solved once it is identified as a building block or successfully decomposed. The set of Open leaf molecules forms the search frontier.

The agent interacts with this memory through two categories of tools. Memory tools advance the search: the agent retrieves candidate reactions for a frontier molecule, selects which and how many to apply to grow the graph, and inspects the updated frontier to decide which molecule to expand next. Chemistry tools provide auxiliary heuristics such as synthetic accessibility and building block availability for prioritizing among candidates. We provide the full tool specifications in 6.

The environment automatically handles status propagation, cycle detection, and frontier computation after each tool call, so the agent never needs to reason about graph consistency and can focus entirely on search strategy. Through this interface, the agent learns to adapt its entire search strategy, including which molecule to prioritize, how many reactions to apply, and when to explore alternatives, rather than following the fixed policies of previous approaches [12], [20].

With this interface, the search becomes a sequential decision-making problem. The graph is initialized with the target \(m_0\) as the sole Open node. At each turn, the agent can flexibly invoke tools to query candidate decompositions, expand one or more molecules, inspect node details, or call chemistry tools for heuristic guidance. The multi-turn interaction between agent responses and environment observations forms a complete trajectory that we train end-to-end with RL. The search terminates successfully when a complete route is found, or fails when the budget is exhausted.

When the agent chooses to expand a molecule, the environment executes the expansion and status propagation (2): it adds new reactant nodes, checks each against \(\mathcal{B}\), marks building block matches as Solved, and propagates status upward, where a reaction node becomes Solved when all children are solved following AND semantics, and a molecule node becomes Solved when any child reaction is solved following OR semantics. Propagation only visits ancestors of newly changed nodes, requiring \(O(d)\) time where \(d\) is the route depth; route completion is then determined by a single \(O(1)\) check of the root node’s status. 3 shows an example: Expanding M7 via R4 where both reactants M8 and M9 are building blocks triggers upward propagation through R4, M7, R3, and ultimately solves the root M1.

Remarks. We highlight several design choices in the tool interface. First, the agent can apply one or multiple candidate reactions in a single call, creating several OR branches at once. Existing search-based methods expand with all available candidates [12], [13], while Retro-R1 [20] commits to exactly one reaction per molecule. Our multi-expand design allows the agent to balance between these extremes: it can explore multiple promising reactions to hedge against dead ends while skipping clearly unpromising candidates, all within a single tool call. Second, candidate decompositions returned by the single-step model include building block annotations for each reactant. This allows the agent to immediately assess how close each reaction gets to completion without additional tool calls, reducing episode length and providing a direct signal for prioritizing candidates. Third, the environment pre-filters single-step model predictions that would create cycles or duplicate existing reactions. Since LLMs are unreliable at tracking graph structure [26], offloading validity checks to the environment ensures that the agent only sees valid, non-redundant candidates and can focus entirely on strategic evaluation. Finally, the environment enforces early termination when the agent makes no progress for several consecutive turns or when all frontier molecules have no remaining viable predictions, preventing wasted computation on unsolvable targets.

2.3 Training with Reinforcement Learning (RL)↩︎

We use the LLM as the policy \(\pi_\theta\) and train it end-to-end with RL using verifiable, outcome-based rewards. Each episode is a complete multi-turn search trajectory; only agent tokens are trainable, while tool observations are masked.

2.3.1 Reward Function↩︎

The reward is verifiable and outcome-based: the agent receives a positive reward only when it finds a complete route to building blocks, with no partial credit for incomplete routes. Beyond this binary outcome, we add reward shaping terms that encourage efficient search behavior by penalizing excessive budget usage and deep routes. The reward at episode end is defined as: \[\label{eq:reward} r = \begin{cases} 1.0 - \delta_N - \delta_D + p, & \text{if route completed,} \\ -1.5, & \text{if no valid tool call is ever made,} \\ -1.0, & \text{otherwise,} \end{cases}\tag{1}\] where \(\delta_N\) penalizes excessive search budget measured as the number of single-step model calls and \(\delta_D\) penalizes deep routes. The \(-1.5\) case applies to degenerate trajectories that never emit a well-formed tool call (pure format failure), penalizing them more heavily than genuine but unsuccessful search attempts.

Both penalties follow the same form: zero below a threshold of 20, then 0.02 per unit above, capped at 0.3: \[\delta_N = \min\left(0.3,\; 0.02 \cdot \max(0,\, N - 20)\right), \quad \delta_D = \min\left(0.3,\; 0.02 \cdot \max(0,\, D - 20)\right),\] where \(N\) is the number of single-step model calls and \(D\) is the maximum route depth in reaction steps from the root. The term \(p\) accumulates minor penalties of \(-0.05\) for turns with tool errors or no valid tool calls, discouraging the agent from generating malformed actions. We note that this reward design is flexible: adjusting the relative weights of \(\delta_N\) and \(\delta_D\) induces different search behaviors, e.g., emphasizing \(\delta_D\) encourages shorter routes while emphasizing \(\delta_N\) encourages budget-efficient exploration.

2.3.2 Policy Optimization↩︎

We use GSPO [25], a critic-free policy gradient method that extends GRPO [27] by applying clipping at the sequence level rather than per token. The training objective combines a clipped surrogate policy loss \(\mathcal{L}_{\text{GSPO}}\) with a KL regularization term \(\mathcal{L}_{\text{KL}}\) against the frozen pretrained model.

For each batch, we sample \(B\) target molecules and generate \(G\) rollout episodes per target, indexed by \(i\) and \(g\) respectively. Advantages are computed by normalizing rewards within each group, eliminating the need for a separately trained critic: \[\label{eq:advantage} \hat{A}_{i,g} = \frac{r_{i,g} - \bar{r}_i}{\max(\text{std}(\boldsymbol{r}_i),\, \epsilon)}, \qquad \text{where } \bar{r}_i = \frac{1}{G}\sum_{g=1}^G r_{i,g}.\tag{2}\] The policy loss uses a sequence-level clipped surrogate objective [28]: \[\label{eq:gspo} \mathcal{L}_{\text{GSPO}}(\theta) = -\mathbb{E}_{i,g}\left[\min\left(s_{i,g}\,\hat{A}_{i,g},\;\text{clip}(s_{i,g}, 1{-}\epsilon, 1{+}\epsilon)\,\hat{A}_{i,g}\right)\right],\tag{3}\] where \(s_{i,g} = \exp\!\left(\frac{1}{|\boldsymbol{y}_{i,g}|}\sum_t \log \frac{\pi_\theta(y_t \mid \boldsymbol{y}_{<t})}{\pi_{\text{old}}(y_t \mid \boldsymbol{y}_{<t})}\right)\) is the length-normalized sequence-level importance ratio, \(t\) indexes tokens in the trajectory, and \(\epsilon\) is the clip ratio. To improve sample efficiency, we reuse each batch of rollouts for multiple gradient steps with Truncated Importance Sampling (TIS), clipping the sequence-level importance ratio to a fixed range and discarding samples outside this range to control off-policy variance.

The KL term regularizes against the frozen pretrained model \(\pi_{\text{ref}}\) using the \(k_3\) low-variance estimator [29] to prevent reward saturation: \[\label{eq:kl} \mathcal{L}_{\text{KL}}(\theta) = \beta \cdot \mathbb{E}_t\left[\frac{r_t - 1}{r_t} - \log r_t\right], \qquad r_t = \frac{\pi_\theta(y_t \mid \boldsymbol{y}_{<t})}{\pi_{\text{ref}}(y_t \mid \boldsymbol{y}_{<t})},\tag{4}\] where \(\beta\) is the KL penalty coefficient.

3 Experiments↩︎

We conduct experiments to evaluate RetroAgent through the following questions:

• RQ1. Does RetroAgent outperform existing search-based and LLM-based methods on both in-distribution and out-of-distribution benchmarks?

• RQ2. Is RL training essential, or does the base model already perform well with the structured memory interface?

• RQ3. How do key design choices, specifically multi-expand and depth penalty, affect the agent’s performance?

• RQ4. What search behavior does the RL-trained agent learn, and how does it differ from ablated variants?

3.1 Experimental Setup↩︎

Datasets. Following prior work [12], we use a building block set of 23,081,633 commercially available molecules from eMolecules.¹ For RL training, we extract 20,626 target molecules from the USPTO-Full reaction database [12], filtered to routes with depth greater than 4, with mean depth 5.93. We evaluate on two benchmarks: (1) USPTO-190: 190 challenging target molecules extracted from USPTO-Full [12] with reference routes (mean depth 6.0), serving as the standard in-distribution benchmark; (2) ChEMBL-1000: 1,000 drug-like molecules from the ChEMBL database [13], an out-of-distribution dataset without reference routes, used to test generalizability.

Single-step model. We follow prior work [12] and use a template-based MLP as the single-step retrosynthesis model. The model predicts over 381,302 USPTO reaction templates based on Morgan fingerprints, and uses RDChiral [30] for template application and verification. We use the pretrained weights from DESP [31]. We do not adopt the Retro-R1 weight variants V2, V3, and V4 [20]: V2 and V3 may produce chemically unrealistic reactions [32], and V4 weights are not public. We also do not adopt their two-branch policy that reranks reactions based on predicted probabilities, as we aim to test the learned search capabilities of the agent. Instead, we shuffle all prediction and frontier orderings during both training and evaluation, so the agent evaluates candidates by content rather than positional bias.

Baselines. We compare with both search- and LLM-based baselines. Greedy DFS always selects the reaction with the highest likelihood [12]. Retro* [12] is an A*-like best-first search where the past cost is derived from single-step model probabilities and the future cost is estimated by a learned value network. Retro*-0 [12] is its non-learning variant where the future cost is set to 0. Retro*+ and Retro*+-0 [11] improve Retro* and Retro*-0, respectively, with self-improved single-step models. EG-MCTS [14] improves Retro*+ with experience-guided Monte Carlo tree search. PDVN [13] trains a single-step model with an online algorithm and uses dual value networks for synthesizability and cost estimation. Furthermore, we compare against the state-of-the-art Retro-R1 [20], a Qwen2.5-7B-Instruct-1M agent with a two-action interface that follows depth-first search without backtracking.

Metrics. Following [20], we report two complementary metrics. Pass@1 measures the fraction of targets solved in a single trajectory, directly reflecting the agent’s decision quality without retry. Success rate at budget \(N\) measures the fraction of targets solved within \(N\) unique single-step model calls, capturing cumulative performance as the search budget increases. While the budget definition is the same for all methods, search-based methods consume the full budget within a single deterministic run. LLM-based methods typically cannot exhaust the budget in one trajectory due to the difficulty of long-context modeling; we therefore follow the iterative evaluation protocol [20] and conduct multiple independent trajectories per target with shuffled prediction and frontier orderings, where repeated predictions do not count toward the budget. Details of our evaluation protocol are provided in 10.

Model and infrastructure. We use Qwen3-4B-Instruct-2507 [33] as the base model, a 4B-parameter instruction-tuned model with native tool-calling support. We deliberately choose a smaller model without thinking capabilities to demonstrate that the structured memory interface, rather than raw model capacity, drives planning performance. Training uses the slime framework [34] with Megatron [35] for distributed training and SGLang [36] for inference on \(8{\times}\)H200 GPUs. The tool server runs as a separate process pool with 16 instances, decoupled from the training pipeline; this allows the environment to scale independently of GPU resourc es and avoids blocking gradient computation during tool execution.

Hyperparameter configuration. We optimize with Adam (\(\beta_1=0.9\), \(\beta_2=0.98\)), a constant learning rate of \(1{\times}10^{-6}\), and weight decay of 0.01. Each batch samples 32 target molecules with \(G=8\) rollouts each at temperature 1.0, yielding 256 episodes per batch; with TIS reusing each batch for 2 gradient steps, the effective batch size is 128. The GSPO clip ratio is \(\epsilon = 3.5{\times}10^{-4}\), the KL coefficient is \(\beta = 0.001\), and the TIS clip range is \([0.5, 2.0]\). Each training episode runs for up to 90 turns with a 56K token budget and a maximum route depth of 30 reaction steps. At evaluation time, we use greedy decoding with temperature 0 to ensure deterministic runs, with more generous limits of 120 turns and 120K tokens. For each molecule, we take the 50 highest-scoring predictions from the single-step model, shuffle them, and present at most 20 to the agent, removing positional bias so the agent judges candidates by content rather than rank. Full hyperparameters and training curves are reported in 9.

3.2 Main Results (RQ1)↩︎

¿tbl:tab:main? summarizes results on the in-distribution USPTO-190 and the out-of-distribution ChEMBL-1000. Since Retro-R1 does not report its V1 results on ChEMBL-1000, we reproduce Retro-R1 using its released code and evaluate on this dataset. All other performance results are directly from Retro-R1 [20].

On USPTO-190, RetroAgent achieves 53.26% pass@1 and 85.05% at \(N = 500\) with a 4B-parameter model, outperforming Retro-R1 with 50.00% pass@1 and 84.32% at \(N = 500\), despite using a \(1.75\times\) smaller model of 4B versus 7B parameters and not relying on model probabilities to rerank predictions. Search-based methods such as PDVN train their value networks on all \(\sim\)​300K available routes in USPTO-Full, while RetroAgent uses only 20K targets, highlighting its data efficiency. At higher budgets (\(N \geq 400\)), the exhaustive search of PDVN surpasses LLM-based methods, reaching 92.11% at \(N = 500\). It is important to note that this comparison is not on equal footing: search-based methods apply all available predictions per expansion, up to 50, whereas at evaluation on USPTO-190 our agent selectively applies a mean of 1.71 reactions per expand call. At the same budget \(N\), search-based methods thus explore a substantially larger portion of the search space, which gives them an inherent advantage at high budgets. Despite this, we achieve the highest pass@1 among all methods, demonstrating stronger single-trajectory decision quality.

Our advantage is most evident on the out-of-distribution ChEMBL-1000, where RetroAgent achieves 73.82% pass@1 and 84.40% at \(N = 500\), outperforming Retro-R1 by 5.30% in pass@1 and 1.98% at \(N = 500\), and surpassing all search-based methods including PDVN (80.60% at \(N = 500\)). This suggests the structured memory interface helps the agent acquire transferable planning strategies rather than memorizing dataset-specific reaction patterns. A detailed out-of-distribution breakdown is provided in 12.

3.3 Ablation Studies (RQ2, RQ3)↩︎

We ablate three aspects of our agent on USPTO-190: the necessity of RL training, the effect of multi-expand versus single-expand, and the impact of the depth penalty \(\delta_D\). All variants use the same evaluation protocol and tool interface; the zero-shot baseline uses temperature 0.6 since greedy decoding with the untrained model produces degenerate outputs.

The results are reported in ¿tbl:tab:ablation?. The zero-shot model achieves only 4.42% pass@1, confirming that RL is essential and ruling out data leakage from pretraining. Multi-expand is the most impactful design choice, improving pass@1 by over 12 points: from 40.63% to 53.26% with depth penalty, and from 39.58% to 51.16% without. The depth penalty further raises success rate at \(N = 500\) from 81.26% to 85.05%. We also report “Shorter Routes” in ¿tbl:tab:ablation?, the number of targets (out of 190) for which the found route is strictly shorter than the reference route, averaged over runs (mean\(\pm\)std). RetroAgent finds the most shorter routes (\(38.3\pm3.1\) per run), ahead of all ablated variants (\(26.6\)–\(32.8\)); removing the depth penalty or multi-expand reduces shorter-route discovery, and removing both reduces it most. Aggregating over runs, RetroAgent produces a shorter route than the reference for 69 distinct targets in total.

To further isolate our method from the choice of base model, we reproduce the Retro-R1 two-action interface on our own Qwen3-4B-Instruct-2507 backbone, holding the model and RL training fixed and varying only the interface. It reaches \(45.05\pm0.29\)% pass@1 on USPTO-190 (five-run mean), \(8.21\) points below RetroAgent and even below the original Qwen2.5-7B Retro-R1 result of 50.00%, confirming that the gain comes from the interface rather than the newer backbone. We also examine the sensitivity of our reward shaping: sweeping the relative weighting and magnitude of the budget and depth penalties moves pass@1 by only a few points, and a harsher penalty magnitude trades first-attempt success for a converging budget curve rather than removing the ability to solve targets, indicating that performance is not overly sensitive to these values. Please refer to 11 for full results of both studies.

3.4 Behavioral Analysis (RQ4)↩︎

Beyond the ablation numbers, we analyze what search strategy the agent actually learns by examining tool usage evolution and search tree structure (4), which helps further explain the performance gaps in ¿tbl:tab:ablation?.

Tool usage evolution. 4 (a) shows how tool call ratios change during training. The agent has access to 8 tools: 4 memory tools for search and 4 chemistry tools for heuristic guidance. Chemistry tools are actively used at step 0 (\(\sim\)​11%) but collapse to near zero by step 10, suggesting that the building block annotations returned by the prediction tool provide sufficient signal for candidate prioritization. The node inspection tool stays below \(\sim\)​1% throughout training and quickly drops to near zero, indicating the agent extracts sufficient information from other tool responses. Three tools remain dominant throughout: prediction, expansion, and frontier retrieval. Multi-expand usage emerges around step 120 and becomes dominant after step 350, with the agent applying multiple reactions per call in over 70% of expansions, demonstrating that the agent discovers this strategy through reward shaping rather than being explicitly guided.

Search tree structure. 4 (b)–(f) plot the search tree structure for each configuration, where each point represents one rollout with its maximum tree depth and total molecule nodes expanded. Comparing across configurations reveals three key findings.

First, RL training is essential for long-horizon planning. As shown in 4 (c), the zero-shot model clusters all rollouts in the lower-left corner, rarely exceeding 15 steps or 75 nodes regardless of outcome, indicating it cannot sustain productive search beyond the first few expansions. This corroborates the success rate plateau beyond \(N=100\) in ¿tbl:tab:ablation?. In contrast, the RL-trained agent in 4 (b) routinely builds trees with 50–100+ nodes, with solved cases concentrated at shallow depth.

Second, multi-expand trades depth for breadth. As seen in 4 (e), removing multi-expand yields narrow trees with fewer nodes but greater depth to find solutions. The full model in 4 (b) creates more nodes at shallower depth by expanding multiple OR branches per molecule, hedging against dead ends without committing to deep search. This breadth-first tendency directly explains why multi-expand yields 12+ points of pass@1 improvement while also finding more shorter routes than any ablated variant. The agent applies this breadth selectively rather than exhaustively: across all expand calls, \(\sim\)​84% apply only one or two reactions (\(w=1\): 54.3% on USPTO-190 and 55.6% on ChEMBL-1000; \(w=2\): 29.3% and 28.2%), with a thin tail beyond \(w=5\) (\(\sim\)​0.6% on both). Solved targets expand slightly wider on average than failed ones (mean width 1.81 vs. on USPTO-190; 1.86 vs. on ChEMBL-1000), indicating that multi-expand contributes through selective local breadth at promising decision points rather than indiscriminate branching.

Third, depth penalty prevents inefficient deep exploration. As illustrated in 4 (d), without the depth penalty, solved cases frequently extend to the maximum allowed depth of 30, wasting budget on unnecessarily long routes. Removing both design choices as in 4 (f) yields the weakest configuration, though the agent can still solve some targets through single-expand depth-first search.

4 Conclusion↩︎

We presented RetroAgent, an RL-trained LLM agent for retrosynthesis planning that operates over an AND-OR graph as structured memory. The environment automatically maintains the search state, while the agent makes strategic decisions through tool calls. With a 4B-parameter model, RetroAgent achieves the highest pass@1 on USPTO-190 and substantially outperforms all baselines on the out-of-distribution ChEMBL-1000, despite using a smaller model and less training data. Ablation studies confirm that multi-expand and reward shaping are key to the agent’s performance, and behavioral analysis reveals that the agent progressively learns to trade depth for breadth through RL. The strong out-of-distribution results suggest that the harness enables generalizable planning strategies rather than dataset-specific memorization. We believe the design principle of combining LLM reasoning with structured environment management is broadly applicable to other long-horizon planning tasks beyond retrosynthesis.

Acknowledgments↩︎

This work was partially supported by NSF Center for Computer Assisted Synthesis (2202693), National Artificial Intelligence Research Resource (NAIRR) Pilot (240280, 240443), and Amazon Research Awards. Y.Z. acknowledges additional support from the Tinker Grant and the Lambda Research Grant during the early stages of this work.

LLM Usage Disclosure↩︎

We disclose that LLMs were used for editing and polishing the language of this manuscript. All scientific content, experimental design, and analysis were conducted by the authors.

5 System Prompt↩︎

The system prompt instructs the agent on the AND-OR graph structure, the available tools, and the search loop. The user prompt provides the target molecule SMILES and instructs the agent to begin with predict_templates("M1").

6 Tool Interface Details↩︎

The agent interacts with the environment through 8 tools divided into two categories. All graph-related tools use symbolic identifiers (M# for molecules, R# for reactions, P# for predictions) rather than raw SMILES strings to prevent copying errors.

Memory tools. These tools allow the agent to query and modify the AND-OR graph:

• predict_templates(molecule_id): Queries the single-step retrosynthesis model for a given molecule node. Its 50 highest-scoring predictions are shuffled and truncated to at most \(k=20\) candidates, each assigned an ID (P1, P2, …) and listing its reactants with per-reactant building block annotations (building_block: true/false). Predictions that would create cycles (a reactant is an ancestor of the molecule) or duplicate an existing reaction are automatically filtered out. Within a rollout, subsequent calls for the same molecule reuse the cached candidates with updated filtering at zero additional budget cost.

• expand(molecule_id, prediction_id): Applies one or more predictions to the graph, creating new reaction and molecule nodes. Multiple prediction IDs can be comma-separated (e.g., "P1,P3,P5") to create several reaction branches in a single call. Returns the expansion results and the parent node status.

• get_frontier(): Returns the current set of Open leaf molecules that have not yet been expanded. Also signals when the root molecule has been solved.

• inspect_node(node_id): Returns detailed information about any node in the graph, including its SMILES, status (Open/Solved), building block status, parent nodes, and child nodes.

Chemistry tools. These tools provide domain-specific heuristics to help the agent prioritize among candidates. Unlike memory tools, chemistry tools take SMILES strings as input rather than node IDs:

• get_molecule_info(molecules): Returns molecular properties including molecular weight, logP, number of atoms, and ring counts for a list of molecules.

• search_building_blocks(molecules): Checks whether given molecules are available in the building block database and returns their availability status.

• calculate_synthetic_accessibility(molecules): Computes the synthetic accessibility (SA) score [37] for each molecule, ranging from 1 (easy to synthesize) to 10 (difficult).

• calculate_scscore(molecules): Computes the SCScore synthetic complexity [38] for each molecule, ranging from 1 (simple) to 5 (complex).

7 Related Work↩︎

Multi-step retrosynthesis planning methods broadly divide into two streams: search-based methods that ground every step in a validated template set, and model-based methods that leverage learned representations to generate or predict routes directly. Our work falls within the search-based paradigm: we use a template-grounded single-step model for chemical validity, but replace the fixed search algorithm with an RL-trained LLM agent that learns its own search strategy.

Search-based (template-grounded) methods. The dominant paradigm combines a trained single-step retrosynthesis model, typically an MLP classifier over reaction templates [6], with a search algorithm over the resulting AND-OR graph. Retro* [12] introduced neural-guided A* search with a learned value function estimating the cost-to-go for each molecule. PDVN [13] improved this with dual value networks that separately predict synthesizability and cost. EG-MCTS [14] uses experience from past searches to train a value network for Monte Carlo tree search. 3N-MCTS [15] combined MCTS with three neural networks for rollout-based evaluation. GNN-Retro [39] improves molecule cost estimation by using a graph neural network that aggregates information from structurally similar molecules. DESP [31] proposed bidirectional search, interleaving top-down retrosynthetic expansion with bottom-up forward synthesis from building blocks, guided by a learned synthetic distance function. Retro-fallback [40] introduced probabilistic route reliability through stochastic OR nodes. Beyond node-level heuristics, several works incorporate domain-specific insights into improve multi-step retrosynthesis planning. [41] learn pathway-level representations that rank and cluster retrosynthesis routes by their strategic quality, while [42] reformulate the single-step prediction itself at a higher level of abstraction, separating strategic disconnections from tactical functional group choices to reduce the effective search space. These methods achieve high solve rates within large iteration budgets (up to 500 single-step model calls) but encode fixed search strategies: the heuristic is always a pretrained value function, and the expansion order is determined by the algorithm, not by per-target reasoning.

Template-free and generative methods. Beyond template-based classifiers [6], [43], [44], template-free methods generate reactants directly. The Molecular Transformer [8] and augmented NLP models [9] frame retrosynthesis as sequence-to-sequence translation, while RootAligned [45] enforces a one-to-one mapping between product and reactant tokens by aligning them to a shared root atom. BatGPT-Chem [46] trains an LLM on chemical instruction data for single-step prediction. LLM-Syn-Planner [19] uses an LLM to mutate retrieved routes into multi-step plans for new targets. These approaches can leverage broad chemical knowledge but typically lack the structural guarantees of template-grounded methods.

LLM agents for retrosynthesis. Recent work has explored LLMs as agentic planners that interact with chemistry tools through multi-turn conversations. Retro-R1 [20] trains a Qwen2.5-7B agent with PPO to select molecules and reactions through a CALL/SELECT interface, achieving state-of-the-art pass@1 success rates. However, Retro-R1 operates in a linear, no-backtracking mode: the agent commits irrevocably to each reaction selection, with no ability to revisit alternative branches. Its state representation is implicit: the agent must infer the search state from the conversation history. AOT* [47] combines an LLM with AND-OR tree search, using the LLM to guide node selection within a traditional search framework. In contrast, our agent has full control over the search process and learns its own strategy end-to-end through RL.

8 Limitations and Discussion↩︎

Practicality of the generated routes. Our evaluation follows the standard template-based protocol: a route is considered valid if every step applies a verified reaction template and all leaves terminate at purchasable building blocks, and quality is measured by success rate, search budget, and route length. Because all routes pass through the same validated template set and building-block catalog, the routes RetroAgent produces are chemically valid by construction, and the comparison to baselines is on equal footing. However, this notion of validity does not capture full synthetic practicality: factors such as reaction yield, regio- and stereoselectivity, reaction conditions, and protecting-group strategy are not modeled, and a shorter or template-valid route is not necessarily the most practical one in the laboratory. These considerations affect all template-based planners equally and depend on information absent from the benchmarks, which provide only target molecules and, for USPTO-190, reference routes. A consultation with expert chemists found the generated routes generally reasonable, but a systematic, large-scale assessment of practical utility, ideally with experimental validation, remains an open problem for the field and an important direction for future work.

Inference cost relative to traditional search. RetroAgent relies on LLM inference at every search step, which is substantially more expensive than the lightweight value-network or policy lookups used by traditional search algorithms such as Retro* and PDVN. A single planning trajectory issues tens of LLM calls and consumes thousands of tokens, whereas a classical planner evaluates its scoring function in milliseconds. This overhead is the price of the explicit, inspectable reasoning that drives our gains, and we regard it as acceptable in the retrosynthesis setting: planning is a one-time, offline computation of seconds to minutes per target, negligible against the days to weeks required to validate a route experimentally. Nonetheless, for latency-sensitive or very high-throughput screening, traditional search remains more economical, and reducing LLM planning cost (e.g., through smaller distilled policies or selective invocation) is a worthwhile direction.

9 Training Details↩︎

Training hyperparameters. ¿tbl:tab:hyperparams? lists the full training hyperparameters for RetroAgent.

Training curves. 5 shows the training dynamics of RetroAgent over 400 steps. Pass@1 on the training set rises rapidly from \(\sim\)​30% to \(\sim\)​80% within the first 50 steps and continues to improve gradually, reaching \(\sim\)​90% by step 400. Mean reward follows a similar trajectory, stabilizing around 0.75. Mean turns per episode increases from \(\sim\)​12 to \(\sim\)​27, reflecting the agent’s ability to sustain longer, more productive search episodes as it learns.

10 Evaluation Protocol↩︎

We evaluate using the standalone eval pipeline with the same tool server as training, with greedy decoding (temperature 0). Each rollout runs for up to 120 turns and a 120K token budget, more generous than training (90 turns, 56K tokens) to allow extra exploration. Following the iterative protocol, each experiment issues successive rollouts for a target until a route is found or the search budget is exhausted; we run 10 independent experiments per target and report the mean and standard deviation, with stochasticity coming entirely from shuffled prediction and frontier orderings in the tool server, not from LLM sampling. A rollout terminates when the root molecule is solved, when maximum turns are reached, when no successful expansion occurs for several consecutive turns (no progress), when the frontier is exhausted (dead frontier), when the token budget is exhausted, or when a single turn exceeds the response limit of 8,192 tokens due to repetition. Reaching the maximum route depth of 30 reaction steps blocks only the offending branch and does not by itself end the rollout.

11 Additional Experiments↩︎

11.1 Same-Backbone Comparison with Retro-R1↩︎

To isolate the contribution of our structured-memory interface from the choice of base model, we reproduce the Retro-R1 two-action (CALL/SELECT) interface on the same Qwen3-4B-Instruct-2507 backbone used by RetroAgent, holding the model and RL training procedure fixed and varying only the interface. We use the released Retro-R1 code, the step-200 checkpoint, and the same iterative evaluation protocol on USPTO-190, reporting five independent runs. As shown in ¿tbl:tab:same-backbone?, the Retro-R1 interface on the Qwen3-4B-Instruct-2507 backbone reaches \(45.05\pm0.29\)% pass@1, \(8.21\) points below RetroAgent(53.26%) on the same backbone and even below the original Retro-R1 result of 50.00% on Qwen2.5-7B-Instruct-1M. Thus moving the Retro-R1 interface onto our backbone does not by itself improve performance; the gain comes from the interface and learned policy rather than the base model. Together with the zero-shot result of 4.42% pass@1 (¿tbl:tab:ablation?), this shows the interface does not help without RL training, and that the gains are not attributable to the choice of base model.

11.2 Reward Sensitivity↩︎

We study the sensitivity of RetroAgent to its reward shaping from two perspectives: the relative weighting of the budget penalty \(\delta_N\) and depth penalty \(\delta_D\), and the overall magnitude of both penalties. All configurations are otherwise identical to RetroAgent and trained from the same base model. ¿tbl:tab:reward? reports USPTO-190 success rates as mean\(\pm\)std over five runs.

On the relative weighting, keeping both penalty terms is best (53.26%), and dropping either one lowers pass@1 only modestly (budget-only 51.16%, depth-only 50.84%), so the two terms are complementary but performance is robust to their balance. On the magnitude, doubling both penalty rates and caps lowers pass@1 to 48.32% but the budget curve converges (\(N=200\) rises to 75.58%, approaching the full model’s 78.63%). Harsher penalties make the agent more conservative, trading first-attempt success for budget efficiency without removing the underlying ability to solve targets.

12 Out-of-Distribution Analysis↩︎

We quantify the distribution shift between the ChEMBL-1000 evaluation set and the USPTO-Full training targets in terms of three properties (¿tbl:tab:ood?):

• Scaffold novelty: we extract the Bemis-Murcko scaffold of each molecule and mark it novel if it does not appear among the training targets; 96.2% of ChEMBL-1000 scaffolds are novel, versus 74.8% for USPTO-190.

• Similarity to training: the mean maximum Tanimoto similarity (Morgan fingerprints, radius 2, 2048 bits) of each target to the training set is 0.345 for ChEMBL-1000 versus 0.483 for USPTO-190, and 85% of ChEMBL-1000 targets fall below the USPTO-190 median.

• Molecular complexity: synthetic accessibility is comparable between the two sets (median 3.18 for ChEMBL-1000 versus 3.55 for USPTO-190).

Together, these show the shift is driven by scaffold novelty and dissimilarity to training rather than by raw complexity. Despite the strong scaffold-level shift, RetroAgent solves the 962 scaffold-novel ChEMBL-1000 targets at 73.8% pass@1 and 84.2% at \(N\le500\) (810/962), essentially matching the full-set rates (74.4% pass@1, 84.6% at \(N\le500\)). This indicates the agent does not rely on memorized scaffold-to-template associations but instead learns a policy driven by distribution-independent signals such as building-block proximity.

13 Failure Case Analysis↩︎

We categorize every failed target on USPTO-190 and ChEMBL-1000 by the per-episode termination reason logged by the environment (¿tbl:tab:failures?). The environment records four mutually exclusive termination reasons:

• max_turns: the episode reaches the turn limit (120) without solving the root molecule;

• no_progress: the agent makes no successful expansion for five consecutive turns and is stopped early;

• dead_frontier: every open molecule has been expanded yet none yields a solved route;

• token_limit: the episode exhausts the 120K-token budget.

Reaching the maximum route depth of 30 reaction steps only blocks the offending branch rather than ending the episode, so such attempts are subsumed within max_turns or no_progress. On both benchmarks, budget exhaustion dominates: max_turns and no_progress together account for \(\sim\)​96% of failures on USPTO-190 and \(\sim\)​94% on ChEMBL-1000, while dead_frontier and token_limit together remain under 7%. Failures are therefore concentrated on hard, deep targets that exhaust the search budget rather than on the agent running out of candidates or hitting action-space limits. We also note that failed targets receive a similar number of first-step predictions as solved ones (15.0 vs. on USPTO-190; 16.6 vs. on ChEMBL-1000); failures thus arise from exhausting the search budget on hard, deep targets, not from a lack of initial candidates.

14 Example Transcripts↩︎

We present two solved and two failed episodes from USPTO-190 evaluation. Each transcript shows the full tool call sequence, where T\(n\) denotes the \(n\)-th turn of the agent-environment interaction. For each predict_templates call, we list the number of returned predictions and the Building Block (BB) count per prediction. For expand calls, we show which predictions the agent selected; \(w=k\) denotes the expansion width, i.e., the number of predictions applied in a single call.

Solved example 1. Solved in 11 turns with 4 predict calls, reaching depth 4 with 18 molecule nodes.

The agent selects predictions with building block reactants: at T5, it picks exactly the 3 predictions with BB annotations out of 20 candidates. At T11, it picks P14 which has 2 BB reactants, completing the route.

Solved example 2. Solved in 14 turns with 5 predict calls, reaching depth 5 with 22 molecule nodes.

The agent adapts its expansion width based on prediction quality: wide (\(w=5\)) at the root where many predictions have BB reactants, narrow (\(w=1\)) when only one prediction has BBs, and moderate otherwise.

Failed example 1. Failed after 112 turns, 38 predict calls, depth 30, 95 molecule nodes.

Failed example 2. Failed after 103 turns, 35 predict calls, depth 30, 88 molecule nodes.

Both failed episodes follow the dominant failure pattern quantified in 13: the agent sustains a long predict-expand-frontier cycle but cannot reach building blocks on these hard, deep targets, eventually triggering early no_progress termination.

References↩︎