1 Introduction↩︎

Molecular optimization is a central problem in chemistry, with applications ranging from drug discovery to catalyst design [1]–[3]. These tasks require identifying molecules that satisfy multiple, often competing objectives from vast chemical space [4]. Most relevant optimization problems are also data scarce: phenomena such as protein-ligand binding or catalytic activity are difficult to simulate accurately, and experimental measurements—particularly those requiring molecular synthesis—are costly and time-consuming. As a result, the real-world impact of an optimization strategy is defined by the number of evaluations required to discover optimal molecules (sample efficiency). Low sample efficiency necessitates reliance on inexpensive, low-fidelity in silico oracles vulnerable to hacking [5], [6]. Improved sample efficiency would enable the use of higher-fidelity simulations or even direct experimental feedback, thereby increasing the practical impact of molecular optimization methods.

ML methods for molecule optimization must traverse chemical space and strategically guide this exploration toward the most promising regions. Traversal methods include deep generative methods that learn a data distribution and can sample from that distribution, including recurrent neural networks (RNNs) [7], [8], transformers [9], [10], and diffusion or flow-based models [11]–[13]. These generative methods can be steered with various optimization techniques including Monte Carlo tree search (MCTS), Markov Chain Monte Carlo (MCMC), reinforcement learning (RL), as well as with Bayesian Optimization (BO). Language models that generate molecules in the form of Simplified Molecular Input Line Entry System (SMILES) [14] steered with RL remain among the most sample efficient methods [15], [16]. However, state-of-the-art generative methods still require between hundreds and thousands of oracle evaluations to identify high-quality candidates [15], [16]. While this cost may permit the use of moderately expensive simulations such as DFT, it remains prohibitive for workflows involving direct experimental evaluation, particularly in the case of prospective campaigns where the target distribution is not known.

BO, an optimization technique that uses a probabilistic surrogate model and an acquisition function to guide sequential experimentation within a candidate library or a continuous latent space [17], has excellent sample efficiency and has seen widespread use and success in chemistry [18]–[20]. Through the fitting of the surrogate model, these methods learn quickly from negative examples. However, performance depends on the presence of suitable candidates within a library or accurate decoding from a latent space to molecule space [21], [22]. Moreover, BO is highly sensitive to the choice of molecular representation, which can be challenging to design, expensive to compute, and may not generalize across diverse regions of chemical space [23]. Recent BO workflows use large language models (LLMs) trained jointly with GP objectives that enable the rapid, flexible featurization only from SMILES with excellent performance across diverse chemical tasks, opening the door to on-the-fly application across chemical space [23]–[25].

We propose the Sample Efficient Generative Optimization (SEGO) framework, a tandem system based on RL and BO for sample efficient search in open-ended molecular space. At each iteration, a probabilistic surrogate forms a hypothesis about where hits lie and steers a generative model to produce a library of candidates in that region. The library is featurized on-the-fly through deep-kernel learning over LLM embeddings [23] and an acquisition function selects the most promising candidate for evaluation; the resulting oracle call both sharpens the surrogate into a new hypothesis and anchors the generator in real reward. SEGO achieves sample efficiency from two mechanisms within this loop: SMILES augmentation in the surrogate training, and an anchoring mechanism for the surrogate-guided generation. Our method outperforms both pure BO and pure RL methods and attains state-of-the-art performance on the practical molecular optimization (PMO) benchmark using only a tenth of the oracle budget of other methods. On a multiparameter docking optimization task, SEGO identifies candidates in half or less of the oracle budget of other state-of-the-art methods. These gains could enable prospective campaigns beyond predefined libraries based on direct experimental feedback.

2 Background and related work↩︎

2.1 De Novo Molecular Design.↩︎

De novo molecular design seeks to generate molecules with desired properties from scratch, bypassing the constraints of fixed libraries [4]. Generative approaches span a broad range of architectures, from SMILES-based recurrent networks and transformers to graph-based, diffusion, and flow-based models. Among these, language-based models operating on SMILES representations have proven to be particularly sample efficient, capable of satisfying even three-dimensional objectives such as docking despite working from one-dimensional string inputs [15], [26]. A key advantage of SMILES is its non-injective nature: a single molecule admits many valid string representations, and this redundancy can be leveraged through augmentation to improve generalization in low-data regimes [27], [28]. As the field has shifted toward stricter oracle budgets that better reflect real-world constraints, sample efficiency has become a central evaluation criterion, and language-based methods consistently rank among the top performers under these conditions. Recent advances such as combining experience replay with SMILES augmentation have pushed sample efficiency further, establishing strong baselines for generative molecular optimization [16].

However, based on the formulation of the loss, these systems do not learn much from negative examples early in the optimization [8]. In challenging optimization scenarios, where early batches yield little to no reward, slow learning causes long lag times which erode sample efficiency. This motivates integrating generative models with principled selection strategies that extract signal from all evaluations, even from negative evaluations early in the optimization trajectory.

2.2 Bayesian Optimization on Molecules.↩︎

Bayesian optimization has been applied to molecular discovery across domains in chemistry. For example, it has been used to identify redox-active molecules from libraries of scaffolds enumerated with substituents and featurized with DFT descriptors [21], to identify candidate protein binders from an Enamine library using fingerprint representations [29], and for materials discovery efforts, where sets of predefined building blocks were evaluated using time-dependent DFT [2]. More broadly, library-based BO has been applied to transition-metal complexes [30], metal-organic frameworks [31], with surrogate models trained on small labeled subsets and acquisition functions used to iteratively select candidates for evaluation. These case studies illustrate the strength of BO for optimization in challenging reward landscapes, but also highlight two recurring limitations: optimization is constrained to expert-crafted libraries that cannot propose structures beyond their initial enumeration, and performance depends heavily on molecular representation. Bespoke descriptors such as DFT features can be expensive to compute, while general-purpose fingerprints, though cheap, may not capture the structure-property relationships relevant to a given task.

Recent work has begun to address the representation challenge through language-model-based molecular embeddings [23]–[25]. Building on this direction, the GOLLuM framework fine-tunes language model embeddings using Gaussian Process (GP) marginal likelihood, yielding task-adaptive representations that outperform both fingerprints and bespoke descriptors across diverse benchmarks [23]. Crucially, because GOLLuM operates directly on SMILES strings without requiring additional descriptor computation, it can featurize novel molecules on-the-fly, making it feasible to apply BO over dynamically generated libraries rather than fixed, predefined candidate sets.

2.3 Hybrid Generative BO methods.↩︎

To move beyond fixed libraries, several works have coupled generative models with BO. A common approach embeds discrete molecular representations into a continuous latent space, performs optimization there, and decodes the resulting vectors back into molecules [1], [32]. However, this latent-optimize-then-decode paradigm faces practical challenges: decoded molecules are frequently invalid or redundant, and latent spaces can be high-dimensional and poorly structured, complicating surrogate modeling and uncertainty quantification. Alternative approaches bypass latent embeddings and optimize acquisition functions directly over discrete representations using grammar-constrained search [33], graph-based kernels [34], or learned generation policies [35], though these methods face their own difficulties in maintaining chemical plausibility and scaling to large batch sizes.

A more recent line of work decouples the generative and optimization stages entirely, first using a generative model to propose candidates and then applying BO to select the most informative subset for evaluation. [36] developed a hybrid framework combining REINVENT with a fingerprint-driven, random-forest-based active learning loop, demonstrating improved sample efficiency over baseline REINVENT, though the modest efficiency of the underlying generative model limits practical applicability. [37] used a genetic algorithm as a molecule generation engine and a Tanimoto kernel on molecular fingerprints to achieve excellent results on the PMO benchmark. [38] also proposed a generate-then-optimize system with a new acquisition function for batch selection. While SEGO shares this hybrid format, it introduces surrogate-guided generation and augmented optimization to improve sample efficiency substantially. Further, SEGO builds on frontier methods in both Bayesian optimization and generative design, enabling new levels of sample efficiency not achieved by prior hybrid approaches.

3 SEGO Design↩︎

SEGO is a framework for generative optimization that attains excellent sample efficiency through the intelligent allocation of oracle calls across dynamically generated libraries (Figure 1). Each oracle call drives updates to a generative agent, a GP surrogate model, as well as the surrogate’s learned featurization. SEGO is organized around the following components:

• An anchor generative agent \(\pi_{\theta_{\text{Agent}}}\) updated only on true-oracle feedback.

• An inner-loop agent \(\pi_{\theta_{\text{Inner}}}\), re-instantiated at each BO iteration from the anchor and trained against the GP posterior mean to produce a dynamic candidate library.

• A deep-kernel GP surrogate over LLM embeddings, which featurizes the dynamic library for selection via an acquisition function.

SEGO’s sample efficiency rests on two contributions: an anchored surrogate-guided inner loop, enabling surrogate-driven generation while retaining information from true-oracle feedback; and SMILES-augmented surrogate updates, which exploits the string-based featurizer to extract additional signal from each oracle call.

3.1 Molecular Generation Engine.↩︎

As our generative engine, we take Saturn [16], a Mamba-based [39] autoregressive SMILES generator trained with reinforcement learning. A SMILES string \(x = (a_1, ..., a_T)\) is generated as a Markov process, \[\label{eq:smiles95prob} P_{\theta}(x) = \prod_{t=1}^{T} \pi_{\theta_{\text{Agent}}}(a_t \mid s_t)\tag{1}\] where \(a_t\) is the token selected at step \(t\), \(s_t\) is the token sequence so far, and \(\pi_{\theta_{\text{Agent}}}\) is the Mamba backbone. The aim in RL is to maximize the expected reward (equation 2 ). \[J(\theta) = \mathbb{E}_{a_t \sim \pi_{\theta_{\text{Agent}}}} \left[ \sum_{t=1}^{T} R(a_t, s_t) \right]. \label{eq:rl95objective}\tag{2}\] Augmented likelihood (equation 3 ) is defined, where the prior is a pretrained model with frozen weights, \(R\) is a reward function, and \(\sigma\) is a scalar modulating the reward’s effect: \[\label{eq:augmented95likelihood} \log \pi_{\theta_{\text{Augmented}}} = \log \pi_{\theta_{\text{Prior}}} + \sigma R(x)\tag{3}\] Maximizing equation 2 is equivalent (up to a factor) to minimizing the squared difference between the Augmented and Agent likelihoods (equation 4 ). \[\label{eq:reinvent95loss} L(\theta) = \frac{1}{|B|} \left[ \sum_{a \in A^*} \left( \log \pi_{\theta_{\text{Augmented}}} - \log \pi_{\theta_{\text{Agent}}} \right)^2 \right]\tag{4}\]

where \(B\) is a batch of SMILES sampled from the agent and \(A^*\) denotes the action sequences across all timesteps in the batch. Sample efficiency in Saturn comes from Augmented Memory: a replay buffer of the top-100 rewarded SMILES is maintained, randomized into equivalent SMILES, and used for \(N_{augmentation}\) additional update rounds per iteration. Mode collapse is mitigated by purging the buffer of scaffolds with low-diversity samples and assigning zero reward to repeated generations beyond a threshold.

3.2 Bayesian Optimization Engine.↩︎

As our BO engine, we take GOLLuM, a flexible framework based on SMILES that jointly trains LLM embeddings and GP hyperparameters [23]. A learned feature transformation is composed with the base kernel: \[\label{eq:deep95kernel} k_{\theta,\phi}(\mathbf{x}, \mathbf{x'}) = k_\theta(g_\phi(\mathbf{x}), g_\phi(\mathbf{x'}))\tag{5}\] where \(g_\phi\) is a parameterized feature extractor with parameters \(\phi\), trained jointly with the GP hyperparameters through the marginal likelihood. Though multiple ways of constructing \(g_\phi\) were proposed in the original paper, we take the projection layer variant: \(g_\phi(\mathbf{x}) = \mathbf{P}\text{LLM}(x)\), where \(\mathbf{P} \in \mathbb{R}^{m \times d}\) is a learned linear projection followed by ELU activation. The GP hyperparameters \(\theta = (\ell, \sigma^2, \sigma_n^2, c)\) and feature parameters \(\phi\) are jointly optimized by maximizing the log marginal likelihood: \[\begin{align} \label{eq:marginal95likelihood95gol} \mathcal{L}(\theta, \phi) &= \log p(\mathbf{y} \mid \mathbf{X}, \theta, \phi) \notag \\ &= -\frac{1}{2} \bigl( \mathbf{y}^\top \mathbf{K}_{\theta,\phi}^{-1} \mathbf{y} + \log |\mathbf{K}_{\theta,\phi}| + n \log 2\pi \bigr) \end{align}\tag{6}\] GOLLuM is particularly well suited as the BO engine in SEGO because it operates directly on SMILES strings, bypassing the need for hand-crafted molecular descriptors. Through joint optimization of the LLM and GP via the marginal likelihood, GOLLuM learns to smooth the representation space so that the GP can generalize effectively from very few observations.

3.3 SEGO Operation↩︎

At iteration \(i\), SEGO maintains the anchor agent \(\pi_{\theta_{\text{Agent}}}^{(i)}\) and a GP \(\mathcal{GP}^{(i)}\) fit to the oracle-evaluated set \(\mathcal{D}^{(i)} = \{(x_j, y_j)\}_{j \le i}\). The loop is initialized by sampling \(L\) molecules from \(\pi_{\theta_\text{Prior}}\), scoring them under the acquisition function, and selecting \(n_\text{init}\) for oracle evaluation; these seed \(\mathcal{D}^{(0)}\) and provide the first updates to \(\pi_{\theta_\text{Agent}}\) and \(\mathcal{GP}\). Each iteration:

1. Initializes \(\pi_{\theta_\text{Inner}}\) from \(\pi_{\theta_\text{Agent}}^{(i)}\).

2. Trains \(\pi_{\theta_\text{Inner}}\) for \(K_\text{inner}\) Saturn updates against \(R_\text{surr}(x) = \mu^{(i)}(x)\), the GP posterior mean.

3. Samples a candidate library \(\mathcal{C}^{(i)}\) of size \(L\) from \(\pi_{\theta_\text{Inner}}\), then discards \(\pi_{\theta_\text{Inner}}\).

4. Selects \(x^\star_i = \arg\max_{x \in \mathcal{C}^{(i)}} \alpha(x)\) under acquisition function \(\alpha\) (we use expected improvement).

5. Queries the oracle to obtain \(y^\star_i = f_\text{oracle}(x^\star_i)\).

6. Constructs an augmented set \(\mathcal{A}_i = \{(x^\star_{i,k}, y^\star_i)\}_{k=1}^{N_\text{aug}}\) from \(N_\text{aug}\) randomized SMILES variants of \(x^\star_i\), updates \(\mathcal{D}^{(i+1)} \leftarrow \mathcal{D}^{(i)} \cup \mathcal{A}_i\), refits \((\theta, \phi)\) via Eq. 6 , and updates \(\pi_{\theta_\text{Agent}}\) by one round of Saturn updates (Eq. 4 ) with reward \(y^\star_i\).

4 Experiments↩︎

4.1 SEGO architecture ablation experiments↩︎

We explore the impact of key hyperparameters in SEGO: the role of the inner loop in sample efficiency, the anchor’s role in training stability, and augmented optimization.

Experimental Setup. Following [27], we optimize molecules according to a multi-property optimization (MPO) objective: molecular weight (MW) \(<\) 350 Da, number of rings \(\geq\) 2, and maximize the topological polar surface area (tPSA). To satisfy this objective, molecules must have rings saturated with heteroatoms. Such saturated molecules are dissimilar from training data, so this experiment also evaluates out-of-distribution optimization. We use an extremely stringent oracle budget of 100 calls, representing 10% of the already stringent budget employed in [16], which better reflects realistic budgets for experimental campaigns. All experiments were run across 10 seeds (0-9 inclusive) using a prior pretrained on the ChEMBL 33 dataset [40].

\begin{table}[!ht]

\vskip

-0.1in \end{table}

Metrics. As our sample efficiency metrics, we take Yield, the number of unique molecules generated above a reward threshold and Oracle Burden (OB), the number of oracle calls required to generate N molecules above a reward threshold. As our threshold, we take 0.7 as molecules start to possess saturated heteroatom rings. We also compute diversity metrics, as there is a tradeoff in most generative methods between yield and diversity. We compute IntDiv, the internal diversity [41], Scaffolds, the number of Bemis-Murcko scaffolds [42], and #Circles, [43] which measures chemical space coverage.

Inner loop anchor strategies. We explore three strategies for instantiating the inner loop, in which the RL agent receives reward from the GP predicted mean. The default strategy initializes each inner loop from an anchor, an RL agent previously trained on true oracle rewards, giving the inner loop a warm start in a promising region of chemical space. To test whether this warm start is necessary, we ablate with two alternatives: a prior strategy, which re-initializes from the pretrained prior at each iteration, testing whether the inner loop can learn to generate high-scoring molecules without any oracle-informed starting point; and a continuous strategy, which retains the inner loop agent across iterations, testing whether it is sufficiently flexible to track the changing GP landscape throughout the campaign.

The prior initialization strategy fails to produce hits in most cases (Table ¿tbl:Inner32Loop32Initialization?). This is unsurprising: at early iterations the GP has been trained on as few as 10 molecules, so its reward landscape is poorly calibrated across chemical space, providing too faint a signal for the agent to learn from scratch. The continuous strategy yields competitive hit counts, but at the cost of reliability. Since the agent is never reset, it tends to commit to a narrow region of chemical space; if that region contains hits, oracle burden is low, but the agent struggles to transition elsewhere in the absence of hits. This is reflected in the high variance of OB1 across seeds. We adopt the anchored framework as default.

\begin{table}[!ht]

\vskip

-0.1in \end{table}

Surrogate-guided inner loop. Given the strong performance of the anchor strategy, we next assess the impact of the number of inner loop iterations. To isolate the effect of surrogate-informed training from the additional augmented memory updates that the inner loop provides, we include a further control: a variant with zero inner loops but with N anchor augmentation rounds set to 210, matching the total number of updates the 20-step inner loop would receive (20 inner loop steps \(\times\) 10 inner loop augmentations, plus 10 outer loop augmentations). Sample efficiency improves and variance across seeds decreases as the number of inner loop iterations increases, and diversity follows suit, reflecting the stronger optimization (Table ¿tbl:inner32loop32iters?). The augmentation-matched control, despite receiving the same number of gradient updates from the replay buffer, suffers in yield, oracle burden, and the number of seeds that successfully identify a hit, confirming that the benefit of the inner loop stems from training on GP-predicted rewards rather than from additional augmented memory updates alone. Though the difference between 0 and 20 inner loops does not reach statistical significance for most metrics, the consistent trends across all metrics suggest that the inner loop improves both the reliability and efficiency of SEGO.

Augmented Optimization. Inspired by Augmented Memory, in which a generative model is updated according to \(N\) augmentation rounds of randomized, high-reward SMILES, we develop augmented optimization, in which library SMILES are augmented \(N\)-fold and the GP is trained on all representations for each molecule. Because the GOLLuM framework learns its latent space through implicit contrastive learning — reorganizing so that similar molecules have more similar representations — we hypothesize that providing many representations of the same molecule will encourage the GP to align them, improving predictive accuracy.

Augmented optimization improves both oracle burden and yield without statistically significant impacts on internal diversity and #Circles (Table ¿tbl:augmentation95factor?). The effect saturates at 5-fold augmentation; no statistically significant difference was observed between 5-, 10-, and 20-fold augmentation. To investigate the mechanism behind this improvement, we examined the impact of augmentation on both representational alignment and predictive accuracy. At each iteration we sampled 500 molecules from the prior and from the anchor and recorded the GP mean prediction across 10 augmentations of each molecule. SMILES augmentation narrows the range of predictions for different representations of the same molecule, and this alignment improves with additional augmentation rounds (Figure 2). Crucially, the effect is observed for the prior as well as the anchor, indicating that augmentation improves global representational alignment rather than only aligning the narrow region of chemical space being explored by the oracle. This corresponds to gains in predictive accuracy — measured by the correlation between mean predicted values and true oracle values — particularly early on, when the GP has very little training data (Figure 2).

Isomer Collapse. The previous experiments showcase the powerful optimization behavior of SEGO, though as is typical of strong optimizers, this comes at the cost of diversity. The Bemis-Murcko scaffold metric even inflates diversity for this oracle: dense saturation means that molecules differing in only heteroatom placement are assigned different scaffolds. To probe diversity collapse for this oracle, we introduce the metric Isomers, defined as the number of unique hit molecular formulas. Despite generating 59 (\(\pm\)​25) unique molecules at this threshold, SEGO produces only 12 (\(\pm\)​9) unique isomers, revealing a form of mode collapse in which the agent repeatedly generates constitutional isomers of the same formula (Table ¿tbl:tab:eixi?).

We address this isomer collapse by modulating the expected improvement acquisition function. Standard expected improvement can favor isomers predicted to score near the previous best over novel formulas whose higher uncertainty does not translate into a higher chance of exceeding the incumbent. Following [44], we inflate the incumbent by a margin \(\xi\) and so require a candidate’s predicted improvement to clear this raised threshold. We observe that \(\xi=0.05\) and \(\xi=0.1\) significantly increase the number of unique hit isomers, with no decrease in the overall number of hits (Table ¿tbl:tab:eixi?, implementation details in section 6.3). We also introduce an isomer filter analogous to a Bemis-Murcko scaffold filter, setting a stringent threshold of 2. This filter offers additional gains when combined with the inflated expected improvement, showing significant improvements over the baseline in isomers and both diversity metrics. Both the \(\xi\) parameter and the isomer bucket size are hyperparameters that can be tuned to balance exploration and exploitation.

\begin{table}[!bt]

\vskip

-0.1in \end{table}

Hybrid versus individual approaches. SEGO is a hybrid architecture that enables sample-efficient search over open-ended chemical space. We compare SEGO to the individual generative and BO engines it is built from to showcase the benefits of the hybrid approach. To benchmark GOLLuM, we take a random sample of 500 molecules from ChEMBL 33 [40], the same database used to train the prior in Saturn and in SEGO, to mimic a real-world prospective campaign in which a desirable library is not known. Given the benefits of augmented optimization observed earlier, we also test GOLLuM with 10-fold SMILES augmentation (GOLLuM-10x). For SEGO, we use the anchored framework with 20 inner loop iterations and an augmented optimization factor of 10. We also use \(\xi=0.05\) as well as an Isomer diversity filter with a bucket size of 2.

Saturn fails to produce any hits at this strict oracle budget (Table ¿tbl:tab:toyres?). With the recommended batch size of 16 and augmentation rounds of 10, it undergoes only 6 rounds of RL updates, which is insufficient to shift the distribution toward hit molecules. Smaller batch sizes in Saturn do not lead to gains in sample efficiency due to the noise in approximating expected reward with small batch sizes [16]. GOLLuM fails because the randomly sampled library does not contain hits. In SEGO, the surrogate-steered generation shifts the distribution toward high reward regions providing the selector with hit-containing libraries to yield 63 (\(\pm9\)) hits.

4.2 PMO benchmark↩︎

Experimental Setup. We evaluate SEGO on the practical molecular optimization (PMO) benchmark, a suite of 23 oracles spanning model-based scoring (e.g., DRD2, JNK3, GSK3\(\beta\)), similarity-based objectives (e.g., Albuterol, Mestranol, Celecoxib rediscovery), and physicochemical and multi-property composites (e.g., QED, isomer matching, scaffold hopping) [15]. PMO reports the sum of each trajectory’s area under the curve (AUC) of the top-10 average performance as a function of cumulative oracle queries, favoring methods that find high values in fewer calls. The PMO takes a relaxed budget of 10,000. For increased relevance to expensive oracles, following [45], we instead report results for baselines on the PMO when restricted to an oracle budget of 1,000 (PMO-1k). For SEGO, we restrict oracle budget to just 100 oracle calls, after which the top-1 and top-10 scores are held fixed for computing metrics at the 1k horizon [46] to showcase sample efficiency. As many objectives are similarity based, and therefore diversity is not a priority, we run SEGO with vanilla expected improvement as the acquisition function. We report both top-1 AUC (consistent with 1% of oracle calls) and top-10 AUC, aggregated across 3 seeds (0–2 inclusive). We compare against published PMO leaderboard methods including REINVENT [8], Augmented Memory [27], Genetic GFN [47], MOLLEO [48], Graph GA [49], and GP BO [37].

\begin{table}[!b]

\vskip

-0.1in \end{table}

\begin{table}[!ht]

\vskip

-0.1in \end{table}

At the top-1 threshold, which mirrors the baselines by evaluating the top 1% of the oracle budget, SEGO achieves state-of-the-art performance despite consuming only a tenth of the oracle budget available to other methods (Table ¿tbl:pmo?). SEGO attains competitive performance at the top-10 threshold which is particularly challenging because top-10 represents 10% of the total budget available to SEGO. Extended results for individual oracles are reported in the appendix (Table ¿tbl:pmo-detailed?). Together, these results demonstrate SEGO’s strong optimization capability under tight budget constraints.

The most closely related method in this benchmark is GP BO [37], which uses BO to select candidates from a library generated by a genetic algorithm. SEGO outperforms GP BO at a fraction of the oracle budget, highlighting the strength of its underlying generative and optimization components. LICO [45] follows a similar framework but replaces the GP surrogate with an LLM. SEGO surpasses LICO with greater sample efficiency, though LICO’s strong results suggest that substituting SEGO’s GP with an LLM is a promising direction for future work. Another language-model approach is MOLLEO [48], which leverages an LLM to perform mutation and crossover operations within an evolutionary algorithm. While MOLLEO achieves competitive performance, it relies on detailed textual task descriptions provided to the LLM, raising concerns about data leakage and limiting its applicability to prospective campaigns where such descriptions may not be available.

4.3 Molecular Docking Task↩︎

Experimental Setup. Following [50], we benchmark SEGO on multiparameter optimization docking tasks against five targets (parp1, fa7, 5ht1b, braf and jak2). We optimize the MPO objective function (equation 7 ), where \(\widehat{\mathrm{DS}}\) is the normalized QuickVina 2 [51] score and \(\widehat{\mathrm{SA}}\) is the normalized synthetic accessibility score. We compare against the goal-aware fragment extraction, assembly and modification (GEAM) model [50] and against Saturn. \[R(\mathbf{x}) = \widehat{\mathrm{DS}}(\mathbf{x}) \times \mathrm{QED}(\mathbf{x}) \times \widehat{\mathrm{SA}}(\mathbf{x}) \in [0, 1] \label{eq:geamscore}\tag{7}\] Metrics. Mirroring a campaign in which the aim is to find several promising candidates for further evaluation and in which experiments are expensive, we take the oracle burden at 10 hits (OB@10). Here, we define a hit as a strict hit with stringent criteria requiring a molecule to have a docking score better than the median of known actives, a \(QED>0.7\) and \(SA<3\) [16]. We take strict hits to test the optimization capacity of the model. SEGO was run for an oracle budget of 600 calls, Saturn and GEAM were run for 3,000 calls.

Results. SEGO finds 10 strict hits in half or less of the oracles required by Saturn and GEAM, showcasing strong optimization performance on a pharmacologically relevant oracle (Table ¿tbl:tab:geam?). Generated molecules for each target are shown in Figure 3.

5 Discussion↩︎

We introduce SEGO, a hybrid generative optimizer for sample-efficient search in open-ended chemical space. Unlike pure generative methods that rely on slow updates to shift the distribution and conventional BO methods that operate on fixed libraries, SEGO leverages the principled exploration of BO across dynamically generated libraries. This design yields gains in hit-finding and sample efficiency compared to the individual parent methods on a toy oracle. SEGO also achieves excellent performance on the PMO benchmark, reaching state-of-the-art top 1% performance in only 100 oracle calls—one-tenth of the oracle budget of PMO-1k. On a more challenging MPO molecular docking oracle, SEGO identifies multiple hits much faster than current state-of-the-art methods.

Our ablations show that our design decisions—namely, guiding an inner loop based on surrogate predictions and anchoring that inner loop—contribute to the sample efficiency of SEGO. Further, data augmentation, known to be beneficial for sample efficiency in generative models, can improve the predictive performance of BO as well and improve representational alignment between diverse SMILES. Together, this setup yields an effective optimizer that is prone to repeat isomer generation. We enhance diversity through the inclusion of an isomer-based diversity filter as well as by inflating the previous best in the expected improvement acquisition function.

Excellent performance on the PMO benchmark and the MPO docking benchmark hint at powerful optimization ability; however, it may not reflect optimization behavior in the complex reward landscapes found in real-world campaigns where competing objectives like synthetic accessibility, solubility, and activity must be navigated simultaneously. Further work should assess SEGO on challenging oracles including physics-based simulations relevant to drug-discovery and catalysis as well as stringent synthesis constraints before considering deployment to wet-lab prospective campaigns [52]. Finally, practical high-throughput campaigns require multiobjective optimization over Pareto fronts and batched acquisition [18] which could be introduced into SEGO via alternative acquisition functions [38]. Despite these limitations, the sample efficiency and hit-finding ability demonstrated here are a promising step toward generative molecular design driven by direct experimental feedback.

Software and Data↩︎

Software to run SEGO can be found at https://github.com/schwallergroup/sego.

Impact Statement↩︎

As AI-driven methods become increasingly integrated into chemical discovery, it is important to acknowledge the dual-use risks inherent in molecular generation, which become more relevant as engines become more powerful. The risks of SEGO are in line with existing performant molecular design engines. Such risks can be mitigated through controlled access to oracles and established safeguards at the institutional level.

Acknowledgments↩︎

This publication was created as part of NCCR Catalysis (grant number 225147), a National Centre of Competence in Research funded by the Swiss National Science Foundation.

6 Implementation details.↩︎

6.1 Saturn Hyperparameters.↩︎

Saturn was used with default hyperparameters as recommended in the paper [16]. The inner loop used 10 augmentation rounds and a batch size of 64, selected to be a good balance between sample efficiency and stability in the surrogate-guided approach which does not consume oracle calls. The outer loop used 10 augmentation rounds based on a batch size of 1, selected by the BO loop. Unless otherwise stated, the ablation studies use only a Bemis-Murcko scaffold diversity filter. The toy-oracle hybrid comparison and the benchmark studies additionally include an isomer diversity filter in both the inner loop and the outer loop, with a default bucket size of 2.

6.2 GOLLuM Hyperparameters.↩︎

GOLLuM was used with default hyperparameters as recommended in the paper [23]. We used the T5 language model [53] as our base featurizer in the PLLM setting, using a Gaussian process [54] as our surrogate model and a Matérn-5/2 kernel. We use the expected improvement acquisition function. For ablation studies and the PMO benchmark, we used a \(\xi\) value of 0.0; for the docking benchmark and the toy oracle we used a \(\xi\) value of 0.05. For other implementation details, refer to [23].

6.3 Expected Improvement Modulation.↩︎

Following [44], we adjust the balance between exploration and exploitation by tuning the incumbent \(f(\mathbf{x}^{+})\) by \(\xi\) (equations 8 , 9 ). \[\mathrm{EI}(\mathbf{x}) = \begin{cases} \bigl(\mu(\mathbf{x}) - f(\mathbf{x}^{+}) - \xi\bigr)\,\Phi(Z) + \sigma(\mathbf{x})\,\phi(Z) & \text{if } \sigma(\mathbf{x}) > 0, \\[4pt] 0 & \text{if } \sigma(\mathbf{x}) = 0, \end{cases} \label{eq:ei}\tag{8}\]

\[Z = \begin{cases} \dfrac{\mu(\mathbf{x}) - f(\mathbf{x}^{+}) - \xi}{\sigma(\mathbf{x})} & \text{if } \sigma(\mathbf{x}) > 0, \\ 0 & \text{if } \sigma(\mathbf{x}) = 0. \end{cases} \label{eq:ei-z}\tag{9}\]

7 Extended Ablation Results↩︎

7.1 Inner Loop Iterations↩︎

\begin{table}[!ht]

\vskip

-0.1in \end{table}

7.2 SMILES augmentation↩︎

7.3 Diversity↩︎

\begin{table}[!ht]

\vskip

-0.1in \end{table}

8 Extended Benchmark Results↩︎

\begin{table}[!ht]

\vskip

-0.1in \end{table}

References↩︎