1 Introduction↩︎

Large language models (LLMs) have achieved strong performance across many language benchmarks [1]–[3], motivating increasing interest in using them to support scientific workflows, from literature synthesis to hypothesis generation [4]. However, many scientific tasks require iterative, feedback-driven experimentation: observations motivate hypotheses; experiments test them; results update hypotheses and inform subsequent experimental design (Fig. 1 (a)). This raises a central question: can LLMs operate in an interactive discovery setting—proposing hypotheses, selecting informative experiments, and updating beliefs under feedback—to recover hidden ground truths?

Many existing benchmarks evaluate isolated research skills, such as writing and coding [5], [6], or domain-specific endpoints in material discovery [7], [8] and synthetic biology [9], [10]. Although recent interactive environments study more realistic discovery behavior, failures in such settings can be difficult to attribute because they conflate exploration, domain knowledge, language grounding, stochastic feedback, and memory. As a result, there remains a need for controlled diagnostics that preserve the closed-loop structure of discovery while isolating specific prerequisite capabilities.

Our benchmark occupies a middle ground between realistic scientific discovery environments and isolated capability tests. Realistic environments such as open-ended laboratory or virtual-world benchmarks offer ecological validity, but failures in such settings can arise from many entangled factors: exploration strategy, domain knowledge, stochastic observations, tool use, language grounding, or memory. At the other extreme, single-shot diagnostic tasks can isolate narrow skills but do not preserve the closed-loop structure of discovery. Our goal is therefore not to simulate real chemistry, sociology, or physics in full fidelity, but to construct controlled discovery abstractions that preserve the iterative hypothesis–experiment–update loop while removing confounds. This allows us to ask a sharper question: when ground truth is deterministic and feedback is clean, can current LLM agents maintain and update structured beliefs over long interaction horizons?

To address this gap, we introduce Auto-Discovery-Bench, a controlled oracle-guided diagnostic benchmark for iterative discovery. In each task, the ground truth (e.g., a graph or an equation) is hidden; the model alternates between proposing a hypothesis and selecting an experiment, and an oracle returns feedback after each step. We instantiate this protocol in three tasks: directed graph discovery, undirected relational discovery, and symbolic equation discovery. Fig. 1 (b) provides a concrete example where the agent must infer hidden relationships (edges) between entities (e.g., Game of Thrones houses). The agent performs a perturbation (the sword icon), changing the state of the target node and propagating to its neighbors. By tracking these state shifts (e.g., \((2,3) \to (2,4)\)) over multiple rounds, the agent recovers the hidden interaction graph.

Empirically, we observe large gaps across models as difficulty increases. In the graph-based tasks, many models degrade substantially when scaling from small graphs to 10-node graphs: some require more rounds to converge, while others fail within the interaction budget. To probe the underlying failure mode, we additionally introduce a trajectory-tracking diagnostic, where models must integrate state changes over extended sequences. Accuracy decreases with horizon length, suggesting that limitations in maintaining and integrating long-range structured information are an important bottleneck for oracle-guided discovery.

This paper makes the following contributions:

• We introduce Auto-Discovery-Bench, a controlled oracle-guided diagnostic benchmark for closed-loop discovery behavior, designed to evaluate structured belief maintenance under repeated feedback while retaining the hypothesis–intervention–update loop.

• We instantiate the protocol in three task archetypes (directed graph discovery, undirected relational discovery, and symbolic equation discovery) with exact ground truth and deterministic evaluation.

• We show that model performance and interaction efficiency degrade with larger structures, longer trajectories, and distractor variables, indicating that clean feedback alone is insufficient for reliable discovery.

• We provide evidence that long-horizon structured state tracking is an important bottleneck, while acknowledging that planning, representation format, and prompt sensitivity may also contribute.

2 Related Work↩︎

Many LLM benchmarks evaluate single-shot reasoning through static queries, including arithmetic and symbolic reasoning tasks [11]–[14]. While useful for measuring local reasoning, these tasks do not preserve the closed-loop structure of discovery, where observations inform later interventions and hypotheses. Our work is also related to LLM-based causal graph discovery [15]–[18], which often queries edge existence or relies on variable metadata. In contrast, Auto-Discovery-Bench requires agents to actively choose interventions, receive oracle feedback, and update structured hypotheses over multiple rounds.

Recent benchmarks study richer interactive discovery behavior, including open-ended scientific environments such as DiscoveryWorld [19] and controlled physics-discovery settings such as PhysGym [20]. PhysGym is closest to our symbolic equation discovery setting, but its main focus is controlled prior knowledge in physics environments, obtained by masking context, variable descriptions, and variable names. Auto-Discovery-Bench is complementary: it studies long-horizon structured belief maintenance across multiple controlled discovery archetypes and includes a trajectory-tracking diagnostic that isolates state tracking from exploration and hypothesis generation.

3 Auto-Discovery-Bench↩︎

3.1 Scope and Design Philosophy↩︎

Auto-Discovery-Bench is designed around three principles.

First, we prioritize diagnostic control over ecological realism. All tasks have deterministic oracles, known ground truth, and exact evaluation metrics. This makes failures easier to attribute than in open-ended environments where language grounding, domain knowledge, stochasticity, and tool-use errors are entangled.

Second, the benchmark preserves the closed-loop structure of discovery. Unlike single-shot matrix or equation tasks, the agent must repeatedly choose an action, observe feedback, and revise a structured hypothesis under an interaction budget.

Third, the benchmark is intended to expose lower-bound failures. If a model cannot maintain structured state in clean, fully observed, noise-free settings, then similar failures are likely to become more severe in realistic settings with noise, partial observability, and high-dimensional observations.

Based on these principles, we introduce three diagnostic tasks: (i) directed graph discovery, (ii) undirected relational discovery, and (iii) symbolic equation discovery.

3.2 Directed Graph Discovery↩︎

In this setting, we use directed acyclic graphs (DAGs) as a controlled abstraction of directed dependency structures, as illustrated in Fig. 3 (a) and (b). Each node is described as a molecule for concreteness, and directed edges indicate dependency relations. For each node, a state value is assigned randomly from the set \(\{0, ..., S\}\), where \(S\) is the maximum state value. An intervention on a molecule causes all children molecules to change their state to a random state, while the state of the intervened molecule remains unaffected.

Let \(H_{ch} \in \{0,1\}^{N \times N}\) be the adjacency matrix representing the relationships between molecules, where \(N\) denotes the number of molecules. Each element of the matrix is either 1 or 0, with 1 indicating the presence of a directed edge and 0 indicating its absence. The ultimate goal for the LLMs is to infer this underlying matrix \(H_{ch}\).

We begin our experiment with an initial prompt that encapsulates all the details such as the problem statement and ultimate goal, and prompts the model to come up with an intervention to learn more about the DAG. More details are provided in the Appendix [Appendix]. In summary, the model is given the current node states and is prompted to suggest an intervention. After receiving a response from the model, based on the suggested intervention, the oracle executes the intervention and provides the new observations. In the consecutive cycles, the prompt includes the previous interventions performed by the LLM and all observations corresponding to those interventions in the form of an observation matrix. We denote the observation matrix as \(G_{ch} \in \{0, ..., S\}^{M \times N}\), where \(M\) represents the number of cycles and \(N\) represents the number of molecules. The cycles continue until \(K_{ch}\) matches the ground-truth \(H_{ch}\) or reaches a predefined number of cycles.

3.2.1 Evaluation↩︎

For a given DAG, multiple adjacency matrices can exhibit similar causal behavior. As shown in Fig. 3 (c) and (d), even though the graph structures differ, interventions may yield similar effects on the remaining nodes. Therefore, it is essential to consider this property when evaluating the similarity between generated matrices and the ground truth. Let \(K_{ch}^{n}\) denote the matrix \(K_{ch}\) raised to the \(n^{\text{th}}\) power, i.e., \(K_{ch}\) multiplied by itself \(n\) times. For all \(i,j\), we define \[H(i,j)= \begin{cases} 1, & \sum_{n=1}^{M} K_{ch}^{n}(i,j) > 0,\\ 0, & \text{otherwise.} \end{cases}\]

Given the hypothesis matrix produced by the model, we compute the corresponding reachability matrix for both the hypothesis and ground truth matrices. Then the resultant matrices are compared to each other to evaluate the similarities.

3.3 Undirected Relational Discovery↩︎

Similar to our directed graph discovery, relationships between individuals are represented as graphs, as illustrated in Fig. 1 (b). The main differences lie in the structure of the connections between nodes and in how they influence each other. To make the task more intuitive, we use the houses in Game of Thrones as an example.

The connections between houses are undirected. Here, an intervention represents an attack on a specific house, while the house state reflects its anger level. An intervention on a particular house increases its state, as well as the states of its neighboring houses by 1. Consequently, the adjacency matrix \(H_{so} \in \{0,1\}^{N \times N}\) should be symmetric, where \(N\) represents the number of houses in the relational network. The observation matrix here is denoted as \(G_{so} \in \{0, 1, 2...\}^{M \times N}\), where \(M\) represents the number of cycles, \(N\) represents the number of houses.

max width=0.9

3.4 Symbolic Equation Discovery↩︎

In this benchmark, we evaluate whether an LLM can recover a hidden symbolic equation through iterative measurements. The setting is physics-inspired but intentionally simplified: variables have generic physical names, the ground-truth formula is generated from a controlled grammar, and evaluation is based on exact input-output agreement.

3.4.1 Task Setup↩︎

In this setting, we simulate physical systems where a target quantity \(F\) (force) depends on a subset of available variables: mass (\(M\)), acceleration (\(A\)), velocity (\(V\)), surface area (\(S\)), temperature (\(T\)), and energy (\(E\)). Each variable takes integer values from the range \(\{1, ..., 10\}\). The ground-truth equation \(H_{ph}\) is hidden from the model.

At each round \(t\), the LLM receives input-output observations \(\{(x_i,F_i)\}_{i=1}^{N_{\text{obs}}}\), where \(x_i\) represents variable assignments and \(F_i\) is the corresponding output. History of previously proposed formulas, accuracy feedback and results from previous actions are also given. It proposes a hypothesis equation \(h_t\) and a measurement action \(a_t\) by assigning values to all active variables. The oracle scores \(h_t\) on random test cases, executes \(a_t\), and returns both predicted and true outputs, enabling hypothesis refinement.

3.4.2 Formula Generation↩︎

To ensure diverse and scalable evaluation, we implement a random formula generator that produces equations of varying complexity: (1) Simple: Pure linear combinations or products without exponents (e.g., \(F = M + A + V\), \(F = M \cdot A \cdot V\)) (2) Medium: Mixed operations with single squared terms (e.g., \(F = M^2 \cdot A + V\), \(F = M \cdot A - V\)) (3) Complex: Multiple squared terms or division operations (e.g., \(F = M^2 + A^2 \cdot V\), \(F = (M \cdot A) / V\))

Additionally, we introduce distractor variables that appear in the observations but do not affect the true formula. This tests whether models can identify irrelevant variables and avoid overfitting.

3.4.3 Evaluation↩︎

We evaluate formula correctness by comparing predictions across 100 random test cases with an error tolerance of \(\epsilon = 10^{-4}\). A formula is considered correct only if all outputs are identical to the ground truth across all test cases.

4 Experiments↩︎

We evaluate several state-of-the-art models, including Gemini 2.5 Pro [21], Grok-4 [22], Claude-3-5 [23], GPT-4o [24], Llama-3.1 (70B) [25], and Qwen2.5 (72B) [26]. All experiments are conducted on the OpenRouter platform [27], and the experimental budget is 365 USD. To ensure robust results and to minimize the impact of random fluctuations, we conduct 20 independent trials for directed graph discovery and 10 trials for undirected relational discovery. We will release the complete codebase upon acceptance. Below, we provide details of the models, experimental setup, and parameter configurations.

Directed graph discovery We evaluate three configurations, \((N,S)\in\{(3,3),(3,5),(10,5)\}\), to probe how LLM reasoning scales with graph complexity. By increasing nodes and state diversity, we test whether performance remains stable and identify thresholds where it degrades.

Undirected relational discovery We evaluate three network sizes—3, 5, and 10 houses to test how LLM reasoning scales with increasing relational complexity. As the number of houses and potential ties grows, we assess whether performance remains robust on larger, more intricate networks.

Symbolic equation discovery We evaluate five settings with varying numbers of variables (Var), three levels of formula complexity (Comp), and different numbers of distractors (Dis). By varying these settings, we test the performance of different LLMs under increasing difficulty.

4.1 Analysis↩︎

The results are reported in Table [Benchmark]. For directed graph discovery and undirected relational discovery, each episode is capped at \(2N\) interventions, where \(N\) denotes the number of nodes/houses. For symbolic equation discovery, the intervention budget is set to 10. If the model fails to recover the underlying ground-truth solution within the budget, we mark the episode as a failure. The success rate is the fraction of episodes solved, and the average number of iterations is computed over successful episodes only. We analyze the results from several angles. If the success rate is 0 (i.e., none of the episodes is solved), we report the average number of iterations as \(\infty\).

Directed graph discovery As shown in Table [Benchmark], GPT-4o and Qwen2.5 perform best in the small-graph regimes, achieving 100% success for both (3 nodes, 3 states) and (3 nodes, 5 states), with similar iteration budgets (GPT-4o: 4.15 vs.; Qwen2.5: 4.15 vs.). Grok-4 and Gemini-2.5-pro are also strong (90% success in both settings), while Claude-3.5-haiku and Llama-3.1 lag behind (success \(\leq\) 35%).

As expected, increasing the number of nodes substantially increases difficulty. In the (10 nodes, 5 states) setting, only Grok-4 and Gemini-2.5-pro retain perfect success (100%) with around 11 iterations on average (11.0 and 11.5, respectively). In contrast, GPT-4o drops sharply to 15% (10.67 iterations over successful episodes), while Claude-3.5-haiku, Llama-3.1, and Qwen2.5 all fail completely (0%). These results suggest that while several LLMs handle richer state dynamics on small directed graphs, robust discovery on larger directed structures remains a major open challenge.

Undirected relational discovery Table [Benchmark] shows that GPT-4o, Qwen2.5, Grok-4, and Gemini-2.5-pro all achieve 100% success on the 3- and 5-house settings, whereas Claude-3.5-haiku and Llama-3.1 remain substantially weaker (Claude: 20% \(\rightarrow\) 0%; Llama: 70% \(\rightarrow\) 30%). As the network size increases to 10 houses, performance diverges sharply: Grok-4 and Gemini-2.5-pro maintain 100% success (10.40 and 10.10 average iterations, respectively), while GPT-4o drops to 30% and Qwen2.5 collapses to 0%. This indicates that scaling to larger relational graphs remains challenging for several models, while the strongest models can still reliably extract and reason over the underlying structure.

In terms of efficiency, the average number of iterations increases with graph size for models that continue to succeed (e.g., Grok-4: 3.10 \(\rightarrow\) 5.00 \(\rightarrow\) 10.40; Gemini-2.5-pro: 3.40 \(\rightarrow\) 5.30 \(\rightarrow\) 10.10), consistent with the need for more probing interventions to disambiguate dense relations. Overall, current LLMs exhibit a clear scaling bottleneck on larger undirected relational graphs, suggesting that improved structural priors, better handling of long relational contexts, and more reliable multi-step reasoning could be crucial for further progress.

Symbolic equation discovery This task is the most polarized one across models. Grok-4 and Gemini-2.5-pro dominate across nearly all settings: both achieve 100% success for Var=3 with Simple/Medium/Complex formulas when no distractors are present (Dis=0), typically requiring only 1–2 iterations (e.g., Grok-4: 1.00–1.30; Gemini-2.5-pro: 1.00–2.10). When distractors are introduced (Var=3, Complex, Dis=2), Grok-4 remains at 100% success (1.40 iterations) and Gemini-2.5-pro remains high at 90% (1.89 iterations). The hardest setting (Var=5, Complex, Dis=2) reveals the key scaling challenge: Grok-4 retains 90% success (3.22 iterations), while Gemini-2.5-pro drops to 60% (4.00 iterations). In contrast, all other models stay at or below 30% in the easiest cases and largely fail (0%) once distractors or additional variables are introduced (often reported as \(\infty\) iterations due to zero success). Overall, larger variable counts and distractors are the main drivers of difficulty, and strong symbolic reasoning appears concentrated in the top-performing models.

Summary across three tasks In the three tasks, all models exhibit clear scaling effects as the problem size/complexity increases (more nodes/houses, more states, more variables, and additional distractors). Among all evaluated models, Grok-4 and Gemini-2.5-pro achieve the most consistently high success rates. However, average iterations provides a more fine-grained view beyond success rate: even when both models solve the same setting, Grok-4 typically requires fewer interventions than Gemini-2.5-pro (e.g., Symbolic equation discovery often solves in \(\sim\)​1–3 iterations for Grok-4 versus \(\sim\)​1–4 for Gemini-2.5-pro), indicating higher efficiency in reaching the correct solution. Overall, success rate captures whether a model can solve the task within budget, while average iterations further distinguishes how efficiently it does so.

4.2 Reasoning Traces↩︎

Figure 4 (a) shows reasoning-keyword frequencies in symbolic equation discovery for relationship, uncertainty, and evidence terms. Gemini-2.5-pro uses the most uncertainty and evidence terms, suggesting a more cautious, data-driven style, whereas Llama-3.1-70b-Instruct shows the lowest uncertainty usage and Claude-3.5-haiku remains consistently concise across categories. Figure 4 (b) reports average tokens per round: Gemini-2.5-pro is the most verbose, Claude-3.5-haiku is the most concise, and GPT-4o shows task-dependent verbosity. Overall, keyword usage and response length help characterize model reasoning styles and partially align with performance trends.

5 Evaluation on Trajectory Tracking↩︎

The main benchmark evaluates end-to-end discovery, where success depends on intervention selection, feedback interpretation, belief maintenance, and hypothesis revision. To isolate structured state tracking, we introduce a trajectory-tracking diagnostic in which the model receives the full observation matrix and only identifies which variables change across consecutive steps. Failures in this setting therefore point to difficulty maintaining and transforming structured observations over long horizons, rather than exploration strategy alone.

5.1 Task Definition↩︎

In each round, the LLM is given an observation matrix \(X \in \{0,\ldots,P\}^{M \times N}\), where \(M\) is the trajectory length, \(N\) is the number of nodes, and \(P\) denotes the number of color states. The goal is to infer a binary change-indicator matrix \(Y \in \{0,1\}^{(M-1)\times N}\), where \(Y_{i,j}=1\) iff node \(j\) changes color between consecutive steps \(i\) and \(i{+}1\), i.e., \(Y_{i,j}=\mathbb{1}[X_{i,j}\neq X_{i+1,j}]\) for \(i=0,\ldots,M{-}2\) and \(j=0,\ldots,N{-}1\). To reduce the impact of outliers, we repeat this procedure for \(R\) rounds.

max width=0.9

5.2 Prompting and Model Evaluation↩︎

To assess LLMs’ ability to comprehend and reason over structured information over long trajectories, we employ a structured zero-shot prompting strategy. The prompt explicitly describes the task, including matrix interpretation and expected output format. We evaluate multiple LLMs, including GPT-4o, Gemini, and Qwen, under identical conditions. Models are queried with the prompt and required to generate structured JSON outputs containing only the change-indicator matrix \(Y\).

We conduct systematic evaluations across various trajectory lengths (\(M \in \{3,5,10,15,20,25,30\}\)) to measure the robustness of each model. To evaluate the performance of each LLM, we develop two types of metrics, denoted as Average Trajectory Accuracy (AT-Acc) and Overall Accuracy (OA-Acc).

Overall Accuracy is computed as the fraction of exactly predicted change-indicator matrices: \[\text{OA-Acc} = \frac{1}{R} \sum_{i=1}^{R} \mathbb{1} (Y_i = \hat{Y}_i)\] where \(\hat{Y}\) represents the model-generated change-indicator matrix, \({Y}\) represents the ground-truth change-indicator matrix, \(R\) represents the number of rounds, and \(\mathbb{1}\) is the indicator function. In our experiment, we use \(R = 100\).

Average Trajectory Accuracy measures how accurately the model predicts color changes between each consecutive pair of rows (we refer to the change from row \(i\) to row \(i{+}1\) as the \((i{+}1)\)-th trajectory). For each round \(r \in \{0,\ldots,R{-}1\}\), let \(Y^{(r)}\) and \(\hat{Y}^{(r)}\) denote the ground-truth and predicted change-indicator matrices, respectively. We construct \(T \in \{0,1\}^{R \times (M-1)}\) where \(T_{r,i}=\mathbb{I}\!\left[Y^{(r)}_{i,:}=\hat{Y}^{(r)}_{i,:}\right]\) for \(i \in \{0,\ldots,M{-}2\}\), i.e., \(T_{r,i}=1\) iff the entire \(i\)-th trajectory (all nodes) is predicted correctly in round \(r\). The Average Trajectory Accuracy is then computed as: \[\begin{align} \text{AT-Acc}_j &= \frac{1}{R} \sum_{i=1}^{R} T_{i,j}, \\ &\forall j \in \{1, \dots, M-1\} \end{align}\]

5.3 Chain-of-Thought Prompting↩︎

To test whether CoT aids long-horizon tracking, we prepend the instruction “Please also include the reason for your answer” before the prediction.

5.4 Experiment Results↩︎

In this section, we present and analyze the results of our long trajectory measurement. The experimental results are presented in Table [Acc95vs95NoO] and Figure 5. We analyze these results from two perspectives:

Overall accuracy vs.trajectory length. Table [Acc95vs95NoO] shows that accuracy drops sharply as trajectories grow longer. For example, Claude falls from 69% at length 3 to near-zero accuracy by length 10. CoT generally helps on short sequences (e.g., GPT-4o: 13% \(\to\) 100% at length 5; Qwen2.5 shows similar gains), but the benefit diminishes on longer inputs, indicating limited robustness to extended evidence.

Average trajectory accuracy across positions. With length fixed at 20, Figure 5 averages 100 trials and shows a monotonic decline in accuracy with later trajectory positions, evidencing temporal attention decay. CoT partially mitigates this for GPT-4o and Qwen2.5, yet performance at late steps (e.g., Qwen2.5 at position 19 \(<70\%\)) remains weak. Future work should strengthen long-horizon reasoning (memory/attention over extended evidence) rather than relying solely on CoT.

6 Conclusion↩︎

We introduced Auto-Discovery-Bench, a controlled oracle-guided diagnostic benchmark for evaluating structured belief maintenance in closed-loop discovery. The benchmark includes three controlled archetypes: directed graph discovery, undirected relational discovery, and symbolic equation discovery. Our results reveal clear scaling limitations as problem size and complexity increase, with only the strongest models achieving consistently high success. Beyond success rate, average iterations provides a finer-grained measure of interaction efficiency, distinguishing models that solve the same instances but require different intervention budgets. Together, the benchmark and trajectory-tracking diagnostic provide reproducible, ground-truth evaluation of long-horizon structured state tracking and highlight the need for improved evidence integration and reliable multi-step inference.

7 Limitations↩︎

Auto-Discovery-Bench is not intended to simulate real scientific practice in full fidelity. The directed graph, undirected relational, and symbolic equation tasks are controlled abstractions with discrete states, deterministic feedback, simplified dynamics, and known ground truth. They omit important aspects of real discovery, including noisy measurements, partial observability, continuous variables, high-dimensional observations, non-stationary systems, and domain-specific background knowledge.

The conclusions should therefore be interpreted narrowly: the benchmark diagnoses whether models can maintain and update structured hypotheses in clean oracle-guided discovery loops. It does not establish that a model is capable of real-world scientific discovery, nor does failure necessarily identify a single causal mechanism. Although the trajectory-tracking diagnostic provides evidence that long-horizon structured state tracking is an important bottleneck, planning, representation format, prompt sensitivity, and tool use may also contribute.

Future versions can introduce controlled noise, missing observations, richer equation grammars, continuous variables, larger graph families, and standardized fixed test sets to bridge diagnostic abstraction and realistic discovery environments.

References↩︎