CIPHER: A Decoupled Exploration-Selection Framework for Test-Time Scaling of Data Science Agents
July 15, 2026
Abstract
Data science tasks span from closed-ended information extraction to open-ended analysis, presenting significant challenges for automation. Recent AI agents powered by language models show promise for handling such complex tasks. However, existing agents typically rely on a single initial state that conditions the entire agent’s execution, making them vulnerable to cascading errors initiated by a suboptimal initial state. To mitigate this, we present CIPHER, an automated data science agent that leverages test-time scaling through the generation and selection of multiple initial states for concurrent execution. Unlike existing works on test-time scaling of AI agents, CIPHER explicitly decouples the generation of candidate initial states from their strategic selection for parallel execution. Through extensive evaluation on two benchmarks (closed-form and open-form tasks), we demonstrate that CIPHER exceeds state-of-the-art performance in matched-model comparisons, and remains competitive against larger-model baselines despite relying on a substantially smaller base LM. Our empirical study characterizes the design space of the Decoupled Exploration-Selection (DES) framework: we quantify how generation strategy, selection strategy, and aggregator model capacity contribute to overall performance, and derive actionable design recommendations for practitioners.
1 Introduction↩︎
Data science [1] is an interdisciplinary field that leverages statistics, computer science, and scientific methods to extract knowledge from data. Data science tasks range from closed-ended where the goal is to extract verifiable information [2], to open-ended where the goal is more ambiguous [3]. For example, formulating an exploratory hypothesis for data analysis is an open-ended task, while extracting specific information from the data to validate such hypothesis is a closed-ended task.
Data science is increasingly influenced by AI solutions, such as the generation of new features with deep learning [4] or pipeline optimization with automated machine learning [5], [6]. More recently, AI agents powered by language models (LM) emerge as a promising approach [7], [8]. Unlike traditional AI solutions that target specific sub-tasks of data science, AI agents can handle fully complex open and closed tasks, with minimal human oversight [8].
Existing data science agents [9]–[11] are designed to mimic how data scientists execute tasks, such as planning subsequent data analysis stages based on prior results [10], or generating code through iterative debugging [11]. These prior works focus on a specific class of AI agents that utilize LMs through predefined execution route [12]. The execution of the route starts from an initial state (e.g. a high-level plan) generated by a LM, based on the task provided by the user. The quality of the initial state heavily influences task success, as improvements at this first step compound throughout the entire execution route. Since even the most capable language models can produce hallucinations or reason sub-optimally [13], [14], initial state generation represents a high-leverage point for improving agent performance — the methodological motivation of this work. The influence of initial states generation is even more pressing when using test-time scaling, which consists of executing multiple independent initial states towards an aggregated final answer to improve performance [15]–[18].
1.0.0.1 Research question and contributions:
We hypothesize that in test-time scaling, the generation of initial states and their selection for execution jointly influence the performance of the agent. To validate this hypothesis, we make three main contributions: (1) We present
CIPHER1 (see Figure 1), a data science agent that demonstrates performance improvements upon state-of-the-art agents on open and
closed-form data science tasks. (2) Through CIPHER, we introduce a Decoupled Exploration-Selection (DES) framework which is a novel approach for test-time scaling based on the generation followed by the
selection of initial states, opening the way for future research on this problem. (3) We propose an extensive empirical study on the optimal design of the DES framework with a focus on the data science domain, which may be of independent
interest for practitioners.
2 CIPHER: a data science agent↩︎
CIPHER’s execution route is illustrated in Figure 1. The agent receives as input a task noted \(\mathcal{T}\) composed of a dataset and a textual user goal (open-ended or
closed ended data science question). Examples of tasks and executions are provided in Appendix 11. The execution route consists of a deterministic directed graph implemented in LangGraph [19]. The output is a textual response to the user goal noted \(s^*\).
A novel approach for initial state generation and selection CIPHER first generates a broad set of \(N\) candidate states, where \(N\) is a hyperparameter to
set by the user. The candidates are generated by a Planner agent instructed to create detailed and actionable plans aiming to solve the task. Subsequently, a subset of \(M<N\) states is selected for
execution (\(M\) is a hyperparameter set by the user as well). Together, parameters \(M\) and \(N\) control the novel Decoupled Exploration-Selection (DES)
framework further detailed in Section 3. This framework conceptually differs from existing works on test-time scaling, which typically conflate the two steps as a finite set of \(M\)
states is typically generated and is then fully executed [20], [21].
Parallel routes execution CIPHER executes the \(M\) selected states in parallel to decrease latency. Each selected state follows the same execution route detailed in Figure 1. The route is composed of a Feature agent (suggests the creation of relevant features), a coding loop (translates the plan into executable code), and a Finalizer agent (analyze code executions and
provide a summary). Finally, an Aggregator agent synthesizes the information from the concurrent routes, enabling the final output.
Code iterative refinement Loop CIPHER instantiates an iterative refinement loop [22] to improve the
quality of the generated code. A Coder agent generates a first code draft, which is then executed. Then, an Analyzer agent evaluates jointly the logs and the code to identify errors or suboptimal patterns. If issues are detected,
an Editor agent modifies locally the code draft without re-generating it all. The loop repeats until the iteration budget \(b\) is reached (\(b\) is a hyper-parameter set by the
user). We provide in Figure 5 in the Appendix descriptive statistics on the effectiveness of the coding loop.
3 The Decoupled Exploration-Selection (DES) Framework↩︎
In this Section, we outline the proposed decoupled exploration-selection test-time scaling framework instantiated in CIPHER. The Decoupled Exploration-Selection (DES) framework draws structural inspiration from the classic
exploration-exploitation trade-off in decision theory and reinforcement learning [23]. By separating candidate generation (\(N\)) from state execution (\(M\)), it maps wide-breadth search space exploration, to a distinct phase of strategic, value-maximizing exploitation.
To our knowledge, explicitly decoupling candidate generation from strategic selection is a novel design point in the test-time scaling design space, distinct from prior work that couples the two steps [24], [25]. The integration of the
framework with CIPHER is illustrated in Figure 1, and is further detailed in Algorithm 2. We highlight in green the main
technical novelty compared to existing frameworks [20].
3.1 Generating candidate states↩︎
The method Generate-Candidates in Algorithm 2 outputs a set \(\mathcal{C}\) (with \(|\mathcal{C}|=N\)) of candidate states. The method’s inputs are the user task \(\mathcal{T}\), the current set of candidate states \(\mathcal{C}\) and a generation mode \(g \in \{ \texttt{ensemble}, \texttt{independent}, \texttt{conditional} \}\), two of which are inspired from the literature and one is a novel approach. Other generation modes certainly exist (e.g., use of different inference configurations; yet results suggest it may be suboptimal [26]). Our goal is to empirically evaluate these three distinct design paradigms to establish a foundational understanding of the decoupled exploration-selection framework.
Ensemble generation consists in sampling \(N\) candidates from a set of \(E\) distinct prompts (parameter set by user), instead of relying only on one single prompt. More details are provided in Appendix 9. Each prompt is used with probability \(p=[p_1,\dots,p_E]\) 2. This mode is embarrassingly parallel. Different prompts may access complementary subspaces of the LM’s capabilities. This is further motivated by prior research on ensemble learning which improves generalization [27], [28]. To our knowledge, this ensembling generation mode is novel in test-time scaling of multi-turn agents. Broader impacts include the design of prompt ensembles (chain of thought, few shot, persona-based prompting [29]), and the learning of optimal \(p\) distributions.
We also consider the following baselines. Independent generation: Consists in using the same prompt and inference configuration \(N\) times to populate \(\mathcal{C}\). This strategy samples independently from the same LM’s stochastic distribution conditioned by the prompt and its inference configuration. This is a baseline strategy previously used in test-time scaling [15], [20]. Independent generation is embarrassingly parallel and has no sequential dependencies. Conditional generation consists in using \(\mathcal{C}\) from prior calls to obtain a new candidate \(c\) on the current call to Generate-Candidate-State. The prior candidates in \(\mathcal{C}\) are used in the prompt instructions to elicit diversity in the upcoming candidates. Conditional generation is a form of adaptive sampling where each new sample is informed by the previous ones. This generation mode is investigated in prior works [3], [20]. It is not possible to parallelize in this mode.
3.2 Selecting states for concurrent execution↩︎
The method Select-States outputs a set \(\mathcal{S}\) (\(|\mathcal{S}|=M\)) of selected states that are to be executed concurrently, receiving the candidate set \(\mathcal{C}\), a budget \(M\), the task \(\mathcal{T}\), and a selection mode \(v \in \{ \texttt{random}, \texttt{entropy}, \texttt{clustering}, \texttt{alignment} \}\). For Random selection, \(M\) states are sampled uniformly at random from \(\mathcal{C}\). For Entropy selection, each candidate is projected into an embedding space where a greedy maximin algorithm maximizes the selection’s entropy using cosine distance [30]. Under Clustering selection, candidates are embedded and grouped into \(M\) clusters via \(k\)-means [31], selecting the single candidate closest to each centroid. Finally, Alignment selection leverages an LLM judge to analyze all \(N\) candidates in \(\mathcal{C}\) and return the \(M\) indices most aligned with the task, though high values of \(N\) scale up inference costs and risk context window cognitive saturation.
Embeddings The embeddings are obtained from Titan model [32], with \(1,024\) as embedding size. To our knowledge, projecting candidate spaces into a latent space for subset selection is a novel algorithmic approach for test-time scaling AI agents. Designing embedding spaces that further capture semantic properties of candidates is a promising avenue for future work.
3.3 Aggregating diverse execution results↩︎
The method \(\texttt{Aggregate}\) takes as input all the execution results, it also needs an aggregation mode \(a \in \{ \texttt{self}, \texttt{leader} \}\) to be set and it returns the final response. We investigate whether the capacity of the language model aggregator influences final performance. Self-aggregation employs the same LM capacity across all agent nodes. Leader aggregation selectively upgrades the aggregator node to a more advanced LM. We use identical agent execution logs; the only difference is the model used at the aggregator node. This ensures that all upstream agent states are identical, isolating specifically the effect of the aggregator node.
Prior works [17] on single-turn LM test-time scaling propose to execute multiple concurrent executions and then use softmax averaging over the distribution of tokens to aggregate and obtain a final response. The limitation of this approach is that it assumes access to the logits of the LM, which is often not the case in real-world applications where LMs are accessed through APIs.
4 Evaluation setup↩︎
We conduct experiments on a total of \(357\) tasks, spanning two open-source benchmarks, namely Infi-DA-Bench [2] and InsightBench [3]. InsightBench [3] evaluates performance on open-ended ambiguous tasks (e.g., hypothesis validation through data analysis). In contrast, Infi-DA-Bench [2] evaluates performance on closed-ended tasks (e.g., exploratory data analysis and modeling). Therefore, Infi-DA-Bench corresponds to only a subspace of the skills required to solve InsightBench. Examples of tasks from both benchmarks are available in Table 1.
| Closed-ended (Infi-DA-Bench) | Open-ended (InsightBench) | |
|---|---|---|
| Task | Identify countries where gdpPercap_1982 values are outliers using the IQR method (\(1.5 \times\) IQR threshold). Return the list in format: @outlier_countries[list]. | Analyze warranty period distribution by model category, cost distribution by model category, and the correlation between cost and warranty period for computer assets. Provide 5 data-driven insights. |
4.1 Open-ended tasks (InsightBench)↩︎
InsightBench [3] includes a set of \(T=100\) tasks. Each task comes with a dataset and a set of ground truth actionable insights. The goal of the agent on each task is to retrieve accurately the ground truth actionable insights in the dataset. There is a total of \(428\) ground truth insights to be retrieved across all the questions.
Task setup To help the agent retrieve the ground truth insights, we provide the agent with a set of exploratory hypotheses (formulated as open-ended questions). This is akin to a data scientist having formulated a set of exploratory hypotheses and using the agent to conduct the analysis. For reproducibility, the exploratory hypotheses we use are those available in the meta-data of each task in the benchmark. Another possible setup consists in evaluating the agent to retrieve the ground truth insights without the guidance of the exploratory hypotheses. As evidenced by our empirical results, both setups are challenging, with the full autonomy setup being harder.
Performance metric The performance is defined as the proportion of insights correctly retrieved by the agent across the \(T\) questions and \(428\) insights: \(\text{Accuracy} = \frac{\sum_{i=1}^{T} \text{Insights Retrieved}_i}{\text{Total Insights}}\). An increase of \(1\)pp corresponds to retrieving approximately \(4.3\) insights. While semantic similarity metrics have been proposed for this benchmark [3], they are difficult to interpret and do not directly reflect the agent’s operational ability to retrieve insights, unlike the accuracy metric we adopt.
Correctness evaluation A LLM-as-a-judge [33] evaluates whether the agent’s generated insights match the ground truth. We use Claude 3.5 Haiku at temperature \(0\) as the primary judge. The judge is held fixed across all methods, so relative comparisons are unbiased with respect to the choice of judge.
State-of-the-art baseline The current state-of-the art on InsightBench is the open-source strategy Agent-Poirot [3]. We evaluate Agent-Poirot with the official repository hyper-parameters.
4.2 Close-ended tasks (Infi-DA-Bench)↩︎
Infi-DA-Bench benchmark includes a total of \(T=257\) tasks. Each question is composed of a dataset and a closed-ended question (e.g. descriptive statistics, modeling). The goal of the agent is to respond correctly to the question.
Performance metric We report the performance as the accuracy over the set of \(T\) tasks, measured as \(\text{Accuracy} = \frac{ 1}{T} \sum_{i=1}^{T} [ \text{Correct Response}_i ]\). An increase of \(1\)pp corresponds to the correct resolution of approximately \(2.57\) tasks.
Correctness evaluation We use the official rule-based evaluation function to evaluate the responses [2]. The function is strict: small numerical deviation from the ground truth (up to \(1e-6\)), partially true responses, or valid responses with wrong formatting are all marked as incorrect. Consequently, performance improvements under this strict evaluation are meaningful and significant.
State-of-the-art baseline The current state of the art is Data-Wise agent [11]. Other agents exist but have been outperformed by Data-Wise agent [9], [34]. We consider Data-Wise agent as a baseline, and we use the official hyper-parameter configuration reported in the open-source repository.
4.3 Hyper-parameters↩︎
All the agents (baselines included) use the same inference configuration across all prompts: the temperature parameter is \(0.5\), the top-p is \(0.99\) and the top-k is \(500\). These parameters were selected after conducting a hyper-parameter sensitivity study on the following configurations: \((0.2,0.99,250)\), \((0.5,0.99,250)\), \((0.5, 0.99, 500)\), \((0.9, 0.99,500)\). The model cards used by the agents are Claude 3.5 Haiku (anthropic.claude-3-5-haiku-20241022-v1:0) and Sonnet 3.7 (anthropic.claude-3-7-sonnet-20250219-v1:0), with the temperature set to \(0\) for reproducibility. Each configuration is evaluated on three random seeds. The maximum number of generated tokens is set to \(20\)k.
Aggregator node. When a leader is used, we employ Sonnet 3.7 + thinking with a reasoning budget of \(5\)k tokens, and a temperature of \(1\) as required for reasoning. In that case the maximum number of generated tokens is set to \(25\)k.
5 Empirical Results↩︎
In this section, we present our empirical results. Our evaluation is the first to evaluate a data science agent able to solve both close-ended and open-ended tasks. We draw findings from this novel evaluation that provide valuable design insights to improve research on data science agents. We investigate three main design axes of the Decoupled Exploration-Selection framework: i) the generation strategy, ii) the selection strategy and iii) the aggregation strategy. We consider \(24\) configurations, stemming from the \(3 \text{ generation strategies} \times 4 \text{ selection strategies} \times 2 \text{ aggregation strategies}\). The \(24\) configurations are evaluated across \(2\) selection budget regimes \(M\in\{3,5\}\) and across two distinct benchmarks (InsightBench and Infi-DA-Bench), totaling \(96\) measurements, each averaged across three random seeds. To our knowledge, this is the first and most comprehensive ablation study on the design choices of test-time scaling.
5.1 Ablation Study on the Decoupled Exploration-Selection Framework↩︎
\(\triangleright\) Influence of budget \(N\) and \(M\) on performance. Figure 3 (a) shows that increasing the selection budget yields a monotonic accuracy gain of \(+1.9\)pp on Infi-DA-Bench and \(+1.4\)pp on InsightBench when going from \(M{=}3\) to \(M{=}5\), on top of an initial \(+7.2\)pp and \(+3.3\)pp jump when going from the single-agent baseline to \(M{=}3\). The marginal gain therefore shrinks with each additional selected plan, mirroring the diminishing-return pattern reported by [15] for single-turn LLM queries and extending it to multi-turn agentic designs on both closed-form and open-form tasks. Panel (b) reveals a qualitatively different picture for the generation budget \(N\): scaling from \(N{=}10\) to \(N{=}30\) has an opposite-sign effect depending on task type. On the open-form InsightBench, the gain over baseline grows from \(+4.7\)pp to \(+6.2\)pp; on the closed-form Infi-DA-Bench, it shrinks from \(+9.1\)pp to \(+8.0\)pp. This suggests that generation diversity helps only when the task admits a broad solution space, and confirms that \(M\) is the reliable performance lever across task types.
\(\triangleright\) Using a powerful leader in the aggregator node improves performance. Tables 2 and 3 show that when the base LM is an efficient model like Haiku 3.5, using a stronger LM for the aggregator node consistently improves performance. This effect is statistically robust: a
Mann-Whitney U test comparing leader versus self aggregation is significant on both benchmarks (Infi-DA-Bench: \(\Delta{=}{+}2.81\) pts, Cohen’s \(d{=}1.59\), \(p{<}0.001\); InsightBench: \(\Delta{=}{+}4.52\) pts, \(d{=}1.65\), \(p{<}0.001\)), and the effect holds uniformly across
generation strategies and budget settings. This finding suggests that agent designs would benefit from executing combinations of diverse LM models conditioned on the node, rather than using a single LM model for the entire execution graph.
\(\triangleright\) The advantage of the decoupled exploration-selection framework emerges only under leader aggregation. We apply a Friedman test to the rank ordering of the three generation strategies
(base, conditional, ensemble) across matched conditions, first on the pooled data (both benchmarks) and then on each benchmark individually. Under self aggregation, no strategy dominates (\(\chi^2{=}0.88\), \(p{=}0.65\); both per-benchmark tests are also non-significant), and the baseline base+random configuration remains competitive. Under
leader aggregation the picture reverses sharply: the pooled test becomes highly significant (\(\chi^2{=}14.00\), \(p{<}0.001\)), with Ensemble ranking first in
\(60\%\) of matched conditions versus \(23\%\) and \(17\%\) for the other two. The effect is directionally consistent on both benchmarks and reaches
significance on InsightBench (\(p{=}0.002\)); on Infi-DA-Bench the same ordering does not reach significance (\(p{=}0.17\)), which we attribute both to reduced per-benchmark power and to the
closed-form nature of the task, where a diverse plan pool offers limited additional headroom. This finding suggests that weak aggregators saturate information [35] and cannot exploit the diversity introduced by the exploration-selection framework, whereas stronger leader models translate that diversity into measurable accuracy gains.
| M = 3 | M = 5 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| 3-6 (lr)7-10 | Accuracy (%) | Accuracy (%) | |||||||
| 3-4 (lr)7-8 Generation | Selection | self-agg. | leader-agg. | Entropy | Tok In/Out (K) | self-agg. | leader-agg. | Entropy | Tok In/Out (K) |
| Base | Clustering | 74.84 ± 0.22 | 77.95 ± 0.45 | 0.069 \(\rightarrow\) 0.083 | 81.9/4.7 | 79.12 ± 0.22 | 82.23 ± 1.62 | 0.069 \(\rightarrow\) 0.081 | 82.1/4.7 |
| Goal Align | 77.82 ± 0.78 | 78.99 ± 0.78 | 0.069 \(\rightarrow\) 0.069 | 82.0/4.7 | 78.47 ± 0.81 | 80.80 ± 1.57 | 0.069 \(\rightarrow\) 0.068 | 82.3/4.7 | |
| Maxent | 74.84 ± 0.45 | 79.25 ± 1.80 | 0.070 \(\rightarrow\) 0.111 | 81.9/4.7 | 77.30 ± 0.98 | 80.42 ± 1.96 | 0.069 \(\rightarrow\) 0.090 | 82.3/4.7 | |
| Random | 77.69 ± 3.12 | 80.03 ± 1.19 | 0.069 \(\rightarrow\) 0.069 | 81.7/4.7 | 78.21 ± 1.40 | 81.45 ± 1.25 | 0.070 \(\rightarrow\) 0.070 | 82.4/4.7 | |
| 2-10 | 76.30 ± 2.06 | 79.05 ± 1.26 | 0.069 \(\rightarrow\) 0.083 | 81.9/4.7 | 78.27 ± 1.06 | 81.23 ± 1.56 | 0.069 \(\rightarrow\) 0.077 | 82.3/4.7 | |
| Conditional | Clustering | 74.32 ± 0.00 | 77.43 ± 1.17 | 0.069 \(\rightarrow\) 0.082 | 82.0/4.7 | 78.60 ± 0.67 | 80.80 ± 1.75 | 0.070 \(\rightarrow\) 0.082 | 82.4/4.7 |
| Goal Align | 75.49 ± 1.95 | 78.86 ± 0.59 | 0.069 \(\rightarrow\) 0.068 | 81.9/4.7 | 79.38 ± 1.03 | 80.80 ± 1.12 | 0.070 \(\rightarrow\) 0.069 | 82.2/4.7 | |
| Maxent | 75.62 ± 0.81 | 78.47 ± 0.22 | 0.068 \(\rightarrow\) 0.108 | 82.3/4.7 | 77.30 ± 1.47 | 80.29 ± 0.81 | 0.070 \(\rightarrow\) 0.091 | 82.3/4.7 | |
| Random | 76.65 ± 0.78 | 78.99 ± 1.40 | 0.068 \(\rightarrow\) 0.069 | 81.7/4.7 | 76.78 ± 1.84 | 79.25 ± 1.25 | 0.069 \(\rightarrow\) 0.069 | 82.3/4.7 | |
| 2-10 | 75.52 ± 1.29 | 78.44 ± 1.04 | 0.069 \(\rightarrow\) 0.081 | 82.0/4.7 | 78.02 ± 1.56 | 80.29 ± 1.28 | 0.070 \(\rightarrow\) 0.078 | 82.3/4.7 | |
| Ensemble | Clustering | 76.65 ± 1.03 | 79.77 ± 0.00 | 0.088 \(\rightarrow\) 0.112 | 85.1/4.9 | 77.17 ± 1.80 | 80.29 ± 1.25 | 0.088 \(\rightarrow\) 0.102 | 85.6/4.9 |
| Goal Align | 77.43 ± 2.43 | 80.42 ± 1.47 | 0.088 \(\rightarrow\) 0.069 | 85.3/4.9 | 78.08 ± 1.80 | 81.06 ± 1.25 | 0.088 \(\rightarrow\) 0.073 | 85.7/4.9 | |
| Maxent | 77.69 ± 2.00 | 78.34 ± 2.25 | 0.088 \(\rightarrow\) 0.138 | 85.2/4.9 | 76.52 ± 2.00 | 81.19 ± 1.25 | 0.089 \(\rightarrow\) 0.110 | 85.7/4.9 | |
| Random | 75.88 ± 2.17 | 78.86 ± 1.37 | 0.088 \(\rightarrow\) 0.088 | 85.1/4.9 | 78.86 ± 0.98 | 82.23 ± 1.62 | 0.087 \(\rightarrow\) 0.088 | 85.3/4.9 | |
| 2-10 | 76.91 ± 1.84 | 79.35 ± 1.53 | 0.088 \(\rightarrow\) 0.102 | 85.2/4.9 | 77.66 ± 1.71 | 81.19 ± 1.36 | 0.088 \(\rightarrow\) 0.093 | 85.6/4.9 | |
| M = 3 | M = 5 | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| 3-6 (lr)7-10 | Accuracy (%) | Accuracy (%) | |||||||
| 3-4 (lr)7-8 Generation | Selection | self-agg. | leader-agg. | Entropy | Tok In/Out (K) | self-agg. | leader-agg. | Entropy | Tok In/Out (K) |
| Base | Clustering | 41.74 ± 3.76 | 42.99 ± 1.07 | 0.107 \(\rightarrow\) 0.131 | 95.5/5.8 | 43.85 ± 0.54 | 49.22 ± 0.75 | 0.103 \(\rightarrow\) 0.122 | 95.8/5.7 |
| Goal Align | 41.36 ± 2.63 | 42.68 ± 1.37 | 0.104 \(\rightarrow\) 0.098 | 95.0/5.7 | 41.28 ± 0.49 | 49.45 ± 1.72 | 0.105 \(\rightarrow\) 0.102 | 96.0/5.7 | |
| Maxent | 40.73 ± 1.64 | 41.82 ± 2.04 | 0.107 \(\rightarrow\) 0.169 | 95.1/5.8 | 42.68 ± 0.94 | 49.14 ± 1.29 | 0.106 \(\rightarrow\) 0.137 | 96.3/5.8 | |
| Random | 42.68 ± 1.35 | 45.09 ± 2.23 | 0.104 \(\rightarrow\) 0.107 | 95.1/5.7 | 44.24 ± 0.59 | 48.99 ± 3.59 | 0.104 \(\rightarrow\) 0.103 | 96.1/5.7 | |
| 2-10 | 41.63 ± 2.28 | 43.15 ± 1.95 | 0.105 \(\rightarrow\) 0.126 | 95.2/5.7 | 43.01 ± 1.33 | 49.20 ± 1.82 | 0.105 \(\rightarrow\) 0.116 | 96.1/5.7 | |
| Conditional | Clustering | 40.81 ± 2.82 | 43.22 ± 1.07 | 0.104 \(\rightarrow\) 0.122 | 95.4/5.7 | 44.31 ± 1.29 | 49.92 ± 1.20 | 0.106 \(\rightarrow\) 0.124 | 96.1/5.7 |
| Goal Align | 41.98 ± 0.75 | 40.34 ± 1.50 | 0.104 \(\rightarrow\) 0.101 | 96.2/5.7 | 42.45 ± 1.75 | 48.29 ± 1.18 | 0.104 \(\rightarrow\) 0.101 | 95.4/5.6 | |
| Maxent | 42.60 ± 0.71 | 43.93 ± 1.42 | 0.105 \(\rightarrow\) 0.165 | 94.7/5.7 | 43.61 ± 0.82 | 49.14 ± 1.05 | 0.103 \(\rightarrow\) 0.132 | 95.9/5.7 | |
| Random | 40.03 ± 3.58 | 42.60 ± 1.78 | 0.105 \(\rightarrow\) 0.103 | 94.7/5.6 | 43.07 ± 2.17 | 49.53 ± 1.17 | 0.104 \(\rightarrow\) 0.105 | 95.6/5.7 | |
| 2-10 | 41.36 ± 2.25 | 42.52 ± 1.88 | 0.104 \(\rightarrow\) 0.123 | 95.3/5.7 | 43.36 ± 1.54 | 49.22 ± 1.17 | 0.104 \(\rightarrow\) 0.116 | 95.8/5.7 | |
| Ensemble | Clustering | 41.59 ± 2.00 | 47.90 ± 2.25 | 0.118 \(\rightarrow\) 0.151 | 99.2/6.7 | 43.93 ± 0.93 | 48.36 ± 0.84 | 0.119 \(\rightarrow\) 0.140 | 99.8/6.6 |
| Goal Align | 40.73 ± 0.94 | 47.59 ± 1.52 | 0.116 \(\rightarrow\) 0.057 | 100.8/6.7 | 43.46 ± 2.00 | 49.53 ± 0.23 | 0.116 \(\rightarrow\) 0.073 | 101.7/6.7 | |
| Maxent | 42.29 ± 0.81 | 46.96 ± 0.84 | 0.117 \(\rightarrow\) 0.182 | 99.1/6.7 | 42.06 ± 1.87 | 50.39 ± 1.87 | 0.116 \(\rightarrow\) 0.145 | 100.3/6.7 | |
| Random | 41.20 ± 0.97 | 47.90 ± 0.81 | 0.117 \(\rightarrow\) 0.122 | 100.8/6.8 | 43.69 ± 3.31 | 49.77 ± 1.82 | 0.116 \(\rightarrow\) 0.113 | 100.1/6.7 | |
| 2-10 | 41.45 ± 1.24 | 47.59 ± 1.32 | 0.117 \(\rightarrow\) 0.128 | 100.0/6.7 | 43.28 ± 2.02 | 49.51 ± 1.40 | 0.117 \(\rightarrow\) 0.118 | 100.5/6.7 | |
\(\triangleright\) The Ensemble generation strategy is the only source of true plan-space diversity. We measure the entropy of the embeddings of generated and selected states to assess the
diversity induced by different design choices. Tables 2 and 3 show that Base and Conditional
generation strategies produce statistically indistinguishable entropy scores (Mann-Whitney with Bonferroni correction, \(p{=}0.76\) on Infi-DA-Bench and \(p{=}0.29\) on InsightBench; \(|d|{<}0.4\)), which is aligned with [20]. In
contrast, the proposed Ensemble generation strategy exhibits significantly higher entropy on every benchmark (both pairwise comparisons vs. Base and Conditional yield \(p{<}0.001\),
with effect sizes \(d{>}5\) on both benchmarks individually). This stems from combining distinct prompt templates to encourage structurally different executions.
\(\triangleright\) The Maxent selection strategy is uniquely effective at identifying diverse plan subsets. Across all generation methods and both benchmarks, the selection strategy
Maxent consistently produces the highest selected-plan entropy. The ordering Maxent\(\geq\)Clustering\(\geq\)Random\(\geq\)Goal-Align is preserved on both benchmarks independently, validating that Maxent’s entropy-based objective successfully identifies maximally diverse subsets from the candidate pool. However, this
highest selected-plan diversity does not translate into better downstream accuracy, suggesting that raw entropy of the selected subset is not by itself a sufficient criterion for improving task performance. This finding supports [36], who show that diversity can in some cases be a performance bottleneck in agent designs.
5.2 Identifying the Most Promising Design↩︎
We consider four complementary properties. Performance is the mean accuracy across seeds; Reliability is the squared distance between a strategy’s worst-case run and the absolute worst run observed on the panel, rewarding methods that avoid catastrophic seeds; Consistency is the top-\(3\) agreement of the selected plans across seeds; and Scalability is the mean accuracy gain when moving from \(M{=}3\) to \(M{=}5\).
Pure performance ranking Ranking the 12 strategies on Performance alone (see Figure 6), Ensemble+Goal-Align and Ensemble+Random share the top spot with an average rank of
3.4 across the four panels, followed by Ensemble+Maxent at 4.8. All three top slots are Ensemble variants, and on raw mean accuracy the four Ensemble variants occupy the top four positions overall (64.1–64.7%), before any Base or Conditional
strategy appears. Between the two co-leaders, Ensemble+Goal-Align stands out as the only strategy among the twelve whose panel-wise rank never drops below 5 (panel ranks \([1, 3, 5, 4]\));
Ensemble+Random, by contrast, falls to rank 7 on Infi-DA-Bench \(M{=}3\). To confirm this pattern statistically, we compare the three generation families under identical experimental conditions:
Ensemble significantly outperforms Base (\({+}1.25\)pp, \(p{=}0.018\)) and Conditional (\({+}1.79\)pp, \(p{<}0.001\)).
Composite ranking To check that this pattern extends beyond raw accuracy and does not depend on any particular weighting of the four metrics, we sample \(10{,}000\) weight vectors uniformly from the
four-metric simplex (\(\mathrm{Dirichlet}(1,1,1,1)\)) and recompute the ranking under each. The Ensemble family occupies \(1.65\times\) its expected share of top-3 slots,
whereas Conditional occupies only \(0.39\times\); both deviations are large relative to a uniform baseline. Within the Ensemble family, Ensemble+Random, Ensemble+Goal-Align and
Ensemble+Maxent each land in the top-3 in more than half of all weight configurations (\(59.5\%\), \(54.6\%\) and \(50.6\%\) respectively),
while Ensemble+Clustering never does (\(0\%\)). The Ensemble family therefore consistently holds the top of the leaderboard across the four-metric weighting space.
5.3 CIPHER performance against existing data science agents and across task types↩︎
We denote CIPHER(1,1) as the simplest instance without test-time scaling and base generation, CIPHER\(^\dagger\)(10,5) as our recommended configuration
(Ensemble+Goal-Align, leader aggregation), and CIPHER\(^\star\)(10,5) as the empirically best configuration per benchmark (see Section 5.2). We compare
against Agent-Poirot [3] and DataWise [11].
\(\triangleright\) CIPHER performance against specialized baselines. On InsightBench (Table [tbl:tab:insightbench95results]), CIPHER(1,1) matches Agent-Poirot at the single-agent level (\(38.32\)% vs.\(38.63\)%), and enabling test-time scaling widens the gap to \({+}10.9\)pp for CIPHER\(^\dagger\) (\(49.53\)%) and
\({+}11.8\)pp for CIPHER\(^\star\) (\(50.39\)%; Welch \(p{<}0.01\)). On Infi-DA-Bench
(Table ¿tbl:tab:infidabench95results?), the specialized DataWise baseline outperforms CIPHER(1,1) at the single-agent level (\(74.06\)% vs.\(69.13\)%) — unsurprisingly, since DataWise instantiates a more complex fixed route with both iterative planning and debugging while our base route only uses iterative debugging. Scaling compute completely reverses
this dynamic under a matched-model comparison: CIPHER\(^\dagger\) reaches \(81.06\)% (\({+}7.0\)pp) and CIPHER\(^\star\) reaches \(82.23\)% (\({+}8.2\)pp, Welch \(p{<}0.01\)). We stress that these are matched-model gains:
DataWise paired with a stronger base LM (GPT-4o) reaches \(85.99\)% on Infi-DA-Bench and remains the absolute state of the art on that benchmark. Our contribution is therefore not to claim best-in-class
performance across all model tiers, but to show that the DES framework unlocks accuracy gains that close most of the gap while relying on a substantially smaller base model. This gain comes at a \({\sim}5\times\)
input-token cost relative to the single-agent baseline (Table ¿tbl:tab:infidabench95results?), consistent with the multiple-execution regime we study. The gap between our recommended and absolute-best configurations remains within \(1.5\)pp on both benchmarks, confirming that Ensemble+Goal-Align is a competitive default even when it is not strictly the top-scoring choice on a given benchmark.
\(\triangleright\) Non-uniform performance gains across task and model type. Figure 4 confirms that test-time scaling is associated with performance gains, which is aligned with the literature [20]. Crucially, our findings extend these insights by demonstrating that these performance gains are non-uniform across data science task regimes. On close-ended questions (Infi-DA-Bench), the gain is more pronounced with efficient LMs like Haiku 3.5 (+10.0pp) compared to more capable LMs like Sonnet 3.7 (+4.3pp). This is because efficient LMs like Haiku 3.5 solve tasks with higher variance, so more executions improve likelihood of solving the task successfully. The pattern is reversed for open-form tasks (InsightBench): scaling continues to yield gains for more capable models, with the gain increasing from +5.5pp to +7.8pp as model capacity increases (from Haiku 3.5 to Sonnet 3.7). The novelty of our observation lies in the task-type dependence of this scaling behaviour, not in the existence of gains for test-time scaling.
6 Conclusion↩︎
The proposed framework opens the door to algorithmic innovations to improve upon the standard test-time approaches (Base generation + Random selection). Although this baseline is competitive, we show that it can be outperformed
by strategies based Ensemble generation and entropy based selection techniques. The results further highlight the role of the aggregator node as a support to the entire framework. Lastly, the proposed agent CIPHER performs
competitively against state-of-the-art baselines, demonstrating the relevance of the decoupled exploration-selection framework to improve the performance of AI agents. Future research avenues include the scaling of ensemble generation, and the selection of
candidate states using gradient based methods [37].
Limitations Our selection strategies are agnostic to the token-level probabilities of the underlying LM; some open-access models expose logits which could enable finer selection. CIPHER generates all \(N\) candidate states in a single upfront batch, whereas adaptive schemes that grow the candidate pool in response to execution failures could unlock further gains. Our validation is focused on data science tasks; whether the
DES framework transfers to other agent domains remains an open question.
7 Related Works↩︎
Our design connects to a broader test-time scaling literature. [38] established parallel sampling as a reliable inference-time technique on reasoning benchmarks. [39] generalize this into structured search over reasoning trees, and [40] propose parallel agent debate for open-ended reasoning. Sequential test-time refinement ([41], [42]) offers a complementary alternative that trades parallel breadth for iterative depth. Recent work by [43] characterizes optimal test-time compute allocation, providing theoretical context for our empirical study of the generation–selection trade-off.
[15] shows that multiple executions of the same prompt in single-turn LM applications improves performance at test-time on closed-ended tasks. [16] proposes a system to leverage parallel execution of LLM prompts with inter-communication to improve performance and share information across threads. [25] compare the returns of parallelization versus reasoning on large language models and finds that reasoning tends to outperform parallelization. However, [18] show that test-time scaling with the right generation, selection and aggregation strategy can outperform larger LM models, highlighting that parallelization is an avenue for cost efficient designs. [24] highlight that adaptive strategies that emulate diverse solution paths improve performance, and propose a method based on exploration-exploitation.
Although test-time scaling properties have been investigated in the realm of single-turn LM applications, its impact on agentic designs remains an emerging avenue of research. [44] propose Flash-Searcher, showing how parallelization can deliver latency speed-ups when sectioning problems into independent parts. [21] operate concurrent executions using diverse LLMs. [20] proposes a seminal framework for test-time scaling but there is no clear distinction between generation and selection. Furthermore, many questions remain open regarding the number of agents to parallelize, and the strategy to aggregate results. The approach SPIO [45] implements test-time scaling at the node level where each node is executed multiple times within an agentic workflow.
8 Discussion and limitations↩︎
8.0.0.1 Compute-matched comparison.
We report input and output tokens for each configuration, showing that CIPHER\(^\star\)(10,5) uses approximately \(5\times\) more input tokens than the single-agent
CIPHER(1,1). A fully compute-matched baseline (e.g., five independent CIPHER(1,1) executions aggregated via majority voting) is not evaluated here, and we acknowledge this as a limitation. We note however that
CIPHER’s gain arises from generating and selecting structurally different plans, not from repeated sampling of the same plan distribution: an equivalent-token replay of CIPHER(1,1) would draw from a single initial plan
and would not exploit the plan-space diversity that Section 5.2 identifies as the primary source of the observed gains.
8.0.0.2 Judge sensitivity
Because Claude 3.5 Haiku serves as both the InsightBench judge and the base LM for every method in Table [tbl:tab:insightbench95results], absolute accuracy values could in principle be biased by shared vocabulary or stylistic preferences between the judge and the models under evaluation. This concern applies uniformly across all methods — every row uses the same base LM and the same judge — so the relative rankings and the significance tests we report are unaffected by the choice of judge.
9 Details on CIPHER and DES design and implementation↩︎
Trade-off related to cost efficiency \(\texttt{CIPHER}\) is instructed to reason exclusively on the text modality. Although the generated code can produce visual artifacts (e.g., charts, plots), the downstream agents only consume the corresponding textual logs (e.g., summary statistics, model metrics, tables) printed during code execution. This design choice is motivated by two key factors: i) high performance on images necessitates multimodal reasoning LMs which are more costly, ii) image tokenization costs are orders of magnitude greater than processing equivalent information through text.
Ensemble strategy The choice of the optimal \(E\) parameter is linked to the parameter \(M\). For a task, if \(M\) states are executed, the set of prompts \(E\) is more likely to be fully covered when \(E<M\). Since in our experiments we focus on \(M=3\) and \(M=5\) regimes, we set \(E=2\) and \(p=[0.5,0.5]\). The two prompts are: prompt 1) is the base prompt used in CIPHER (1,1) the base agent, 2) the second prompt is a paraphrase obtained from a LLM. The key difference is that the paraphrase in prompt no. 2 imposes a fixed structure to the plan generated, while prompt no. 1 does not.
Aggregator node We adopt an open-source prompt previously proposed to aggregate outcomes from concurrent executions [20].
9.1 Implementation details on the selection strategies↩︎
All embedding-based methods use Amazon Titan Text Embeddings v2 (amazon.titan-embed-text-v2:0), which produce unit-normalized vectors; no additional normalization is applied.
Random. \(M\) plans are sampled uniformly at random without replacement.
MaxEnt (Greedy Maximin). We compute pairwise cosine distances between all \(N\) plan embeddings. Selection proceeds greedily: the two most distant plans are chosen first, then each subsequent plan is the one maximizing its minimum cosine distance to any already-selected plan. This greedy maximin procedure approximates maximum entropy over the selected set.
Clustering. We apply \(k\)-means with \(k{=}M\) clusters over the plan embeddings (scikit-learn implementation, \(k\)-means++ initialization, \(n\_init{=}10\) restarts, seeded with the experiment random seed). Within each cluster, the plan closest to the centroid in Euclidean distance is selected as the representative.
Goal Align (LLM Judge). An LLM judge (Claude 3.5 Sonnet, temperature 0.5) receives the original question and all \(N\) plan texts, and is prompted to select the \(M\) most promising plans based on three criteria: goal alignment, feasibility, and diversity of perspectives. The judge outputs a JSON array of \(M\) plan indices. The full prompt context comprises the question and all plans without truncation; the response is limited to 200 tokens. If parsing fails, the method falls back to random selection.
10 Additional results↩︎
Ablation on CIPHER coding loop design Within each plan step, if code execution fails, the router sends the failed cell to the editor for a fix; the retry loop coder \(\rightarrow\) executor \(\rightarrow\) analyzer \(\rightarrow\) editor \(\rightarrow\)
executor can repeat up to three times before the step is routed to the finalizer. We set max_iterations\(=3\) based on the observed distribution of retries under our baseline
CIPHER(1,1) configuration (Figure 5): \(86.9\%\) of steps on Infi-DA-Bench and \(59.7\%\) on InsightBench succeed on the first attempt
(iteration \(0\)), and by the second retry (iteration \(2\)) the cumulative success rate reaches \(\geq99\%\) on both benchmarks. Only \(\leq1\%\) of steps ever exhaust the loop without success. A cap of three iterations therefore captures essentially all recoverable failures while bounding the worst-case token cost per step.
![Figure 6: Table: Strategy rankings across benchmarks (Infi-DA-Bench, InsightBench) and selection budgets (M \in{3, 5}) with leader aggregation, N{=}10 generated plans, and 3 random seeds. Ranks derived from mean accuracy. Medals denote top-3 overall average rank. Panel: Mean accuracy per strategy with leader aggregation (N{=}10, 3 seeds), min-max normalized within each benchmark (pooling M{=}3 and M{=}5 values to [0, 100]). Strategies sorted by normalized score within each panel. (a) Infi-DA-Bench, M{=}3. (b) Infi-DA-Bench, M{=}5. (c) InsightBench, M{=}3. (d) InsightBench, M{=}5.](/article/2607.14386/_pdflatex/spgxlkzn.png "fig:")
Performance based ranking In Figure 6 we report the ranking of the design choices when the key metric is the performance.
Different gain patterns across question types From Figure 7, we observe different gains across task types. On Infi-DA-bench, test-time scaling benefits most medium and hard tasks, while on InsightBench it benefits most easy and hard tasks. On InsightBench, the classification of difficulty (easy, medium, hard) is based on the dataset that accompanies the tasks rather than the tasks themselves – this can explain the counter-intuitive U-shaped distribution.
11 Execution examples↩︎
