1 Introduction↩︎

Prompt optimization has emerged as an increasingly practical alternative to parameter-efficient fine-tuning for large language models (LLMs) [1]–[4]. Instead of updating model weights, recent frameworks such as TextGrad [3], GEPA [4] automatically search the prompt space to improve downstream task performance. These approaches have become particularly attractive for modern LLM systems because they are lightweight, modular, and directly compatible with LLM and agentic applications [5]–[7] and retrieval-augmented workflows [8], [9].

Despite their empirical success, prompt optimizers often exhibit unstable behavior across tasks and model backbones [10]–[12]. In practice, prompt revisions that improve one benchmark frequently fail to transfer to another, and optimizers that perform well on logical or sequential reasoning tasks may substantially degrade performance on mathematical or multi-hop reasoning benchmarks. Importantly, this pattern appears consistently in our experiments across multiple optimization frameworks and across diverse LLM backbones, including GPT-5.2, GPT-4o [13], Qwen3-32B [14], and Deepseek [15] models. Such instability raises a fundamental question: what kinds of prompt edits are modern optimizers actually learning to apply, and why do some edit patterns help certain task types while harming others?

Existing work has largely studied prompt optimization at the aggregate benchmark level [16]. Prior studies analyze optimizer success rates [17], optimization dynamics [3], [18], or embedding-level optimization directions [19], [20], but provide limited insight into the edit-level behaviors underlying optimizer failures. As a result, current evaluations often reveal whether optimization succeeds, but not which prompt modifications systematically contribute to improvement or degradation across different task settings. Yet understanding these edit-level behaviors is important for both diagnosis and optimizer design: two optimizers may achieve similar average gains while relying on fundamentally different editing strategies, and seemingly beneficial prompt modifications may interact differently with different reasoning tasks.

In this work, we investigate prompt optimizer behavior through an observational multi-view analysis of optimizer-induced prompt edits, as illustrated in Figure 1. Rather than treating optimized prompts as indivisible artifacts, we analyze consecutive prompt revisions within real optimization trajectories and examine how different edit families are associated with downstream performance changes across task groups.

To reduce dependence on any single representation of prompt edits, we probe optimizer behavior through three complementary views: (1) GPT-4o-annotated cognitive and instructional features, (2) deterministic surface-level text statistics, and (3) literal text-diff motifs extracted from consecutive prompt revisions. These complementary representations are subsequently integrated into a propensity-adjusted associational analysis framework to characterize heterogeneous optimizer behaviors across reasoning tasks.

Methodologically, our analysis adopts a causal inference-inspired observational framework. We use propensity-adjusted associational estimation [21] to reduce measured selection bias arising from differences in prior prompt states, while explicitly avoiding strong causal claims. Because the analysis involves many simultaneous feature-task comparisons, we distinguish between statistically robust findings that survive false-discovery-rate (FDR) correction and exploratory directional patterns that provide corroborative but non-confirmatory evidence. Throughout the paper, we therefore organize results using a unified two-tier evidence hierarchy. Tier 1 refers to associations surviving Benjamini–Hochberg false discovery rate correction. Tier 2 refers to directionally consistent corroborative patterns reproduced across multiple representations or frameworks but not surviving full multiple-testing correction.

Across around 20 thousand real world prompt optimization’s (prompt, question, correctness) tuples, we observe consistent edit-level heterogeneity across task groups. In particular, complexity-increasing and meta-instructional edits tend to be negatively associated with mathematical and multi-hop reasoning performance, whereas metacognitive and step-structured edits are positively associated with logical and sequential reasoning tasks. Several of these associations remain significant after false-discovery-rate correction, while others appear consistently across multiple independent prompt representations.

Our main contributions are summarized as follows:

• We present an observational edit-level analysis of prompt optimizer behavior across multiple optimization frameworks, LLM backbones, and reasoning task groups.

• We identify statistically robust heterogeneous associations between prompt optimizer-induced edits and downstream task performance.

• We provide evidence that prompt optimization failures are not purely random, but are systematically associated with interactions between edits and benchmark-specific characteristics, motivating future task-conditioned optimizer design.

2 Related Work↩︎

2.0.0.1 Automated prompt optimization.

Automated prompt optimization treats the prompt as a learnable variable to maximize task performance. Early work learns soft prompts or searches discrete prompt tokens/templates for few-shot adaptation and probing [19], [20], [22], [23]. Later black-box methods optimize natural-language instructions directly, including instruction generation and ranking [24], gradient-free edit search [25], reinforcement-learning-based prompt editing [26], [27], and textual-gradient revision [3], [17]. More recent systems use LLMs as optimizers or search operators, through natural-language proposal selection [18], evolutionary/self-referential mutation [4], [28], [29], and program-level prompt compilation [1], [2]. Despite their different optimization mechanisms, these methods converge to a common empirical failure mode in our setting: consistent performance degradation on math and multi-hop tasks. This convergence motivates a feature-level diagnosis of prompt changes rather than a method-specific analysis.

2.0.0.2 Prompt sensitivity and format effects.

A complementary line of work studies how non-semantic prompt properties affect LLM performance. [30] shows that few-shot calibration can substantially reduce format-induced bias, while [31] finds that demonstration order can change accuracy by up to 30 points. Related studies further show that in-context learning depends on label space, input distribution, and sequence format more than exact input-label mappings [32], that models can be insensitive to instruction semantics [33], that answer surface forms distort likelihood-based scoring [34], and that minor formatting choices can induce large performance swings [35]. Together, these findings demonstrate that prompt surface properties matter independently of semantic content. We extend this perspective from isolated format variation to optimizer trajectories, examining consecutive optimizer-induced edits and their task-type-conditioned performance effects. The Coin Flip paper [10] uses ANOVA-based analysis to identify run-level conditions under which optimization succeeds; we complement this with feature-level and edit-level causal analysis across task types.

2.0.0.3 Causal inference for NLP.

Causal inference has been increasingly used in NLP for debiasing, model behavior analysis, and text-based causal estimation [36]. Recent LLM-era work further studies LLMs both as objects of causal analysis and as tools for causal discovery or effect estimation [37], [38]. For example, recent studies use LLMs to estimate causal effects from unstructured text [39], build causal graphs and perform counterfactual inference from natural language [40], generate or match counterfactual texts for faithful model explanation [41], [42], and evaluate LLMs’ formal causal and counterfactual reasoning abilities [43]–[45]. The heterogeneous treatment effect literature [46] provides the statistical framework for conditional causal effects across subpopulations, while the double machine learning framework [47] enables nuisance-adjusted effect estimation with flexible machine learning models. We adapt inverse probability of treatment weighting (IPTW) [48] to prompt optimization traces, where the treatment is an optimizer-induced change in prompt features and the confounder is the prior prompt state. This framing extends causal-effect estimation tools to optimizer behavior analysis, while our evidence-tier design distinguishes FDR-controlled associations from exploratory corroboration.

2.0.0.4 Closest prior work.

CPO [49] applies double machine learning to whole-prompt embeddings to estimate optimization effects, whereas our focus is diagnostic and feature-level: we analyze how interpretable prompt edits are associated with gains or losses across task groups. While the CausalNLP survey [36] reviews text-as-treatment methods, we instantiate this perspective at the optimizer edit-motif level. Recent work on embedding-based causal inference [50] shows dense representations can reduce selection bias; in contrast, we emphasize interpretable surface features that support direct comparison across annotation-based and annotation-free views. The Coin Flip paper [10] studies optimizer success at the run level via ANOVA, whereas we perform edit-level IPTW-adjusted analysis to identify which specific edit types are associated with optimizer success or failure.

2.0.0.5 Cognitive load in instructional design.

Our annotation scheme draws on cognitive load theory, which distinguishes intrinsic load from task complexity, extraneous load from irrelevant or poorly structured information, and germane load from task-relevant cognitive processing. We use GPT-4o to annotate these constructs at the prompt level and validate them against deterministic text proxies (§3.2), following partial-validity assessment practices in computational social science.

3 Data and Annotation↩︎

3.1 Pairwise Prompt Comparison Dataset↩︎

We collected optimization logs from 3 frameworks, i.e., DSPy [1] (with MiPROv2 optimizer [2]), TextGrad [3], and GEPA [4]; 5 LLM backbones, i.e., GPT-5.2, GPT-4o, Qwen3-32B [14], Deepseek-v3, and Deepseek-R1 [15]; and 11 NLP benchmarks spanning 5 reasoning task categories:

Commonsense reasoning (including Commonsense QA [51], causal judgment, disambiguation QA datasets [52], with a mean base accuracy 0.70),
Mathematical reasoning (including GSM8K [53], MultiArith datasets [54], with a mean base accuracy 0.97),
Logical reasoning (including boolean expressions, coinflip [52] datasets, with a mean base accuracy 0.81),
Sequential reasoning (including last letters [52], ListOps [55] datasets, with a mean base accuracy 0.75),
Multi-hop reasoning (including Strategy QA [56], date understanding [52] datasets, with a mean base accuracy 0.75).

Especially, Math benchmarks exhibit particularly high initial performance,raising the possibility of a ceiling effect that we further examine in §4.

For each consecutive optimization step, we record the performance gain, defined as \(\Delta\mathrm{acc}= \mathrm{acc}(p_2) - \mathrm{acc}(p_1)\), on a fixed evaluation set. \(p_1\) and \(p_2\) are the former and updated prompts, and the prompt optimization applies the edit on \(p_1\) to produce \(p_2\). This process yields 2,095 pairwise comparisons from DSPy for the main analysis, and an additional 17,708 comparisons from TextGrad and GEPA for cross-framework replication.

3.2 Prompt Feature Annotation↩︎

Each prompt is annotated with 12 features derived from cognitive load theory and instructional design, each scored on a 1–10 scale by GPT-4o.

3.2.0.1 Construct validity.

We correlate these annotations against 13 text-derived features (§5.1) using Spearman rank correlation. 7 of 19 expected correlations are validated (\(|\rho| > 0.10\), \(p < 0.10\)): Metacognition \(\leftrightarrow\) meta-cognitive word density (\(\rho = +0.691\), \(p < 0.001\)); Intrinsic_load \(\leftrightarrow\) word count (\(\rho = +0.566\), \(p < 0.001\)); Structural_logic \(\leftrightarrow\) numbered list presence (\(\rho = +0.457\), \(p < 0.001\)). Annotation validity is partial but sufficient for coarse-grained analysis, and we address the remaining validity concern through annotation-free replication in §5.

4 Confounding-Adjusted Associational Analysis↩︎

4.1 Estimand and Assumptions↩︎

We estimate Inverse Probability of Treatment Weighting [21] (IPTW)-adjusted conditional mean gain differences (ACMGD) for optimizer-selected edit regimes. ACMGD is therefore interpreted as an IPTW-adjusted observational association rather than as an interventional causal estimand. For a given feature indicator \(T\), we set treatment \(= 1\) when the revision from \(p_1\) to \(p_2\) increases that feature, and \(0\) otherwise. Within each dataset/task group, the estimand compares expected performance gain between edit regimes with and without \(T\) , under three standard assumptions:

• Consistency: The observed outcome equals the potential outcome under the observed treatment.

• Positivity: Each unit has positive probability of both treatment values given covariates.

• Conditional exchangeability: Given pre-edit covariates (prompt \(p_1\) state: length, demo count, headroom, framework, backbone, dataset, optimization step), treatment assignment is conditionally independent of untreated potential outcomes.

4.1.0.1 Plausible unmeasured confounders.

Conditional exchangeability is unlikely to hold exactly in this setting. Several optimizer-internal variables remain unmeasured: (1) search trajectory history may confound the estimates, because optimizers tend to insert meta-instructions after observing prior failures, leaving “meta-instruction inserted” correlated with “prior prompt was struggling”; (2) selection pressure may arise because high-performing prompts at later optimization steps are more likely to receive additional complex features; (3) LLM-specific instruction following may matter because backbones vary in how they process instruction-level directives, and backbone is partially conditioned on rather than fully controlled. IPTW adjusts for measured pre-edit state but cannot eliminate these residual dependencies. Accordingly, the results should be interpreted as adjusted associational contrasts rather than causal effects.

4.1.0.2 Treatment bundling.

Prompt optimizers often modify multiple features simultaneously, so treatments are edit bundles rather than isolated feature changes. Although the propensity model conditions only on pre-edit covariates, residual bundle heterogeneity remains a limitation. Thus, the analysis provides observational associational evidence rather than causal identification.

4.2 Method: IPTW within Task Types↩︎

For each feature \(f\) and task group \(\tau\), we proceed as follows. First, we define treatment indicator as \(T = \mathbf{1}[f\text{-change} > 0]\). Second, we fit a logistic propensity model \(P(T{=}1 \mid X)\) using only pre-edit covariates. Third, we compute stabilized IPTW weights, \[w_i = \frac{T_i \,\bar{P}(T{=}1)}{e(X_i)} + \frac{(1-T_i)\,\bar{P}(T{=}0)}{1-e(X_i)},\] with weights capped at 10 to limit the influence of extreme propensity scores.

Fourth, we estimate the stabilized-IPTW weighted average conditional mean gain difference (ACMGD), \[\widehat{\mathrm{ACMGD}}_{f,\tau}^{\mathrm{SIPW}} = \frac{\sum_i w_i T_i Y_i}{\sum_i w_i T_i} - \frac{\sum_i w_i (1-T_i) Y_i}{\sum_i w_i (1-T_i)},\] where \(Y_i\) denotes the observed performance gain for revision pair \(i\).

Fifth, we calculate block-bootstrap standard errors using 500 resamples, with blocks defined by dataset \(\times\) backbone stratum to account for within-run and within-dataset correlation. All reported p-values and BH-corrected significance stars are based on these block-bootstrap uncertainties. BH-FDR correction is applied simultaneously across all 60 (feature \(\otimes\) task-group) tests.

The average confounding magnitude across all combinations is \(|\text{SIPW}- \text{naive}| = 0.018\), confirming that selection bias is present in naive estimates.

4.3 Results↩︎

We organize results into two tiers: Tier 1 (Confirmatory) contains BH-FDR corrected findings (\(q < 0.05\)), whereas Tier 2 (Exploratory) contains uncorrected directional patterns that corroborate Tier 1 but do not independently support inference.

4.3.0.1 Tier 1: Confirmatory findings (BH-FDR corrected).

Table 2 reports IPTW-adjusted ACMGD estimates by feature and dataset/task group. Across 60 simultaneous tests, 11 reach uncorrected significance (\(p < 0.05\)) and 2 survive Benjamini–Hochberg FDR correction (\(q < 0.05\)):

• Extraneous_load \(\otimes\) sequential reasoning: ACMGD \(= -0.060^{\bigstar}\)². Increases in extraneous load are significantly associated with performance degradation in the sequential task group (LOO-stable across 5/5 splits). Here, leave-one-out (LOO) stability refers to re-estimating the sign and direction of an association after iteratively excluding one dataset at a time, providing a robustness check against single-dataset-driven effects.

• Metacognition \(\otimes\) sequential reasoning: ACMGD \(= +0.062^{\bigstar}\). Metacognitive prompting is significantly associated with performance gains in the sequential task group (LOO-stable across 2/3 splits; one flip in disambiguation_qa).

Per-dataset robustness details for the Tier 1 findings are provided in Appendix 13.

3pt

4.3.0.2 Tier 2: Exploratory directional patterns.

Beyond the two BH-corrected effects, several uncorrected patterns appear directionally consistent and corroborate Tier 1: the Extraneous_load sign reversal (commonsense \(+0.032^*\) vs.sequential \(-0.060^{\bigstar}\)) is stable in 5/5 LOO splits. Demos (math \(+0.017\) vs.sequential \(-0.044^*\)) is directionally consistent with the surface feature analysis in §5. In addition, math task groups show predominantly negative point estimates for complexity-increasing features across 9/12 features, a pattern consistent with the motif analysis but uncorrected. These results are reported as exploratory hypothesis-generating observations only and do not independently support inferential claims.

4.4 Ceiling Effects↩︎

We examine whether dataset/task-group heterogeneity can be explained by ceiling effects alone. Headroom. defined as \(1 - \text{base\_accuracy}\), captures the remaining potential for improvement. The association between headroom and performance gain is weak and not statistically significant, with Spearman \(\rho(\text{headroom}, \Delta\mathrm{acc}) = 0.047\) (\(p = 0.053\)). In addition, the partial \(R^2\) of task group conditional on headroom is small but nonzero, \(R^2(\text{task group} \mid \text{headroom}) = 0.003\), indicating that task-group membership explains residual variation beyond headroom. These results suggest that ceiling effects are only a minor contributor and that dataset/task-group moderation is not reducible to baseline performance differences alone.

5 Annotation-Free Corroboration↩︎

To assess whether the BH-corrected findings in §4 depend on GPT-4o annotation choices, we re-examine the same prompt pairs using annotation-free text representations. These analyses produce Tier 2 corroborating: they align with Tier 1 findings but are not independently confirmatory.

5.1 Surface Complexity Features↩︎

We extract 14 deterministic text features from raw instruction text, including word count, demonstration count extracted from the JSON structure, step-word density based on terms such as “step,” “first,” and “then,” type-token ratio, compression ratio, sentence count, and related surface statistics. We then apply the same IPTW-adjusted associational analysis on DSPy pairs (\(N = 2{,}095\)).

3pt

All four major LLM-annotated directional patterns are reproduced by text-native features (Table 3). Specifically, n_demos algins with Demos (math positive, sequential negative); type_token_ratio algins with Clarity (math negative); step_words algins with Structural_logic (logical/sequential positive, multihop negative); word_count algins with Extraneous_load (sequential positive, multihop negative).

This convergence reduces the concern that the observed patterns are artifacts of GPT-4o annotation choices. If the sign reversals were driven solely by GPT-4o annotation choices, deterministic surface proxies would be unlikely to reproduce the same directional patterns. Although this comparison does not fully rule out annotation bias, becauseboth representations are derived from the same underlying prompts,it provides annotation-independent corroboration. Figure 3 visualizes this multi-view agreement for five key feature comparisons.

5.1.0.1 Cross-framework directional corroboration (TextGrad, GEPA, \(N = 17{,}708\)).

We apply the same surface-feature extraction to TextGrad and GEPA pairs as a directional corroboration check. Because TextGrad and GEPA uses a different optimization framework and prompt structure, this analysis is a partial replication of directional patterns only, rather than a full replication of the IPTW adjustment protocol. The results show consistent directional pattern: word_count (sequential \(+0.017^*\), multihop \(-0.014^*\)); sentence_count (commonsense \(+0.027^*\) vs.sequential \(-0.021^*\) vs.multihop \(-0.018^*\)); reasoning_words (sequential \(+0.039^*\) vs.commonsense \(-0.014^*\)). The n_demos feature is less interpretable for TextGrad (which does not use structured few-shot demos in the same format). We therefore interpret these results as cross-framework directional corroboration, instead of independent replication.

5.2 Edit Motif Effects↩︎

We compute text-diff for each consecutive prompt pair, extract inserted word spans of at least \(\geq 4\) tokens, and classify them into four pre-specified motif categories using regex patterns defined prior to analysis: chain_of_thought (“step by step”, “think through”, “reasoning”, etc.), meta_instruction (“make sure to”, “do not”, “ensure”, “remember to”, etc.), step_by_step (“step 1”, “first...then”, numbered instruction lists), and clarity_constraint (“concisely”, “briefly”, “simple”, “avoid”, etc.).

We then estimate IPTW-adjusted ACMGD for each motif \(\times\) dataset/task-group combination using pre-treatment covariates, and apply BH-FDR correction across all 15 motif \(\times\) group tests simultaneously.

4pt

5.2.0.1 Tier 1: BH-corrected motif findings.

Meta-instruction insertion is significantly associated with lower performance in the math dataset group (ACMGD \(= -0.103^{\bigstar}\), BH-corrected). Per-dataset caveat: This effect is partially concentrated in MultiArith (naive CATE \(= -0.277\), \(n = 64\)) vs.GSM8K (\(-0.019\), \(n = 14\)); the aggregated math estimate should be interpreted with this per-dataset heterogeneity in mind (see Appendix 12). Meta-instruction is also directionally negative in multihop (\(-0.044^*\), uncorrected, Tier 2).

Clarity-constraint insertion is significantly associated with lower performance in the logical dataset group (ACMGD \(= -0.083^{\bigstar}\), BH-corrected). Instructions to “be concise/brief/simple” may suppress step-by-step reasoning on boolean and coin-flip tasks.

5.2.0.2 Tier 2: Exploratory directional pattern.

Chain-of-thought shows a preserved directional pattern across representations: negative in math groups (motif: \(-0.048\); surface step_words: \(-0.044\); LLM Structural_logic negative), positive in sequential/last_letters (motif: \(+0.045^*\); surface step_words: \(+0.075^*\); LLM Structural_logic positive). This pattern is consistent but uncorrected across all representations and should be treated as Tier 2 (exploratory corroboration). Dataset caveat: “sequential” is a single-dataset group (last_letters); this is dataset-level evidence, not multi-dataset task-type evidence.

5.2.0.3 Motif validity.

We manually audited 50 classified-positive spans per key motif to assess precision. For meta_instruction, representative inserted spans include:

• “Ensure that both outputs clearly explain how the final solution was reached based on the arithmetic operations involved.”

• “Make sure to articulate the reasoning process clearly, even if it doesn’t require detailed breakdowns.”

These patterns are consistently genuine meta-instructional directives added by the optimizer. For chain_of_thought, the most common false positive is the JSON key "reasoning": "..." appearing in structured few-shot demo fields, i.e., the regex matches the word “reasoning” but the pattern appears in a non-instruction field not visible to the model as a directive. This false positive inflates the treatment indicator for chain_of_thought without representing an actual user-visible instruction change. We estimate meta_instruction precision at \({\gtrsim}90\%\) based on this audit (50 random positive spans, stratified by dataset; labeled independently by one author; ambiguous cases counted as false positives). Chain_of_thought precision is lower and the pattern should be treated as noisier. Motif precision was not formally estimated at scale and the audit involved one labeler; results remain preliminary.

6 Conclusion↩︎

We present a multi-view observational analysis of edit-level heterogeneity in prompt optimizer behavior. Using three complementary representations of 2,095 pairwise prompt comparisons, i.e., GPT-4o-annotated cognitive-load features, deterministic surface text statistics, and literal text-diff motifs, we find four BH-FDR-corrected associations: Extraneous_load \(\otimes\) sequential (\(-0.060^{\bigstar}\)), Metacognition \(\otimes\) sequential (\(+0.062^{\bigstar}\)), meta-instruction \(\otimes\) math (\(-0.103^{\bigstar}\)), and clarity-constraint \(\otimes\) logical (\(-0.083^{\bigstar}\)). All four involve at least two benchmark datasets per task group. Beyond these confirmatory findings, directionally consistent corroborating evidence appears in surface text features and in 17,708 TextGrad/GEPA pairs, and 83% of headline sign patterns survive LOO exclusion.

These patterns suggest that edit-family-level analysis, especially when conditioned on benchmark-specific task demands, offers a more informative diagnosis than aggregate success or failure metrics alone. The pre-optimization question “does my target benchmark resemble groups where meta-instruction and complexity-increasing edits are negatively associated with gains?” has practical diagnostic value even under the observational limitations of this study, and points toward edit-conditioned optimizer design as a fruitful direction for interventional follow-up.

6.0.0.1 Implications.

For practitioners, the ACMGD tables in this paper serve as a preliminary diagnostic: before running an expensive optimizer, consider whether the target task/dataset resembles groups (math-like, multihop-like) where meta-instruction and complexity-increasing edits show negative associations with gains. This does not guarantee failure, but the BH-corrected associations are consistent across frameworks, representations, and LOO folds. The consistent negative association between meta-instruction insertion and math-type benchmarks suggests that future prompt optimizers may benefit from task-conditioned edit control, particularly by discouraging unnecessary complexity-increasing or meta-instructional edits during optimization.

6.0.0.2 Reproducibility.

All analysis code and pairwise comparison dataset will be released upon publication. The TextGrad processing pipeline is compatible with TextGrad public official repository.

Limitations↩︎

Our analysis is observational rather than interventional. Although IPTW reduces measured selection bias using pre-edit prompt states, residual confounding may remain due to optimizer trajectory history, backbone-specific instruction following, and bundled prompt edits that modify multiple features simultaneously. Accordingly, the reported ACMGD estimates should be interpreted as adjusted associational contrasts rather than causal effects.

In addition, several findings remain benchmark-sensitive. Some task-group effects are partially concentrated in specific datasets (e.g., MultiArith within math), and cross-framework corroboration is directional rather than a fully controlled replication because different optimization frameworks use different prompt structures. Finally, motif extraction relies on pre-specified regex patterns and therefore introduces unavoidable labeling noise, particularly for chain-of-thought-related edits.

7 Statistical Significance (Uncorrected p-values, BH-FDR Correction)↩︎

Our analysis involves multiple simultaneous hypothesis tests across prompt features, edit motifs, and dataset/task groups. In such settings, interpreting raw statistical significance without correction can substantially inflate the probability of false discoveries. This appendix clarifies the distinction between uncorrected p-values and Benjamini–Hochberg false discovery rate (BH-FDR) corrected results used throughout the paper.

7.1 Uncorrected p-values↩︎

For each feature–task-group combination, we estimate an IPTW-adjusted associational effect (ACMGD) and compute a corresponding hypothesis test: \[H_0 : \mathrm{ACMGD} = 0,\] where the null hypothesis assumes no adjusted association between the optimizer-induced edit regime and performance gain.

The reported uncorrected p-value measures the probability of observing an effect at least as extreme as the estimated one under the null hypothesis: \[p = \Pr\left(|T| \ge |T_{\mathrm{obs}}| \mid H_0 \right),\] where \(T\) denotes the test statistic computed using block-bootstrap uncertainty estimates.

Throughout the paper, an uncorrected significance marker \[{}^{*} : p < 0.05\] indicates that the corresponding association is statistically significant when considered as an isolated test.

However, because our analysis evaluates many hypotheses simultaneously (60 feature \(\times\) task-group tests in Table 2 and 15 motif \(\times\) group tests in Table 4), interpreting uncorrected p-values alone would inflate the probability of false positives due to the multiple comparisons problem.

7.2 Multiple Comparisons Problem↩︎

Suppose \(m\) independent null hypotheses are tested at significance level \(\alpha = 0.05\). Even if all null hypotheses are true, the expected number of false positives is: \[m \times \alpha.\]

For example, with \(m=60\) tests, approximately \[60 \times 0.05 = 3\] false discoveries are expected purely by chance under naive thresholding.

Consequently, some apparently significant associations may arise from stochastic variation rather than systematic optimizer behavior.

7.3 Benjamini–Hochberg False Discovery Rate Correction↩︎

To mitigate inflated false positives, we apply the Benjamini–Hochberg (BH) false discovery rate correction [57] across each family of simultaneous tests.

Unlike family-wise error rate procedures (e.g., Bonferroni correction), which control the probability of any false positive, BH-FDR controls the expected proportion of false discoveries among all rejected hypotheses: \[\mathrm{FDR} = \mathbb{E} \left[ \frac{ \#\text{ false discoveries} }{ \#\text{ total discoveries} } \right].\]

This criterion is substantially less conservative and is therefore widely used in exploratory high-dimensional analyses where moderate discovery power is desirable.

7.4 BH Procedure↩︎

Given \(m\) hypothesis tests with ordered p-values \[p_{(1)} \le p_{(2)} \le \cdots \le p_{(m)},\] the BH procedure compares each ordered p-value against the adaptive threshold: \[p_{(i)} \le \frac{i}{m}\alpha,\] where:

• \(i\) is the rank of the ordered p-value,

• \(m\) is the total number of simultaneous tests,

• \(\alpha\) is the target false discovery rate (0.05 in this paper).

The largest index satisfying the inequality is identified, and all hypotheses with smaller-ranked p-values are declared significant after correction.

The resulting BH-adjusted significance level is reported as a q-value throughout the paper.

7.5 Interpretation in This Work↩︎

We distinguish between two evidence tiers:

7.5.0.1 Tier 1 (Confirmatory).

Associations surviving BH-FDR correction: \[q < 0.05\] are treated as statistically robust findings. These effects remain significant even after accounting for the full multiple-testing burden.

7.5.0.2 Tier 2 (Exploratory).

Associations satisfying: \[p < 0.05\] but not surviving BH correction are reported as directional or corroborative patterns only. These findings may still reflect meaningful optimizer behavior, but they carry a substantially higher false discovery risk and therefore do not independently support strong inferential claims.

Our analysis evaluates a large number of simultaneous hypotheses across prompt features, edit motifs, task groups, and representation spaces. In such exploratory settings, relying solely on uncorrected p-values would substantially increase the probability of reporting false-positive associations that arise purely from random variation. BH-FDR correction provides a principled compromise between statistical rigor and discovery sensitivity: rather than controlling the probability of any false positive, it controls the expected proportion of false discoveries among all reported significant findings. This is particularly suitable for our setting because the goal of the analysis is not to establish a small number of tightly controlled confirmatory effects, but to characterize heterogeneous optimizer behaviors across multiple complementary views. Accordingly, we treat BH-corrected findings (\(q < 0.05\)) as confirmatory evidence, while uncorrected findings (\(p < 0.05\) only) are explicitly framed as exploratory and hypothesis-generating observations. This distinction reduces the risk of over-interpreting noisy associations while still preserving sufficient statistical power to identify potentially meaningful optimizer behavior patterns.

8 Conditional Independence Structure (Exploratory)↩︎

We pool prompt observations per task type (\(N\) ranging from 30 to 92) and apply:

• PC algorithm (constraint-based, Fisher-Z test, \(\alpha = 0.05\)) on feature vectors

• DirectLiNGAM (non-Gaussian directed acyclic graph estimation)

8.0.0.1 Results.

Average cross-task Jaccard similarity \(= 0.249\)—task types share only \(\sim\)​25% of their inferred graph edges, confirming structural heterogeneity. Pairwise similarity ranges from 0.053 (commonsense vs.math) to 0.667 (logical vs.sequential). Math tasks have the most distinctive structure. These results are reported as exploratory analysis of conditional independence patterns; no mechanism claims are made.

9 Causal Receptivity Score (Descriptive)↩︎

We construct a per-task-type receptivity score as the weighted sum of BH-significant and directionally consistent CATE estimates (positive sign = optimizer-compatible). Spearman \(\rho = -0.866\) between receptivity and optimizer success rate at the task-type level (\(n = 5\), \(p = 0.058\)). This correlation is descriptive and based on five task-type aggregates; it does not constitute a validated predictive diagnostic. LOO-CV classifier AUC \(= 0.260\) (below chance) confirms that the receptivity score does not generalize to held-out task types at current sample sizes. We report it as a descriptive summary only.

10 SBERT Edit-Direction Sensitivity Analysis↩︎

We compute SBERT [58] embedding differences \(\Delta\mathbf{e} = \mathbf{e}(p_2) - \mathbf{e}(p_1)\) for each prompt pair, then project onto the first 20 principal components via PCA. For each component, we compute SIPW-CATE stratified by task type.

Results: 10/20 PCA components show sign reversals (spread \(> 0.05\)) across at least two task types. PC7 shows the clearest pattern: commonsense \(+0.061^*\), math \(-0.098^*\), sequential \(+0.084^*\) (spread \(= 0.181\)). This embedding-level evidence is consistent with the annotation-free robustness findings in §5 and provides an additional representation-independent corroboration.

11 Full LOO Stability Table↩︎

Table 5 reports all 18 LOO splits covering six headline sign reversals. A split is “stable” (✔) if the sign reversal direction matches the full-data baseline.

4pt

12 Per-Dataset Motif Effect Breakdown↩︎

Table 6 reports naive (unweighted) CATE for chain_of_thought and meta_instruction insertion by dataset.

4pt

13 Per-Dataset Robustness for Tier 1 Findings↩︎

Table 7 reports per-dataset estimates for all four BH-corrected Tier 1 associations. For each finding, we report the IPTW-adjusted ACMGD, treated/control counts, and sign consistency across datasets within the task group.

4pt

14 Prompt Feature Extraction↩︎

Below we provide the prompt template used in Prompt Feature Extraction.

15 Discussion↩︎

15.1 Evidence Hierarchy↩︎

We use a three-tier structure to prevent misreading:

Tier 1. Four BH-corrected associations across two analysis families. From the GPT-4o feature analysis: (a) Extraneous_load/sequential: ACMGD =\(-0.060^{\bigstar}\), 100% LOO-stable; (b) Metacognition/sequential: ACMGD =\(+0.062^{\bigstar}\), 67% LOO-stable. From the motif analysis: (c) meta_instruction \(\otimes\) math: ACMGD =\(-0.103^{\bigstar}\), partially concentrated in MultiArith; (d) clarity_constraint \(\otimes\) logical: ACMGD =\(-0.083^{\bigstar}\). The sequential effects involve last_letters only; the math effect involves two datasets with heterogeneous within-group estimates. These are statistical associations, not causal effects.

Tier 2. Corroborated but exploratory. Math-vs-sequential directional reversals in surface features (n_demos, step_words); chain-of-thought direction pattern consistent across representations; TextGrad directional corroboration. These are consistent with Tier 1 but do not independently support inference.

Optimizer-design suggestions (task-conditioned edit priors, complexity penalties, validation gates); causal mechanisms for the meta-instruction effect; generalization to unseen task types. Grounded in the observed associations but require controlled validation before implementation.

15.2 Observational Edit-Effect Heterogeneity↩︎

Across three representations of prompt edits, i.e., annotated cognitive-load features, deterministic surface statistics, and literal edit motifs, a consistent directional pattern emerges: optimizer-induced edits that add meta-instruction language (“make sure to”, “do not”) and structural scaffolding (chain-of-thought framing) are negatively associated with math performance (BH-corrected for meta-instruction), while logical and sequential task groups tend to benefit from step-by-step and structural features.

15.2.0.1 What this paper establishes.

The associations between edit types and dataset/task-group performance are directionally consistent across three annotation-independent representations, survive IPTW adjustment, and are robust to leave-one-dataset-out exclusion for the BH-corrected effects. This consistency is compatible with an optimizer-task mismatch account, though residual confounding cannot be ruled out (see §4.1 for specific unmeasured confounders).

15.2.0.2 What this paper does not establish.

Whether these associations are causal. Conditional exchangeability is not fully defensible; treatments are bundles; interventional validation at scale was not possible. These results characterize optimizer behavior patterns rather than definitively explaining failure.

15.3 The Meta-Instruction Effect in Math↩︎

The strongest FDR-corrected finding is the negative association between meta-instruction insertions and math dataset/task-group performance (ACMGD \(= -0.103^{\bigstar}\), partially concentrated in MultiArith, see Appendix 12). This is notable because meta-instructions (“make sure to show your work”, “do not skip steps”, “you must include a final answer”) might seem helpful for math. One hypothesis: models fine-tuned on math instruction-following may have already internalized these guidelines; redundant meta-instruction may increase prompt complexity without benefit. A competing explanation consistent with the unmeasured confounders in §4.1: optimizers may tend to insert meta-instructions specifically when prior prompts are struggling, so the association partly reflects selection into struggling trajectories rather than a direct effect. Targeted interventional validation would be required to distinguish these accounts.

The chain-of-thought directional reversal (negative for math groups, positive for sequential \(\otimes\) last_letters) is consistent across representations (Tier 2) but does not survive FDR correction. The difference from the well-established finding that CoT helps math [59] likely reflects the distinction between optimizer-generated CoT meta-instructions in system prompts (vague “reason step by step” phrases) vs.user-designed structured CoT with explicit answer markers.

15.3.0.1 Sanity check: Structured CoT intervention.

To verify that the chain-of-thought motif finding is not simply an artifact of optimizer trajectories containing CoT-style content without actual instruction-level changes, we ran a minimal controlled sanity check using Qwen2-VL-7B-Instruct on 25 GSM8K and 25 last_letters questions: base prompt vs.base + structured CoT with explicit “#### answer” markers. Results: GSM8K base \(= 28\%\) \(\to\) CoT \(= 60\%\) (\(\Delta = +32\%\), McNemar \(\chi^2 = 4.90\), \(p = 0.027\)); last_letters base \(= 0\%\) \(\to\) CoT \(= 4\%\) (\(\Delta = +4\%\), NS). Explicit structured CoT helps math, consistent with [59]—confirming that our chain-of-thought motif estimate (\(-0.048\) for math groups, uncorrected) does not reflect a general CoT-hurts-math phenomenon. Caution: This sanity check does not validate or invalidate the observational motif estimate; it shows only that the motif labels do not correspond to clean user-designed structured interventions. The experiment used one model and 25 questions per task; generalization is not established.

15.4 Benchmark Sensitivity↩︎

Table 8 summarizes leave-one-dataset-out stability for six headline sign reversals across 18 LOO splits. 15/18 (83%) preserve the sign reversal from the full-data baseline, exceeding our 80% success criterion. The two BH-corrected effects (Extraneous_load and Demos) are 100% stable. The word_count multihop reversal appears driven by date_understanding and should be treated with caution.

4pt

15.5 Implications↩︎

15.5.0.1 For practitioners (Tier 3 — hypothesis-generating).

The ACMGD tables in this paper serve as a preliminary diagnostic: before running an expensive optimizer, consider whether the target task/dataset resembles groups (math-like, multihop-like) where meta-instruction and complexity-increasing edits show negative associations with gains. This does not guarantee failure, but the BH-corrected associations are consistent across frameworks, representations, and LOO folds. These suggestions are Tier 3 hypotheses derived from observational associations; implementation without controlled validation would be premature.

15.5.0.2 Concrete optimizer-design hypotheses.

(1) Task-conditioned edit priors: Optimizers could down-weight meta-instruction insertion for arithmetic or multi-step reasoning tasks, which are implementable as a regex-based edit filter or a classifier over proposed edits. (2) Complexity-penalized search: The BH-corrected meta_instruction \(\otimes\) math association (\(-0.103^{\bigstar}\)) suggests complexity-penalizing objectives may be worth piloting for math-type benchmarks. (3) Validation gates: Before accepting an optimizer-proposed edit, evaluate on a held-out subset classified by task group, which is motivated by but not validated in this paper.

These are mechanism-level hypotheses, not established causal rules.

References↩︎