1 Introduction↩︎

Large language models (LLMs) have demonstrated impressive capabilities across a wide range of reasoning tasks [1], [2]. Recent work suggests they can perform causal reasoning with high accuracy, surpassing existing methods on standard benchmarks [3]. These findings have fueled optimism about using LLMs as assistants for causal inference and policy analysis.

However, most existing evaluations rely on synthetic causal graphs [4], commonsense scenarios, or abstract logical problems [5]. The real-world domain of policy evaluation—where economists and social scientists use quasi-experimental methods to estimate causal effects—presents a fundamentally different challenge. Policy effects are messy, contested, and, critically, sometimes counter-intuitive: the data show the opposite of what most people would expect.

We introduce a benchmark of 40 empirical policy evaluation cases drawn from peer-reviewed economics and social science research. Each case is classified along an intuitiveness dimension:

• Obvious: The empirical result aligns with common prior expectations (e.g., Medicaid expansion increases healthcare utilization).

• Ambiguous: Reasonable arguments exist for multiple directions (e.g., welfare reform may increase or decrease employment depending on the mechanism).

• Counter-intuitive: The empirical result contradicts naïve expectations (e.g., introducing fines for late daycare pickup increases lateness).

This classification is inspired by dual-process theory from cognitive psychology [6]–[8]: obvious cases can be solved by fast, intuitive “System 1” processing, while counter-intuitive cases require slow, deliberative “System 2” reasoning that overrides the initial heuristic response.

We evaluate four frontier LLMs—GPT-5.2, Claude Sonnet 4.5, Claude Opus 4.6, and GPT-4.1—across five prompting strategies (naïve, expert persona, chain-of-thought, structured reasoning, and adversarial framing) with 10 repetitions per condition, yielding 8000 experimental trials. We analyze the results using mixed-effects logistic regression to properly account for the nested structure of the data.

Our key findings are: (1) a CoT paradox—chain-of-thought prompting substantially improves accuracy on obvious cases (main-effect OR = 17.3) but this benefit is attenuated on counter-intuitive ones (interaction OR = 0.278, \(p\) \(<\)​0.001); (2) intuitiveness as the dominant factor, with case-level random intercepts capturing the majority of variance (ICC = 0.671); and (3) a knowledge–reasoning dissociation, where citation-based familiarity is unrelated to accuracy (\(p =\) 0.84), suggesting models possess relevant knowledge but fail to reason with it when findings contradict intuition. Overall accuracy of 79.6% masks critical failures precisely where counterfactual reasoning matters most.

2 Related Work↩︎

2.0.0.1 Causal Reasoning in LLMs.

[3] report 92% accuracy on counterfactual reasoning but rely on commonsense and synthetic scenarios. [4] propose CLadder using synthetic causal graphs with oracle-derived ground truth. [9] argue LLMs are “causal parrots” reciting memorized knowledge. Our work bridges these perspectives: LLMs can leverage causal knowledge, but fail when doing so requires overriding intuitive priors—a distinction invisible in synthetic benchmarks.

2.0.0.2 Counterfactual Reasoning Benchmarks.

[5] propose CLOMO for generative counterfactual modification; [10] use VAE-based counterfactual reasoning in narratives; [11] develop Tree-of-Counterfactual prompting. Unlike these works, we focus on evaluative counterfactual reasoning in real-world policy evaluation, where ground truth is established by peer-reviewed empirical research.

2.0.0.3 Cognitive Parallels in LLM Behavior.

[12] classify LLMs’ response styles as “intuitive” or “dependent”; their “intuitive” refers to behavioral style, whereas ours refers to cognitive accessibility of task content. [13] find that LLM accuracy does not necessarily reflect reasoning validity—a conclusion our findings reinforce. [14] show that LLMs follow the Yerkes-Dodson law under stress-inducing prompts.

2.0.0.4 Chain-of-Thought Reasoning.

CoT prompting [15], [16] is widely assumed to improve reasoning, but [17] show that reasoning chains are only as strong as their weakest step. Our CoT paradox extends this: the weakest step may be the very first one—the intuitive prior that anchors subsequent reasoning. [18] show that test-time compute scaling depends critically on prompt difficulty, consistent with our finding.

3 Benchmark Design↩︎

3.1 Case Selection↩︎

We curate 40 empirical policy evaluation cases from economics/social science, each satisfying three criteria: (1) at least one peer-reviewed publication establishing the causal effect via a credible identification strategy (difference-in-differences, regression discontinuity, instrumental variables, or randomized controlled trial); (2) an unambiguous directional finding (increase, decrease, or no significant change); and (3) sufficient prominence that the study is likely present in LLM training data. Cases span 11 countries, 10 policy domains, and publication years from 1990 to 2020. Table 1 summarizes the distribution.

3.2 Intuitiveness Classification↩︎

Each case is classified as obvious, ambiguous, or counter-intuitive based on whether the established empirical finding aligns with, is unclear relative to, or contradicts the prior that an informed layperson would hold. This classification was performed by the author and validated by three independent human annotators (Fleiss’ \(\kappa\) = 0.32, indicating fair agreement; see Appendix 12). The categorical classification is further supported by its convergent validity with the experimental results: CoT prompting yields large gains on obvious cases (+26.7 pp) but substantially attenuated gains on counter-intuitive cases (+13.8 pp), consistent with the dual-process interpretation (§5.4).

Examples of counter-intuitive cases include:

• Introducing fines for late daycare pickup increases lateness [19]: monetary penalties replace moral obligation with a market transaction.

• “Right-to-carry” concealed handgun laws increase violent crime [20]: contrary to deterrence theory.

• Violent movie releases decrease violent crime [21]: voluntary incapacitation outweighs any media-effects priming.

3.3 Test Types↩︎

We employ two test types. One-sided tests present a directional hypothesis and ask the model to answer YES or NO (e.g., “Did the minimum wage increase decrease employment? YES or NO”). Two-sided tests ask the model to predict the direction of the counterfactual (INCREASE / DECREASE / NO SIGNIFICANT CHANGE). Two-sided tests are strictly harder, as the model must both identify whether an effect exists and determine its sign.

3.4 Prompt Design↩︎

We design five prompting strategies to probe different aspects of reasoning:

1. P1 – Naïve: A direct question with no guidance, capturing the model’s default “System 1” response.

2. P2 – Expert Persona: The model is told it is an “experienced econometrician” to test whether domain framing improves accuracy.

3. P3 – Chain-of-Thought (CoT): The model is asked to “think step by step,” listing causal mechanisms and confounders before answering.

4. P4 – Structured Reasoning: A more detailed template requiring explicit enumeration of causal channels, confounders, and finally a prediction.

5. P5 – Adversarial Framing: The prompt introduces a fictitious “analyst consensus” suggesting the policy had no effect, testing epistemic robustness against anchoring bias [22].

4 Experimental Setup↩︎

4.1 Models↩︎

We evaluate four frontier LLMs accessed via API:

• GPT-5.2 (OpenAI)

• Claude Sonnet 4.5 (Anthropic)

• Claude Opus 4.6 (Anthropic)

• GPT-4.1 (OpenAI)

All models are accessed via default inference parameters, held constant across all experimental conditions. To prevent data contamination during evaluation, we sandbox each call: file-system tools are disabled, the working directory is set to a temporary folder, and custom instructions are suppressed.

4.2 Procedure↩︎

Each of the \(40{} \times 4 \times 5 \times 10 = 8000{}\) trials is executed independently. Responses are parsed using regular expressions to extract the final answer (INCREASE / DECREASE / NO SIGNIFICANT CHANGE / YES / NO), with a lightweight LLM fallback parser for the \(<1\%\) of responses where regex extraction fails.

4.3 Statistical Analysis↩︎

We analyze the binary outcome (correct/incorrect) using mixed-effects logistic regression [23]:

\[\label{eq:model} \text{logit}(P(\text{correct}_{ij})) = \mathbf{X}_{ij}\boldsymbol{\beta} + u_j\tag{1}\]

where \(i\) indexes observations and \(j\) indexes cases. The fixed effects \(\boldsymbol{\beta}\) include model identity, prompt strategy, intuitiveness category, and their interactions (prompt \(\times\) intuitiveness). The random intercept \(u_j \sim \mathcal{N}(0, \sigma_u^2)\) captures case-level heterogeneity. This specification follows recommendations in [24] for confirmatory hypothesis testing. We report odds ratios (OR) with 95% confidence intervals. All analyses are implemented in R using lme4 [23].

5 Results↩︎

5.1 Overall Performance↩︎

Across all 8000 trials, overall accuracy is 79.6% (6369/8000). However, this aggregate masks substantial heterogeneity. One-sided tests yield 98.7% accuracy (near ceiling), while two-sided tests yield only 69.3%. Within two-sided tests, accuracy is uniformly low across intuitiveness categories (obvious 68.2%, counter-intuitive 69%, ambiguous 72.9%), suggesting that the challenge lies not in raw accuracy differences but in how prompting strategies interact with intuitiveness (§5.4).

5.2 Model Comparison↩︎

Table 2 shows that GPT-5.2 substantially outperforms all other models, particularly on two-sided counter-intuitive cases, where GPT-4.1 falls to 43.2%—only 10 percentage points above the 33% chance baseline. Notably, Claude Sonnet 4.5 outperforms Claude Opus 4.6 on counter-intuitive cases (73.5% vs.%) despite being a smaller model, suggesting that model scale alone does not determine susceptibility to intuitive priors (see also Figure 3 in Appendix 11).

5.3 Prompt Strategy Comparison↩︎

Structured reasoning (P4) and CoT (P3) dominate overall, but these aggregates conceal the critical interaction with intuitiveness (§5.4).

5.4 The CoT Paradox↩︎

The central finding of this paper emerges from the interaction between prompting strategy and intuitiveness. Table [tab:interaction] presents the key interaction terms from our mixed-effects model.

4pt

The CoT \(\times\) counter-intuitive interaction OR of 0.278 means that CoT’s large benefit on obvious cases (main-effect OR = 17.3) is substantially attenuated on counter-intuitive cases. Structured reasoning shows a similar pattern (OR = 0.244). CoT still helps on counter-intuitive cases (effective OR \(\approx\) 17.3 \(\times\) 0.278 \(\approx\) 4.8), but the attenuation is substantial (\(p\) \(<\)​0.001).¹

The adversarial prompt (P5) shows a significant positive interaction with counter-intuitive cases (OR = 2.83, \(p\) \(<\)​0.001). However, this result is confounded by test type: 13 of 21 counter-intuitive cases are one-sided with ground truth NO, and P5’s “no effect” anchor coincidentally aligns with the correct answer. On two-sided counter-intuitive cases—where the ground truth is INCREASE or DECREASE—P5 reduces accuracy from 70% (P1) to 41.6%, the worst of any prompt. The apparent positive interaction thus reflects accidental alignment between the anchor and the correct answer, not genuine debiasing.

Figure 1 illustrates the CoT paradox: the accuracy gain from chain-of-thought prompting diminishes substantially as intuitiveness decreases.

5.5 Self-Consistency Analysis↩︎

Majority-vote accuracy [25] across ten repetitions improves marginally from 79.6% to 80.5% overall, but provides minimal improvement on two-sided counter-intuitive cases (69% \(\to\) 71.9%). Errors are systematic, not stochastic: models consistently give the same wrong answer, reflecting stable but incorrect causal beliefs.

6 Analysis↩︎

6.1 Error Patterns↩︎

Among incorrect two-sided responses where the ground truth is INCREASE, 72% of errors predict DECREASE (directional reversal) rather than NO SIGNIFICANT CHANGE—consistent with models applying an intuitive causal model that predicts the opposite direction. The expert persona (P2) shows a distinctive pattern: 65% of its errors are NO SIGNIFICANT CHANGE, suggesting that “econometrician” framing induces excessive hedging.

6.2 Case Difficulty↩︎

The random intercepts (\(u_j\)) reveal substantial case-level heterogeneity. The hardest cases are Case 23 (daycare fines; 39%) and Case 36 (right-to-carry laws; 38%), both counter-intuitive and two-sided.

Figure 2 reveals the full pattern: while most cases are solved reliably by at least one model, the hardest counter-intuitive cases produce systematic failures across all models.

6.3 Familiarity as a Non-Factor↩︎

Adding log(citations) as a moderator yields a non-significant effect (OR = 1.08, \(p =\) 0.84). Models are not simply “looking up” well-known results: Case 23 (daycare late pickup fines), based on one of the most widely cited behavioral economics experiments [19], achieves only 39% accuracy. This speaks to the “causal parrots” debate [9]: if LLMs merely recited memorized facts, accuracy should correlate with citation frequency. The null result suggests models have absorbed causal knowledge broadly, but deploying it correctly requires a reasoning step that fails when intuitions conflict.

7 Discussion↩︎

7.1 Slow Talking, Not Slow Thinking↩︎

Our CoT paradox challenges the assumption that CoT elicits deeper reasoning. On counter-intuitive cases, the interaction OR of 0.278 attenuates CoT’s main-effect log-odds benefit by 45% (from \(\ln(17.3{}) = 2.85{}\) to an effective \(\ln(4.8{}) = 1.57{}\)). CoT still helps—the effective OR of \(\approx\)​4.8 remains significant—but the attenuation is substantial and highly significant (\(p\) \(<\)​0.001). The reasoning chain’s first step appears anchored by the same intuitive prior that produces errors, and subsequent steps partially—but not fully—overcome this anchor. This is consistent with [17], who show that reasoning chains are only as strong as their weakest step.

From the dual-process perspective [6], effective System 2 reasoning requires inhibiting System 1. Our results suggest that LLMs’ CoT achieves partial inhibition—enough to help on many cases, but insufficient when intuitive priors are strong. The result is “slow talking” that only sometimes constitutes genuine “slow thinking.”

7.2 Implications for Policy Applications↩︎

The overall accuracy of 79.6% is inflated by the one-sided test format (98.7% accuracy), where the binary YES/NO choice simplifies the task. On two-sided cases—where careful directional prediction matters most—accuracy drops to 69.3%, and specific counter-intuitive cases fall as low as 38% (right-to-carry) and 39% (daycare fines). Policymakers relying on LLM-assisted analysis may receive confident but incorrect guidance precisely where it is most consequential.

8 Conclusion↩︎

We introduce a benchmark for evaluating LLM counterfactual reasoning in policy evaluation, organized around an intuitiveness dimension inspired by dual-process theory. The CoT paradox reveals that chain-of-thought prompting helps primarily when the correct answer aligns with intuition; its benefit is substantially attenuated on counter-intuitive cases. Combined with the failure of self-consistency and the null effect of familiarity, this points to a fundamental limitation: current LLMs can retrieve causal knowledge and partially reason with it, but their deliberative processes are insufficient to fully override strong intuitive priors. Improving counterfactual reasoning may require innovations that enable more robust inhibitory control—the ability to recognize when one’s first instinct is wrong and to sustain reasoning past the initial anchor.

Limitations↩︎

8.0.0.1 Benchmark Scale.

Our benchmark comprises 40 cases—sufficient for mixed-effects modeling with case-level random intercepts (ICC = 0.671), but limited in breadth relative to the space of possible policy evaluations. While we cover 10 policy domains across 11 countries, some domains (e.g., crime, education) are more heavily represented than others. Expanding the benchmark would improve generalizability but is constrained by the requirement for established causal evidence.

8.0.0.2 Intuitiveness Classification.

Our primary intuitiveness classification was performed by the author and validated against human annotator accuracy (3 annotators, Fleiss’ \(\kappa\) = 0.32, indicating fair agreement). The continuous human accuracy measure (proportion of annotators answering correctly) yields qualitatively identical results in robustness analyses. However, “intuitiveness” is inherently subjective and may vary by population—what is counter-intuitive to a layperson may be obvious to a domain expert.

8.0.0.3 Temporal Validity.

We evaluate models via API at a single point in time (May 2026). Model performance may change with updates, RLHF iterations, or expanded training data. Our results represent a snapshot of current capabilities rather than a permanent characterization.

8.0.0.4 Prompt Coverage.

We test five prompting strategies, but the space of possible prompts is infinite. Techniques such as self-debate, reflection, or multi-agent discussion might yield different patterns. Our prompts are designed to probe specific theoretical mechanisms (System 1 vs. System 2 activation) rather than to exhaustively optimize accuracy.

8.0.0.5 Ground Truth.

While all cases are grounded in peer-reviewed research, some findings remain debated within their fields (e.g., the effect of right-to-carry laws on violent crime). We use the majority finding from the most methodologically rigorous studies, but acknowledge that some “ground truths” may be revised by future research.

Ethics Statement↩︎

This study evaluates proprietary LLMs via their public API interfaces. Human annotators provided factual judgments on published research findings for validation purposes only; this does not constitute human subjects research. The benchmark cases are drawn entirely from published, peer-reviewed research in economics and social science; no private or sensitive data were collected or used. All experimental prompts and model outputs are stored locally and do not contain personally identifiable information.

We acknowledge that LLM-assisted policy analysis carries risks if deployed without appropriate scrutiny. Our finding that LLMs systematically fail on counter-intuitive cases—precisely where careful analysis is most valuable—underscores the importance of human oversight in policy-relevant applications. We caution against using LLM outputs as a substitute for rigorous empirical analysis and expert judgment in policy evaluation.

Acknowledgements↩︎

GitHub Copilot was used for code implementation (experimental scripts, statistical analysis pipelines) and for polishing paper text (grammar and style editing). All research ideas, scientific claims, analyses, and interpretations were conceived and verified by the author.

9 Full Case List↩︎

Table 4 presents the complete list of 40 benchmark cases. Each case is grounded in at least one peer-reviewed publication using a credible causal identification strategy. Intuitiveness classifications are validated against human annotator accuracy (§3.2).

3pt

10 Prompt Templates↩︎

We use five prompting strategies, each with variants for one-sided (YES/NO) and two-sided (INCREASE/DECREASE/NO SIGNIFICANT CHANGE) test formats. Template variables: {policy}, {country}, {year}, {outcome}, {hypothesis} (one-sided only).

P1 — Naïve (Two-sided)↩︎

Consider the following policy: {policy}, implemented in {country} in {year}. If this policy had NOT been implemented, what would have happened to {outcome}? Please answer with exactly one of: INCREASE, DECREASE, or NO SIGNIFICANT CHANGE. Answer:

P2 — Expert Persona (Two-sided)↩︎

You are an experienced econometrician specializing in policy evaluation. You are familiar with difference-in-differences, synthetic control, regression discontinuity, and instrumental variable methods. Consider the following policy: {policy}, implemented in {country} in {year}. Based on your understanding of the causal mechanisms involved, if this policy had NOT been implemented, what would have happened to {outcome}? Please answer with exactly one of: INCREASE, DECREASE, or NO SIGNIFICANT CHANGE. Answer:

P3 — Chain-of-Thought (Two-sided)↩︎

Consider the following policy: {policy}, implemented in {country} in {year}. If this policy had NOT been implemented, what would have happened to {outcome}? Think step by step: 1.What are the main causal channels through which this policy affects {outcome}? 2.Are there any confounding factors or alternative explanations? 3.Based on your analysis, what is the most likely counterfactual direction? Final answer (exactly one of: INCREASE, DECREASE, or NO SIGNIFICANT CHANGE):

P4 — Structured Reasoning (Two-sided)↩︎

Consider the following policy: {policy}, implemented in {country} in {year}. I want you to analyze the counterfactual: what would have happened to {outcome} if this policy had NOT been implemented? Please structure your response as follows: Causal mechanisms: List the main channels. Confounders: List any confounding factors. Counterfactual prediction: The most likely direction. Final answer (exactly one of: INCREASE, DECREASE, or NO SIGNIFICANT CHANGE):

P5 — Adversarial Framing (Two-sided)↩︎

Consider the following policy: {policy}, implemented in {country} in {year}. Some analysts argue that this policy had NO meaningful effect on {outcome}—that the observed changes were driven entirely by other factors such as pre-existing trends, concurrent policies, or macroeconomic conditions. Do you agree with this assessment? If this policy had NOT been implemented, would {outcome} have been essentially the same? Please answer with exactly one of: INCREASE, DECREASE, or NO SIGNIFICANT CHANGE. Answer:

One-sided variants follow the same structure but replace the outcome question with the pre-formulated {hypothesis} and use YES/NO as response options.

11 Additional Results↩︎

12 Human Annotation Details↩︎

Three annotators with diverse academic backgrounds (economics, statistics, medicine) independently predicted the outcome direction for all 40 cases without access to the answers. Table 5 summarizes annotator characteristics.

The continuous intuitiveness measure (human_acc = proportion of annotators answering correctly) takes values in \(\{0, 1/3, 2/3, 1\}\) and is used as a robustness check for the categorical classification (see §3.2).

References↩︎