1 Introduction↩︎

Large language models have moved, in only a few years, from autocompleting single lines to drafting, reviewing, and merging whole software changes. As they were deployed, a folk engineering discipline grew up around them: practitioners discovered that how you ask matters as much as what you ask, and they began packaging reusable instructions (“prompt skills”) that tell a model to plan before coding, to enumerate edge cases, to criticize its own draft, or to reason in the style of some expert. These skills are cheap to write, easy to share, and routinely reported to lift quality. The most intellectually attractive of them borrow the language of science itself: they ask the model to act as a Popperian falsificationist, to treat each candidate program as a conjecture and to attack it with the most severe tests it can imagine before declaring success [1], [2]. The appeal is clear: a model that internalized the scientist’s habit of seeking refutation rather than confirmation might catch its own bugs.

The trouble is in the measurement. A skill’s reported benefit is almost always established by an LLM-as-a-judge: a second model scores the output against a rubric, and the skill “wins” if its scores are higher. But the judge is a fragile instrument. At the coarsest level it is sensitive to the order in which candidates are presented: [3] find that simply swapping two answers flips the preferred one in \(82.5\%\) of comparisons for one judge and \(46.3\%\) for a stronger one. Judges prefer outputs from their own model family [4], [5] and (most relevant here) they reward surface features such as fluency, length, and recognizable jargon over substance: [6] show that a longer answer carrying minor factual errors is scored above a shorter correct one (Elo \(1206\) vs.\(1096\)). The pathology reaches into code specifically. [7] report that code judges shift their verdicts by up to \(26.7\) percentage points under content-preserving, cosmetic perturbations, and [8] report that injecting authority- or reasoning-flavored cues into a judge prompt drags accuracy on identical code from \(77.5\%\) down to \(61.0\%\). A judge that can be moved this far by words alone is the wrong instrument for deciding whether words, a skill’s vocabulary, improved a program.

Zoom out from judges to the broader prompting literature and the picture sharpens into a warning. A growing body of careful re-evaluations finds that celebrated prompt-engineering effects shrink or vanish once a proper control is introduced. [9] re-run five popular techniques across six models and five benchmarks and find that most, including chain-of-thought, produce no reliable gain (chain-of-thought: \(\approx\!0\%\) on average, \(p{=}.8\)), with only isolated, architecture-specific exceptions. [10] show that on high-capability models the value of elaborate prompting collapses, and what survives is not phrasing but execution feedback. The constructive counterpart to these null results is an emerging consensus about where prompting effects actually come from: structure, not wording. [11] demonstrate that meaning-preserving formatting choices alone (spacing, separators, casing) move accuracy by as much as \(76\) points; [12], in a factorial study of \(124{,}800\) Copilot prompts, find that structural features (worked examples, a task summary) significantly improve correctness (\(p{=}10^{-4}\) for examples) while stylistic features (mood, tense) do not; and [13] confirm that adding a persona or a chain-of-thought flourish on top of a structural scaffold yields no statistically significant correctness gain. [14] sharpen the boundary: vocabulary perturbations bite only when a prompt is structurally thin (a \(-7.1\%\) average drop on the lightly specified HumanEval, but only \(-0.2\%\) on the richly specified LiveCodeBench); once the scaffold is rich, the words barely matter.

Read together, this literature poses a clean, unanswered question for any methodology-flavored skill. Suppose a Popperian coding skill appears to help. The apparent help could be a judge artifact (the rubric rewards the words “falsifiability” and “severity”); it could be a structural effect (the skill, like any scaffold, makes the model enumerate steps and cases); or it could be the genuine article (the falsification content causes better code). Disentangling these requires three controls that prior code-generation studies almost never deploy together: a length-matched placebo that preserves prompt length and best-practice tone while removing all Popperian content; a labels-only scaffold that keeps the Popperian section headers but strips the procedure beneath them, so that the full skill can be compared against a fixed labelled scaffold rather than against vanilla alone; and an execution oracle that scores functional correctness with unit tests rather than rubric impressions. The components of this design exist separately in the literature (surface-form judge bias [7], self-preference in same-model judges [4], [5], structure-versus-style ablations [12]–[14], and formatting sensitivity [11]) but we are not aware of a prior pre-registered study that combines a length-matched placebo, a labels-only scaffold, an execution oracle, a vocabulary-halo sentinel, and a same-model self-judge audit to ask whether a Popperian falsification skill adds value beyond plain structure. That integration is the gap we fill.

We approached the problem philosophically as well as empirically, because the object of study is itself a philosophical claim. Popper’s program holds that a hypothesis earns scientific standing not by accumulating confirmations but by surviving sincere attempts at refutation; the degree of corroboration \(C(h,e)\) is explicitly not a probability of truth [1], and progress is framed as increasing verisimilitude [2], [15]. The error-statistical school makes the operative idea precise: data warrant a hypothesis only if the test had a high probability of exposing the hypothesis as false had it been false—Mayo and Spanos’s severity requirement [16], [17]. A coding skill that merely names severity, falsifiability, and predictive risk has done none of this work; whether the naming nevertheless changes behavior is an empirical question. There is an irreducible reflexivity here that we embrace rather than hide: we evaluate a Popperian skill using a Popperian method—pre-registered hypotheses, executable counterexamples, and blinded outcome prediction. The skill is the object of measurement; the method is the instrument, and our results show the two diverge.

A second design choice deserves its own justification, because it inverts the usual reflex to test on the strongest available model. We test not only a frontier model but, centrally, a very small one (a 0.5B-parameter local model). A weak model sits far below the benchmark ceiling, which makes it a low-baseline setting with substantial headroom to look for a real effect: a prompt component that genuinely carries signal has ample room to produce a large, unmistakable gain there. Our own data confirm the sensitivity of this testbed: the labels-only and structured scaffold conditions lift best-of-eight performance by roughly \(20\)–\(22\) points, so if the incremental Popperian content carried a correctness signal, this is a setting where it would have ample room to show itself. Its absence at low capability (the full skill does not separate from the labels-only scaffold) is therefore more informative about a separable content effect than a high-capability ceiling result alone could be. The same logic supplies a constructive screening principle that we make explicit as a screen, not a proof of transfer: a sensitive small-model testbed is a reasonable place to screen a candidate prompt component before committing high-capability compute, and a component that does not move a low-baseline weak model with ample headroom is, at the least, not demonstrated to be an active ingredient. We are careful about the limits of this principle (transfer is not guaranteed in either direction [18], [19], and capability-dependent effects in other models or tasks are not ruled out) but as a negative screen it is useful, and it organizes our two-tier design.

This reframing turns a vague “does it work?” into a falsifiable program. We pre-register an ablation ladder—vanilla (V), labels-only (L), labels\(+\)discipline (LD), labels\(+\)discipline\(+\)severity (LDS), and the full skill (F)—together with a length-matched placebo (P) and a verificationist anti-skill (ANTI). We score a high-capability model and a low-capability model against the same execution oracle, add a reliability stage with a vocabulary-only halo sentinel, and, at the low tier, test whether the model can use its own Popperian rubric to select the best of several samples—a miniature of test-time reward selection. Every hypothesis and threshold was committed to public version control before the corresponding data were collected, following recent calls for registered protocols in machine learning and NLP [20], [21].

1.0.0.1 Contributions and findings.

This paper makes the following contributions; we state up front that the headline result is a calibrated negative one, and we are explicit about the conditions under which the method fails.

1. A content-versus-structure ablation design for methodology-flavored coding skills: an additive ladder (V \(\to\) L \(\to\) LD \(\to\) LDS \(\to\) F) plus a length-matched placebo and a verificationist anti-skill, with the \(\mathrm{F}-\mathrm{L}\) contrast estimating the effect of the full Popperian procedural content added on top of a fixed labels-only scaffold—a content-beyond-labels contrast, not a pure-vocabulary one (§2).

2. A reliability stage with a vocabulary-halo sentinel that detects a confound intra-rater reliability misses entirely: our validated judge (\(\mathrm{ICC}{=}0.959\)) still awards \(10/20\) substance points to an output that contains the Popperian labels and nothing else.

3. A two-tier correctness result. At high capability the full skill shows no separable correctness benefit over vanilla, the placebo, or labels-only (all within \(2\) points, \(N{=}163\)); the pre-registered \(+5\)-point improvement hypothesis is not supported, though this is a ceiling-limited non-detection rather than a demonstration of zero effect.

4. A low-capability dissection, positioned as a sensitive screen. Every structured arm we tested lifts a \(0.5\)B model’s best-of-eight correctness by roughly \(20\)–\(22\) points, but the full skill shows no separable benefit over labels-only (aggregate \(\mathrm{F}@8{=}\mathrm{L}@8\)) and only a small, non-significant edge over a placebo—so the lift is carried by scaffold structure rather than by the incremental Popperian content, in this setting.

5. A small-scale self-judge audit: a \(0.5\)B model applying its own Popperian rubric to select among samples does not beat a random draw and concentrates its picks on one candidate index, a result consistent with a verifier–generator capability gap and with position bias (an order-randomized replication, which we did not run, would be needed to identify the latter cleanly).

6. A reusable disambiguation protocol (placebo \(+\) labels-only \(+\) oracle \(+\) halo sentinel) and a public release of pre-registrations, raw outputs, and analysis code, so that future skill-effectiveness claims can be stated—and tested—on firmer ground.

The boundary of the claim matters. We do not conclude that Popperian reasoning is worthless for code; we conclude that, in the two settings we tested, the full skill’s procedural content showed no separable correctness benefit once structure was controlled and correctness was measured by execution. Where the skill helped (the weak model, under best-of-eight evaluation), a labels-only scaffold matched the aggregate lift, and the corresponding single-sample labels/placebo comparison was not collected; where it might have helped (the strong model), the benchmark ceiling left little room to detect it.

2 Method↩︎

2.1 The proposed framework↩︎

Our object of study is a single Markdown “skill” loaded as the system prompt of a coding agent. The skill instructs the agent to externalize a hypothesis about the required program, to design severe tests against that hypothesis, to quantify “severity” and “predictive risk,” and to revise before answering. The engineering claim attached to such skills is that this Popperian content improves the code. The framework we propose is built to decide whether that claim survives once two well-known confounds (judge bias and prompt structure) are removed, and it does so by combining an ablation ladder, two confound-removing controls, two capability tiers, and an execution oracle into a single pre-registered measurement instrument, sketched in Figure 1.

The first part of the framework decomposes the skill into the seven nested conditions of Table 1. Each is a Markdown skill loaded as the system prompt of a coding subagent; the ladder is additive, so that consecutive conditions differ by exactly one component and the marginal effect of that component is the difference of their outcomes. Condition L keeps Popperian terminology in section headers only, with no procedure beneath them; LD adds a numbered procedural scaffold; LDS adds a severity-quantification formula; and F is the full skill. P is a length-matched best-practices placebo that contains no Popperian content, and ANTI is a verificationist inverse that instructs the model to seek confirmations. The single most important comparison in the paper is F versus L: because L and F share the same labels-only scaffold (the same headers, the same nominal sections) and differ in the content placed beneath those headers (the full falsification procedure, severity formula, and predictive-risk narrative in F; nothing in L), their difference estimates the marginal effect of that Popperian content on top of the scaffold that carries it. As Proposition 1 makes explicit, this is a content-beyond-labels contrast rather than a pure-vocabulary one (F and L also differ in procedure, instruction density, and length) while the length-matched placebo P guards separately against a prompt-length confound.

The seven conditions enter the two measurement stages asymmetrically, which we flag here rather than leave implicit. The full ladder (including the intermediate rungs LD and LDS and the anti-skill ANTI) is scored only at the rubric/anatomy stage (Tier 1.A1), where its purpose is to localize which component of the skill moves a rubric judge; the execution-oracle stage (Tier 1.A2 and Tier 2) reports the four conditions—V, L, F, and P—for which complete generation artifacts were collected and graded under the oracle. We make no execution-correctness claim for LD, LDS, or ANTI, because they were not run through the oracle at scale; this is a deliberate scope limit, restated in §4.

The second part of the framework states, precisely, what each contrast estimates. Let \(c \in \mathcal{C}=\{\mathrm{V},\mathrm{L},\mathrm{LD},\mathrm{LDS},\mathrm{F},\mathrm{P}, \mathrm{ANTI}\}\) index a condition, realized as a system prompt \(s_c\). For a problem \(q\) drawn from a benchmark \(\mathcal{Q}\) and a generator \(M\), let \(Y(c,q)\in\{0,1\}\) be the event that a sample produced under \(\langle s_c, q\rangle\) passes the execution oracle. The correctness of a condition is \[\label{eq:pi} \pi(c)=\mathbb{E}_{q\sim\mathcal{Q}}\,\mathbb{E}_{M}\,[\,Y(c,q)\,].\tag{1}\] Write each prompt as \(s_c=(\sigma_c,\kappa_c)\), where \(\sigma_c\) is a labels-only scaffold (the Popperian section headers and the output contract) and \(\kappa_c\) is the procedural content placed beneath those headers (the numbered procedure, the severity formula, the predictive-risk narrative, and the revision discipline). The ladder holds the labels-only scaffold fixed across L and F (\(\sigma_{\mathrm{L}}=\sigma_{\mathrm{F}}\)) and adds content in steps, while V is the empty scaffold; this yields a structure estimand and a content-beyond-labels estimand, \[\begin{align} \Delta_{\mathrm{struct}} &= \pi(\mathrm{L})-\pi(\mathrm{V}), &&\text{(labels-only scaffold over vanilla)} \tag{2}\\ \Delta_{\mathrm{content}} &= \pi(\mathrm{F})-\pi(\mathrm{L}). &&\text{(full Popperian content over labels-only)} \tag{3} \end{align}\] We are deliberately conservative about how \(\Delta_{\mathrm{content}}\) is read.

Correctness under sampling is reported with the unbiased \(\mathrm{pass}@k\) estimator of [22]: generating \(n\ge k\) samples per problem and counting \(c_q\) correct, \[\label{eq:passk} \mathrm{pass}@k=\mathbb{E}_{q\sim\mathcal{Q}}\!\left[\,1-\binom{n-c_q}{k}\big/\binom{n}{k}\,\right],\tag{4}\] so that \(\mathrm{pass}@1\) is the single-sample (greedy) rate and \(\mathrm{pass}@n\) is best-of-\(n\). To make the philosophical target operational we adopt the error-statistical definition of severity, which the skill’s vocabulary gestures at but does not implement.

An execution oracle on a dense suite of adversarial unit tests approximates an operational severe-test regime in the sense of 5 : it raises the probability that an incorrect program is exposed, because a wrong program is likely to fail at least one augmented test. We are careful not to overstate the correspondence—the augmented tests are an engineering analogue of severity, not a full instantiation of Mayo–Spanos severity, which additionally requires a specified test distribution, an alternative error model, and an explicit probability of detecting a discrepancy. A skill that merely writes the word “severity” supplies none of this, which is why we measure correctness by execution and treat rubric scores as suspect. That suspicion is itself quantified through a halo ratio.

Finally, “no meaningful difference” is operationalized by an equivalence rule rather than a failure to reject. With a pre-registered margin \(\delta=5\) points, a directional hypothesis of the form \(\pi(\mathrm{F})-\pi(\mathrm{V})\ge\delta\) is declared falsified when the observed difference is below \(\delta\) and the upper confidence limit excludes it, and two conditions are declared equivalent when the two one-sided tests (TOST) for paired proportions reject at the \(\pm\delta\) boundary [23]: \[\label{eq:tost} \text{equivalence}(c,b)\;\Longleftrightarrow\; \big[\widehat{\pi}(c)-\widehat{\pi}(b)\big]_{\text{CI}_{1-2\alpha}} \subset (-\delta,+\delta).\tag{6}\] Paired pass/fail outcomes use McNemar’s test [24], and all intervals are bootstrap \(95\%\) confidence intervals [25].

The rubric judge that produces the Tier-0 evidence is itself validated before any reliance on it. Tier-1.A1 rubric scores come from a Claude Sonnet 4.6 subagent applying a ten-dimension rubric to each output, blinded to the condition. We measure intra-rater reproducibility by the intraclass correlation coefficient under a two-way random-effects, absolute-agreement, single-measures model (ICC(2,1) in the nomenclature of [26], treating the repeated scoring occasion as the random replicate factor) and interpret it with their thresholds and those of [27]. Cross-rater agreement across Sonnet, Opus, and Haiku judges uses Krippendorff’s ordinal \(\alpha\) [28]. We then construct vocabulary-only sentinels (Definition 2) and score them, because intra-rater reliability, however high, cannot reveal a judge that reliably rewards the labels themselves. All hypotheses and thresholds were committed to the project’s git history before the corresponding data were collected, following registered-protocol practice [20], [21]; because the registered protocol fixes the decision rules in advance, every negative verdict reported later is the firing of a pre-committed probe rather than a post-hoc reading. We close the framework with a boundary we keep throughout: the skill names three Popperian constructs (falsifiability and the non-probabilistic degree of corroboration [1], the severity of a test in the error-statistical sense [16], [29], and verisimilitude [2], [15]) but we evaluate the coding skill that invokes these constructs, and our experiments neither validate nor refute Popper’s philosophy.

2.2 The proposed algorithm↩︎

The framework is realized by two procedures. Algorithm 2 is the per-condition correctness pipeline used in both tiers; it is deliberately judge-free downstream of generation, so that the verdict on whether vocabulary helps never passes through a biased rubric. For each problem the generator produces \(n\) samples (greedy plus stochastic), the entry function is recovered from each sample by parsing the abstract syntax tree, and the function is executed against the augmented HumanEval+ tests under a process timeout; the per-condition correctness \(\pi(c)\) and its \(\mathrm{pass}@k\) follow from 1 –4 , with a bootstrap interval. A mandatory pre-flight step aborts the run unless every canonical reference solution passes the oracle, which guards against a grader that silently mis-scores.

Algorithm 3 is the test-time reward-selection procedure used in Tier 2 to ask whether a model can grade itself. The generator’s own Popperian rubric is applied to its \(n\) samples to pick a single “best” one (\(F_{\mathrm{judge}}\)); the same selection is also made by drawing one of the \(n\) samples uniformly at random (\(F_{\mathrm{rand}}\)). Both choices are then scored by the execution oracle. A self-judge that carries signal must beat the random draw by the pre-registered margin; we additionally record the selected index and the rating pattern so selection-index concentration can be audited as evidence consistent with position sensitivity.

3 Results↩︎

The proposed framework was tested in two distinct execution environments that bracket the capability spectrum, summarized in Table [tab:setup]. Correctness in both environments is adjudicated on HumanEval and its augmented variant HumanEval+. HumanEval [22] is a set of \(164\) hand-written Python programming problems released by OpenAI (github.com/openai/human-eval); each problem pairs a function signature and docstring with a small set of hidden unit tests, and correctness is decided by executing the generated function against those tests. Because the original test suites are sparse enough to let incorrect programs pass, we use HumanEval+ from the EvalPlus framework [30] (github.com/evalplus/evalplus), which augments each problem with roughly \(80\times\) more automatically generated tests (an EvalPlus-reported average of \(764.1\) per problem) and which Liu et al.showed reduces measured \(\mathrm{pass}@k\) by up to \(19.3\)–\(28.9\%\) across \(26\) LLMs relative to vanilla HumanEval—direct evidence that the original suite overestimates functional correctness. Provenance and usage are listed in the upper block of Table [tab:setup]. The high-capability surface comprises \(N{=}163\) problems (we exclude HumanEval/\(32\), whose augmented test contains an unpacking bug incompatible with its single-float canonical return), while the low-capability surface comprises the full \(N{=}164\) problems (HumanEval/\(32\) is admissible under the Tier-2 grader path); together the two tiers cover \(327\) problem instances. All correctness numbers come from a deterministic pytest oracle (no LLM judge is in the loop downstream of generation) that parses the fenced code, extracts the entry function by AST, and executes it against the augmented tests under a multiprocessing timeout.

The two tiers differ primarily in generator capability, serving path, sampling protocol (\(1\) greedy sample vs.\(8\)), the inferential surface (\(N{=}163\) vs.\(N{=}164\) and \(\mathrm{pass}@1\) vs.\(\mathrm{pass}@8\)), one benchmark-handling detail (HumanEval/\(32\)), and the presence of a self-judge stage in Tier 2 only; these differences are part of the intended bracketing of two practically relevant serving regimes and are not a controlled scaling curve, so we do not draw direct tier-to-tier causal comparisons. Model identifiers and serving are stated as recorded in our run logs. The high-capability tier uses Claude Sonnet 4.6 (model identifier claude-sonnet-4-6 as logged) as the generator, with Claude Haiku 4.5 (claude-haiku-4-5-20251001) and Claude Opus 4.7 (claude-opus-4-7) as auxiliary generators and judges, all invoked as subagents through the Claude Code agentic CLI. This serving path is itself a boundary on Tier-1 internal validity that we state plainly: because generation runs through the agentic CLI rather than the raw model API, every Tier-1 condition—“vanilla” included—is a condition within that harness and may carry operator or system scaffolding we neither set nor observe. “V” therefore denotes a vanilla prompt inside the Claude Code path, not a raw-API baseline; raw-API behavior could differ, and a raw-API replication is a natural follow-up (§4). The low-capability tier uses Qwen2.5-Coder-0.5B-Instruct (fp16; Ollama tag qwen2.5-coder:0.5b-instruct-fp16) served locally through the Ollama HTTP API on a 6 GB-VRAM laptop; the HTTP path is mandatory because the full skill prompt reaches \(\approx\!4{,}874\) tokens and the CLI path silently truncates at \(2{,}048\), which would convert the full skill into vanilla. The Tier-2 self-judge is the same 0.5B model. Reliability and rubric ablation additionally use three hand-authored cases scored on a ten-dimension rubric by the Sonnet judge, validated in Tier 0 and audited by an Opus secondary judge.

We first validate the rubric judge (Tier 0). The Claude Sonnet 4.6 judge is highly reproducible: intra-rater \(\mathrm{ICC}=0.959\) (\(95\%\) CI \([0.904,0.987]\)), “excellent” by the [26] thresholds. Cross-rater agreement across the three judge models is only moderate (Krippendorff \(\alpha=0.544\), itself an \(N{=}25\) estimate whose sampling spread is wide), in fact below Krippendorff’s conventional \(0.667\) threshold for even tentative reliance [28]. High intra-rater reproducibility is consistency, not validity: a judge can repeat itself precisely and still reward the wrong features. For both reasons we treat the entire rubric stage as exploratory and diagnostic (useful for locating which component of the skill moves a judge, not for any correctness claim) and every correctness result below rests on the execution oracle instead. Even granting the judge its excellent intra-rater reliability, a pre-registered counterexample fired: a vocabulary-only sentinel containing the severity labels but no code and no procedure received \(10\) out of \(20\) on the substance dimensions; measured against the genuine LDS condition mean of \(14.0\) (Definition 2, \(\bar J(\mathrm{LDS}){=}14.0\)), this is a halo ratio of \(\rho=10/14.0=0.714\), exceeding the pre-registered \(\tau=0.70\) threshold. The judge is therefore a reliable instrument that nonetheless lets Popperian wording leak into substance scores, a local instance of a general pathology [6], [7].

This halo is exactly what makes the rubric ladder misleading, as the anatomy ablation (Tier 1.A1) shows. On the ten-dimension rubric the ladder climbs monotonically (Table 2, Figure 4), which at face value looks like cumulative evidence for the skill. The decomposition tells a different story. The single large, significant increment is the procedural scaffold: \(\mathrm{LD}-\mathrm{L}=+4.11\) points (\(p=0.004\)). Adding the severity machinery on top is not significant (\(\mathrm{LDS}-\mathrm{LD}=+0.78\), \(p=0.137\)), and its sentinel is exactly the one that triggers the halo (\(\rho=0.714\)); the full skill’s final increment over LDS is descriptive only (\(+0.38\)). What a naive reading would credit to Popperian machinery is, on inspection, the scaffold plus a vocabulary halo, not the severity content. This reading is consistent with small-sample robustness checks: re-judging in isolated single-cell context reproduces the scores (Pearson \(r=0.998\) on \(n{=}5\) re-judged cells), a secondary Opus judge agrees (\(r=0.935\), \(n{=}6\)), and having a different model (Haiku) blindly re-author the variants preserves the ordering—though, given the rubric stage’s exploratory status and these small samples, we lean on it only for anatomy, not for any correctness claim.

When the rubric is replaced by the execution oracle at the high-capability tier (Tier 1.A2, \(N{=}163\), Claude Sonnet 4.6), the apparent rubric-stage contribution does not appear in execution correctness. As Table 3 and Figure 5 show, all four conditions fall within \(2\) points and their \(95\%\) confidence intervals overlap heavily: \(\mathrm{V}=95.1\%\), \(\mathrm{F}=95.7\%\), \(\mathrm{L}=95.7\%\), \(\mathrm{P}=96.9\%\). The full Popperian skill does not exceed vanilla by the pre-registered \(5\)-point margin: the observed gap is \(+0.6\) points, the paired difference is not significant (McNemar \(p=1.0\), discordant \(3/2\) of \(163\)), and the upper bound of its paired \(95\%\) bootstrap CI is \(+3.1\) points, which excludes \(+5\) (Table [tab:paired]). By the pre-registered rule the directional hypothesis HA1 (\(\mathrm{F}-\mathrm{V}\ge 5\)) is therefore not supported; we read this specifically as a ceiling-limited non-detection rather than a demonstration of zero effect, because a baseline already at \(95.1\%\) leaves almost no room for a \(+5\)-point gain—reaching the threshold would require near-perfect generated-solution accuracy under the augmented oracle, after excluding the known HumanEval/\(32\) issue from the high-tier surface. Two further pre-registered predictions hold in the same data: labels-only does not separate from the full skill (\(\mathrm{L}=\mathrm{F}=95.7\%\), HA4), and, rather than the skill beating the placebo, the placebo is numerically highest (\(96.9\%\), so HA2’s confound prediction holds with the sign reversed). A pilot reading of \(\mathrm{F}>\mathrm{V}\) by \(+14\) points at \(N{=}7\) was a small-sample artifact, not reproduced once the surface grew to \(163\) problems. At a near-ceiling benchmark there is little room for any condition to separate, and what little movement exists slightly favors the shorter prompts.

The low-capability tier (Tier 2, \(N{=}164\), Qwen2.5-Coder-0.5B) is where a real effect could have appeared, because the \(0.5\)B model is far from any ceiling, and it is the low-baseline screen our design depends on—a more sensitive setting than the ceiling-limited high tier. Table 4 and Figure 6 report the eight arms. Two quantities must be kept distinct. The single-sample rate \(\mathrm{pass}@1\) (greedy) is the deployment-relevant number (what a user would receive from one generation) whereas \(\mathrm{pass}@8\) is an oracle-selected best-of-eight quantity that measures the model’s latent capacity to produce at least one correct program in eight samples; it is not an output a user could obtain without an external selector. With that distinction in place: every structured arm delivers a large best-of-eight lift, \(\mathrm{F}@8=56.7\%\) versus \(\mathrm{V}@8=34.8\%\), a \(+22.0\)-point gain (McNemar \(p\approx2\times10^{-7}\), discordant \(43/7\); paired \(95\%\) CI \([+14.0,+29.9]\)), confirming the pre-registered HB1. The greedy single-sample rate is \(\mathrm{F}@1=37.2\%\), so best-of-\(8\) sampling supplies \(+19.5\) points of headroom (HB4, which narrowly misses its \(+20\)-point threshold at \(+19.51\) pt, paired CI \([+13.4,+26.2]\), so we record it as not met rather than confirmed). The incremental Popperian content again shows no separable benefit: the aggregate best-of-eight rate of labels-only coincides with the full skill (\(\mathrm{L}@8=\mathrm{F}@8=56.7\%\), \(93/164\) each), though the two conditions disagree on \(24\) individual problems (Jaccard \(0.77\), McNemar \(p=1.0\)), so they do not separate rather than being literally identical; the length-matched placebo trails by only \(2.4\) points (\(\mathrm{P}@8=54.3\%\), paired \(p=0.60\)), so HB2 (the skill beats the placebo by \(\ge5\) points) is not met—though its paired CI \([-4.3,+9.2]\) does not exclude a \(+5\)-point effect, so we treat HB2 as non-confirmatory rather than as ruling such an effect out. The lift is consistent with the presence of any generation-discipline scaffold rather than to severity, falsifiability, or predictive-risk framing—in the low-baseline setting where, with ample headroom, a real content effect should have been easiest to see. We note one gap in this comparison: single-sample (\(\mathrm{pass}@1\)) rates were collected only for V and F, not for L or P, so the deployment-relevant labels/placebo comparison is available at \(\mathrm{pass}@8\) only (§4).

The self-judge result is the most diagnostic. When the same \(0.5\)B model rates the full skill’s eight samples with the Popperian rubric and we keep its top pick, the selected sample passes \(25.6\%\) of the time (\(F_{\mathrm{judge}}\))—at essentially the random-selection level rather than above it: a single seeded random draw from the same eight passes \(26.8\%\) (\(F_{\mathrm{rand}}\)) and the analytic expectation of a uniform single draw—the mean over problems of the fraction of the eight samples that pass—is \(24.9\%\). The \(1.2\)-point gap to the seeded draw is within noise (McNemar \(p\approx0.82\), discordant \(9/11\); paired CI \([-6.7,+4.3]\)), so HB3 (self-judge beats random by \(\ge5\) points) is not supported—the structured-reward signal does not improve on chance selection, but neither is it reliably worse, so we avoid the stronger “anti-informative” reading. The selection pattern is consistent with position bias (Figure 7): \(60\%\) of judge calls pick sample index \(1\), far above the \(12.5\%\) a content-blind uniform selector would place there, and the three most frequent rating patterns account for \(119/164\) (\(73\%\)) of all calls. We are careful here: because the eight candidates were presented to the judge in a fixed sample order (the greedy sample first), an order confound cannot be separated from genuine position bias without an order-randomized replication, which we did not run; we therefore report the concentration as evidence consistent with position bias rather than as identified position bias (§4). The pattern is, at small scale, an instance of the verifier–generator gap documented for larger models that cannot reliably self-correct without an external signal [31]–[33].

Table [tab:paired] reports the paired statistics behind these verdicts, computed directly from the released per-problem pass matrices. The resampling unit is the problem: each of \(10{,}000\) paired bootstrap resamples draws problems with replacement (seed \(20260601\)), clustering the two conditions’ outcomes by problem, and the reported interval is the resulting \(2.5\)–\(97.5\) percentile range of the paired difference; McNemar’s exact two-sided binomial test uses the discordant pairs only. We apply no multiple-comparison correction, so each \(p\)-value should be read per-contrast. The pre-registered equivalence margin is \(\delta=5\) points. We use the word “equivalent” only when a paired interval lies inside \((-\delta,+\delta)\): this holds for every Tier-1 contrast (all six pairwise intervals fall within \(\pm5\) points, so F, L, P, and V are statistically equivalent at the \(\pm5\)-point margin), but it does not hold for either Tier-2 contrast: the \(\mathrm{F}@8\)-versus-\(\mathrm{L}@8\) interval \([-6.1,+6.1]\) slightly exceeds \(\pm5\), and the \(\mathrm{F}@8\)-versus-\(\mathrm{P}@8\) interval \([-4.3,+9.2]\) exceeds it at the high end, so a \(+5\)-point skill-over-placebo effect is not formally ruled out there. We therefore describe both Tier-2 pairs as “not separating” rather than as equivalent.

5pt

Table 5 collects the eight pre-registered hypotheses and their verdicts. Three were confirmed, one confirmed with the sign reversed (HA2), two are not supported with the margin also statistically excluded (HA1, ceiling-limited; HB3), and two are not met—the criterion unreached but the margin not statistically excluded (HB2 and HB4); the non-confirmations are not noise but the firing of probes committed before the data existed. The method delivered signal in both directions: it confirmed that structure helps a weak model and that the judge is reproducible, and it found no separable correctness benefit from the full Popperian content over a labels-only scaffold, no advantage over a placebo, and no usable reward signal from a \(0.5\)B self-judge.

4 Discussion↩︎

The framework asks whether a methodology-flavored prompt skill carries content or only its labels, and its three controls each neutralize a specific rival explanation: the labels-only condition removes the structure confound by holding the scaffold fixed while stripping the procedure; the length-matched placebo removes the length and tone confound; and the execution oracle removes the judge-bias confound that would otherwise let recognizable vocabulary inflate the score. Run across two capability tiers, the design returned a consistent verdict: once the labels-only scaffold is held fixed and correctness is measured by execution, the full skill’s Popperian procedural content shows no separable correctness benefit. At the ceiling, F, L, P, and V sit within two points (statistically equivalent at the \(\pm5\)-point margin); at the floor, F and L do not separate (aggregate \(\mathrm{}@8\) identical, paired \(p=1.0\)) and P is not significantly lower. The one place the skill “worked” under best-of-eight evaluation—the weak model’s \(+22\)-point lift—was matched in aggregate by the labels-only scaffold; the corresponding single-sample labels/placebo comparison (\(\mathrm{L}@1\), \(\mathrm{P}@1\)) was not collected.

Why a Popperian vocabulary might be expected to help (and why it nonetheless need not) is sharpened by the very philosophy the skill borrows. The advantage that falsificationism gives a methodology is specific: in the error-statistical reconstruction, evidence warrants a hypothesis only when the test was severe—when it would very probably have failed the hypothesis had the hypothesis been false (5 ; [16], [17]). Severity is what separates a program that passed sincere tests from one that merely was asserted to be correct, and a skill that could induce a model to construct severe tests against its own code would be genuinely valuable. The limit, equally precise, is that naming severity is not having it. Lakatos’s critique of “naive falsificationism” is exactly that attaching the vocabulary of refutation to an inference, without the operational scaffolding that points the modus tollens at a real target, yields a label and not an operation [34]; and Popper himself insisted that corroboration is not a probability of truth [1], [29], so invoking it confers no inductive warrant. Our empirical finding is the engineering counterpart of this point: the prompt supplies the words of severity but neither the test distribution nor the error probability \(\alpha\) of 5 that would give them force, and an LLM consuming those words gains only the discipline of a scaffold.

The same conclusion can be stated in the language of the estimands, which also clarifies why the oracle beats the judge and why the self-judge fails. The quantity an engineer cares about is the content-beyond-labels effect \(\Delta_{\mathrm{content}}\) of 3 ; our data put it at \(0.0\) points at high capability and \(0.0\) at low capability, with paired intervals that lie within the pre-registered \(5\)-point margin at the high tier (Table [tab:paired]). A rubric-only study cannot recover this number, because the judge’s response is approximately \(\pi(c)+h(c)\) where \(h\) is the halo term of ?? ; with \(\rho=0.714\) the halo is large enough to manufacture an apparent content effect from labels alone, which is precisely what the monotone rubric ladder of Figure 4 would have shown a naive reader. The execution oracle sets \(h\equiv0\) by construction, which is why we trust it. The self-judge result has a similar formal cause: test-time reward selection is useful only when the selector’s accuracy exceeds chance, and a verifier no stronger than the generator has no such edge. This is the small-scale instance of a documented gap (LLM verifiers accept invalid solutions at high rates [32], generation skill does not transfer to judgment skill [35], and intrinsic self-correction degrades reasoning accuracy without an external signal [31], [33]) and it predicts that useful self-selection needs a verifier strictly stronger than the generator, or an external oracle, exactly the ingredient that lifts code agents in practice [10], [36].

The two-tier design also lets us reason carefully about transfer, which is the practical payoff of testing on a weak model. Prompt-surface effects are capability-modulated: [10] document that the value of structured prompting collapses on advanced models once baseline capability saturates the benchmark, while [19] show that larger, more instructable models can become less reliable overall—so neither upward nor downward transfer can be assumed for free, and [18] remind us that a reasoning trigger that unlocks large gains at scale can be inert on small models. We use the weak tier accordingly, as a negative screen rather than a guarantee of transfer: structure clears that screen by a wide margin (\(+22\) points), demonstrating the testbed is sensitive, while the incremental Popperian content does not move it. The honest inference is asymmetric and we state it as such: the absence of a separable content effect in this low-baseline setting weakens the claim that the Popperian procedural content is the active ingredient (claiming it as the active ingredient would now require independent positive evidence of transfer that we do not have), but it does not rule out capability-dependent effects in other models, tasks, or skill implementations. We likewise do not claim the converse, that the content would help a large model, since the ceiling argues against detecting any such effect there in the first place.

Table 6 situates our result against the closest prior studies along the axis that matters: which confound-removing controls each one deploys, and what each obtained. Two patterns are visible. First, the field has repeatedly found, in adjacent settings, that surface form moves judges and that prompt structure rather than wording moves correctness—our result is continuous with [12], [11], [14], and [13], and extends them by naming a specific, popular skill family and showing that its incremental Popperian content does not separate from a labels-only control. Second, and this is where we differ, prior code-generation studies rarely combine a length-matched placebo, a labels-only scaffold, and an execution oracle, and we are not aware of prior prompt-skill evaluations pairing these controls with a small-model self-judge audit; that combination is our methodological contribution, and it is why our negative result is harder to dismiss than a single-control study. We are equally explicit about where others are stronger: studies such as [12] (a \(124{,}800\)-prompt factorial) and [9] (six models, five benchmarks) have far greater external validity than our two substrates and one benchmark family. Where we are stronger is in isolation, not breadth: none of these holds a labels-only scaffold fixed while varying the full Popperian content (our \(\mathrm{F}\)-versus-\(\mathrm{L}\) non-separation), adds a vocabulary-halo sentinel to catch the judge confound directly, or audits a self-judge at the \(0.5\)B scale where reward selection is most tempting and, we find, least informative. Our placebo design also imports a control (length-matched sham text) that is standard outside code generation [37], [38] but, as far as we are aware, novel for an LLM-judged coding skill.

4pt

Two boundaries of the design deserve to be named precisely, because they bound what the contrasts can and cannot say. First, \(\mathrm{F}-\mathrm{L}\) estimates the effect of the full Popperian content over a labels-only scaffold, not of terminology in isolation (Proposition 1); isolating vocabulary as such would require a further family of ablations we did not run (the same procedure phrased in generic, non-Popperian terminology; the Popperian terminology with the procedure removed; a neutral filler of matched length; the same headings carrying different behavioral instructions; the same procedure under several jargon families; and token-count-matched variants of L, F, and P) each holding procedure, length, or behavioral content fixed while varying only the words. Second, the execution-oracle comparisons are complete only for V, L, F, and P: the intermediate rungs LD and LDS and the anti-skill ANTI were scored only at the exploratory rubric stage, and single-sample (\(\mathrm{pass}@1\)) rates were collected only for V and F, so the deployment-relevant labels/placebo comparison and the full additive ladder under the oracle remain to be measured. Because \(\mathrm{L}@1\) and \(\mathrm{P}@1\) were not collected, the single-sample deployment-relevant source of the \(\mathrm{F}@1\) gain remains unresolved, and the F/L/P dissection is currently available at best-of-eight only. We list these as concrete next experiments, since each could sharpen—or overturn—a specific reading above.

These results carry practical lessons for anyone measuring a prompt skill. The most important is that a length-matched placebo should be a default control rather than an afterthought: a rubric gain that evaporates against inert text of equal length was never a skill-content effect to begin with [37], [38]. A correctness oracle should sit alongside any rubric, because, as our \(\rho=0.714\) sentinel shows, an LLM judge can manufacture an apparent vocabulary effect out of labels with no code behind them, and only execution closes that gap. The vocabulary-halo sentinel itself is worth adopting as a cheap standing probe, since it exposes precisely the confound that an otherwise excellent intra-rater reliability would conceal. And where a skill proposes to select among its own outputs, the self-judge result is a caution against same-capacity grading: the verifier has to be stronger than the generator, or external, or it collapses into position.

Finally, the strongest claims are bounded to the tested settings, and we state those bounds precisely here. The high-capability tier is a single model at a near-ceiling benchmark, which structurally caps any detectable effect; we report an equivalence there, not a proof of zero, and we are careful not to read an absence of evidence as decisive evidence of absence. The low-capability tier is a single 0.5B substrate with one seed budget (\(8\) samples \(\times\,164\) problems \(\times\,4\) conditions), and the self-judge is tested only at \(0.5\)B; whether a stronger judge recovers signal is untested and is the natural next experiment. The Tier-1.A1 rubric used a single primary judge, with only small-sample robustness support—isolate-rejudge (\(r{=}0.998\), \(n{=}5\)), an Opus secondary judge (\(r{=}0.935\), \(n{=}6\)), and blind re-authoring—and some early per-cell raw scores were lost before an archival pattern was adopted (aggregates survive); the three-judge cross-rater \(\alpha=0.544\) is itself below Krippendorff’s conventional threshold. For all these reasons the rubric stage is exploratory and no correctness-bearing claim rests on it. We exclude HumanEval/\(32\) from the high-capability surface owing to an augmented-test bug, and “vanilla” baselines in agentic settings can be silently contaminated by operator scaffolds and system prompts [39], [40], which if anything would shrink the apparent benefit of an added skill and so does not threaten the negative direction. Pre-registration rests on the project’s git commit chain; we provide the commit identifiers but recommend an external timestamp for replication. We report the negative result precisely because the field’s documented bias against null findings would otherwise leave the question open [41], [42].

5 Conclusion↩︎

We set out to decide whether a coding skill that attaches Popper’s vocabulary (severity, falsifiability, predictive risk) to prompts improves the code or only its apparent quality. The method was a pre-registered, two-tier ablation with three controls: a labels-only condition that fixes the scaffold while removing the procedure, a length-matched placebo that fixes length and tone, and an execution oracle that scores correctness by unit tests rather than by a biased rubric—all audited by a reliability stage with a vocabulary-halo sentinel, and anchored on a small model where a real effect would be easiest to detect. The study evaluates one prompt-skill implementation; it is not a test of Popperian methodology, whose meta-level use was in fact load-bearing in our own design.

The result is a calibrated, setting-specific negative. On the frontier model (Claude Sonnet 4.6) the full skill matched vanilla, labels-only, and the placebo to within two points on \(163\) HumanEval+ problems—statistically equivalent at the \(\pm5\)-point margin—so the pre-registered \(+5\)-point improvement hypothesis was not supported, with the caveat that a \(95.1\%\) baseline makes that threshold a ceiling-limited target rather than a fair test. On the small model (Qwen2.5-Coder-0.5B) structured arms lifted best-of-eight correctness by roughly \(20\)–\(22\) points, but the full skill’s Popperian content showed no separable benefit over a labels-only scaffold (aggregate \(\mathrm{F}@8{=}\mathrm{L}@8\)) and a placebo difference that was not statistically significant, while a same-model self-judge applying the Popperian rubric did not beat random selection and concentrated its picks on one index, a pattern consistent with position bias. The strongest positive finding is that a structured scaffold substantially helps a weak model under best-of-eight evaluation; the strongest negative is that the full Popperian skill does not separate from a labels-only control in aggregate best-of-eight; and the most practical contribution is the disambiguation protocol itself (placebo \(+\) labels-only \(+\) execution oracle \(+\) halo sentinel), which we recommend as a default for prompt-skill claims. In the settings tested, the measurable best-of-eight gains are carried by scaffolded generation rather than by the incremental Popperian procedural content, but the result remains bounded by one benchmark family, two serving regimes, and incomplete single-sample labels/placebo measurements.

The natural next step turns the method on a sharper target. If the present skill showed no effect because it names severe tests without executing them, its obvious successor is a skill built around executable falsification—generating, running, and selecting against real counterexamples rather than rubric impressions—under the same pre-registered, oracle-grounded protocol, and screened first on the same low-baseline small-model tier. We are developing this successor as a companion line of work; the same disambiguation protocol used here—placebo, labels-only, execution oracle, and halo sentinel—should be reused so that any measured gain is attributed to executable testing rather than to additional rhetoric. Whether that sharper skill clears the screen the present one did not is a question we leave to that future evaluation.

Reproducibility and Data Availability↩︎

The data, code, and the Popperian coding skill under test are available in the project repository, which you can access at https://github.com/PhiniteLab/popperian-coding-skill.

Ethics and Generative-AI Use Disclosure↩︎

This study evaluates an AI coding skill using AI generators and, in the Tier-0/Tier-1.A1 reliability stage, an AI judge. The skill under test was authored by an LLM and is the object of measurement, not a methods tool. The study’s design, hypotheses, pre-registration, literature review, statistical analysis, and claim calibration are the author’s own work. A generative-AI assistant was used only to improve the readability of the manuscript prose (copy-editing and phrasing); it did not design the experiments, produce or select the results, choose the citations, or set any conclusion. No AI system is listed as an author.

Supplementary Materials and Planned Appendices↩︎

The repository (see Reproducibility) already contains the per-problem pass/fail matrices for both tiers, the condition prompts (V, L, LD, LDS, F, P, ANTI), the \(164\) self-judge rating arrays with selected indices, the deterministic oracle, and the bootstrap and figure scripts. The following analyses are recommended as appendices before journal submission and are not yet compiled into the manuscript: (i) a prompt-token-count table for all conditions (needed to make the length-matching of P quantitative); (ii) a full per-problem \(\mathrm{F}/\mathrm{L}/\mathrm{P}\) overlap (Jaccard) table beyond the headline values reported here; (iii) an error taxonomy of failures (syntax error, extraction failure, wrong entry point, timeout, assertion failure, type error, partial correctness); (iv) the self-judge rating-pattern frequency table (the three modal patterns already account for \(73\%\) of calls); and (v) a concrete order-randomized shuffle-test protocol for the self-judge, which would convert the position-bias observation from “consistent with” to “identified.” These are listed explicitly so that the gap between what is released and what remains to be added is visible to the reader.

6 The Popperian coding skill under test (condition F)↩︎

For full reproducibility we reproduce, verbatim, the Markdown skill that defines the full condition F: it is loaded as the coding agent’s system prompt and is the object of measurement throughout this study. Listing [lst:skill] is the skill’s entry-point file (SKILL.md); under F it routes to a generation-mode file, and the four mode files (generation, review, verification, test_construction) ship in the ancillary anc/ directory of this submission, with the entry-point file reproduced below; the helper scripts and the full loadable package are available in the stand-alone repository at https://github.com/PhiniteLab/popperian-coding-skill. The labels-only control L retains this file’s section headers while removing the procedure beneath them; the length-matched placebo P replaces the Popperian content with generic best-practice text of comparable length; neither is reproduced here but both are included in the repository. The skill’s structured-reporting step—the report_falsification_result tool referenced in the listing—is part of the loadable package but is not invoked in the HumanEval+ generation harness used for the oracle evaluation: correctness in Tiers 1.A2 and 2 is decided by the deterministic pytest oracle on the function extracted from each generation (Algorithm 2), independently of that tool, which is accordingly not among the file/shell tools declared under the skill’s allowed-tools. We stress, consistent with the body of the paper, that this skill is the specific implementation we evaluate, not Popperian methodology in general.

---
name: popperian-coding
description: |
  Switches the coding agent from verification ("does my answer look right?")
  to falsification ("what severe test would break my answer if it were wrong?").
  Applies to four coding modes — code review, code generation, task verification,
  and test construction — and enforces three contracts: severity-scored test
  design, N-of-N severe-test survival as the output gate, and an MDL-based
  ad hoc rescue blocker. Replaces self-refine / Reflexion verification loops.
when_to_use: |
  Activate for any of:
  - High-assurance code tasks (security-critical, correctness-critical, prod-deploy)
  - Independent verification of another agent's claim of completion
  - Test suite quality audit and structural gap detection
  - Early clarification discipline against ambiguous specifications
  - Cases where self-refine / Reflexion-style verification loops are inadequate
allowed-tools:
  - Read
  - Write
  - Edit
  - Bash
  - Grep
  - Glob
mode_router:
  patterns:
    review: ["review", "PR review", "code review", "audit this diff"]
    generation: ["implement", "write", "code", "generate", "build a function"]
    verification: ["verify", "is it done", "check completion", "accept"]
    test_construction: ["write tests", "test suite", "coverage", "test design"]
---

# Popperian Coding Skill — Protocol

You operate in one of four modes — `review`, `generation`, `verification`, or
`test_construction`. The mode is selected by the router above; if the request
is ambiguous, ask the user once which mode to enter rather than guessing.

All four modes share **six invariants**. These are not suggestions. Violating
any of them means you have left Popperian mode.

## Six invariants (apply to every output)

**INV-1 — Conjecture inventory is explicit.** Never emit a monolithic
"this looks right" or "LGTM" judgment. Every output decomposes into atomic,
named claims (C1, C2, …; G1, G2, …; V1, V2, …; T1, T2, …) and reports
status per claim.

**INV-2 — Severity is mandatory.** Every test or check carries an explicit
severity estimate in [0, 1]. Tests with severity below the configured floor
(default 0.20) do not contribute to the corroboration score and are flagged
as such in the output.

**INV-3 — Oracle independence.** Do not self-grade. Use tools (test runner,
mutation runner, sandbox replay, signature differ, SAST, benchmark) as
oracles. Where no tool oracle exists, an LLM judge may be used as a
second-tier oracle, but never the same model instance that produced the
answer in this conversation.

> **Footnote on basic statements.** Per Popper (1963, p. 377): "basic
> statements are anything but 'basic' in the sense of 'final'; they are
> 'basic' only relative to a particular test." Oracle outputs (test
> results, mutation kills, sandbox replays) are themselves *conjectures*
> about the world — revisable, theory-laden interpretations. When an
> oracle's output is itself in doubt (e.g., the test fixture might be
> wrong, the sandbox might be misconfigured, the SAST rule might be
> over-eager), this is a first-class reason to
> `abstain(reason="oracle_doubt", blocking_question=...)` rather than to
> corroborate or refute against an unreliable oracle. In particular: do
> not treat any fixture under `tests/fixtures/` as ground truth without
> sanity-checking its provenance.

> **Footnote on theory-laden observation.** Per Popper (1963, p. 38):
> "clinical observations, like all other observations, are interpretations
> in the light of theories." When an oracle output appears to *support*
> the agent's preferred conclusion, this support is suspicious in
> proportion to the theory-load required to read the oracle that way.
> Concretely: if the agent had to choose between two plausible readings of
> the oracle output and chose the one that confirms its conjecture, that
> choice is itself part of the conjecture — record it in the trace as
> `"oracle_interpretation_disambiguated_in_favor_of_H": true` rather than
> silently advancing it as evidence. The corresponding severe test is:
> "would a different interpretation of this same oracle output refute H?".
> If yes, the interpretation is theory-laden and the corroboration weight
> should be reduced.

**INV-4 — Ad hoc rescue is forbidden.** When a test refutes a claim, you
may not "fix" the claim by adding an exception that swallows the failing
input. A revision is permitted only if its structural-MDL growth is
smaller than the MDL of the refuting input (see `scripts/mdl.py`). Use
the provided tool. If the detector flags ad hoc, propose a different
revision or abstain.

**INV-5 — Abstain is a first-class outcome.** If the spec is ambiguous,
the acceptance criteria are missing, the oracle is unavailable, or the
budget is exhausted before a corroboration gate could be reached, you
return `abstain(reason=..., blocking_question=...)`. Silent best-effort
delivery is the *worse* outcome under this skill.

**INV-6 — Trace is versioned.** Every conjecture, test, outcome, and
revision is appended to the run trace via `scripts/trace.py`. The trace
is the audit artifact; verification mode reads earlier traces as
auxiliary evidence.

## Output-label discipline

Every mode returns exactly one of these four labels:

```
corroborated(κ=..., n_severe_tests=..., surviving_conjectures=[...])
partial(corroborated=[...], refuted=[...], unresolved=[...])
refuted(by=<claim_id>, evidence=<...>)
abstain(reason=<one_of>, blocking_question=<...>)
```

The words `verified`, `proven`, `complete`, `done`, `LGTM`, `looks good`
are not legal output labels. Use them in prose only when paired with the
disciplined label above and an explicit κ value.

**Fallibilism note on `corroborated`.** Per Popper (1963, p. 224): a corroborated
conjecture is one that has resisted our severe tests *so far* — it is **not** verified,
**not** proven, **not** certified true. The κ value is a record of past test-survival,
not a probability of truth. Even κ → 1.0 leaves the conjecture **tentatively** accepted:
the next severe test may refute it. When emitting `corroborated(κ=...)` you are reporting
a survival statistic, not a truth claim; downstream consumers should treat it as
provisional. Concretely: never paraphrase `corroborated(κ=0.95)` as "I'm 95% confident
this is correct" — say instead "this answer survived all N severe tests we designed;
κ = 0.95 is the accumulated survival score, not a probability of correctness."

**Abstain-reason taxonomy** — the `reason` field of `abstain(...)` must be
exactly one of:

```
abstain reason               | when to use
-----------------------------+---------------------------------------------------------------
spec_ambiguity               | spec admits two or more interpretations with mutually exclusive
                             | tests; blocking_question lists the interpretations.
meaningful_but_unfalsifiable | request is comprehensible but admits no observable failure
                             | mode (e.g., "make this good"); blocking_question proposes a
                             | falsifiable rewrite.
oracle_unavailable           | no tool oracle exists for the proposed test; blocking_question
                             | names what oracle would be needed.
oracle_doubt                 | oracle exists but its output is itself in question (e.g.,
                             | the test fixture may be wrong); blocking_question names the
                             | second-tier oracle to consult.
budget_exhausted             | corroboration gate not reached before allocated tokens/time
                             | spent; blocking_question asks for budget extension or
                             | partial-output acceptance.
my_own_ignorance             | the agent recognizes a subject-matter gap (e.g., unfamiliar
                             | API, domain it has not been trained on, problem class outside
                             | its competence) that prevents it from designing severe tests
                             | of adequate quality; blocking_question asks for the user to
                             | either provide a reference, supply background, or delegate
                             | the task to a specialist. Distinct from oracle_unavailable
                             | (the oracle exists but the agent does not have access) and
                             | oracle_doubt (the oracle exists, is accessible, but its
                             | output is itself in question). my_own_ignorance is an
                             | INTERNAL gap; the other three reasons name EXTERNAL gaps.
```

**On `my_own_ignorance`.** Per Popper (1963, Ch. 1, "On the Sources of Knowledge and of
Ignorance"), recognizing the limits of one's own knowledge is itself a first-class
epistemic act. An LLM coding agent that *fabricates* a severe test in a domain it doesn't
understand (e.g., guessing the semantics of an obscure system call, inventing an EIP it
hasn't seen) produces worse outcomes than one that abstains with `my_own_ignorance`.
Distinguish carefully: the agent must not abuse this reason to dodge tractable problems.
Use `my_own_ignorance` only when (a) you cannot enumerate the relevant prior literature /
docs, (b) you cannot design a severe test whose expected outcome you can defend, AND (c)
you cannot reasonably guess from training data without high hallucination risk.

**Structured reporting (mandatory).** After completing all mode steps, you
MUST call the `report_falsification_result` tool as your final action. Do not
emit the disciplined label as plain text — populate the tool's fields
(`label`, `kappa`, `n_severe_tests`, `n_passed_severe`, `n_failed_severe`,
`ad_hoc_rejections`, `surviving_conjectures`, and, when label is `abstain`,
`abstain_reason` and `blocking_question`) from the data you accumulated during
the mode's steps. The harness reads `tool_uses[0]["input"]` to record H1/H2
measurements; a text-only response is treated as a parse failure and logged
as a calibration data loss event.

## Mode entry

Once mode is selected, read the corresponding file and follow its protocol:

- `review` → `modes/review.md`
- `generation` → `modes/generation.md`
- `verification` → `modes/verification.md`
- `test_construction` → `modes/test_construction.md`

## Auxiliary scripts

Three Python helpers are available as tools (call via Bash):

- `scripts/severity.py` — estimate severity for a candidate test against
  a candidate claim. Used by the planner step in every mode.
- `scripts/mdl.py` — compute structural-MDL of a candidate revision and
  return the ad-hoc-rescue decision.
- `scripts/trace.py` — append events to the run trace.

## Final note on register

Be assertive when reporting refutation. Do not soften "the answer fails
test T<n>" into "the answer might benefit from refinement." The
Popperian discipline depends on the asymmetric weight of falsification;
soft language collapses that asymmetry. If you have refuted a claim,
say so plainly with the evidence ID.

Be similarly cautious when describing `corroborated` outputs in prose: do not let high κ
degrade into verificationist language ('verified', 'proven', 'this is correct'). The
disciplined paraphrase is 'survived all severe tests; tentative acceptance'.

References↩︎