Published molecular docking scores are workflow outputs, not standalone facts: they depend on the receptor, ligand, software, search box, random seed, and preparation choices. We ask whether such claims can be re-executed from their own published records. We introduce MERS-Dock, a 16-field Minimum Executable Reporting Set; a deterministic E0-E4 ladder assigned over audited field states; and a re-execution protocol that benchmarks observed deviations against AutoDock Vina’s run-to-run noise and a 2.0 kcal/mol operational tolerance. In 236 open-access, text-extractable SARS-CoV-2 main-protease docking papers, 8.1% met the prespecified essential-field rule for direct re-execution (E3; 95% CI 5.2-12.2), 44.1% required explicit assumptions (E2), 47.9% were blocked by a missing foundational field (E1), and none reached E4 status; the paper-level distribution is byte-reproducible from the released field-state matrix under the frozen rule. Mean field completeness was 49.1%; the search-box centre was reported by only 33.9% of papers. We validated the automated field-state audit against two independent human reviewers on a 65-paper stratified sample (all directly-executable papers plus sampled blocked and assumption-dependent ones): the reviewers agreed with each other on 92% of field states (pooled Cohen kappa 0.87), and the automated agent that produced the corpus states agreed with humans on the execution-blocking fields (PDB 89%, software 92%, box-centre prevalence 34 vs 32%) while systematically over-calling two non-blocking fields (numeric-result reported 88% by agent vs 37% by humans; validation-redocking 42 vs 12%). Because those over-called fields do not enter the executability rule, the E-class was 68% concordant between agent and humans and, where it differed, human review lowered the executable count — so the low-executability headline is confirmed, not inflated, by human validation. Reporting did not improve over five years (16-field completeness vs publication year Spearman rho = -0.01, p = 0.92; no execution-blocking field improved). As a bounded, mechanistic re-execution, we re-docked one claim from each of 13 fully-reported papers at its reported centre under three boxes — its reported box, a 25 Å default, and a ligand-volume-informed box. A reported box reproduced the published score better than the default (median absolute deviation 1.85 vs 2.51 kcal/mol; exact Wilcoxon signed-rank P = 0.027, nine complete pairs), but the effect is fragile (it hinges on one paper; P = 0.055 without it) and null for on-pocket focused-box papers (P = 0.44); a sensibly ligand-sized box did no better than the default (P = 0.20, an exploratory post-hoc arm), and the two papers no small box could reproduce are exactly those whose reported centre lies 33–38 Å off the pocket. The difference is therefore box-coverage geometry, not box-size disclosure: reproducing a wide blind-docking result requires the entire reported box, centre and size together. E-class acts as an executability gate, not a reproducibility predictor. We release Mpro-DockExec as a traceable measurement layer for digital-library and evidence-synthesis systems deciding what is checkable in published computational claims.

Scope note. This study audits the open-access, text-extractable SARS-CoV-2 main-protease docking literature. It measures reporting executability, validates the audit against independent human reviewers on a stratified sample, and includes a bounded re-execution proof of concept; it does not verify the literature at scale.

1 Significance↩︎

A docking paper can report a score without reporting the workflow needed to regenerate it. MERS-Dock turns that problem into a measurement: 16 execution-relevant fields, a deterministic E0-E4 label, and a deep-audit ledger that records method and result spans at claim level. In 236 open-access Mpro docking papers, nearly half are execution-blocked by at least one missing foundational field, and none meet the E4 robustness-ready rule — a gap that has not narrowed in five years. An independent two-reviewer human validation on a stratified 65-paper sample confirms the audit is reliable (92% inter-reviewer agreement) and, importantly, shows the automated agent’s largest errors fall on fields that do not affect executability, so the low-executability finding survives human scrutiny. The main fix is simple: disclose the search box, configuration, and seed. A bounded re-execution then shows the harder half of the problem is geometric: a published score is regenerable only when the full search box — centre and size together — is recoverable, and for wide blind-docking boxes placed at off-pocket centres no standard reconstruction reaches the pocket, so an unreported box is often irrecoverable. The result is a reusable reporting and verification substrate: it tells a reader or automated agent which claims can be re-executed cleanly before any scientific correctness claim is made.

2 Introduction↩︎

A molecular docking prediction depends jointly on the receptor structure and chain, the ligand identity and protonation state, the software and its version, the search-box geometry, the search effort, the random seed, and the handling of waters and co-factors (Trott and Olson 2009; Eberhardt et al. 2021; Guedes et al. 2013). Change any of these and the score changes. A paper that reports only the score has not made a testable claim; it has published a number with an unknowable provenance. Mpro docking is an ideal place to measure this: the COVID-19 response produced hundreds of Mpro virtual-screening studies in a few years (Mandour et al. 2020; Sisakht et al. 2021; Ambrosio et al. 2023; Peralta-Moreno et al. 2023).

The problem is recognised, and two prior audits have examined the SARS-CoV-2 Mpro docking literature specifically — but each measured validation quality, not re-executability. Llanos and colleagues found that a docking protocol was validated “at some level” in 57.7% of 168 Mpro virtual-screening investigations, with only three complete retrospective analyses (Llanos et al. 2021); a companion critique of 61 Mpro repurposing manuscripts concluded that most do not validate their methodology or their scores against activity (Macip et al. 2021). These establish that the field under-validates. They do not measure whether a paper’s reported record is sufficient to regenerate its docking result, nor do they provide a machine-readable schema or a computable label. More broadly, methodological audits have flagged selective reporting in docking (Jain 2007), and large redocking and scoring-function benchmarks show that performance is strongly conditional on protocol choices that are often left implicit (Flachsenberg et al. 2024; Zajaček et al. 2024; Warren et al. 2006; Cross et al. 2009; Su et al. 2019). Recent guidance now specifies practical reproducibility tips and minimum reporting items for docking (Martis and Teletchea 2025; Kittelson et al. 2026), but it does not provide a deterministic literature-audit instrument or estimate how often published records satisfy those items. Across computational biology, most published workflows cannot be regenerated from the published record (Samuel and Mietchen 2023; Rule et al. 2019), part of a reproducibility problem documented across science (Baker 2016; Sandve et al. 2013; Peng 2011; Stodden et al. 2016). Adjacent computational fields have answered with minimum-documentation standards, including FAIR data principles (Wilkinson et al. 2016), datasheets (Gebru et al. 2021), model cards (Mitchell et al. 2019), ARRIVE (Percie du Sert et al. 2020), and MIAME (Brazma et al. 2001). The issue is now urgent for a new reason: language models extract and propagate published claims at a scale beyond human review (Gartlehner et al. 2024; Konet et al. 2024; Yisha et al. 2026), and a claim that cannot be re-executed is one whose errors cannot be caught before they spread.

What has been missing is an execution-oriented instrument that converts published reporting into a computable label and an empirical distribution. We introduce three coupled components. First, MERS-Dock, a 16-field Minimum Executable Reporting Set for molecular docking, tiered by the role each field plays in execution and designed as an author checklist, reviewer tool, and machine-readable schema for scholarly-document audit. Second, an E0–E4 ladder assigned by a deterministic rule over the audited field-state matrix, which we validate against two independent human reviewers on a 65-paper stratified sample. Third, a re-execution protocol that contextualises score deviations with the docking engine’s measured run-to-run noise floor (median 0.04 kcal/mol) and a declared 2.0 kcal/mol operational tolerance. We field-test the assembled method on 236 open-access, text-extractable SARS-CoV-2 Mpro docking papers, localise which reported fields most often block execution, and release the standard, classifier, derived data, and scripts as Mpro-DockExec. Ranking claims by trustworthiness and re-executing them at scale remain explicitly downstream companion work.

Contributions. (1) MERS-Dock, an execution-oriented Minimum Executable Reporting Set for molecular docking: 16 fields tiered by the role each plays in execution (Tier-1 blocks execution when absent, Tier-2 forces explicit assumptions, Tier-3 enables robustness checks). It is released as a reusable author checklist, reviewer tool, and machine-readable schema for digital-library audit. (2) An E0–E4 executability ladder with a deterministic decision rule assigned over source-span-verified field states, reproducible from the released matrix by construction and validated against two independent human reviewers on a 65-paper stratified sample (inter-reviewer agreement 92%, pooled Cohen kappa 0.87). The validation localises where the automated agent is trustworthy (execution-blocking fields) and where it is not (numeric-result and redocking, both non-blocking), and shows human review does not raise — and slightly lowers — the executable fraction. (3) A five-year temporal analysis showing execution-metadata reporting has not improved (completeness vs year rho = -0.01, p = 0.92), a result robust to a consistent agent bias because such a bias cancels in a trend. (4) A bounded re-execution audit with an engine-noise reference: deviations are read against Vina’s measured self-noise (median 0.04 kcal/mol) under a declared 2.0 kcal/mol operational tolerance. Re-docking one claim from each of 13 fully-reported papers at its reported centre under three boxes (reported, 25 Å default, and a post-hoc ligand-volume box) shows a reported box beats a 25 Å default (paper-level median \(\Delta\) 1.85 vs 2.51 kcal/mol; exact Wilcoxon p = 0.027, nine pairs), but the effect is fragile (drop-one-paper p = 0.055; null for on-pocket focused boxes, p = 0.44) and is not reproduced by a sensibly ligand-sized box (p = 0.20 vs the default) — the difference is box-coverage geometry at off-pocket blind-docking centres, not box-size disclosure, so the full box (centre and size) is execution-critical. The executability label is read as an executability gate, not a reproducibility predictor. (5) A source-span-anchored, deterministically verified audit method, released as Mpro-DockExec, demonstrated as a field-scale readout on 236 open-access Mpro papers. The search-box centre is the most common execution-blocking omission (missing from two-thirds of papers; mean completeness 49.1%). Because the corpus is open-access and text-extractable, the figures are an upper bound for that subset, not a field-wide constant.

3 Results↩︎

3.1 A field-test corpus: open-access Mpro docking papers↩︎

From 1,400 screened records (1,018 after deduplication), 241 had retrievable open-access full text and 236 were confirmed original Mpro docking studies (Table 1; PRISMA 2020 flow, Fig. S1). These papers carry roughly 12,466 ligand-level claim rows, but a single high-throughput screen contributes ~9,600 of them, so we treat that total as a rough upper bound rather than a count of independent docking claims and report paper-level statistics as primary throughout. The corpus is therefore restricted to open-access, text-extractable literature: every audited paper was processed from genuine full text (PMC JATS, publisher open-access, preprints), never from a paywalled or PDF-only route. This scope is a deliberate strength for reliability and an explicit limitation for generality, addressed in the Discussion.

3.2 The method: MERS-Dock, a deterministic executability ladder, and a computable label↩︎

We define MERS-Dock, the Minimum Executable Reporting Set for docking: 16 fields grouped by execution role. Tier-1 fields block execution when absent: the receptor/PDB, the search-box centre, and the docking software. Tier-2 fields do not block execution but force assumptions when absent, and govern the step from assumption-dependent (E2) to directly executable (E3): box size, software version, search effort, receptor and ligand preparation, protonation state, water/ion handling, protein chain, and ligand identity. Tier-3 fields enable robustness checks: random seed, validation redocking, reusable configuration. The deterministic E-class rule below uses exactly this structure: a missing Tier-1 field forces E1, full Tier-1-plus-essential reporting reaches E3, and the disclosed seed and configuration of Tier-3 mark E4.

Each paper is placed on a five-level ladder by a deterministic rule over the verified field states (Fig. 1). Define the blocking set, essential set, and a shorthand for complete reporting as

\[\begin{align} \mathrm{BLOCK}&=\{\mathrm{pdb},\mathrm{grid\_centre},\mathrm{software}\},\\ \mathrm{ESS}&=\{\mathrm{pdb},\mathrm{grid\_centre},\mathrm{grid\_size},\mathrm{software}\}\\ &\quad\cup\{\mathrm{version},\mathrm{receptor\_prep},\mathrm{ligand\_prep}\},\\ R(S)&\equiv\forall f\in S:\;\mathrm{state}(f)=\mathrm{reported}. \end{align}\]

\[E(\mathrm{paper})=\begin{cases} E_1 & \exists f\in \mathrm{BLOCK}:\;\mathrm{state}(f)=\mathrm{missing}\\ E_4 & R(T_1\cup T_2)\wedge \mathrm{seed}\wedge \mathrm{config}\\ E_3 & R(\mathrm{ESS})\wedge \mathrm{ligand\_id}\ne\mathrm{missing}\\ E_2 & \mathrm{otherwise}. \end{cases}\]

E-class describes the reporting record, not the science: an E2 result may be entirely correct, it simply does not uniquely determine its workflow. Scoring is strict — only reported counts; partial is treated as not-reported — and the label is reproducible from the released field states by construction (the 236-paper distribution below re-derives byte-for-byte from the matrix under this rule).

3.3 Applying the standard: which fields are reported, and which block execution↩︎

Applying MERS-Dock across the corpus localizes the reporting gap to specific execution-critical fields rather than to reporting in general. Only two fields were near-universal: docking software (97.9%) and receptor PDB (94.9%). Completeness then fell sharply, and it fell precisely where re-execution depends most (Table 2, Fig. 2). The search-box centre, the field that most often blocks execution, was reported in only 33.9% of papers; both box parameters appeared together in only 27.5% (65/236). The random seed, the cheapest field to deposit and the one that enables exact reproduction, was disclosed by a single paper. Rather than average all sixteen fields into one number (which would weight a PDB identifier and a random seed equally), we read completeness by execution tier, and it collapses monotonically down the tiers that matter for re-execution: the execution-blocking Tier-1 fields are reported 75.6% of the time, the assumption-forcing Tier-2 fields 47.5%, and the robustness Tier-3 fields — seed, configuration, validation — only 12.3% (the overall 16-field mean is 49.1%). The fields that make a result re-executable and checkable are precisely the least reported.

3.4 The executability label across the corpus (E1–E4), and what it localizes to fix↩︎

Assigned by its deterministic rule at the paper level (N = 236, the manuscript’s main endpoint after a logged post hoc unit-of-analysis deviation; Table 3, Fig. 4), the executability ladder resolves a clear gradient: 47.9% of papers are execution-blocked (E1; 95% CI 41.6–54.2), 44.1% are executable only after explicit assumptions (E2; 37.9–50.4), 8.1% are directly re-executable (E3; 5.2–12.2), and none reach full reproducibility (E4; 0/236, 0.0–1.6). No paper was E0, the floor class for a record with no extractable docking claim. The label thus localizes what to fix: directly executable papers are about one in twelve, and nearly half cannot be run at all because a single Tier-1 field is missing. This distribution is a frozen-rule recomputation over the released field states and reproduces exactly.

No paper achieved E4: the one paper that reports a seed omits its configuration. Full reproducibility is achievable with only a few extra disclosed fields, but in this corpus no one achieves it. At the claim-weighted level the distribution is E1 10.6%, E2 86.9%, E3 2.4%, E4 0%, but this is dominated by one high-throughput screen (~77% of all claim rows, ~88% of E2 claims): removing that single paper moves the claim-weighted distribution to E1 45.6% / E2 43.9% / E3 10.5%, close to the paper-level figures. Completeness does not increase with output (Spearman \(\rho\) \(\approx\) 0.04, n.s.), so the claim-weighted concentration reflects one bulk paper, not a reporting tendency; the paper-level distribution is primary.

The directly-executable fraction depends on how strict the E3 bar is. Across an exhaustive sweep of rule variants scored against the reference labels, the execution-blocked fraction is highly stable (E1 47.5–47.9%), whereas the directly-executable fraction ranges 1.3–8.1% across the principal reference-aligned rules — all of which score fields under strict reported-only coding. Relaxing that coding so partially-reported essential fields count as adequate raises E3 to about 30%, so the 8.1% figure is conditional on strict scoring; only the blocked-majority finding (E1 ~48%) is robust to the coding choice.

3.5 Validating the audit: two independent human reviewers on 65 papers↩︎

The audit’s field states are produced by an automated extraction agent, so the load-bearing question is whether those states are correct. We answered it with a prospective, two-reviewer human validation. From the 236-paper corpus we drew a stratified 65-paper sample — every directly-executable (E3) and E4-candidate paper plus a random sample of blocked (E1) and assumption-dependent (E2) papers (sampling manifest released) — and had two independent human reviewers re-assess all 16 MERS-Dock field states for each paper from the paper’s own text, blind to each other, with the extraction agent’s values shown only as a checklist to accept or overrule. Disagreements were adjudicated by a third human. All E-classes in this section are recomputed from the human-adjudicated states with the identical frozen rule used for the corpus (Fig. 5).

The two human reviewers agreed on 92% of the 1,040 field states (pooled Cohen kappa 0.87 — “almost perfect” on the Landis–Koch scale). Per-field kappa was \(\geq\) 0.80 for eleven of sixteen fields, including every execution-blocking one (docking software 1.00, grid-centre 0.86; the receptor-PDB field reads kappa 0.56 only because it is almost always reported — a prevalence artefact for which the prevalence-robust Gwet AC1 is 0.97 at 92% agreement). This is the independent human inter-rater reliability the corpus previously lacked, and it certifies that the field states — not merely the codebook — are reproducibly judged (Table 4b).

The validation also prices the automated agent. On the fields that determine executability, the agent tracked the humans closely: it matched the human on the receptor-PDB 89% of the time and on the docking software 92%, and its box-centre prevalence (34%) essentially equalled the human value (32%). But it systematically over-called two non-execution fields: it marked a usable numeric result as reported in 88% of papers where humans found only 37% (a 52% per-paper disagreement), and marked redocking validation reported in 42% versus the human 12% (Fig. 5B). Crucially, neither field enters the E-class rule — numeric_result is not in the ladder and validation_redocking is a Tier-3 robustness field that affects only the (empty) E4 class. Because the agent’s largest errors miss the executability-determining fields, the E-class agreed between agent and humans on 68% of the 65 papers, and where it differed the net effect of human review was to lower the executable count (agent E3 = 10 \(\rightarrow\) human E3 = 8), with the remaining changes being E1\(\leftrightarrow\)E2 boundary churn that nearly cancels (Fig. 5C). Human validation therefore confirms the headline rather than inflating it: the true executable fraction is, if anything, slightly below the agent’s estimate, and the over-reported fields are ones a downstream user must human-verify regardless (see Limitations).

3.6 Reporting has not improved over five years↩︎

Because a single consistent agent bias cancels in a temporal trend even if it shifts the level, the change in reporting over time is the most bias-robust readout in the corpus. It shows no improvement. Across the 236 papers spanning 2021–2026, 16-field completeness versus publication year has Spearman rho = -0.01 (p = 0.92): flat. None of the execution-blocking fields improved — grid-centre rho = -0.09 (p = 0.15), grid-size -0.08 (p = 0.21), random-seed +0.08 (p = 0.22, and it stays near 0%), reusable-config -0.11 (p = 0.09) — and the flat trend is not a composition artefact, because recent papers are smaller screens, not bulk ones (claims-per-paper vs year rho = -0.35, p < 0.001), and bulk papers report the grid-centre slightly more often, not less. Only two downstream practice fields rose (redocking validation rho = +0.16, p = 0.013; protonation +0.13, p = 0.05). Reporting guidelines and reproducibility checklists published across this window (Martis and Teletchea 2025; Kittelson et al. 2026) have not moved the fields that block re-execution. We state this as flat / no improvement, not a decline: a period table can look like a grid-centre drop (41%\(\rightarrow\)​27%), but the monotone trend test is not significant.

3.7 Re-executing an unreported search box: a coverage-geometry problem, not a box-size effect↩︎

Whether a published score can be regenerated is only cleanly testable on the reported subset, because re-running an under-reported claim requires supplying the very field whose absence defines it. We therefore designed the primary re-execution test as a within-paper paired comparison. A two-arm protocol — the paper’s own reported box versus a standardized 25 Å default box, both at the identical reported centre — was frozen before the runs were inspected (protocol frozen 2026-07-11); a third ligand-volume-informed box (isotropic edge = ligand heavy-atom diameter + 10 Å, clamped to 22.5–40 Å) was added post hoc, after the two-arm result was known, as an exploratory control for whether a sensible size guess recovers the score, and is labelled exploratory throughout. The unit of analysis is one paper. From every open-access paper reporting a complete AutoDock Vina Mpro run — receptor PDB, box centre, box size, an identifiable ligand, and a numeric score, each anchored to a verified source span — we selected one claim before docking and re-docked it under the three conditions, holding receptor, ligand, engine (AutoDock Vina 1.2.7), search effort, and three deterministic seeds (11, 29, 53) fixed across arms. Nineteen papers passed the span-verified eligibility gate. One was excluded when its non-specific “compound 17” label mis-matched an unrelated iodinated contrast agent, and five more had no ligand resolvable to a public structure, leaving thirteen with a docked structure (PubChem, CAS or ZINC identity, each confirmed by InChIKey; ivermectin, returned as a homolog mixture, was docked as its major component). Of these, one produced no bound pose under either box and drops out, so twelve papers contributed a pose in at least one arm and nine produced a pose under both boxes — the nine complete pairs that carry the primary analysis (full claim- and paper-level attrition in Fig. S3). Each paper is thus its own control, which removes the confounding that limits a between-paper comparison.

Three findings together show that the difference between reconstructions is a matter of pocket coverage, not of box size and not of the mere availability of the box. First, the reported box did reproduce the published score better than the 25 Å default (paper-level median absolute deviation 1.85 vs 2.51 kcal/mol; paired median benefit 0.52; exact Wilcoxon signed-rank P = 0.027, nine complete pairs; 20,000-resample bootstrap 95% CI of the benefit -0.005 to 0.96, i.e. touching zero), but that effect is fragile: it hangs on a single paper — dropping the one paper with an off-pocket centre (index 481) moves P to 0.055 — and it vanishes in the clean regime, where for the five papers whose box is focused (\(\leq\) 40 Å) and whose centre lies on the pocket the paired benefit is 0.04 kcal/mol (P = 0.44). Second, the benefit scales with the reported box size (Spearman \(\rho\) = 0.80, P = 0.009): it is confined to four wide blind-docking boxes (60–72 Å) and is absent for focused boxes. Third, and decisively, the exploratory ligand-volume-informed box — the box a re-executor would sensibly choose — did no better than the naive 25 Å default (median difference 0.02 kcal/mol, P = 0.20) and was statistically indistinguishable from the reported box (P = 0.30). Had the effect been about box size, a correctly sized box should have recovered the reported score; it did not. The engine is not the source of the spread (within-paper seed range median 0.058, max 0.82 kcal/mol; AutoDock Vina is near-deterministic).

The mechanism is geometric. The papers whose scores no small box could regenerate are those whose reported centre lies far from the actual binding pocket: measured against the co-crystallised-ligand centroid, paper 59’s reported centre sits 33 Å from the 6LU7 pocket and paper 324’s 38 Å, so a 25 Å cube — or a 30 Å ligand box — placed at those centres (half-width 12–15 Å) physically cannot reach the site and returns no bound pose under either standardized box. (Paper 253, whose centre lies 17 Å off-pocket, is a boundary case: it failed the 25 Å default but did dock under the ligand-volume box in all three seeds, so it is not in the no-pose set.) The McNemar test on the “reported-box-only pose” pattern is not significant (P = 0.25). In these blind-docking studies the reported centre alone does not locate the pocket; only the paper’s own wide box happened to span it. Reproducing such a claim therefore requires the entire reported box geometry — centre and size together — not the centre plus any sensible size guess (Fig. 6). This is a bounded, mechanistic result on a small, difficult subset, not evidence that disclosing the box size per se improves reproduction.

A larger claim-level case series points the same way but is not an independent test. Re-docking every curated ligand claim yielded 48 quality-controlled claims across three targets (6LU7, 6Y2E, 6yb7); those run with a reported or capped box size reproduced to a median of 0.33 kcal/mol (11/12 within tolerance) versus 0.91 for the 36 forced onto a 25 Å default (28/36; Mann–Whitney P = 0.024; Table 5, Fig. S2; per-claim CID/InChIKey scores in Table S1). These 48 claims are clustered in only eight papers and — unlike the paired design — no single paper carried both box conditions, so box availability is confounded with paper identity and the comparison is pseudoreplicated. Collapsed to those eight source papers, the contrast stays directional (paper-level median 0.34 vs 1.15 kcal/mol) but is not significant (exact Mann–Whitney P = 0.20, four vs four papers): the claim-level P = 0.024 is inflated by pseudoreplication. We therefore report the series only as exploratory, direction-consistent support; the paired within-paper design, not this case series, carries the inference. The executability class co-varies with the same signal (E3 median 0.24, n = 6, vs E2 0.83, n = 42) but is confounded with box-size source and rests on an E3 stratum of six claims drawn largely from one paper (P0006; P = 0.07), so it is not an independent predictor. Execution-blocked (E1) papers, lacking a box centre, cannot be reconstructed without inventing one, which we did not do; the re-execution set spans only E2 and E3.

We therefore read the executability label as an executability gate, not a reproducibility score: it identifies which claims can be re-executed at all — the precondition an automated agent needs before deciding whether re-execution is even an applicable verification tool. And the re-execution itself yields a narrower, mechanistic lesson rather than a headline effect: a published docking score is regenerable only when the complete search geometry is recoverable, and for the wide blind-docking boxes at off-pocket centres in our eligible set, neither the disclosed centre nor a sensibly sized box guess suffices — the full box, centre and size together, is execution-critical. This is what justifies MERS-Dock treating both box fields as required; it is not a claim that box-size disclosure alone improves regeneration. A clean re-execution is in any case only ever possible on the reported subset, because re-running an under-reported claim requires supplying the very field whose absence defines it — a boundary intrinsic to the approach.

3.8 The Mpro-DockExec release: standard, classifier, and re-execution scripts as a measurement layer↩︎

We release Mpro-DockExec: the 236-paper corpus with paper-level E-class and 16-field states; a span-free 283-claim table carrying method/result span-verification flags; the 65-paper human-validation package (per-paper dual-reviewer field states, adjudicated states, per-field agreement/kappa, sampling manifest, and reviewer attestation); the reporting standard; the reproducible analysis and re-execution scripts, including the frozen two-arm paired protocol, per-paper paired inputs (receptor/ligand SHA-256, resolved SMILES/InChIKey/CID) and the three-box docking outputs (paired_manifest.csv, input_provenance.csv, paired_runs.csv, volume_runs.csv) and locked statistics with pocket-offset geometry (reexec_v2_paper.csv, reexec_v2_stats.json) alongside the exploratory case series (reproduction_outcomes.csv, self_consistency.csv); and an RO-Crate provenance package. The dataset is a measurement layer for scholarly records: it records what each paper reports and therefore what is checkable. It does not re-run the literature at scale. Copyrighted full texts and verbatim evidence spans are not redistributed; derived metadata and span-verification flags are released.

4 Discussion↩︎

MERS-Dock and the executability ladder give the field the instrument it lacked: a way to measure, not just assert, whether a published docking score can be re-run, and to localize exactly what blocks re-execution. Prior audits of this same Mpro literature measured whether protocols were validated and whether scores predict activity (Llanos et al. 2021; Macip et al. 2021); MERS-Dock instead measures whether the reported record suffices to regenerate the result, and turns that into a computable label with an independent human-validated reliability. Applied here, the instrument shows that the gap is a workflow-description problem, not a data-sharing problem — most papers deposit figures and numbers but omit the parameters that determine the score — and it prices the remedy. The fix is almost free: a single configuration file and one disclosed seed would move most E2 papers to E3 or E4, yet no paper deposits both (E4 = 0/236) and about 97% report neither a reusable configuration (2.5%) nor a seed (0.4%). The standard, the infrastructure and the near-zero cost all exist; the measurement layer we release supplies the one missing piece — a field norm, and a tool to enforce it, that require them. And the gap is not closing on its own: over five years, execution-metadata reporting is flat (rho = -0.01), so a passive expectation that the field will self-correct is not supported by the data.

The audit is human-validated, and its errors are localized. Two independent human reviewers agreed on 92% of field states (pooled kappa 0.87), giving the corpus the independent inter-rater reliability that a codebook-only reproducibility argument cannot. The same validation shows the automated agent is trustworthy exactly where it matters — the execution-blocking fields that drive E-class — and untrustworthy on two non-blocking fields it over-reports (numeric result, redocking validation). This has a practical consequence: the executability headline is safe to read at scale, but any specific claim about whether a paper reports a usable numeric result or a redocking validation must be human-checked, because the agent inflates both. Because the over-called fields sit outside the E-class rule, human adjudication left the executable fraction unchanged-to-lower (E3 10\(\rightarrow\)​8 in the enriched sample), so the low-executability finding is confirmed under human scrutiny.

Reproducing a docking score requires the complete search geometry, not just the reported centre. Re-docking one claim from each fully-reported paper at its reported centre under three boxes shows the reproduction gap is a coverage-geometry effect, not a box-size effect: a ligand-sized box reproduces no better than a 25 Å default (P = 0.20), the reported box’s advantage over the default is fragile (P = 0.027, but 0.055 without a single off-pocket paper, and null on-pocket, P = 0.44), and the papers no small box can reproduce are those whose reported centre lies 33–38 Å off the pocket. Against the engine’s near-determinism (seed range median 0.058 kcal/mol) these failures are geometric, not noise: in wide blind-docking studies only the paper’s own box spanned the pocket, so the full box — centre and size together — must be reported to regenerate the score. A larger 48-claim case series of mostly focused 6LU7 small-molecule claims reproduces tightly (0.33 vs 0.91 kcal/mol) but is pseudoreplicated (eight papers, no paper carrying both conditions; paper-level P = 0.20) and is reported as exploratory. We do not elevate any of this to a claim that the executability class predicts reproducibility: that contrast is confounded with box-size source and rests on an E3 stratum of six claims from largely one paper (p = 0.07). The honest reading is narrower and more useful: the executability label is an executability gate — it tells a reader, or an automated verification agent, whether a claim can be cleanly re-executed at all, the precondition for any re-execution-based check. Re-execution can only ever validate the well-reported subset, because re-running an under-reported claim requires supplying the very field whose absence defines it; that boundary is intrinsic to the approach, not a shortfall of this dataset.

This is a measurement layer, not a runner. The supported claim is precise: the labels are source-span-anchored, human-validated on a stratified sample, and reproducible from the released matrix, while the re-execution proof of concept identifies a box-coverage-geometry effect but does not validate E-class as a reproducibility predictor. The system does not re-execute papers at scale, and the audit tells a reader what is checkable, not what has been checked.

The estimates likely overstate real-world executability. The corpus is open-access and text-extractable. Open-access full-text papers may report methods and artifacts more completely than paywalled or PDF-only literature, and at web scale, with noisier text extraction, field-state accuracy and therefore the E-class would degrade. The 8.1% directly-executable figure should be read as an upper bound for the open, clean-text subset, not as a field-wide constant.

Generalization is partially shown. Because the Tier-1 execution-blocking fields (receptor, box geometry, software, ligand identity) are common to all grid-based docking, the missing-field taxonomy is expected to transfer across targets and engines. The re-execution test spans three Mpro structures, but still one target class and one open-source rerun engine. A distinction matters here: the MERS-Dock reporting audit is engine-agnostic — it measures whether execution-critical fields are reported, which is defined identically for Glide, GOLD, MOE or Vina — whereas the re-execution proof of concept is Vina-specific. We use AutoDock Vina precisely because it is near-deterministic (seed-range median 0.058 kcal/mol), making it a conservative test bed that isolates the box-coverage effect from engine noise; commercial engines with proprietary search and scoring, and their own cross-version drift, would if anything widen the reported-vs-default gap, so our contrast is a lower bound. Re-executing those workflows directly is blocked by licensing and closed source — itself part of the reporting-and-access problem this standard documents. Extension to other targets, engines and method classes (omics, folding, molecular dynamics) requires an adapter-specific field checklist, claim schema, executability ladder, codebook and benchmark for each; that cost is an open question, not a solved one.

Limitations. The corpus is open-access and text-extractable only, biasing it toward more complete reporting. The frozen protocol specified a claim-level primary endpoint, but the scale-up operationalised E-class at paper level because the audited method fields describe a shared workflow; this post hoc deviation is logged, and the claim-weighted distribution is secondary. The human validation covers a stratified 65-paper sample, not all 236, and its E-class distribution is therefore not a re-weighted population estimate but an agent-versus-human agreement measure; we use it to certify the field states and to bound the agent’s per-field error, and we propagate the population headline from the frozen-rule recomputation over the full matrix. The two field-state reviewers are members of the study team; their assessments were independent and blind to each other and adjudicated by a third, but this is an internal, not an external, validation, and reviewer identities are recorded in the released attestation. The automated agent over-reports the numeric-result and redocking-validation fields (agent 88%/42% vs human 37%/12%), so any use of those two fields for a specific paper requires human verification; these fields do not affect the E-class. The re-execution is a bounded proof of concept (one open-source engine, three Mpro targets) whose boundary is structural, not merely statistical: an under-reported claim cannot be re-executed without substituting its missing search box, so a clean re-execution is only possible on the fully-reported subset — itself small, itself a consequence of the reporting gap documented here. The re-execution is small (nine complete pairs) and its reported-box advantage over a default is fragile — significant only across all nine pairs (P = 0.027; bootstrap CI touches zero), non-significant on dropping one off-pocket paper (P = 0.055) and null for on-pocket focused-box papers (P = 0.44); a sensibly ligand-sized box did no better than the default (P = 0.20, a post-hoc exploratory arm), so the effect is box-coverage geometry concentrated in wide blind-docking papers, not a general box-size effect, and we frame it as a bounded mechanistic finding rather than a reproducibility result. The supporting 48-claim case series is pseudoreplicated and non-significant once clustered (P = 0.20). The executability class is confounded with box-size source and its directly-executable stratum is small (n = 6, five from one paper), so it is not read as a reproducibility predictor. For any claim used as quantitative evidence, the parsed box centre and, where available, size warrant per-claim human verification.

5 Methods↩︎

Protocol. A pre-specified protocol, frozen before auditing, fixed eligibility, the 16-field checklist, the E0–E4 codebook (E0 = no extractable docking claim), a claim-level primary endpoint, and human-review rules. During scale-up, E-class was operationalised at paper level because execution-critical method fields apply to the shared workflow rather than to each ligand row. This post hoc unit-of-analysis deviation is logged; paper-level distribution is the manuscript’s main endpoint and the claim-weighted distribution is secondary. Reporting follows PRISMA 2020 (Page et al. 2021). The 2.0 kcal/mol re-execution tolerance came from the later re-execution analysis plan, not the original audit freeze.

Formal framework. Each MERS-Dock field \(f\) has an audited state mapped to a value,

\[s(f)=\begin{cases}1 & \text{reported}\\ 0 & \text{partial or missing}\end{cases}\qquad(1)\]

(“partial” is retained as a distinct field state for the E-class rule, but scored 0 under the strict completeness definition that yields the reported figures.)

with not-assessed fields dropped from every mean (renormalisation). Paper-level reporting completeness is the mean of \(s(f)\) over the assessed fields \(\mathcal{F}_p\),

\[\mathrm{Comp}(p)=\frac{1}{|\mathcal{F}_p|}\sum_{f\in\mathcal{F}_p}s(f).\qquad(2)\]

The executability class \(E(p)\in\{E_0,\dots,E_4\}\) is assigned by the deterministic ladder rule of the Results (the E-class equation) from the verified field states. Its use as a monotone feasibility weight in an execution-aware priority score is developed separately in the companion Paper 2 and is not computed here.

For re-execution, given AutoDock Vina’s within-ligand self-consistency ranges \(\{r_i\}\) (max \(-\) min across seeds), the operational tolerance rule is

\[\tau=\max\!\big(2.0,\;\mathrm{P}_{95}(\{r_i\})\big)\;\text{kcal/mol};\qquad(3)\]

a claim with reported score \(y\) and re-executed score \(\hat{y}\) reproduces iff \(|y-\hat{y}|\le\tau\), and a dock is excluded as a technical failure iff \(\hat{y}>-2\) kcal/mol (no bound pose, reproducible across seeds). Label reliability uses Cohen’s \(\kappa\),

\[\kappa=\frac{p_o-p_e}{1-p_e},\qquad(4)\]

with observed and chance agreement \(p_o,p_e\); every proportion reports a Wilson 95% interval,

\[\frac{\hat{p}+z^2/2n\;\pm\;z\sqrt{\hat{p}(1-\hat{p})/n+z^2/4n^2}}{1+z^2/n},\quad z=1.96,\qquad(5)\]

and executability-class medians are compared with a two-sided Mann–Whitney \(U\) test and 1,000-resample bootstrap confidence intervals.

Search and corpus. PubMed, Europe PMC, OpenAlex and Crossref were queried with stored exact queries and dates (Table 1); only open-access, retrievable full text was admitted.

Audit pipeline. Open-access full text was retrieved (PMC JATS, publisher OA, preprints; no paywalled routes), each field state proposed with a verbatim span by an automated extraction agent, every span verified by a deterministic whitespace-normalised substring check (unsupported spans set to missing), and the E-class assigned by the rule in Results from verified fields only. Completeness uses strict (reported = 1) scoring with Wilson 95% intervals. Paper-level E-class is from the 236-paper audit; claim-level from the 283-claim deep audit. The 236-paper E-class distribution reported here is a fresh recomputation of the frozen rule over the released field-state matrix (an earlier paper_e_class column computed with a non-frozen variant is superseded and not used).

Human field-state validation. To validate the automated field states, we drew a stratified 65-paper sample from the 236-paper corpus: all E3 papers and E4 candidates (exhaustive stratum) plus a random sample of E1 and E2 papers (seed 2026; sampling manifest released). Two reviewers from the study team independently re-assessed all 16 field states for each paper from the paper’s own full text (or abstract where full text was unavailable, recorded as the distinct state unavailable), blind to each other’s labels; the extraction agent’s proposed values were shown only as a checklist to accept or overrule. Inter-reviewer reliability is reported per field as raw agreement and Cohen’s \(\kappa\) over the four-state vocabulary (reported/partial/missing/unavailable), with Gwet’s AC1 alongside \(\kappa\) for high-prevalence fields where \(\kappa\) is deflated by low marginal variance. Disagreements were adjudicated by a third reviewer to produce a consensus state per cell; E-class was then recomputed from the consensus states with the identical frozen rule. Agent accuracy is the per-field disagreement between the agent’s state and the consensus human state. Reviewer independence, blinding, adjudication scope, and reviewer identities are recorded in the released reviewer_attestation.json. This is an internal validation by the study team, not an external review.

Temporal analysis. Publication year was joined to each paper from the corpus metadata. For each field and for 16-field completeness we tested a monotone trend against year with Spearman’s \(\rho\); a period table (year bins) is reported for description only, because the binned pattern can suggest a decline that the monotone test does not support. To rule out a composition artefact we also correlated claims-per-paper with year. A consistent agent extraction bias shifts the level of every year equally and therefore cancels in the trend, which is why the temporal result is reported as the most bias-robust readout even though point prevalences depend on agent accuracy.

Reliability of the E-class label. E-class is assigned by the deterministic codebook rule over the verified field states, so the label is reproducible from the released matrix by construction (the 236-paper distribution re-derives exactly). As a check that the codebook itself is unambiguous, the rule reproduces an independent reference-label set on the 33-paper deep-audit subset at Cohen’s \(\kappa\) = 0.926 (raw 0.97; one borderline disagreement, P0005), clearing the protocol’s \(\geq\) 0.70 gate; a language model applying the same codebook reaches \(\kappa\) = 0.736 and the same model judging without the codebook only 0.075, i.e. the explicit codebook, not the rater, carries the label (Table 4). This codebook-reproducibility check is distinct from, and secondary to, the field-state human validation above, which is the primary reliability evidence.

Re-execution — paired primary. A conda-forge environment (AutoDock Vina 1.2.7, RDKit, Meeko) was used throughout. The primary re-execution is a paper-level paired test whose two arms — reported box and 25 Å default, both at the reported centre — were frozen before runs were inspected (protocol frozen 2026-07-11); a third ligand-volume arm was added post hoc after the two-arm result was seen and is reported as exploratory. An eligible paper reports, with verified method and result source spans, a complete Mpro AutoDock Vina run: receptor PDB, three box-centre coordinates, three box dimensions, an identifiable ligand, and a numeric score. One claim was selected per paper before docking (highlighted or rank-1 ligand; otherwise the first named ligand with a resolvable public identifier and explicit score), independent of any re-execution result; nineteen deep-audit papers met this bar. Ligand identity was resolved to a public structure via PubChem (name or CID), CAS or ZINC with a CACTUS fallback, and confirmed by an RDKit-computed InChIKey; one paper was excluded because its ligand label (“compound 17”) mis-matched an unrelated triiodinated contrast agent, and ivermectin, returned by PubChem as a defined homolog mixture, was docked as its largest fragment — leaving thirteen papers with a docked structure. Receptors (chain A, waters and any co-crystallised inhibitor removed, alternate locations resolved to conformer A) were prepared with Meeko (Gasteiger charges); ligands were embedded with RDKit ETKDGv3 (seed 2026). Each selected claim was re-docked under three conditions at the identical reported centre — its reported box (capped at 60 Å per axis, flagged), a standardized 25 Å default box, and a ligand-volume box (isotropic edge = ligand heavy-atom diameter + 10 Å, clamped 22.5–40 Å) — with receptor, ligand, engine, search effort and three deterministic seeds (11, 29, 53) held identical across arms. The primary outcome is the per-paper, per-arm median absolute deviation from the reported score across seeds; each paired contrast is tested by an exact two-sided Wilcoxon signed-rank test with a 20,000-resample paper-bootstrap 95% interval, which takes precedence over the p-value. A run is invalid if Vina returns no pose or a non-negative score, if the ligand or receptor cannot be prepared, or if a required span fails verification; such failures remain in the flow table and are not converted into divergent reproductions. Robustness of the reported-vs-default contrast was assessed by a leave-one-paper-out sensitivity, a focused-box subgroup (reported box \(\leq\) 40 Å with an on-pocket centre), the Spearman correlation of the paired benefit with reported box size, and — as the decisive post-hoc control — the two paired contrasts against the ligand-volume box; the pocket offset per paper is the distance from the reported centre to the centroid of the receptor’s co-crystallised ligand. Across the three seeds Vina’s within-paper score range was median 0.058 and at most 0.82 kcal/mol. The 2.0 kcal/mol tolerance is a fixed declared threshold on the conventional docking scoring-error scale; the measured self-noise (P95 0.32, max 0.82 kcal/mol) never approaches it, so the tolerance does not vary and is not engine-noise-calibrated in practice. Receptor and ligand PDBQT SHA-256 hashes, resolved SMILES/InChIKey/CID, and the exact Vina commands are released.

Re-execution — exploratory case series. As non-independent supporting evidence we also re-docked every curated ligand claim across 6LU7, 6Y2E and 6yb7, yielding 48 quality-controlled claims after excluding three failed-to-bind docks (re-executed score > \(-\)​2 kcal/mol, reproducible across seeds), eight no-bound-pose claims (including an entire 5R7Y paper), and ligands unresolvable to a PubChem CID. Box centres were span-parsed for all 48 claims (Table S3); box sizes were recoverable for 12 (5 reported, 7 blind-docking boxes capped at 40 Å), the remaining 36 (75%) defaulting to a 25 Å cube. Because these 48 claims cluster in eight papers and no paper carried both box conditions, box availability is confounded with paper identity; the stratified Mann–Whitney contrast (Table 5, Fig. S2) and the E-class comparison are reported as exploratory. A sensitivity analysis counting the three failed-to-bind docks as non-reproductions is in Table S5.

Provenance. Typed PROV-compatible objects and an RO-Crate 1.1 package (Groth and Moreau 2013; RO-Crate Community, n.d.) accompany the release.

6 Tables↩︎

Table 1. Corpus funnel (open-access only).

Stage	N
Records screened	1,400
After deduplication	1,018
Open-access full text retrieved	241
Confirmed original Mpro docking studies	236
Deep-audit subset (span-verified)	33 papers / 283 claims
Human-validation sample (stratified, dual-reviewer)	65 papers

Table 2. Reporting completeness, 16 MERS-Dock fields, N = 236.

Numeric-result and validation/redocking prevalences are agent-derived and, per the 65-paper human validation (Table 4b), are over-reported by the agent (human values 37% and 12%); the execution-blocking fields that drive E-class match human review.

Table 3. Executability classes, criteria, and distribution (N = 236).

Recomputed from the released field-state matrix under the frozen MERS-Dock v1.1 rule; reproduces exactly.

Table 4a. Codebook reproducibility of the E-class label (N = 33 deep-audit papers). A secondary check that the codebook is unambiguous; the primary reliability evidence is the human field-state validation (Table 4b).

Table 4b. Human field-state validation (N = 65 stratified papers; two independent human reviewers). Primary reliability evidence.

Table 5. Bounded three-box re-execution (paper-level; AutoDock Vina 1.2.7; each claim re-docked at its reported centre under its reported box, a 25 Å default box, and a post-hoc ligand-volume box; 3 seeds). Thirteen fully-reported papers docked; nine posed under both reported and default boxes (complete pairs; full attrition Fig. S3, Table S6).

The reported box beats a 25 Å default only marginally and fragilely (Wilcoxon P = 0.027, but 0.055 without one off-pocket paper and null on-pocket), and a sensibly sized ligand-volume box does no better than the default (P = 0.20) while being indistinguishable from the reported box (P = 0.30): the difference is box-coverage geometry at off-pocket blind-docking centres, not box size. Per-paper outcomes are in the released reexec_v2_paper.csv (Table S6). Exploratory case series (non-independent): re-docking every curated claim (48 QC claims in eight papers, mostly focused 6LU7 small molecules) reproduces tightly (reported/capped box 0.33 vs 25 Å default 0.91 kcal/mol; Mann–Whitney P = 0.024) but is pseudoreplicated and, once clustered to its eight source papers, non-significant (0.34 vs 1.15 kcal/mol; exact Mann–Whitney P = 0.20, 4 vs 4); Fig. S2, Table S5.

Table 6. AutoDock Vina self-consistency (noise floor; 15 ligands \(\times\) 3 seeds).

Within-ligand score range	kcal/mol
Median	0.04
Mean	0.10
Maximum	0.38
Operational reproduction tolerance	2.0

7 Figures↩︎

Fig. 1 MERS-Dock tiers to E-class ladder; Fig. 2 reporting completeness; Fig. 3 missing-parameter matrix; Fig. 4 executability distribution; Fig. 5 65-paper dual-human validation; Fig. 6 three-box re-execution (coverage geometry, paper-level). Source: analysis/render_*; paired re-execution from analysis/paired_validation/; validation figure from analysis/render_fig5_validation.py over _dev_handoff/validated_wave1/; case-series (Fig. S2) from 09_scoring_benchmark/runner/out/.

8 Data, Code & Materials↩︎

The Mpro-DockExec release contains paper-level E-class and 16-field states for 236 papers; a span-free 283-claim table with method/result verification flags; the 65-paper human-validation package (dual-reviewer and adjudicated field states, per-field agreement/\(\kappa\), sampling manifest, reviewer attestation); the MERS-Dock guideline; the re-execution environment and scripts; per-ligand CID/InChIKey provenance; reproduction_outcomes.csv; and self_consistency.csv. The release is available at https://github.com/giapha/mpro-dockexec. E-class is reproducible from the released field states (rerun the deterministic rule); field-state reliability is the released dual-human validation. Copyrighted full texts and verbatim evidence spans are not redistributed. Supplementary Information (Figures S1–S3; Tables S1–S6) accompanies this manuscript and is generated from the released derived data.

9 Ethics, Competing Interests, Funding↩︎

No human participants or individual-level data. Author contributions & validation reviewers. The two independent human validators of the field-state audit are members of the study team; their identities, independence, blinding, and adjudication roles are recorded in the released reviewer_attestation.json. Competing interest: V.G. (Vincent Giap) is a scientific cofounder of NewScience, a company developing verification- and executability-aware tools for computational scientific claims; all authors are affiliated with NewScience Lab. The MERS-Dock standard and executability ladder reported here could inform such tools. The apparatus is named generically, the NewScience platform is not a study object, and the derived data, code and reporting standard are released openly so the findings can be assessed independently of any commercial interest. Funding: no external funding was declared for this study.

10 References↩︎