ScratchWorld: Evaluating If World Models Compute Executable Consequences
June 30, 2026
Abstract
World-model evaluations often score a predicted future by overlap with a target state or observation. In sparse-change worlds, this can turn copied persistent state into apparent accuracy. We introduce ScratchWorld, an offline diagnostic benchmark that treats Scratch projects as executable worlds and uses a pinned Scratch VM to produce replay-verified transitions, hidden variables, causal traces, and counterfactual outcomes. ScratchWorld evaluates next-state prediction, long-horizon tracking, causal event attribution, and counterfactual prediction; each replay-verified target can be presented under raw-program, structured-state, natural-language, or rendered input modalities, and our experiments use the structured-state condition. Its primary state metric is value-aware changed-field \(F_1\), which gives credit only for the changed field and its executed value. In a \(659\)-example release, seven prompted language/reasoning models reach at most \(13.8\%\) value-aware \(F_1\) in a state-only partial-observation stress test. A same-instance copy diagnostic makes the overlap confound concrete: copying the input state reaches \(98.0\%\) implied full-state field accuracy and \(0.0\%\) changed-field \(F_1\), with the largest inflation on real projects. Auxiliary diagnostics separate hidden-state rollout drift, intervention sensitivity, causal attribution, and perturbation robustness. Across these settings, models often react to actions or interventions without following the executable rule that determines the changed value.
1 Introduction↩︎
A useful world model should predict how an environment changes when an agent acts. For planning, control, and interactive decision making, this requires more than matching a future image or preserving a plausible state: the model must identify which state variables change, which executable rule produces the transition, and how the outcome would differ under an intervention [1], [2]. We study this capability in program-defined worlds whose dynamics are executable, inspectable, and replayable.
Current evaluations only partially isolate this capability. Video benchmarks provide rich observations, but actions, hidden state, and inexpensive counterfactuals are often unavailable or weakly specified [3], [4]. Robotics and embodied-agent benchmarks provide action-conditioned trajectories, but data collection and controlled perturbation are costly [5], [6]. Game benchmarks expose interaction, but evaluation is usually through reward, task success, or observation quality [7]. These signals can obscure the source of a world-model error. A model may preserve most of the state while missing the action-induced change, lose a hidden variable that controls future behavior, identify that an event occurred but not the rule chain that caused it, or react to an intervention without computing its exact consequence.
We introduce ScratchWorld, an execution-grounded diagnostic benchmark for action-conditioned world-model evaluation. ScratchWorld uses Scratch projects as executable visual worlds. In these worlds, sprites define visible entities; variables, lists, clones, broadcasts, and active scripts define hidden state; event handlers define transition rules; and the renderer maps executable state to observation. Given a program, a state, and an action window, ScratchWorld obtains the next state by replaying execution on a pinned Scratch VM. Given a target event, it extracts the executed runtime trace that produced the event. Given a counterfactual intervention, it replays the modified execution and records the resulting outcome. The benchmark therefore turns each example into a replay-verified action–state–outcome fact rather than a surface-level observation match.
This design targets a central confound in world-model evaluation: overlap with a future state can reward persistence instead of transition prediction. Scratch states contain many fields, most of which remain unchanged after a local action. A model that copies the input state may therefore appear accurate under full-state overlap while failing to predict the executable change. ScratchWorld instead scores state transitions by value-aware changed-field \(F_1\) (\(F_1^{\mathrm{VA}}\)), which gives credit only when a model identifies a changed field and predicts its executed value within tolerance. Presence-only changed-field \(F_1\) separates field selection from value computation, while full-state field accuracy is used as a copy-sensitive diagnostic rather than the primary measure of transition prediction.
Experiments show that executable state-change prediction remains difficult for current prompted language and reasoning models. On a shared next-state subset under structured state-only evidence, seven models achieve only \(7.5\)–\(13.8\%\) value-aware changed-field \(F_1\) (Table 3). Presence-only scores are substantially higher than value-aware scores, indicating that models often identify an affected field without computing the executed value. The difficulty is especially pronounced on real Scratch projects: real-corpus \(F_1^{\mathrm{VA}}\) is only \(3.8\)–\(10.1\%\), compared with \(20.8\)–\(27.1\%\) on synthetic mechanism families.
The metric changes what looks successful. On the same next-state instances, a no-change copy prior obtains \(98.0\%\) implied full-state field accuracy but \(0.0\%\) changed-field \(F_1\) (Table 4). This inflation is largest on real projects, where large persistent states surround small action-conditioned change sets. Additional diagnostics expose the same issue from different angles: hidden-state accuracy collapses under long-horizon rollout, models show intervention sensitivity without exact counterfactual consequence prediction, event detection is much easier than rule-chain attribution, and perturbation robustness is uneven (Table 5). Across these settings, models often detect that an action or intervention matters, but fail to follow the executable rule that determines the changed value.
We make three contributions:
We introduce ScratchWorld, an execution-grounded benchmark construction pipeline that turns Scratch programs into replay-verified action–state–outcome instances with visible state, hidden state, causal events, counterfactual interventions, semantic perturbations, and rendered observations.
We define value-aware changed-field evaluation for action-conditioned state-change prediction, and use a same-instance copy diagnostic to show that permissive full-state overlap can reward state echoing rather than executable transition prediction.
We evaluate seven prompted language/reasoning models, revealing low changed-value accuracy, a large real/synthetic gap, hidden-state rollout drift, intervention sensitivity without exact consequence prediction, and event detection without reliable rule-chain attribution.
2 Motivating Example↩︎
Consider a Scratch game with a player sprite, a coin sprite, a hidden score flag, a clone counter, and a broadcast handler. The rendered frame exposes the visible sprites, but the executable state also contains variables, list entries, clone metadata, pending scripts, broadcasts, and event-handler state. A single key press may move the player, trigger a collision, broadcast a collection event, hide a coin clone, and update a hidden counter. Most fields stay fixed. A copied state may preserve the visual scene and almost every persistent field while missing the transition caused by the action.
This transition exposes the scoring problem. Suppose VM replay shows that an anonymized sprite coordinate changes from \(x{=}10\) to \(x{=}0\). A full-state prediction that repeats the input receives credit for unchanged fields while missing the executed update. A delta prediction that names the sprite’s \(x\) field and predicts \(x{=}5\) identifies the affected field but fails the value computation. Only the prediction \(x{=}0\) captures both the changed field and the executed value. Full-state overlap, presence-only changed-field \(F_1\), and value-aware changed-field \(F_1\) separate these three behaviors.
The pattern is common in the release. Release inputs average \(235.8\) fields, while next-state target deltas average \(8.5\) changed fields. A full-state metric mostly measures persistent-state retention. A changed-field metric asks whether the model identified the action’s consequences. A value-aware changed-field metric asks whether it computed those consequences. In sparse-change worlds, the easiest way to look accurate may be to predict that almost nothing changed.
The same transition supports causal and counterfactual evaluation. If the coordinate update came from a broadcast chain, event detection asks whether the event occurred; causal attribution asks for the executed handler sequence that produced the change. If an intervention changes a threshold, starting coordinate, or hidden counter, the target comes from replaying the modified execution. The motivating question is not whether a model can produce a plausible future state, but whether its prediction follows the executable rules that generated it.
3 Related Work↩︎
3.0.0.1 World models and action-conditioned dynamics.
World models learn predictive representations of environment dynamics for imagination, planning, and control. Classic and recent systems support behavior through learned dynamics, from latent-dynamics control [1], [2], [8], [9] to controllable generative environments learned from video [10]. Standard evaluations emphasize reward, task success, or observation prediction. ScratchWorld focuses on the transition structure behind those outcomes: hidden variables, executable rules, and counterfactual dependencies.
3.0.0.2 Physical, causal, and video-generation benchmarks.
Many benchmarks probe physical and causal reasoning. IntPhys and PHYRE evaluate intuitive physics and physical problem solving [11], [12]; CLEVRER separates descriptive, explanatory, predictive, and counterfactual reasoning over collision videos [3]. For generative video, VBench and VideoPhy move from perceptual quality toward physical consistency [4], [13]. Recent world-model evaluations make related points: World-in-World emphasizes closed-loop controllability [14], and VisPhyWorld converts predictions into executable simulator code for physics-based checking [15]. ScratchWorld supplies the executable world directly and tests action-conditioned state prediction, event attribution, and controlled interventions.
3.0.0.3 Interactive environments, games, and embodied-agent benchmarks.
Interactive benchmarks couple actions with consequences. Procgen evaluates generalization across procedurally generated games [7]; Habitat and ALFRED evaluate embodied interaction and instruction following [6], [16]; RoboNet aggregates large-scale robot interaction data [5]. These settings expose action-conditioned feedback through reward, success rate, or task completion. ScratchWorld keeps the transition mechanism inspectable: actions, broadcasts, variables, event handlers, clone states, and field-level updates are logged and checked against the executed program.
3.0.0.4 Scratch, block-based programming, and executable media.
Scratch is a block-based, sprite-based, event-driven programming environment [17]. Prior Scratch work [18]–[22] uses the same environment to study different questions. Testing tools such as Whisker and model-based Scratch testing execute projects to check program behavior against abstract state machines [23], [24]. The NAACL benchmark studies multimodal visual-programming reasoning with Scratch-style questions [25]. The newer repair benchmark evaluates debugging and patching of buggy Scratch programs using bug descriptions, fixes, executable tests, block-level edit constraints, and multimedia assets [26]. Another work evaluates multimodal GUI agents that construct Scratch programs through drag-and-drop [27]. ScratchWorld uses a different prediction unit: an existing executable world, a current state, an action window, and a VM-derived next state, causal chain, counterfactual outcome, or rollout target. It treats Scratch as a world-model substrate with program-derived ground truth for visible outcomes, hidden state, causal chains, counterfactual labels, and executable state-change metrics.
4 Benchmark Construction↩︎
ScratchWorld turns Scratch projects into execution-verified world-model instances through six stages (Figure 1). Every action–state–outcome label used for evaluation comes from VM replay.
We begin with a local corpus of Scratch project files. Public projects enter the benchmark only as filtered derived sources. Each project contains sprites, blocks, variables, costumes, sounds, and metadata. We retain projects with meaningful state-changing behavior, including games, simulations, animations, quizzes, drawing programs, and interactive stories. We remove projects that fail to execute, depend on unsupported VM extensions, contain missing assets, terminate immediately, or behave unstably under repeated controlled replay. We also remove near-duplicates using structural similarity (block graphs, sprite layouts, event-handler patterns) and surface similarity (titles, names, and visual assets).
For each retained project, we instrument the Scratch runtime to log structured execution traces without changing program behavior. Visible state includes sprite positions, directions, sizes, costumes, visibility flags, and layer order. Hidden state
includes global and local variables, lists, clone states, active scripts, broadcasts, and control-flow status. User actions such as green-flag events, key presses, mouse clicks, and sprite clicks are injected through a controlled interface; broadcasts,
collisions, threshold branches, and script activations are recorded from the runtime trace. All reported labels use a pinned scratch-vm@5.0.300 runtime. An instance is retained only if replay reproduces the recorded target under the same
project, action sequence, and seed.
Each instrumented run yields a trace \[\tau = \{(s_t, a_t, o_t, e_t)\}_{t=0}^{T},\] where \(s_t\) is structured state, \(a_t\) is the user action or runtime event, \(o_t\) is the rendered observation, and \(e_t\) records executed blocks, triggered events, and active scripts. The execution record supports causal attribution by linking state changes to program rules.
From traces we extract one-step transitions \((s_t,a_t,s_{t+1})\), long-horizon transitions \((s_t,a_{t:t+H-1},s_{t+H})\), causal event windows, and counterfactual pairs. Empty-delta outcomes are valid executable labels: for state-delta tasks, the correct prediction is an empty change set when replay shows that no listed field changed. We retain no-change outcomes in counterfactual and perturbation diagnostics because invariance under a controlled intervention is itself a meaningful target. For causal attribution, ordered runtime traces link a target event to the event handler, condition, broadcast, or script activation that produced the observed change. We calibrate the program-derived causal chains against \(200\) human-annotated instances, reporting agreement between human-reconstructed and VM-derived chains so that the automatically extracted labels are validated against human judgement. Negative examples are retained when the event does not occur; the correct behavior is then to predict non-occurrence and abstain from inventing a rule chain. Instances whose causal chains cannot be recovered reliably under replay are excluded.
For counterfactual instances, we construct paired executions that differ by one controlled intervention. Interventions may replace an action, modify an initial variable, move a sprite, or change a threshold or event condition. We replay the intervened execution on the same VM and score against the resulting counterfactual state. Each intervened instance receives its own replay-verified target.
To audit surface memorization, ScratchWorld constructs semantic perturbations that modify names, coordinates, thresholds, costumes, or independent script order and preserve the intended evaluation structure. The modified project is re-executed and re-verified, giving each perturbation its own target. The paired sets compare performance on original and perturbed variants from the same source project.
Each instance is packaged under five input modalities: raw project JSON, structured state tables, natural-language summaries, rendered screenshots, and short execution clips. The same replay target can be presented under any modality, so performance can be disaggregated by input modality. This design separates missing information from errors in simulating executable dynamics.
5 Evaluation↩︎
ScratchWorld evaluates executable dynamics through four tasks: next-state prediction, causal event attribution, counterfactual prediction, and long-horizon state tracking. The tasks move from local transition modeling to causal and temporal reasoning. Throughout, \(\mathcal{W}\) denotes the executable project, and the evidence condition specifies which part of that project the model can inspect.
5.0.0.1 Task 1: Next-state prediction.
Given a Scratch world \(\mathcal{W}\), current state \(s_t\), action or event window \(a_t\), and optional visual observation \(o_t\), the model predicts the state after execution: \[\hat{s}_{t+1} = f(\mathcal{W}, s_t, a_t, o_t).\] The prediction is compared with the execution-derived ground truth \(s_{t+1}\). Outputs take the form of a full structured state or a state delta \(\Delta s_t = s_{t+1} - s_t\); an empty delta is correct only when replay shows no listed field changed.
5.0.0.2 Changed-field scoring.
The primary metric is value-aware changed-field \(F_1\) (\(F_1^{\mathrm{VA}}\)). A prediction receives credit when it names a changed field and predicts its value within tolerance. Presence-only changed-field \(F_1\) credits the field name alone. Full-state field accuracy measures overlap with the whole state and is used as a copy-sensitive diagnostic. Field identities are canonicalized as sprite, stage, variable, list, clone, or event paths. Numeric comparisons use tolerance \(10^{-3}\) for sprite kinematics and \(10^{-6}\) otherwise; string, costume, Boolean, and list-head values require exact match. Outputs that fail schema parsing are scored as incorrect.
5.0.0.3 Task 2: Causal event attribution.
Given a Scratch world, an action sequence \(a_{t:t+H}\), and a target event \(q\), the model predicts whether the event occurs and identifies the responsible rule chain: \[(\hat{y}, \hat{c}) = f(\mathcal{W}, s_t, a_{t:t+H}, q),\] where \(\hat{y}\) denotes event occurrence and \(\hat{c}\) denotes the predicted causal chain. The ground truth \((y,c)\) is extracted from the runtime trace and executed block sequence. We evaluate event prediction using accuracy and \(F_1\), causal attribution using exact rule-chain match calibrated against \(200\) human-annotated instances, and no-hallucination, the rate at which a model abstains from inventing a causal chain when the event does not occur. The task separates event detection from causal attribution.
5.0.0.4 Task 3: Counterfactual prediction.
Given an observed trajectory and an intervention \(\iota\), the model predicts the outcome under the modified execution: \[\hat{s}^{\iota}_{t+H} = f(\mathcal{W}, s_t, a_{t:t+H}, \iota).\] The prediction is compared against the counterfactual ground truth \(s^{\iota}_{t+H}\) obtained by replaying the intervened execution. Interventions may replace an action, move a sprite, change a threshold, modify a variable value, or alter an event condition. We report intervention-sensitivity accuracy (whether the prediction changes when the outcome should), the over-invariance rate (predicting no change when the intervention has an effect), intervention \(\Delta\)-match (the predicted change equals the true change), and counterfactual changed-field \(F_1\). The task measures consequence prediction under controlled changes.
5.0.0.5 Task 4: Long-horizon state tracking.
Given an initial state \(s_t\) and an action sequence \((a_t,\ldots,a_{t+H-1})\), the model predicts the final state without intermediate ground truth: \[\hat{s}_{t+H} = f(\mathcal{W}, s_t, a_{t:t+H-1}).\] We evaluate horizons such as \(H \in \{2,5,10,15,20\}\). Metrics include full-state field accuracy, hidden-state accuracy, exact full-state match, and degradation as \(H\) increases. The task exposes temporal drift when hidden variables, delayed broadcasts, clone states, or repeated collisions affect later behavior.
5.0.0.6 Input modalities.
Input modalities define how the executable world is presented to the model. ScratchWorld supports five: raw project JSON, structured state tables, natural-language summaries, rendered screenshots, and short execution clips. The experiments below report primarily on the structured-state modality, under which the current state and action are given while the full program rules are hidden, making next-state prediction a partial-observation test. Because the same replay target can be presented under any modality, results can be disaggregated by input modality.
5.0.0.7 Splits and leakage control.
We partition the benchmark at the project level into development and test splits. All instances derived from the same Scratch project, including perturbed variants, remain in the same split. This prevents leakage from evaluating on a surface-modified version of a project seen during development. Test results are reported separately on original and perturbed instances.
5.0.0.8 Memorization audit.
To audit surface familiarity, we compare performance on original instances and semantic perturbations. Let \(M_{\mathrm{orig}}\) and \(M_{\mathrm{pert}}\) denote scores on original and perturbed instances. The perturbation gap is \[\Delta_{\mathrm{pert}} = M_{\mathrm{orig}} - M_{\mathrm{pert}}.\] A large positive gap indicates reliance on surface features such as names, layouts, or visual motifs. A small or negative gap argues against surface familiarity as the main driver under this perturbation protocol.
6 Experiments↩︎
We evaluate prompted models on action-conditioned executable transitions. The common crossed comparison is a state-only partial-observation stress test: seven models answer the same \(265\) next-state instances from structured state and action evidence. The remaining diagnostics probe temporal drift, causal attribution, intervention effects, and surface robustness.
We evaluate seven contemporary language/reasoning models: GPT-5.5, GPT-5.4, DeepSeek-V4-Pro, Qwen3.7-Max, MiMo-V2.5-Pro, MiMo-V2.5, and MiMo-V2-Omni. We use them as prompted predictors for action-conditioned world-model tasks; trained control-oriented dynamics models are a natural next use of the benchmark. For providers with an explicit reasoning-effort control, we use the strongest available setting; for the rest, we use the standard provider API configuration with a large output budget. Provider-run provenance and limitations are summarized in the supplementary material. Prompts never contain the target, gold label, or verification metadata; unparseable outputs are scored as incorrect.
The unified test set contains \(659\) replay-verified examples: \(291\) real-corpus examples derived from public Scratch projects and \(368\) synthetic examples spanning nine controlled mechanism families. It contains \(285\) next-state instances with state-delta targets and \(374\) long-horizon instances with full final-state targets. Empty deltas are valid executable outcomes: \(53\) next-state labels contain no changed field, and an empty predicted change set is correct only for those cases. Changed-field \(F_1\) measures positive changed-field correctness; scorer reports also record no-op accuracy. Counterfactual and perturbation diagnostics retain no-change outcomes because invariance under an intervention is a meaningful label. The main next-state comparison uses the \(n{=}265\) subset with outputs from all seven models.
Tables 1 and 2 summarize task coverage and release statistics.
| Task | Models | What it probes |
|---|---|---|
| Next-state | \(7\) | Changed-field transition prediction |
| Perturbation | \(7\) | Surface-perturbation robustness |
| Causal | \(7\) | Event vs.rule-chain attribution |
| Counterfactual | \(7\) | Intervention-consequence prediction |
| Rollout | \(4\) | Long-horizon hidden-state drift |
5pt
| Aspect | Statistic | Value |
|---|---|---|
| Examples | Total examples | examples: 285 next-state and 374 long-horizon. |
| Projects | Anonymized project hashes | projects: 31 real-corpus and 122 synthetic. |
| Source | Example-level split | real-corpus examples and 368 synthetic examples. |
| Synthetic families | Program mechanisms | families covering motion, backdrop/costume changes, broadcast, clone, collision, delay, list operations, multi-change transitions, and hidden variables. |
| Action triggers | Event types | Green flag, key down/up, and broadcast events. |
| Project density | Instances per project | Mean 4.3, median 3, maximum 35. |
| Input state size | State fields per input | Mean 235.8, median 27, maximum 2621. |
| Observed transitions | Changed fields per next-state example | Mean 8.5, median 3, maximum 268. |
| Hidden-state changes | Hidden fields per next-state delta | Mean 2.0 hidden changed fields. |
3pt
The experiment tables use the metrics defined in the Evaluation section. For next-state prediction, the primary metric is value-aware changed-field \(F_1\) (\(F_1^{\mathrm{VA}}\)). We also report presence-only changed-field \(F_1\) (\(F_1^{\mathrm{pres}}\)). For long-horizon prediction, full-state field accuracy is a diagnostic for copy-sensitive overlap.
Table 3 reports the state-only next-state task. Models receive the current structured state and action/event window. Across the seven-model common set, value-aware \(F_1\) ranges from \(7.5\) to \(13.8\%\). Presence-only scores are roughly twice as high as value-aware scores (mean \(22.1\%\) vs.\(10.4\%\)), showing that field selection is easier than value computation. The real/synthetic split shows where this gap matters. Synthetic scores cluster between \(20.8\) and \(27.1\%\) on controlled mechanism families. Real-corpus scores stay much lower (\(3.8\)–\(10.1\%\)), where large persistent states, hidden variables, clone effects, broadcasts, and project-specific logic interact in the same transition.
| All (\(n{=}265\)) | Real | Synthetic | |||
|---|---|---|---|---|---|
| 2-4(lr)5-5(lr)6-6 Model | VA \(F_1\) | Pres. \(F_1\) | Hidden Acc. | VA \(F_1\) | VA \(F_1\) |
| GPT-5.4 | 13.8 | 27.5 | 13.8 | 10.1 | 25.2 |
| Qwen3.7-Max | 12.5 | 22.7 | 14.1 | 8.2 | 27.1 |
| GPT-5.5 | 11.9 | 27.5 | 16.1 | 9.6 | 22.8 |
| DeepSeek-V4-Pro | 10.7 | 22.2 | 12.8 | 7.8 | 22.1 |
| MiMo-V2-Omni | 8.3 | 18.3 | 11.6 | 4.2 | 22.8 |
| MiMo-V2.5-Pro | 8.0 | 19.2 | 9.5 | 4.0 | 20.8 |
| MiMo-V2.5 | 7.5 | 16.9 | 10.6 | 3.8 | 22.2 |
9pt
Table 4 makes the copy shortcut explicit. On the same \(285\) next-state examples, copying the input state as the full next state obtains \(98.0\%\) implied full-state field accuracy and \(0.0\%\) changed-field \(F_1\). The real split drives this effect: real next-state inputs average about \(496\) fields and \(7.8\) changed fields, giving copied-state overlap of \(98.5\%\); synthetic inputs average about \(18\) fields and \(3.2\) changed fields, giving \(82.1\%\). Open-ended project dynamics give overlap metrics the largest copy reward.
The long-horizon full-state diagnostic shows the same risk at the task level. The copy prior reaches \(83.6\%\) field accuracy on full-state targets. Model scores show the same mismatch. GPT-5.4 scores high on value-aware next-state \(F_1\) (\(13.8\%\)) and low on long-horizon full-state field accuracy (\(50.2\%\)). MiMo-V2.5-Pro scores low on value-aware next-state \(F_1\) (\(8.0\%\)) and highest among prompted models on full-state field accuracy (\(63.3\%\)). Overlap-heavy reporting looks strong when action-conditioned change prediction is weak.
| Method / model | Subset | \(F_1^{\mathrm{VA}}\) | \(F_1^{\mathrm{pres}}\) | Field-Acc. |
|---|---|---|---|---|
| Copy prior | Next, same-instance | 0.0 | 0.0 | 98.0 |
| Copy prior | Long-horizon | – | – | 83.6 |
| GPT-5.4 | Next common | 13.8 | 27.5 | – |
| MiMo-V2.5-Pro | Long-horizon | – | – | 63.3 |
3pt
A concrete real-corpus instance illustrates the distinction. After a broadcast action, the VM changes one anonymized sprite’s \(x\) coordinate from \(10\) to \(0\). Copying the input state receives no changed-field credit even though most fields are unchanged. Predicting the right field with value \(5\) receives presence-only credit but not value-aware credit. Only the field–value pair “\(x{=}0\)” is a value-aware true positive.
Figure 2: 
.
Table 5 summarizes the auxiliary diagnostics.
The perturbation audit tests whether the real-corpus errors mainly reflect surface familiarity. Six of seven evaluated models have non-positive gaps, and GPT-5.4 shows a modest positive gap. Absolute perturbed scores remain low. Changing names, coordinates, thresholds, costumes, or independent script order leaves the transition problem difficult; robustness to irrelevant surface changes is uneven.
The remaining diagnostics separate reactivity from executable consequence prediction. In counterfactual examples, models change their answers on \(65.7\)–\(85.1\%\) of affected pairs, yet exact \(\Delta\)-match reaches only \(16.0\)–\(26.0\%\). For the five non-OpenAI models, factual changed-field \(F_1\) is \(49.3\)–\(54.4\%\), and counterfactual changed-field \(F_1\) drops to \(20.9\)–\(28.7\%\). The models often notice that an intervention matters and still miss the executed consequence. On the synthetic causal-attribution slice, event accuracy is \(62.7\)–\(79.1\%\); exact rule-chain match reaches at most \(10.0\%\). Event accuracy alone is weak evidence on this set: a no-event baseline reaches \(72.7\%\) accuracy and \(0.0\) event \(F_1\). Long-horizon rollout exposes temporal drift. Full-state field accuracy declines gradually, while hidden-state accuracy falls from \(40.3\)–\(44.4\%\) at \(H{=}2\) to \(2.8\)–\(18.1\%\) at \(H{=}20\). Persistent fields keep overlap moderate, but they do not preserve the hidden variables needed for future dynamics.
| Diagnostic | Observed pattern | Interpretation |
|---|---|---|
| Perturbation | Perturbation changes little: 6/7 models have \(\Delta_{\mathrm{pert}}\le 0\), and GPT-5.4 increases by only \(+2.9\). Perturbed presence remains low: \(8.0\)–\(12.7\). | Surface-form changes are not the main cause of low real-corpus performance; the bottleneck is deeper than prompt sensitivity. |
| Counterfactual | Models react to interventions, with sensitivity \(65.7\)–\(85.1\), but rarely match the exact resulting delta: exact \(\Delta\)-match \(16.0\)–\(26.0\), CF-\(F_1\) \(20.9\)–\(31.0\). | Models often notice that an intervention matters, but fail to compute its concrete execution consequence. |
| Causal attribution | On the synthetic causal slice, event accuracy is moderate: \(62.7\)–\(79.1\), with event \(F_1\) \(51.0\)–\(65.0\). Exact rule-chain attribution remains near zero: \(0.0\)–\(10.0\). | Identifying the triggering event does not mean the model can recover the executed causal chain. |
| Rollout | Multi-step rollout degrades sharply. Field accuracy drops from \(70.6\)–\(71.8\) to \(53.7\)–\(56.9\); hidden-field accuracy drops from \(40.3\)–\(44.4\) to \(2.8\)–\(18.1\). | Persistent visible fields preserve some overlap, but hidden state collapses over long-horizon execution. |
5pt
7 Discussion↩︎
7.0.0.1 Metric lesson.
Full-state overlap rewards copying in worlds where actions change only a small part of a large state. Real Scratch projects have this structure: persistent fields dominate the state, and local actions change a small field set. Full-state field accuracy mixes persistent-state retention with transition computation. Value-aware changed-field scoring isolates transition computation by asking whether the model found the changed field and computed its new value.
7.0.0.2 Failure modes exposed by execution.
Execution traces separate errors that visual inspection would merge. Field identification outpaces value computation: presence-only scores are higher than value-aware scores. Reactivity outpaces correctness: counterfactual sensitivity is much higher than exact \(\Delta\)-match, and event detection is much easier than trace-faithful rule-chain attribution. Temporal persistence is fragile in hidden variables: full-state overlap stays moderate while hidden state collapses under rollout.
7.0.0.3 Why real projects matter.
Real Scratch projects create the regime where overlap metrics are most misleading. In the same-instance copy diagnostic, the real next-state split contains \(164\) examples with about \(496\) fields per input and \(7.8\) changed fields per transition. Copying the input state reaches \(98.5\%\) implied full-state field accuracy on this split. The synthetic split averages \(17.8\) fields and \(3.2\) changed fields, and the same copy prior reaches \(82.1\%\). The real split is harder for models and easier for overlap metrics. Reporting the real/synthetic split with changed-field scores keeps persistent-state retention separate from executable transition prediction.
7.0.0.4 Benchmark use.
ScratchWorld is most informative when systems are compared by error profile. A program parser may improve field presence and causal chains; a learned transition model may improve value updates on repeated mechanism families; a visual model may improve scene recovery and still miss state updates. Shared replay sources and state schemas keep these differences comparable across perception, rule access, state update, intervention handling, and temporal persistence.
7.0.0.5 Practical implication.
Action-conditioned world-model evaluation should separate state persistence, changed-field identification, and value computation. Full-state overlap detects gross state loss. Presence-only changed-field \(F_1\) measures affected-field identification. Value-aware changed-field \(F_1\) measures executed-value computation. A copied state preserves persistence and misses change; a guessed delta finds a field and misses arithmetic, thresholding, or event propagation.
8 Limitations↩︎
ScratchWorld targets action-conditioned executable state prediction, not general physical understanding, embodied control, or open-ended video prediction. Its worlds are program-defined: hidden variables, lists, broadcasts, clones, and event handlers are executable state variables rather than latent physical quantities. This makes the benchmark precise and replayable, but bounds the scope of its claims.
Replay fidelity and scoring are central to validity. We use a pinned Scratch VM, retain instances only after replay agreement, canonicalize field paths, score scalar zero changes explicitly, and treat schema errors as incorrect. External validity is limited by the current release scale: \(153\) anonymized project hashes and \(659\) replay-verified examples. Scratch projects vary sharply in state size, clone behavior, event structure, and hidden variables. Larger releases can add broader real-project coverage, project-macro scores, trained baselines, and mechanism-specific analyses.
Causal labels require special care because Scratch concurrency, clones, and nested broadcasts can make ordered explanations ambiguous. We exclude projects whose ordered chains cannot be recovered reliably and report causal attribution as a synthetic diagnostic slice unless stated otherwise. Our experiments focus on prompted language/reasoning and multimodal systems; trained control-oriented world models remain an important target for future benchmark use.
9 Ethical Considerations↩︎
Scratch projects are created by young learners and are treated as sensitive user-generated content. The release omits raw project identifiers, withholds identifier mappings, avoids redistributing raw Scratch projects, and anonymizes free-form public strings where released. Semantic object, costume, and variable names are retained only when needed as model-visible evidence. ScratchWorld is intended for world-model evaluation, not for profiling users, assessing students, identifying authors, or redistributing user-created assets.
10 Conclusion↩︎
We introduced ScratchWorld, an execution-grounded diagnostic benchmark for action-conditioned world models in executable Scratch worlds. By replaying actions, hidden state, event handlers, causal traces, and counterfactual outcomes under a pinned VM, ScratchWorld evaluates whether models compute executable state changes rather than merely preserve plausible futures. The current release shows that full-state overlap can reward copied persistence, while value-aware changed-field scoring separates persistence from executed change. Across next-state, counterfactual, causal, and rollout diagnostics, prompted models often react to the provided evidence but fail to compute the changed values or rule chains produced by execution. ScratchWorld provides a reusable diagnostic target for trained dynamics models and program-aware predictors.
References↩︎
Corresponding author: Jialu Zhang.↩︎