1 Introduction and Related Work↩︎

A living world is only as alive as what can be authored for it. This paper starts from a world that keeps its promises. The GEST-Engine [1] executes formal Graphs of Events in Space and Time (GESTs) [2] deterministically inside a 3D game world: every specified event happens, at a known place, over a known span of frames, and the engine records per-frame spatial relation graphs, event-to-frame mappings, and textual descriptions while it renders (Fig. 1). In the human studies on its corpus, GTASA [3], annotators rated engine-executed videos 0.69 valid, against 0.18 for VEO 3.1 and 0.13 for WAN 2.2, with semantic match 4.09/5 against 2.50 and 1.75 (16 annotators). Execution, in short, is a solved problem in this world. Authoring is not. The engine runs any valid GEST, and until now only two things could produce one: a human, or a random procedural generator with no story to tell.

Who could author it? LLM agent frameworks are the natural candidates, and a wave of them already makes videos. StoryAgent [4], MAViS [5], and DreamRunner [6] orchestrate specialized agents across scriptwriting, casting, and rendering; VideoDirectorGPT [7] and Dysen-VDM [8] condition diffusion on LLM-built plans. Every one of these pipelines ends the same way: a neural generator gets the final word, objects appear and disappear, actors morph, temporal order breaks [9], [10], and since no plan-level artifact survives generation, there is nothing against which the output could be checked. None reports an executability figure; in their setting the concept does not exist.

Agents have driven engines before. GPT4Motion [11] writes Blender scripts for physics-guided generation, but commands three basic motion primitives. FilmAgent [12] stages multi-agent film production in Unity, choosing from predefined stage positions, actions, and camera shots. Its world is a menu: enumerable, stateless, every selection valid, so executability is not a question one can even ask there. The concurrent Cutscene Agent [13] authors Unreal cutscenes through tools and checks call trajectories against an API-derived dependency graph, which is trajectory validity rather than world-state validity, with no baseline comparison or ablation. The synthetic-data tradition, from Playing for Data [14] to BEDLAM [15], VirtualHome [16], and Action Genome [17], proved long ago that simulation buys perfect annotation at scale. In every one of those platforms, a researcher authored the specification, by hand or by code.

Our world is neither a menu nor a script library. A GEST is a directed graph whose nodes are events (an action, a performer, participating entities, a location) and whose edges carry temporal constraints from Allen’s interval algebra [18], along with logical and semantic relation types; the engine grounds it in GTA San Andreas through the Multi Theft Auto framework and orchestrates it by Floyd-Warshall constraint satisfaction [1]. Earlier text-to-GEST prototypes [2] demonstrated the translation in principle, without targeting executability. To author an executable GEST, an agent must keep a story coherent and simulator-valid at the same time, across the 50–200 events of a multi-actor story: action chains, object lifecycles, capacities of points of interest (POIs), cycle-free temporal constraints. LLMs are excellent at the first requirement and fail at the second. We built the obvious system first, a staged pipeline driving GPT-5 through six validated stages, and watched it fail 50 times out of 50 (Section 2).

The architecture that works separates concerns: the LLM decides what should happen, a programmatic state backend decides what is allowed to happen. The agents act only through tool calls validated by the engine’s own procedural generator, repurposed as a state backend. Invalid operations are rejected before they touch the graph, so every emitted specification is executable by construction. And although the videos are generated offline, the authoring session itself is an agent-environment loop. The capability registry (roughly 14,000 lines) cannot fit in a context window, so the agent has partial observability and explores through paginated tools. Validity is state-conditional: an action is legal only as a valid next action at a POI given the chain so far. Every tool call either advances the world state or returns a rejection the agent must react to. The world model here is explicit and symbolic, but the interaction problem, perceiving, acting, and recovering from failure in an environment the agent cannot fully see, is the one embodied agents face.

Our contributions are:

• A constraint-enforcing authoring environment (more than 30 validated tools over a stateful backend with transactional chains, capacity tracking, object lifecycles, cycle detection, and synchronized interactions), and a hierarchical Director / Scene Builder architecture that authors multi-scene stories through it in a round-based protocol, with Relation Subagents populating the logical and semantic edge types that procedural generation leaves empty.

• The staged-pipeline negative result and its failure-mode analysis: a frontier model with staged prompting and per-stage validation produces 0 executable specifications in 50 attempts, while a far smaller model behind the tool layer reaches 80%.

• A reconstruction-fidelity evaluation exploiting a property unique to this setting: the input text is derived from a source graph, so a plan-level ground truth exists. The agent reconstructs events at 0.83 F1 against a 0.55 matched-random floor, and the residual gap is dominated by information the text channel drops.

2 The Staged Pipeline and Why It Fails↩︎

Before the agentic architecture, we built a staged pipeline: a LangGraph workflow that drives GPT-5 through a fixed graph of six specialized stages. Concept drafts an abstract hierarchical GEST with parent and leaf scenes and actor archetypes; Casting assigns skins from the engine’s catalog; Episode Placement maps scenes to valid simulation episodes; Setup plans off-camera preparation and backstage positioning; Screenplay translates the abstract narrative into concrete action sequences; Scene Detail expands each leaf scene into a complete executable GEST. Every stage is a prompt-and-parse step, its structured output validated via Pydantic. The pipeline produced narratively coherent GESTs with logical structure and character motivation, and across 50 attempts zero executed.

The failure modes compound across stages: hallucinated actions (“WalkTo” instead of “Move”), invalid action chains (“TypeOnKeyboard” without a prior “SitDown” and “OpenLaptop”), object lifecycle violations (putting down objects never picked up, two actors holding the same object), temporal cycles the Floyd-Warshall solver cannot resolve, and context overflow from the 14,000-line capability registry. These map onto exactly the two properties that make this world hard to author for: the registry is not enumerable in context, and validity is state-conditional. Each stage makes locally reasonable decisions, but accumulated state (who is where, what they hold, which POIs are occupied, which constraints are in effect) drifts from validity as the GEST grows. And the errors cannot be repaired afterwards: fixing one violation invalidates the state that later events depend on, so correction requires backtracking through the graph at superlinear cost. Validity must be enforced where the specification is written, not checked after it is finished. The lesson: the LLM should decide what should happen; a programmatic backend should decide what is allowed to happen.

3 Architecture↩︎

Two LLM agents do the planning, but the substance of the system is the layer beneath them. Unlike the staged pipeline’s fixed workflow, the Director and Scene Builder are reactive agents (Claude Haiku 4.5, built on LangGraph’s deepagents with hierarchical subagent delegation): they choose their next tool call from the current world state rather than following a predetermined stage graph, and they operate over an authoring environment of more than 30 validated tools backed by a dedicated state-tracking and validation backend. The agents never manipulate the GEST directly; every operation passes through the backend, which checks it against the accumulated world state before it can affect the graph (Fig. 1, top). A complete multi-scene story costs approximately $0.25 in API calls.

The Director Agent receives an optional seed text and a generation configuration, then works in four phases. Exploration: read-only, paginated tools expose the capability registry (episodes, regions, POI action chains, and character skins with visual descriptions). The explore-before-plan pattern prevents hallucination: an action enters a plan only after a tool call has confirmed it exists. Casting: the Director creates the story and its actors (name, gender, skin, starting region). Scene building: the Director processes scenes sequentially; for each, it selects episode, region, and participants, delegates to the Scene Builder with a natural-language brief, and moves actors between regions afterwards. Finalization: the Director links scene boundaries with cross-scene temporal constraints.

The Scene Builder Subagent receives an isolated context (scene, episode, region, actors, and the brief) and never sees the rest of the story. It constructs events through a round-based state machine. A round opens with the current state of every actor (posture, held objects, location). For each actor, the subagent starts a chain at a POI; every continuation returns the valid next actions at that point, so the agent chooses only among moves the world affords. Synchronized two-actor interactions, camera control, and cross-actor temporal dependencies are separate validated calls, and closing the round commits cross-actor ordering. Chains commit atomically: events accumulate in a buffer, only an explicit commit writes them into the GEST, and failed explorations leave no trace.

Relation Subagents. After each scene and after final assembly, the Director optionally delegates to two further subagents: a Logical Relations Agent adds causal and dependency edges (causes, enables, prevents, requires), and a Semantic Relations Agent adds narrative-coherence edges with free-text types (observes, interrupts, motivates). These edges are optional for execution, but they populate the logical and semantic edge types defined in the GEST formalism that procedural generation never produces; to our knowledge this is the first system to exercise the full expressive capacity of the representation in one pipeline.

Constraint enforcement. The state backend is the engine’s procedural random generator, extended with a delegation interface that exposes its operations as validated tools. The building tools implement a transaction-based state machine over four states (IDLE, STORY_CREATED, IN_SCENE, IN_ROUND); each call verifies its state before executing. The backend tracks actor posture, held objects, and location; enforces POI capacity for exclusive-use objects; walks the dependency graph before accepting any before relation, rejecting edges that would create a cycle; coordinates two-actor interactions (co-location, posture, an explicit receiver for Give with an automatically synchronized receiving event); and locks atomic object sequences ( TakeOut\(\to\)Use\(\to\)Stash) until completion. Rejections return explanatory errors the agent uses to re-plan: the environment’s dynamics, seen from the agent’s side. Because the same generator guarantees executability for random generation, agentic authoring inherits the same guarantee.

4 Evaluation↩︎

4.1 Setup↩︎

The rendered channel is already validated. GTASA’s human studies scored this engine’s videos at scale against the strongest neural generators (Section 1), and our agent drives the same executor, so output quality is inherited rather than re-proven. What is new is the authoring, and that is what we evaluate: does an authored specification execute, and does it specify what the text asked for.

The seeded evaluation draws directly from GTASA: 25 corpus stories, diverse in scene type (classroom, gym, garden, house, mixed) and actor count (2–6). For each story we derive a textual description with the engine’s description pipeline and hand it to the agent as seed text; the Director authors a new GEST re-interpreting that narrative. An attempt counts as successful only if the authored GEST simulates. This scale is the norm of the field, where FilmAgent [12] evaluates on 15 story ideas and the concurrent Cutscene Agent [13] on 65 scenarios; across the staged baseline and the seeded runs, we evaluate 75 authoring attempts end to end. The system also runs with no input at all (the Director explores the world, conceives a narrative, and builds the GEST, as in Fig. 1); autonomous mode succeeds roughly half the time and we treat it as a capability demonstration rather than a measured condition.

4.2 Executability↩︎

Table 1 tells the architectural story. The staged pipeline, GPT-5 with per-stage structured validation, wrote narratively coherent GESTs of which none executed. The tool-constrained agent, driving the far smaller Claude Haiku 4.5, reached 80%. The gap cannot be a model effect, because the winning side uses the weaker model. What changed is where constraints live: in the prompt as instructions, or in the tool layer as checks no output can bypass. Failures are never constraint violations, since emitted graphs are valid by construction; they are LLM-side (context exhaustion, repetitive loops) or engine-side simulation errors.

4.3 Reconstruction Fidelity↩︎

Because every seed text is derived from a source graph \(G\), a plan-level ground truth exists for the agent’s output \(\hat{G}\): we can measure how much of the specified story survives the round trip graph \(\to\) text \(\to\) agent \(\to\) graph. To our knowledge no other system in this space has a plan-level ground truth to reconstruct against.

Both \(G\) and \(\hat{G}\) are leaf-level event graphs in the same schema. We describe each event by its action, the multiset of participating entity types, and its location, and score precision/recall/F1 under exact-identity optimal matching (which reduces to multiset intersection) at three levels: action; action + entities; action + entities + location. Sequential structure is scored as F1 over per-actor action bigrams from the chain ordering. As a chance floor we score \(G\) against three random corpus graphs matched by scene type and actor count: the empirical agreement between two independent stories over the same capability registry.

Table 2 reports the scores. Fig. 2 shows one reconstructed story in full: in that pair, every event of the source plan reappears (recall 1.0; F1 0.70, as the agent also adds a phone interlude and a second scene), and the drink-handoff motif survives the round trip with its synchronization intact, grounded in a different room because the seed text never names one. The agent beats its per-pair floor mean in 19 of 20 pairs on events and 20 of 20 on sequential structure (sign test, \(p < 10^{-4}\)). Four pairs reconstruct perfectly (F1 \(=\) 1.0 at every level), verified as genuine rebuilds with different node identities rather than copies, so the ceiling is attainable when the text preserves the information. The location level explains the residual: across all 20 pairs, the agent grounds the story in the source region precisely when the seed text names that region, and the five classroom texts, which never name the room, are exactly the five pairs that score 0 on location. What the language channel drops, no agent can recover; what it keeps, the agent reconstructs. Only the seeded arm has a source graph; autonomous generation cannot be scored this way.

5 Limitations and Future Work↩︎

The distance between what the Director narrates and what the engine renders stems from expressive channels the platform does not expose: fixed cameras (no tracking shots or rack focus), no frame-precise spatial triggers (“as she enters, he looks away”), no atmospheric change during execution, and atomic rather than blended animations. These are platform constraints, not architectural ones.

Looking forward, the authoring layer is the transferable component: any engine that exposes its capabilities as an explorable registry and its constraints as validated state transitions can sit beneath the same agents, and richer channels (camera placement, atmospherics, animation blending) would make the system a controllable video generator in its own right. The representation also invites a human hand: because an authored GEST is inspectable and editable before execution, we can imagine cinematographers programming a film through AI-assisted graph authoring, with the engine guaranteeing that whatever they write will run. And since the system maintains an explicit, inspectable world state for every story it authors, that state is a natural conditioning signal for neural renderers, pairing the semantic reliability of an explicit world with the visual realism of a learned one.

6 Conclusion↩︎

We presented an agentic system that authors executable specifications for a living world: from free text or from nothing, a Director and a Scene Builder construct multi-scene, multi-actor GESTs through a constraint-enforcing tool layer, and the engine executes them into videos with exact per-frame ground truth. The architecture is the result. Moving constraint enforcement from the model into more than 30 validated tools over a transactional state backend raises executability from 0 of 50 (a frontier model in a staged workflow) to 20 of 25 for a far smaller model, and makes invalid specifications impossible rather than improbable. Reconstruction against plan-level ground truth, a measurement unique to this setting, reaches 0.83 event F1 over a 0.55 matched-random floor, with the residual exactly where the text drops information. Authoring here is itself an agent-in-a-world problem. To our knowledge, the system is the first to exercise the full expressive capacity of the GEST formalism (hierarchy, synchronized interactions, semantic and logical structure) in a single pipeline, closing the loop from story, to an explainable scene representation, to video with ground truth, and back to text through the GEST-derived description.

Acknowledgements.↩︎

This work was supported by Büchi Labortechnik AG and by the project “Romanian Hub for Artificial Intelligence — HRIA”, Smart Growth, Digitization and Financial Instruments Program, 2021–2027, MySMIS no. 351416.

References↩︎