0.0.1 Introduction↩︎

Visual narratives promise compact storytelling, but visual clarity alone is not enough. A generated storyboard may preserve characters, objects, and plausible scenes while losing the transition meaning that explains why one scene follows another. Recent story visualization systems improve visual memory, context modeling, identity preservation, layout control, and long-range consistency [1]–[17], but these improvements do not ensure that source-story transition meaning survives generation.

Fig. 1 illustrates the failure. In the left panel, the poor man finds treasure, shares it fairly, and earns respect, so the recovered meaning remains honesty. In the right panel, the storyboard keeps the man, treasure, and final respect scene but omits the sharing event; the sequence still looks coherent, yet the meaning shifts to wealth or luck. We call this transition-level loss semantic trajectory collapse.

Semantic trajectory collapse is the transition-level narrative information loss studied in this work. It occurs when source-supported transition meaning, such as sharing fairly before earning respect, is no longer recoverable from generated images alone; we diagnose this loss using the Semantic Trajectory Gap (STG). A semantic trajectory is the ordered chain of scene-to-scene transitions connecting actions, causes, emotions, consequences, temporal order, and moral-semantic targets. Here, a moral-semantic target is a source-defined recoverability target, such as honesty earning trust or greed causing loss, not a claim of objective moral truth as illustrated in Fig. 1.

We define the Semantic Trajectory Gap (STG) as text-only recoverability minus image-only recoverability. Image-only evaluation is not artificial: in storyboards, comics, wordless children’s books, accessibility settings, low-literacy communication, and film previsualization, images are often the final medium and the source text may not be available to the viewer. If a transition is recoverable only when the source text is supplied again, then that meaning has not survived visualization.

Existing benchmarks and metrics evaluate coherence, discourse consistency, commonsense grounding, character continuity, visual logic, action-state consistency, and prompt faithfulness [10], [18]–[24]. Fine-grained text-to-image metrics such as TIFA, Davidsonian Scene Graphs, GenEval, GenAI-Bench, and VQAScore-style evaluation show that question-answering can provide interpretable image-text faithfulness signals [25]–[29]. However, diagnosing semantic trajectory collapse requires a stricter set of protocol elements: source stories to define intended meaning, transition annotations to localize adjacent-scene targets, recoverability questions to test inference, contrastive variants to expose near-miss semantic drift, image-only evaluation to avoid source-side leakage, and paired text/image scoring to measure STG. Table 1 compares support for these diagnostic requirements. The comparison shows that prior benchmarks cover useful parts of story evaluation, such as coherence, visual logic, or prompt faithfulness, but none jointly provide source-story transition annotations, recoverability QA, contrastive variants, image-only testing, and paired text/image STG scoring.

We introduce KathaTrace, a generator-agnostic recoverability protocol for diagnosing semantic trajectory collapse in generated visual narratives. KathaTrace is multimodal in design but image-only in its primary diagnostic condition: text-only recoverability estimates source-side support, image-only recoverability tests whether the generated storyboard communicates transition meaning without source leakage, and text+image recoverability filters ambiguous or invalid items. We also construct KathaBench-25K, a transition-aware benchmark of 5,000 rights-compatible classical narratives from Kathasaritsagara/The Ocean of Story, Aesop’s Fables, and Panchatantra, normalized into structured scenes, transition annotations, recoverability questions, accepted-answer sets, contrastive variants, evidence-isolated scoring, ambiguity filtering, and held-out splits.

Beyond the leaderboard, controls show that the recoverability gap is not explained by generic QA, renderer quality, or extra frames. In evaluator ablations, the strongest adapted DSG-style baseline reports \(28.9\) STG and \(0.58\) human correlation, while the full KathaTrace protocol reports \(23.5\) STG and \(0.71\), indicating stronger human alignment on causal gaps, hidden consequences, and moral-target drift. With FLUX fixed, the Gemma-ST planner reduces STG from \(35.2\) to \(24.0\), showing that transition-aware planning matters beyond rendering. Semantic Compass further reduces STG from \(28.4\) to \(21.4\) on held-out evaluation, while non-semantic extra frames yield only small gains.

4.2pt

Story: source narratives are included. Trans.: explicit adjacent-scene transition annotations. Recov. QA: recoverability questions for testing inferable transition meaning. Contrast.: semantic variants that expose near-miss transition drift. Image-only: evaluation without source text, prompts, captions, labels, or annotations. STG: paired text-side and image-only scoring for Semantic Trajectory Gap.  = full support;  = partial or proxy support;

 = absent.

Our contributions are:

• We define semantic trajectory collapse as the failure where generated storyboards preserve visible content but lose the transition meaning needed to understand how one scene follows another.

• We propose KathaTrace, a generator-agnostic image-only recoverability protocol for testing whether source-story transitions can be inferred from generated images alone.

• We release KathaBench-25K, a transition-aware benchmark with structured scenes, transition annotations, accepted-answer sets, recoverability questions, contrastive variants, ambiguity filtering, and held-out splits.

• We introduce Semantic Trajectory Gap and actionability to quantify text-to-image loss of recoverable narrative meaning, localize failures by transition type, and test whether these signals can guide reranking and bridge-scene repair.

0.0.2 Related Work↩︎

0.0.2.0.0.1 Story visualization, narrative benchmarks, and the dataset gap.

Story visualization systems generate image sequences from narrative text using story continuation, visual memory, context modeling, bidirectional generation, diffusion rendering, multimodal history, identity preservation, layout control, and long-range consistency [1]–[17], [30], [31]. These methods improve visual coherence, subject consistency, and generation control, but they do not directly test whether viewers can recover why one scene follows another from images alone. Multi-shot video and narrated-story systems add agents, memory, cinematic structure, and world consistency [32]–[39], yet visual continuity can still improve while causal, emotional, consequence, or moral-target meaning is omitted.

Existing benchmarks address related but different goals. As summarized in Table 1, StoryBench, VinaBench, and ViStoryBench evaluate faithful and consistent visual narratives, but only partially support transition-localized recoverability and do not report paired STG [18]–[20]. OpenStory++ / Cohere-Bench and DS-500 / DreamStory support open-domain or consistent story generation, but do not isolate image-only recovery of adjacent-scene transition meaning [10], [21]. LogicTale / LogiStory moves closer through visual logic and causal coherence, but lacks the full diagnostic chain of source transition annotations, fixed recoverability QA, contrastive variants, strict image-only testing, and paired text/image STG [22]. KathaBench-25K fills this gap by combining source meaning, transition targets, recoverability questions, contrastive drift, ambiguity filtering, and STG in one benchmark.

0.0.2.0.0.2 QA-based faithfulness evaluation and KathaTrace.

QA-based evaluation provides interpretable signals for generated images. TIFA tests text-to-image faithfulness with generated QA pairs [25]; Davidsonian Scene Graphs evaluate atomic scene facts and dependencies [26]; GenEval and GenAI-Bench focus on object-level and compositional alignment [27], [28]; GenEval 2 studies benchmark drift [29]; and interleaved or open-ended benchmarks measure image-text structure, coherence, or human-aligned multimodal judgment [40]–[42]. These metrics provide useful QA-based diagnostics, but they do not define source-story transition targets and then test whether those targets remain recoverable from generated images without source text, prompts, captions, labels, or annotations.

KathaTrace keeps QA interpretability but changes the evidence design: each transition target is evaluated under text-only, image-only, and text+image conditions. Text-only scoring estimates whether the target is recoverable from the source story; image-only scoring tests whether the generated storyboard communicates the same target without source leakage; and text+image scoring filters items that remain ambiguous even with full evidence. This produces a paired estimate of text-to-image transition loss, localizes failures by dimension, and supports actionability analysis through Semantic Compass reranking and bridge-scene repair rather than only aggregate coherence.

0.0.3 Method↩︎

We introduce KathaTrace, a generator-agnostic protocol for testing whether source-story transition meaning remains recoverable after visual generation. Instead of scoring only visual plausibility, KathaTrace asks whether a viewer can infer why one scene follows another from images alone. Fig. 2 summarizes the framework. See Appendix Secs. A–D for dataset construction, scoring, ambiguity filtering, judge aggregation, Semantic Compass, and metric details.

0.0.3.1 Task Formulation

A source story is represented as an ordered scene sequence: \[S=\{s_1,s_2,\ldots,s_T\}. \label{eq:source95sequence}\tag{1}\] A storyboard generator \(G\) produces ordered images: \[\hat{X}=G(S)=\{\hat{x}_1,\hat{x}_2,\ldots,\hat{x}_T\}. \label{eq:generated95storyboard}\tag{2}\] KathaTrace requires only three inputs: the source story \(S\), the generated storyboard \(\hat{X}\), and a fixed set of transition-level recoverability questions. It does not use generator internals, model weights, training data, hidden prompts, or intermediate plans. The goal is to test whether transition meaning that is recoverable from the source story is also recoverable from the generated image sequence. This includes causal, emotional, consequence-bearing, temporal, and moral-target transitions.

0.0.3.2 KathaBench-25K Construction

We instantiate KathaTrace with KathaBench-25K, a transition-level recoverability benchmark built to test whether source-supported narrative meaning survives storyboard generation. The benchmark uses 5,000 rights-compatible source stories from three traditions: 1,702 from Kathasaritsagara/The Ocean of Story, 1,676 from Aesop’s Fables, and 1,622 from Panchatantra. This mixture is needed to avoid evaluating only one narrative style, moral pattern, or cultural source.

Each story is normalized into five ordered scenes, producing 25K structured scenes and 20K adjacent transitions. Five scenes are sufficient to represent setup, action, complication, consequence, and resolution, while keeping evaluation comparable across stories. The adjacent transitions are the core unit of KathaTrace because semantic trajectory collapse occurs when the viewer cannot recover why one scene follows another.

For each transition, we store transition fields for action, causality, emotion, consequence, and the benchmark-specific moral-semantic target. These fields are needed to localize failures rather than report only an aggregate score. We then derive 28,712 recoverability questions with canonical answers and fixed accepted-answer sets, so evaluation checks recoverable meaning rather than free-form judge preference.

Finally, 10K contrastive variants preserve major entities and settings while changing one meaning-bearing transition. These variants test whether an evaluator distinguishes true transition preservation from surface similarity. Human validation checks source-scene fidelity, scene order, transition validity, QA answerability, accepted-answer coverage, ambiguity flags, moral-target consistency, and contrastive validity. Together, the source diversity, transition annotations, recoverability questions, contrastives, and validation metadata make KathaBench-25K a benchmark for semantic recoverability, not just a collection of story prompts.

0.0.3.3 Semantic Trajectory Representation

We define a semantic trajectory as the ordered set of meaning-bearing transitions between adjacent scenes: \[\tau=\{r_1,r_2,\ldots,r_{T-1}\}, \qquad r_t=\Phi(s_t,s_{t+1}). \label{eq:semantic95trajectory}\tag{3}\] Each transition stores six source-side fields: \[r_t=(a_t,c_t,i_t,e_t,o_t,m_t), \label{eq:transition95fields}\tag{4}\] where \(a_t\) is visible action, \(c_t\) is causality, \(i_t\) is intention, \(e_t\) is emotional shift, \(o_t\) is consequence or outcome, and \(m_t\) is the benchmark-specified moral-semantic target when applicable. These fields define what meaning should survive story-to-image translation. They are not all scored QA dimensions: intention is retained for annotation and planning, while temporal-order questions are derived from scene order.

0.0.3.4 Recoverability Questions and Evidence Conditions

KathaTrace evaluates six scored QA dimensions: \[\begin{align} \mathcal{K}_{\mathrm{QA}}=\{& \mathrm{action},\mathrm{causal},\mathrm{emotional},\\ &\mathrm{consequence},\mathrm{temporal},\mathrm{moral} \}. \end{align} \label{eq:qa95dimensions}\tag{5}\] These correspond to action visibility, cause-effect recovery, emotional-state recovery, consequence recovery, event-order recovery, and benchmark-specified moral-target recovery. Moral labels are semantic recoverability targets, not claims of objective moral truth.

Each item is indexed by story, transition, QA dimension, and question instance: \[q=(n,t,k,u), \label{eq:question95index}\tag{6}\] where \(n\) is the story, \(t\) is the adjacent-scene transition, \(k\in\mathcal{K}_{\mathrm{QA}}\) is the scored dimension, and \(u\) indexes multiple questions for the same transition-dimension pair. Each question has a canonical answer \(y_q\) and a frozen accepted-answer set \(\mathcal{A}(q)\).

Each question is evaluated under three evidence conditions:

• Text-only \((R_{\mathrm{text}})\): source story \(S\) only; estimates the source-side recoverability ceiling.

• Image-only \((R_{\mathrm{image}})\): generated storyboard \(\hat{X}\) only, without source text, prompts, captions, annotations, labels, or metadata; tests whether meaning survives in images.

• Text+image: both \(S\) and \(\hat{X}\); used only for ambiguity, defect, and contradiction control.

Image-only is the primary diagnostic condition because KathaTrace measures whether transition meaning is recoverable from the generated visual narrative itself.

0.0.3.5 Validity Filtering and Meaning Preservation

Meaning is treated as preserved when the accepted answer for a source-supported transition remains recoverable from generated images alone. Before STG computation, KathaTrace removes questions whose intended answer is not recoverable even with text+image evidence. For each scored dimension \(k\), let \(Q_k\) be the full question set. The valid set is: \[\mathcal{V}_k= \left\{ q\in Q_k: \mathrm{match}\big(J(q,(S,\hat{X})),\mathcal{A}(q)\big)=1 \right\}, \label{eq:valid95question95set}\tag{7}\] where \(J(q,z)\) is the judge answer under evidence condition \(z\). Questions outside \(\mathcal{V}_k\) are excluded from STG and counted in the ambiguity rate, preventing unclear questions or image-text contradictions from being treated as generator failures.

For evidence condition \(z\in\{S,\hat{X},(S,\hat{X})\}\), dimension-specific recoverability is: \[R^k_z = \frac{1}{|\mathcal{V}_k|} \sum_{q\in \mathcal{V}_k} \mathbb{I}\!\left[ \mathrm{match}\big(J(q,z),\mathcal{A}(q)\big)=1 \right]. \label{eq:recoverability}\tag{8}\] Overall recoverability averages the six scored dimensions equally: \[R_z= \frac{1}{|\mathcal{K}_{\mathrm{QA}}|} \sum_{k\in\mathcal{K}_{\mathrm{QA}}}R^k_z. \label{eq:overall95recoverability}\tag{9}\] The ambiguity rate is: \[\rho_{\mathrm{amb}}= \frac{1}{|Q|} \sum_{q\in Q} \mathbb{I}\!\left[ q\notin \mathcal{V}_{k(q)} \right], \label{eq:ambiguity95rate}\tag{10}\] where \(k(q)\) is the scored QA dimension of question \(q\).

0.0.3.6 Semantic Trajectory Gap: Diagnosing Meaning Collapse

The primary metric is the Semantic Trajectory Gap: \[\mathrm{STG}=R_{\mathrm{text}}-R_{\mathrm{image}}, \label{eq:stg}\tag{11}\] where \(R_{\mathrm{text}}=R_S\) and \(R_{\mathrm{image}}=R_{\hat{X}}\). \(R_{\mathrm{text}}\) measures source-side recoverability, while \(R_{\mathrm{image}}\) measures what remains recoverable after visual generation. Low STG indicates strong meaning preservation; high STG indicates semantic trajectory collapse, where source-supported transition meaning is missing from the image-only storyboard. The same definition applies to VLM and human judgments.

KathaTrace also reports dimension-specific gaps: \[\mathrm{STG}_k= R^k_{\mathrm{text}}-R^k_{\mathrm{image}}, \qquad k\in\mathcal{K}_{\mathrm{gap}}. \label{eq:dimension95stg}\tag{12}\] For latent-transition analysis, we use: \[\mathcal{K}_{\mathrm{gap}}= \{ \mathrm{causal}, \mathrm{emotional}, \mathrm{consequence}, \mathrm{moral} \}. \label{eq:gap95dimensions}\tag{13}\] Action visibility and temporal order remain part of overall recoverability but are reported separately because they test local depiction and ordering control rather than latent transition loss.

0.0.3.7 Contrastive Semantic Variants: Isolating Meaning from Appearance

KathaBench-25K includes contrastive variants to test whether recoverability tracks transition meaning rather than shared objects, characters, or settings. Each variant preserves major entities where possible but changes one meaning-bearing transition, such as consequence, motivation, outcome, emotional shift, or causal link. Thus, visual appearance is held close while the semantic trajectory changes. Pairwise contrastive evaluation tests whether recovered meaning matches the source trajectory rather than a visually similar but semantically different alternative. The counterfactual minimal-pair category contains 400 source stories, while the 10,000 contrastive variants are constructed from all 5,000 narratives.

0.0.3.8 Judge Aggregation and Calibration Validation

KathaTrace supports human or VLM judges. For scalable evaluation, it uses fixed evidence packets, fixed prompt templates, deterministic decoding where supported, structured output schemas, frozen accepted-answer sets, and majority-vote aggregation after answer normalization. Ties are scored conservatively as incorrect. Human calibration tests whether VLM recoverability follows human interpretation rather than a single-model artifact, using agreement, rank correlation, calibration error, confidence intervals, and inter-rater agreement. See Appendix Secs. E–F for judge protocols and human calibration details. These checks validate STG as a recoverability measure rather than treating any one judge as ground truth.

0.0.3.9 Semantic Compass: Validating STG as an Actionable Signal

Semantic Compass tests whether KathaTrace diagnostics are actionable. It is not a new generator; it is a post-generation reranking and bridge-repair probe that asks whether localized unrecoverable transitions can guide better storyboard selection or minimal repair.

Given candidate storyboards \(\{\hat{X}^{(1)},\ldots,\hat{X}^{(N_c)}\}\) for the same source story, Semantic Compass selects: \[\begin{align} \hat{X}^{*} = \arg\max_{\hat{X}^{(j)}} \Big[ &\lambda_r R^{\mathrm{val}}_{\mathrm{image}}(\hat{X}^{(j)}) + \lambda_t C_{\mathrm{trans}}(\hat{X}^{(j)},\tau) \\ &- \lambda_p P_{\mathrm{copy}}(\hat{X}^{(j)}) \Big], \end{align} \label{eq:semantic95compass}\tag{14}\] where \(R^{\mathrm{val}}_{\mathrm{image}}\) rewards validation image-only recoverability, \(C_{\mathrm{trans}}\) rewards visual support for annotated transitions, and \(P_{\mathrm{copy}}\) penalizes repeated or near-duplicate frames.

If the weakest adjacent transition is localizable and falls below a validation-selected threshold, Semantic Compass may insert a bridge frame: \[\hat{X}_{\mathrm{bridge}} = \{\hat{x}_1,\ldots,\hat{x}_t,\hat{x}_{t+\frac{1}{2}}, \hat{x}_{t+1},\ldots,\hat{x}_T\}. \label{eq:bridge95sequence}\tag{15}\] All weights, thresholds, candidate counts, and repair decisions are selected on validation data and frozen before final evaluation. Final reporting uses held-out questions and held-out judge prompts, so results test whether STG exposes usable recoverability failures rather than whether Semantic Compass is a new generator.

0.0.4 Experiments↩︎

We evaluate whether generated visual narratives preserve transition-level recoverable meaning under image-only evidence. The experiments test three claims: semantic trajectory collapse is distinct from visual quality, transition-specific evaluation is necessary beyond generic QA, and localized recoverability failures provide actionable signals. All STG values are paired within the same valid examples after ambiguity filtering; raw, filtered, and common-valid audits test whether conclusions depend on filtering.

0.0.4.1 Experimental Setup

Benchmark and split. We evaluate held-out KathaBench-25K examples with method-valid accounting. The release contains 5,000 narratives, 25,000 scenes, 20,000 transitions, 28,712 QA pairs, and 10,000 contrastive variants, with fixed train/validation/public-test/hidden-test splits of 3,500/750/500/250 stories. Validation is used for Semantic Compass selection, public-test for main reporting, and hidden-test for leaderboard evaluation. Common-valid results check whether rankings depend on method-specific usable-storyboard failures.

Evaluated systems. We evaluate StoryDiffusion [7], StoryGPT-V [9], DreamStory [10], Story-Iter [31], ViSTA [11], LogiStory [22], and Gemma-ST + Semantic Compass. Semantic Compass is an actionability probe, not a new generator. Extended experiments vary planners and renderers, including direct prompting, rule planning, caption planning, scene planning, Gemma-ST, SDXL, FLUX, ConSistory, and closed API references.

Judges and reporting. Main results use an ensemble of Gemini 2.5 Flash, Qwen2.5-VL-7B/3B, and SmolVLM2-2.2B/500M with fixed evidence packets, frozen accepted answers, structured schemas, and deterministic decoding where supported. On the strict 400-story human-gold subset, the calibrated VLM ensemble reaches \(77.1\%\) agreement, Spearman \(\rho=0.74\), ECE \(=0.081\), and \(\kappa=0.77\). Human evaluation confirms the same pattern with \(\mathrm{STG}^{\mathrm{human}}=15.8{\pm}2.7\).

0.0.4.2 Main Result

Table 2 shows that visual quality does not imply transition recoverability. Prior methods cluster in a narrow STG range of \(28.4\)–\(34.8\) despite different architectures, planners, and rendering strategies, ruling out a single-method artifact. Story-Iter has the highest controllable visual score, \(4.18{\pm}0.05\), but \(29.7{\pm}1.6\) STG. LogiStory has the strongest controllable image recoverability, \(49.4{\pm}1.3\), but still retains \(28.4{\pm}1.5\) STG. On semantic contrastive variants, LogiStory rises to \(41.2\) STG versus \(28.4\) on source trajectories, showing that KathaTrace is sensitive to meaning changes rather than shared appearance. Gemma-ST + Semantic Compass lowers STG to \(21.4{\pm}1.3\) under held-out evaluation.

0.0.4.3 Does Transition Recoverability Require More Than Generic QA?

Table ¿tbl:tab:evaluator95baselines? shows that transition recoverability is not captured by generic image-level question answering alone. Higher values are better for image recoverability, localization, and human correlation, whereas lower values are better for STG because STG measures the gap between source-story transitions and generated visual transitions.

The strongest adapted generic baseline, multi-frame DSG-style QA, reaches \(48.0{\pm}1.4\) image recoverability, \(28.9{\pm}1.5\) STG, \(0.50{\pm}0.03\) localization, and \(0.58{\pm}0.04\) human correlation. Full KathaTrace improves image recoverability to \(54.6{\pm}1.2\), reduces STG to \(23.5{\pm}1.3\), improves localization to \(0.68{\pm}0.02\), and increases human correlation to \(0.71{\pm}0.03\). This corresponds to a \(6.6\)-point gain in image recoverability, a \(5.4\)-point reduction in STG, a \(0.18\) gain in localization, and a \(0.13\) gain in correlation with human judgments.

These results indicate that the evaluator is measuring more than whether individual images contain answerable objects or events. Generic QA can identify visible entities and some frame-level actions, but it does not explicitly test whether a visual sequence preserves the causal, temporal, motivational, and consequence-bearing transitions required by the source story. The ablation results support this interpretation: transition structure, dimension-specific recoverability questions, and contrastive checks each contribute additional alignment with human recoverability judgments. The improvement is therefore cumulative rather than attributable to a single prompt-format change or a generic increase in QA coverage.

2.5pt

Visual: 1–5 human visual-quality rating. Text / Image Rec.: text-only / image-only recoverability. Moral-target Rec.: benchmark-specific moral-semantic target recovery, not objective moral truth. \(^{\ddagger}\)

Internal prompt rewriting, safety filtering, hardware, and model versioning are not fully controllable.

3.5pt

0.0.4.4 Planning, Rendering, and Failure Signatures

Planner–generator ablations show that planning quality affects narrative recoverability beyond the choice of renderer. Holding the renderer fixed as FLUX, replacing direct prompting with Gemma-ST increases image recoverability from \(40.8\) to \(53.8\) and reduces STG from \(35.2\) to \(24.0\). This is a \(13.0\)-point gain in image recoverability and an \(11.2\)-point reduction in STG. The result shows that an explicit story-transition plan helps the generator preserve the information needed to recover the source narrative from the generated image sequence.

In contrast, when the planner is fixed as Gemma-ST, changing the renderer reduces STG by only \(4.1\) points on average. This does not mean renderer quality is irrelevant; renderer choice still affects visual fidelity, composition, and object depiction. However, the smaller STG change indicates that renderer substitution alone does not solve the main failure mode. The larger improvement comes from improving the intermediate narrative plan, which controls what transition information is made available to the image generator.

The failure signatures further show that different methods fail in different transition dimensions. DreamStory exhibits larger consequence gaps, meaning that generated sequences often fail to preserve what changes after an event. StoryDiffusion exhibits larger causal and action gaps, meaning that actions and their causal links are more often omitted, weakened, or visually underspecified. LogiStory produces a more balanced profile, but its nonzero gaps show that it still loses transition-level information. These patterns are inconsistent with a purely generic visual-quality explanation. If failures were only caused by low image quality, the errors would be expected to affect all transition dimensions more uniformly. Instead, the observed dimension-specific gaps support the need for transition-specific diagnosis.

0.0.4.5 Generalization, Scope, and Hard Cases

Table 3 summarizes controls for generalization and scope. Results are stable across source families, VLM judges, and dataset subsamples. Fig. 3 illustrates the core failure mode: a storyboard can preserve visual plausibility and local character consistency while omitting the rescue transition needed to recover the intended reciprocity moral. KathaTrace localizes this missing transition and moral-target drift, matching the human judgment that the generated storyboard does not preserve the intended meaning. More generally, KathaTrace is most reliable when meaning is expressed through ordered scenes, transition-level questions, and fixed accepted answers. Symbolic, ironic, and long-range narratives show higher ambiguity, so STG should be reported with ambiguity rates and confidence intervals.

3.0pt

See Appendix Secs. G–M for implementation details, additional baselines, leaderboards, ablations, qualitative cases, extensibility examples, and robustness checks.

0.0.5 Conclusion and Future Work↩︎

We introduced KathaTrace, an image-only recoverability benchmark for evaluating whether generated storyboards preserved the transition-level meaning of source narratives. The benchmark was designed to test whether a viewer could recover not only visible entities and actions, but also the causal, temporal, emotional, consequence-bearing, and moral-target information that connected one story state to the next. This addressed a limitation of generic visual-quality and image-QA evaluations, which often treated frame-level correctness as sufficient evidence of narrative faithfulness. Experiments on KathaBench-25K showed that visually plausible storyboards still lost key transition information. Generated sequences often preserved characters, settings, and isolated events, but weakened causal, emotional, consequence-bearing, and moral-target relations. These failures showed that semantic trajectory collapse was distinct from poor visual quality, object omission, or generic QA failure. Evaluator ablations showed that transition structure, dimension-specific questions, and contrastive checks aligned better with human judgments than generic multi-frame QA. Semantic Compass localized the collapsed dimensions, making recoverability failures actionable for storyboard repair. Future work would need to address symbolic, ironic, and long-range transition recovery. These cases remained difficult because their meanings were often indirect, contrastive, or dependent on events separated across multiple scenes. Stronger planning modules, transition-aware generation objectives, and repair mechanisms would be needed to produce storyboards whose narrative meaning could be reliably recovered from images alone.

Appendix↩︎

0.0.6 Dataset Construction and Human Validation Details↩︎

This section specifies the construction, annotation schema, validation protocol, split design, provenance controls, release structure, and quality-control procedures for the proposed KathaBench-25K benchmark. KathaBench-25K is designed for transition-level recoverability evaluation in generated visual narratives: given a source story and a generated storyboard, the benchmark tests whether the story-transition meaning remains recoverable from images alone. We separate source-side transition fields, which describe the intended semantic trajectory, from scored QA dimensions, which define the frozen recoverability-question inventory used during evaluation.

0.0.6.1 Construction Overview

The KathaBench-25K construction pipeline has five stages: source narrative selection, scene structuring, transition annotation, recoverability-question construction, and contrastive-variant construction. Source narratives are drawn from three public-domain or rights-compatible classical narrative families: Kathasaritsagara / The Ocean of Story (1,702 stories), Aesop’s Fables (1,676 stories), and Panchatantra (1,622 stories). These sources were selected because they contain compact, transition-dependent narratives in which causal action, consequence, emotional change, motivation, and moral-semantic interpretation are central to story understanding.

Each story is normalized into five ordered scenes. This fixed length supports controlled comparison across generators while exposing four adjacent transitions per story. The KathaTrace protocol is length-general, but the released KathaBench-25K instance fixes \(T=5\) so that scene boundaries, transition targets, recoverability questions, and accepted-answer sets remain comparable across records. Each record stores provenance metadata, source-rights status, validation status, five scene records, four transition records, and recoverability questions with frozen accepted-answer sets.

Table 4 gives the released KathaBench-25K component counts used throughout the benchmark.

0.0.6.2 Narrative Category Taxonomy

Each KathaBench-25K narrative receives one primary category for balancing, stratified sampling, and category-specific error analysis. Table 5 reports the eight-way category taxonomy and the semantic stress test associated with each category.

0.0.6.3 Scene Representation Schema

Each narrative in KathaBench-25K is decomposed into ordered scenes. A scene is treated as a local visual unit, but it also stores narrative-state fields needed to interpret the scene within the larger semantic trajectory. Table 6 defines the scene-level schema released with each benchmark record.

0.0.6.4 Transition Annotation Schema

The transition between adjacent scenes is the central unit of KathaBench-25K. Each record stores a coarse transition_type for provenance and balancing, together with source-side semantic fields that describe the intended relation between scenes. These fields are not identical to the scored QA inventory: intention is retained for annotation and planning, whereas temporal-order questions are derived from scene order. Table 7 defines the transition annotation schema.

4pt

4pt

4pt

4pt

Table 8 reports the coarse transition-type distribution in KathaBench-25K. These coarse types are used for provenance and balancing, not as scored QA dimensions.

4pt

Table 9 reports the source-side field coverage for the 20,000 adjacent-scene transitions. All transitions store action, causality, intention, emotion, consequence, and expected-change fields, while moral-target fields are stored for the 5,000 story-level moral targets.

4pt

0.0.6.5 Moral-Target Taxonomy

Each KathaBench-25K narrative receives one primary benchmark-specified moral label. These labels are semantic targets for recoverability analysis, not claims of objective moral truth. Table 10 defines the 12-label moral-target taxonomy, and Table 11 reports the final balanced distribution.

4pt

5pt

0.0.6.6 Recoverability QA Schema

Recoverability questions in KathaBench-25K test whether a viewer can infer narrative-relevant information under a specified evidence condition. The released scored QA inventory contains action_visibility, causal, emotional, consequence, temporal_order, and moral; it does not contain a separate intention QA type. Table 12 defines the QA record schema, and Table 13 reports the released QA-type distribution before ambiguity filtering.

4pt

4pt

0.0.6.7 Top-Level Record Status Fields

Each annotations.jsonl record in KathaBench-25K stores two top-level status fields in addition to provenance. The field validation_status is a string and is set to human_review_complete for every released record. The field evidence_status is an object rather than a string. It stores human_evidence_accepted as a boolean and resolution as a categorical value with either unanimous or human_adjudicated. A typical released value is {"human_evidence_accepted": true, "resolution": "unanimous"}.

0.0.6.8 Contrastive Variant Schema

KathaBench-25K includes 10,000 contrastive variants, two per base story. These variants preserve surface entities while changing a meaning-bearing transition. Table 14 defines the schema of contrastive_variants.jsonl.

4pt

0.0.6.9 Human Validation Summary Schema

Human validation is stored separately from the initial annotation fields so that pre-review labels, reviewer votes, consensus decisions, adjudicated labels, ambiguity flags, and correction metadata remain auditable. Table 15 defines the schema of human_validation.jsonl, which contains one story-level summary record for each of the 5,000 released KathaBench-25K narratives.

4pt

0.0.6.10 Construction Transparency

Table 16 reports how each KathaBench-25K component was produced and finalized. The released benchmark annotations are human-validated and human-adjudicated annotations. Trained human annotators define the final released fields through independent review, correction, consensus, and adjudication. Teacher models are used only as scalable candidate-construction aids for selected components, never as ground-truth annotators. No teacher-produced transition field, recoverability question, accepted-answer set, or contrastive variant is released without human review.

Every component is independently audited by three trained reviewers under the protocol in Sec. 0.0.6.11. Edit and rejection rates are reported before adjudication, while final counts reflect post-review, post-adjudication, and post-exclusion released totals.

4pt

Notes. Pass/fail rates are derived from the section-check averages and per-label Fleiss’ \(\kappa\) reported in Secs. [sec:app:human95validation] and [sec:app:inter95annotator95agreement]; edit/rejection rate is \(1-\text{pass rate}\). Records with unresolved ambiguity, invalid QA, invalid contrastives, unusable scenes, unsafe content, or rights uncertainty are excluded from release or from strict gold evaluation per Table [tbl:tab:appendix95adjudication95rules] and are not double-counted against the final counts above, which reflect the released, schema-validated, human-reviewed totals.

The construction process separates candidate generation from benchmark annotation. Teacher models are used only to draft candidate transition fields, recoverability questions, accepted-answer sets, and contrastive variants at scale. These candidates are then checked against the source story by independent human annotators, who may accept, correct, mark ambiguous, or reject each component. Only records that pass human review and, when needed, adjudication are included in the released KathaBench-25K counts. Thus, the released benchmark fields are defined by human validation and adjudication, while teacher models serve only as candidate-construction tools.

0.0.6.11 Human Validation Protocol

Each KathaBench-25K record receives an independent three-reviewer audit after initial construction. Reviewer decisions are stored separately from pre-review fields, consensus labels, adjudicated labels, ambiguity flags, and final release metadata. Majority agreement is used when at least two reviewers give the same valid decision. Conflicting corrections, explicit ambiguity, unusable scenes, invalid QA, invalid contrastives, unsafe content, or rights uncertainty are routed to adjudication. Table 17 gives the reviewer form used during the audit, and Table 18 specifies the adjudication rules applied after independent review.

3.5pt

3.5pt

The review campaign ran from December 3–19, 2025. Three reviewers independently completed all 5,000 records, producing 15,000 completed response records. Of the 5,000 records, 4,101 reached unanimous consensus and 899 were resolved by adjudication. Adjudication concluded on December 22, and gold subsets were finalized on December 24. Mean reviewer confidence was 4.26–4.45 on a 1–5 scale. Section-check pass rates were high for narrative category, scene sequence, transition annotation, recoverability QA, and contrastive validity; moral-label checks had the lowest pass rate, reflecting genuine semantic ambiguity rather than a schema failure.

0.0.6.12 Human Validation Split Design

KathaBench-25K uses the tiered validation design shown in Table 19. The full release receives three independent reviews per record. The 1,000-narrative human-gold subset is stratified by the eight narrative categories, with 125 narratives per category. The strict 400-narrative gold subset is sampled from the human-gold subset with 50 narratives per category, excluding unresolved ambiguity, invalid QA, invalid contrastives, unusable scenes, and rights uncertainty.

3.5pt

Protocol. Campaign protocol version: kathaBench_human_review_v1. Review window: 2025-12-03 to 2025-12-19.

Main-paper human-calibrated results are computed on adjudicated gold records rather than raw pre-review records. Split identifiers, reviewer decisions, consensus labels, adjudicated labels, ambiguity flags, and exclusion reasons are included in the release metadata.

0.0.6.13 Annotation Workload, Annotator Identity, and Ethics

Annotation was performed by a pool of five trained research annotators with English fluency and experience in short narrative interpretation. Each released KathaBench-25K record received three independent reviews. Annotators were independent of model development, metric design, pre-review label production, and experimental evaluation; their role was limited to reviewing, validating, and adjudicating benchmark records according to written guidelines. They received calibration examples before annotation, were informed about task purpose, data handling, workload, compensation, and withdrawal rights, and were compensated at USD 20/hour or above the applicable local living-wage equivalent. Table 20 reports the annotation workload and reliability-audit units.

4pt

0.0.6.14 Inter-Annotator Agreement

We report pre-adjudication agreement to distinguish raw annotator reliability from the adjudicated labels used in headline calibration. Moral-target agreement is the primary chance-corrected categorical reliability measure because moral-target recoverability is the most subjective released annotation. Table 21 reports categorical moral-target agreement and separately lists ordinal rating diagnostics. The ordinal 1–5 ratings are not used as evidence for categorical dataset-label reliability because their score ranges are compressed and concentrated around similar values. Therefore, the KathaBench-25K reliability claim rests on categorical agreement, consensus review, adjudication, and strict-gold filtering, rather than on compressed ordinal preference ratings.

5pt

The ordinal ratings should not be interpreted as evidence of dataset-label reliability. The \(1\)–\(5\) human-study scores are subjective diagnostic ratings, and their restricted score ranges and high means make chance-corrected ordinal agreement unstable. We therefore use ordinal ratings only as diagnostic human-study outputs. Dataset-label reliability is instead supported by categorical moral-target agreement, consensus review, adjudication, and strict-gold filtering. Moral-target labels are interpreted as benchmark-specified semantic recoverability targets, not as universal ethical judgments.

0.0.6.15 Release Structure

KathaBench-25K is publicly available at:

The public release preserves the frozen benchmark schema, split structure, evaluation subsets, metadata files, licensing information, and package-level source provenance manifest described in this appendix. Table 22 lists the released files and the role of each file in making the benchmark inspectable, reusable, and reproducible.

The release contains benchmark annotations, contrastive variants, human-validation summaries, adjudicated gold subsets, aggregate statistics, moral-label balance audits, machine-readable metadata, licensing information, dataset documentation, and a package-level source provenance manifest. These artifacts support verification of the reported counts, split membership, source-family traceability, validation decisions, and rights-screening status. Generated storyboard images, third-party source editions, reference websites, publisher pages, and non-anonymized administrative records are not redistributed as part of the benchmark package. For split reporting, the 750-story held-out test set is represented as an aggregate test partition and is further subdivided into a 500-story public-test subset and a 250-story hidden-test subset.

4pt

Story identifiers follow the format kb25k_XXXX, where XXXX is a globally sequential four-digit index from 0001 to 5000. The identifier ranges correspond to the benchmark narrative categories: moral-semantic (0001--1000), causal-transition (1001--1800), emotional trajectory (1801--2500), procedural state-change (2501--3100), social interaction (3101--3700), hidden consequence (3701--4200), cultural/folk moral (4201--4600), and counterfactual minimal-pair (4601--5000). Each record stores a unified benchmark identifier together with the corresponding narrative category and benchmark split.

0.0.6.16 Dataset Provenance, Licensing, and Annotation Production

KathaBench-25K is constructed from three classical narrative source families: Kathasaritsagara / The Ocean of Story, Aesop’s Fables, and Panchatantra. These collections were selected because they contain compact, visually renderable narratives with explicit event progression, consequence-bearing actions, moral-semantic targets, and recoverable transition structure. Package-level source provenance, representative edition references, rights-scope notes, and traceability links are documented in source_manifest.json. Dataset-level purpose, intended use, limitations, ethical considerations, and licensing are documented in dataset_card.md. The released dataset is available at https://huggingface.co/datasets/iamjamuna/KathaBench-25K.

Each released record stores normalized benchmark content together with a provenance object containing source_dataset (the source collection: kathasaritasagara, aesop, or panchatantra), source_prompt_hash, source_annotations_sha256, source_contrastives_sha256, contrastive_variant_ids, created_at, source_rights_status (set to public_domain_or_rights_compatible for released records), and dataset_license (set to CC-BY-4.0). Story title is stored at the top level of the record as title; record-level validation_status and evidence_status are also stored at the top level. The benchmark release should be interpreted as a normalized evaluation artifact derived from screened source pools, not as a redistribution of any specific third-party edition or website text.

Table 23 reports the source-family mix used to construct KathaBench-25K. Table 24 summarizes the provenance, production, licensing, and validation controls applied before release.

3.5pt

3.5pt

0.0.6.17 Source-Rights Status

The KathaBench-25K release separates record-level provenance from package-level source documentation. Each record stores the source family through source_dataset, the rights-screening result through source_rights_status, and the benchmark artifact license through dataset_license. The accompanying source_manifest.json documents collection-level, book-level, and representative edition-level references consulted during construction, including Aesop/Perry-index references, Panchatantra recension and translation references, and The Ocean of Story references for the Kathasaritsagara.

The source manifest supports provenance, traceability, and source-family verification. It should not be interpreted as a blanket redistribution license for third-party edition text, reference websites, or publisher pages. The benchmark package contains normalized evaluation artifacts: benchmark narratives, scene decompositions, transition annotations, recoverability questions, accepted-answer sets, contrastive variants, metadata, aggregate statistics, dataset documentation, and validation records. These benchmark artifacts are released under CC BY 4.0. Evaluation code and validation scripts are released under Apache-2.0.

As summarized in Table 24, records are included only if they pass schema validation, source-family traceability checks, rights-status screening, scene and transition validation, recoverability-QA validation, and human-validation review.

Ethics Statement and Responsible Release↩︎

0.0.6.17.0.1 Data sources and rights.

KathaBench-25K is constructed from three classical narrative source families: Kathasaritsagara / The Ocean of Story (1,702 stories), Aesop’s Fables (1,676 stories), and Panchatantra (1,622 stories), with the source mix reported in Table 23. The benchmark records contain normalized five-scene representations, scene decompositions, transition annotations, recoverability questions, accepted-answer sets, contrastive variants, metadata, and validation records. Each record stores source_dataset, source_rights_status, and dataset_license inside its provenance object, while title, validation_status, and evidence_status are stored at the top level. Package-level source references, rights-scope notes, and traceability links are documented in source_manifest.json. Records that fail source-traceability, rights-screening, safety, ambiguity, schema-validation, or human-validation checks are excluded from the benchmark package according to the release controls in Table 24.

0.0.6.17.0.2 License and availability.

KathaBench-25K is publicly available at https://huggingface.co/datasets/iamjamuna/KathaBench-25K. The release includes normalized benchmark narratives, scene decompositions, transition annotations, recoverability questions, accepted-answer sets, contrastive variants, metadata, aggregate statistics, validation records, dataset documentation, licensing information, and source-provenance metadata under CC BY 4.0. Evaluation code and validation scripts are released under Apache-2.0. Third-party source texts, source editions, reference websites, generated storyboard images, and model outputs are not relicensed by the benchmark release and remain subject to their own applicable rights, licenses, and provider terms.

0.0.6.17.0.3 Human annotation and ethics review.

The annotation protocol was reviewed under the authors’ institutional ethics process and determined to be non-human-subjects research because annotators performed dataset labeling and were not themselves the subject of behavioral analysis. Annotators were informed about task purpose, data handling, workload, compensation, and withdrawal rights. No personally identifiable annotator information is released. The validation form in Table 17, adjudication rules in Table 18, and workload audit in Table 20 document the human-review process.

0.0.6.17.0.4 Annotators.

Annotation was performed by a pool of five trained research annotators independent of model development, metric design, pre-review label production, and experimental evaluation. Each released record received three independent reviews. Annotators were recruited for English-language narrative interpretation and dataset-labeling experience, compensated at USD 20/hour or above the applicable local living-wage equivalent, and allowed to withdraw.

0.0.6.17.0.5 Content and PII.

Narratives contain no personally identifiable information and were screened for harmful, offensive, sexually explicit, or targeted abusive content. The source collections are classical folk, fable, and narrative traditions; however, cultural and moral interpretations may vary across editions, translations, and communities. Moral labels are therefore treated as benchmark-specified semantic targets, not normative moral claims or objective moral truth.

0.0.6.17.0.6 Intended use and limitations.

The benchmark is intended for evaluating semantic-trajectory recoverability in generated visual narratives. It is not validated for content moderation, real-world moral judgment, psychological assessment, legal reasoning, or non-narrative visual domains. Cultural and folk narratives may encode culturally specific interpretations; such cases are flagged when identified.

0.0.6.17.0.7 Maintenance.

The dataset will be maintained at a public release repository after review. Errata, versioning, schema changes, and issue reports will be tracked through repository releases and issue tracking. During anonymous review, contact information is anonymized; the camera-ready release will include a maintained contact address.

0.0.6.18 Limitations of the Annotation Process

KathaBench-25K focuses on semantic recoverability rather than photorealism or aesthetic quality. Some narratives admit multiple plausible visual interpretations, especially when meaning depends on culture, emotion, motivation, irony, or delayed consequence. We therefore distinguish invalid items from ambiguous-but-valid items. Ambiguous cases are retained with flags when useful for analysis, but excluded from strict gold evaluation when a single accepted answer is required. This design supports diagnostic evaluation while preventing ambiguous or underspecified records from being counted as image-side semantic loss.

0.0.7 Recoverability Scoring and Semantic Compass Details↩︎

This section specifies the operational scoring details used by KathaTrace on the proposed KathaBench-25K benchmark. The main paper defines the task and the Semantic Trajectory Gap (STG); here we define valid-set construction, evidence isolation, ambiguity filtering, judge aggregation, and the validation-only Semantic Compass selection protocol. Semantic Compass is used as an actionability probe for localized KathaTrace failures, not as a new generator.

0.0.7.1 Recoverability Scoring

Let the released scored QA dimensions in KathaBench-25K be \[\mathcal{K}_{\mathrm{QA}} = \{\mathrm{action}, \mathrm{causal}, \mathrm{emotional}, \mathrm{consequence}, \mathrm{temporal}, \mathrm{moral}\}.\] These correspond to action_visibility, causal, emotional, consequence, temporal_order, and moral in the released QA inventory. The transition field intention is retained for annotation and planning, but is not a separate scored QA dimension.

For each dimension \(k\), let \(Q_k\) be the released question set. Each question \(q\) has a canonical answer \(y_q\) and a frozen accepted-answer set \(\mathcal{A}(q)\). For a method \(m\) with generated storyboard \(\hat{X}_m\), we define the method-specific valid set using the text+image condition: \[\mathcal{V}_{k,m} = \left\{ q \in Q_k : \mathrm{match}\!\left(J(q,(S,\hat{X}_m)), \mathcal{A}(q)\right)=1 \right\}. \label{eq:app95valid95set}\tag{16}\] Questions outside \(\mathcal{V}_{k,m}\) are treated as ambiguous, contradictory, or defective for that method and are excluded from filtered STG. This prevents unclear questions, unusable generated evidence, or source-image contradictions from being counted as image-side semantic loss.

For evidence condition \(z \in \{S,\hat{X}_m,(S,\hat{X}_m)\}\), dimension-specific recoverability is \[R^k_{z,m} = \frac{1}{|\mathcal{V}_{k,m}|} \sum_{q \in \mathcal{V}_{k,m}} \mathbb{I}\!\left[ \mathrm{match}\!\left(J(q,z), \mathcal{A}(q)\right)=1 \right]. \label{eq:app95dimension95recoverability}\tag{17}\] Multiple-choice answers use exact label matching. Free-form answers are normalized before matching against \(\mathcal{A}(q)\). Unsupported, unrelated, or unclear responses are counted as incorrect unless uncertainty is explicitly included in the accepted-answer set.

Overall recoverability is the equal-weighted average across the six scored QA dimensions: \[R_{z,m} = \frac{1}{|\mathcal{K}_{\mathrm{QA}}|} \sum_{k \in \mathcal{K}_{\mathrm{QA}}} R^k_{z,m}. \label{eq:app95overall95recoverability}\tag{18}\] Equal weighting prevents frequent question types from dominating the score. Causal, emotional, consequence, and moral-target gaps are the main latent-transition diagnostics; action visibility and temporal order remain part of full recoverability but are reported separately as local-visibility and ordering-control dimensions.

0.0.7.2 Evidence Conditions and Ambiguity Filtering

KathaTrace evaluates each KathaBench-25K recoverability question under three evidence conditions:

• Text-only: the judge receives only the source story \(S\).

• Image-only: the judge receives only the generated storyboard \(\hat{X}_m\), with no source story, prompts, captions, moral labels, transition annotations, or metadata.

• Text+image: the judge receives both \(S\) and \(\hat{X}_m\); this condition is used only for ambiguity and contradiction control.

For method \(m\), the ambiguity indicator is \[b_m(q) = \mathbb{I}\!\left[ \mathrm{match}\!\left(J(q,(S,\hat{X}_m)), \mathcal{A}(q)\right)=0 \right]. \label{eq:app95ambiguity95indicator}\tag{19}\] The ambiguity rate is \[\rho_{\mathrm{amb},m} = \frac{1}{|Q|} \sum_{q \in Q} b_m(q). \label{eq:app95ambiguity95rate}\tag{20}\]

Table 25 audits the effect of ambiguity filtering on KathaBench-25K. Raw STG is computed before ambiguity filtering, filtered STG is the primary reported metric, and common-valid STG uses only questions valid for all compared methods. The similar filtered and common-valid improvements show that the Semantic Compass gain is not an artifact of method-specific valid-set selection.

4pt

0.0.7.3 Semantic Trajectory Gap

For method \(m\), STG is computed as \[\mathrm{STG}_m = R_{\mathrm{text},m} - R_{\mathrm{image},m}. \label{eq:app95stg}\tag{21}\] A low STG indicates that source-supported transition meaning remains recoverable from generated images. A high STG indicates semantic trajectory collapse under image-only evidence. Negative STG is treated as a diagnostic warning before filtering because it can indicate unstable judging, contradiction, or defective questions.

Dimension-specific gaps are \[\mathrm{STG}_{k,m} = R^k_{\mathrm{text},m} - R^k_{\mathrm{image},m}, \qquad k \in \mathcal{K}_{\mathrm{gap}}, \label{eq:app95dimension95stg}\tag{22}\] where \[\mathcal{K}_{\mathrm{gap}} = \{\mathrm{causal}, \mathrm{emotional}, \mathrm{consequence}, \mathrm{moral}\}.\] These four dimensions are emphasized because they test latent transition meaning rather than only visible object presence or chronological order.

0.0.7.4 Judge Aggregation

KathaTrace reports individual-judge and ensemble scores on KathaBench-25K. For classification-style answers, the ensemble prediction is majority vote: \[\hat{y}_{\mathrm{ens}}(q,z) = \mathrm{mode} \left( \hat{y}_1(q,z),\ldots,\hat{y}_M(q,z) \right), \label{eq:app95judge95majority95vote}\tag{23}\] where \(M\) is the number of judges. Ties are scored conservatively as incorrect. For free-form answers, each response is first normalized and mapped to an accepted-answer match or an unmatched category; the same majority rule is then applied.

Human calibration is reported separately using agreement, Spearman correlation, pairwise agreement, and calibration error. These checks test whether image-only recoverability follows human interpretation rather than a single-model artifact.

0.0.7.5 Semantic Compass Reranking

Semantic Compass uses validation-time KathaTrace signals to select among candidate KathaBench-25K storyboards. Given \(N_c\) candidates \[\{\hat{X}^{(1)},\ldots,\hat{X}^{(N_c)}\},\] it selects \[\begin{align} \hat{X}^{*} = \arg\max_{\hat{X}^{(j)}}\, \Big[ &\lambda_r R^{\mathrm{val}}_{\mathrm{image}}(\hat{X}^{(j)}) + \lambda_t C_{\mathrm{trans}}(\hat{X}^{(j)},\tau) \\ &- \lambda_p P_{\mathrm{copy}}(\hat{X}^{(j)}) \Big]. \end{align}\label{eq:app95semantic95compass}\tag{24}\] Here, \(R^{\mathrm{val}}_{\mathrm{image}}\) rewards validation image-only recoverability, \(C_{\mathrm{trans}}\) rewards visual support for annotated transitions, and \(P_{\mathrm{copy}}\) penalizes repeated or near-duplicate frames. All weights and thresholds are selected on the validation split and frozen before held-out evaluation.

The transition-coverage term is \[C_{\mathrm{trans}}(\hat{X},\tau) = \frac{1}{T-1} \sum_{t=1}^{T-1} \mathbb{I}\!\left[ \max_{u \in \{t,t+1\}} \mathrm{sim}\!\left(\psi(\hat{x}_u),\psi(r_t)\right) > \delta \right], \label{eq:app95transition95coverage}\tag{25}\] and the copy penalty is \[P_{\mathrm{copy}}(\hat{X}) = \frac{1}{T-1} \sum_{t=1}^{T-1} \mathbb{I}\!\left[ \mathrm{sim}\!\left(\psi(\hat{x}_t),\psi(\hat{x}_{t+1})\right) > \gamma \right]. \label{eq:app95copy95penalty}\tag{26}\]

0.0.7.6 Single-Bridge Repair

Bridge repair is intentionally limited to at most one inserted frame per storyboard. This isolates whether the weakest localized transition can be repaired without confounding the result with arbitrary sequence-length expansion. The weakest transition is \[t^{*} = \arg\min_t R^{\mathrm{val}}_{\mathrm{image}} \left(r_t \mid \hat{x}_t,\hat{x}_{t+1}\right). \label{eq:app95weakest95transition}\tag{27}\] If the transition score is below a validation-selected threshold, Semantic Compass generates one intermediate bridge frame from the corresponding transition annotation: \[\hat{X}_{\mathrm{bridge}} = \{\hat{x}_1,\ldots,\hat{x}_{t^*}, \hat{x}_{t^*+\frac{1}{2}}, \hat{x}_{t^*+1},\ldots,\hat{x}_T\}. \label{eq:app95bridge95storyboard}\tag{28}\] The repaired storyboard is accepted only if it improves the frozen selection score without violating the copy-penalty threshold. Random bridge insertion, non-semantic extra-frame insertion, and caption-only reranking are used as controls to separate semantic repair from the effect of adding frames.

0.0.7.7 Evaluation Separation

Validation data are used only for Semantic Compass weights, thresholds, candidate selection, and bridge-repair decisions. Final reporting uses held-out questions, held-out judge prompts, fixed accepted-answer sets, fixed story splits, fixed model settings, and frozen thresholds. The KathaBench-25K splits are train (3,500), validation (750), and held-out test (750). The held-out test split is further subdivided into public-test (500) and hidden-test (250) subsets: public-test is used for main reporting, while hidden-test is reserved for leaderboard-style evaluation. Thus, references to test (750) denote the aggregate held-out test set.

0.0.7.8 Semantic Compass Algorithm

Algorithm 4 summarizes the validation-selected Semantic Compass procedure for KathaBench-25K. The algorithm first generates multiple candidate storyboards for the same source story, scores each candidate using frozen validation-selected recoverability, transition-coverage, and copy-penalty terms, and selects the highest-scoring candidate. It then identifies the weakest adjacent transition and performs at most one bridge-frame repair. The repaired storyboard is accepted only if it improves the frozen score and does not violate the copy threshold. This separation ensures that Semantic Compass is evaluated as an actionability probe for localized KathaTrace failures, not as a separately trained generator or an unconstrained sequence-length expansion method.

0.0.8 Dataset and Recoverability Protocol Details↩︎

This section gives implementation-level details for the released KathaBench-25K protocol that are not repeated in Secs. 0.0.6–0.0.7. Section 0.0.6 describes construction, schema, validation, and release files. Section 0.0.7 defines recoverability scoring, ambiguity filtering, STG, and Semantic Compass. Here we specify the dataset-design rules used to select stories, decompose scenes, write transition-level questions, construct accepted-answer sets, and create contrastive semantic variants.

0.0.8.1 Benchmark Design

KathaTrace is defined for an ordered story with \(T\) scenes, but the proposed KathaBench-25K instance fixes \(T=5\) for controlled comparison across generators. Thus each story contains five ordered scenes and four adjacent transitions. The benchmark design follows three requirements: the intended meaning must depend on relations across scenes, the target must be testable from images alone, and failures must be localizable to a scored QA dimension. Table 26 summarizes these design requirements and the corresponding implementation mechanisms.

4pt

0.0.8.2 Why the Annotation Design Matches the Task?

The annotation design follows directly from the KathaTrace task definition. The benchmark does not ask whether a generated storyboard merely contains the right characters, objects, or settings; it asks whether the transition meaning connecting adjacent scenes remains recoverable from images alone. For this reason, each story is represented as an ordered five-scene sequence with four adjacent transition records. The transition records store source-side fields for action, causality, intention, emotion, consequence, and the benchmark-specified moral target, while the scored QA inventory tests action visibility, causal recoverability, emotional shift, consequence, temporal order, and moral-target recoverability.

This structure makes the benchmark diagnostic rather than only descriptive. If a generated storyboard fails, KathaTrace can localize whether the failure comes from a missing visible action, a broken causal link, a flattened emotional change, an omitted consequence, an ordering error, or a shifted moral-target interpretation. Accepted-answer sets make open-ended scoring deterministic while allowing validated paraphrases, and contrastive variants test whether evaluation tracks transition meaning rather than surface entity overlap. Finally, the text+image condition filters questions that remain ambiguous even with full evidence, preventing ambiguous or defective records from being counted as image-side semantic loss.

0.0.8.3 Narrative Selection and Scene Decomposition

A candidate story is included in KathaBench-25K only if its interpretation depends on event progression. It must be decomposable into five visually renderable scenes, contain at least four meaningful adjacent transitions, support recoverability questions with accepted answers, and allow a semantic contrastive variant. Stories are excluded when they are primarily descriptive, depend mainly on non-visual wordplay, require essential external knowledge not present in the story text, contain unresolved rights or provenance uncertainty, or have multiple equally plausible primary moral targets that cannot be adjudicated.

Scene decomposition follows a compact story arc: setup, action, complication, consequence, and resolution. This arc is not treated as a claim about all narrative structure. It is a normalization choice that makes transition-level evaluation comparable across KathaBench-25K records.

0.0.8.4 Transition Annotation Rubric

Each adjacent scene pair is annotated with source-side fields that separate visible events from latent narrative meaning. These fields support recoverability-question construction and failure localization. They are not identical to the released scored QA dimensions: intention is retained as an annotation and planning field, while temporal_order is derived from scene order. Table 27 defines the source-side transition rubric used to construct KathaBench-25K transition records.

4pt

0.0.8.5 Recoverability Questions and Accepted Answers

Recoverability questions are derived from transition annotations rather than isolated captions. Each question targets one released scored dimension: \[\mathcal{K}_{\mathrm{QA}} = \left\{ \begin{array}{l} \texttt{action\_visibility},\;\texttt{causal},\;\texttt{emotional},\\ \texttt{consequence},\;\texttt{temporal\_order},\;\texttt{moral} \end{array} \right\}.\] Each question has a canonical answer and a frozen accepted-answer set. Accepted answers include semantically equivalent paraphrases but exclude answers that are underspecified, visually generic, or shifted to a different transition meaning. Table 28 gives representative KathaBench-25K recoverability questions and accepted-answer patterns for each scored QA dimension.

4pt

0.0.8.6 Evidence-Condition Interpretation

Text-only recoverability checks whether the source story supports the intended answer. Image-only recoverability is the primary diagnostic condition. Text+image recoverability is used only to identify ambiguity, contradiction, or defective questions before STG is computed. Table 29 defines how the three evidence-condition outcomes are interpreted in KathaBench-25K scoring.

4pt

0.0.8.7 Contrastive Semantic Variants

Contrastive variants test whether recoverability tracks transition meaning rather than shared characters, objects, or settings. Each of the 5,000 KathaBench-25K stories has two contrastive variants, producing 10,000 variants in total. A contrastive variant preserves surface entities where possible but changes at least one meaning-bearing transition, such as causality, motivation, emotional shift, consequence, temporal order, or moral-target interpretation.

These variants are distinct from the counterfactual_pair narrative category. The category contains 400 source stories whose original narratives are built around minimal semantic differences, while the 10,000 contrastive variants are additional evaluation artifacts constructed from the full benchmark. Table 30 summarizes the semantic operations used to build these contrastive variants.

4pt

0.0.8.8 Pairwise Contrastive Scoring

Pairwise contrastive scoring asks whether a judge recovers the source trajectory rather than a visually similar contrastive trajectory. For each pair, the source and contrastive outputs share entities or settings but differ in transition meaning. Pair accuracy is \[\mathrm{PairAcc} = \frac{1}{N} \sum_{n=1}^{N} \mathbb{I}\!\left[ \hat{y}^{(n)}_{\mathrm{pair}}=y^{(n)}_{\mathrm{source}} \right], \label{eq:pair95accuracy}\tag{29}\] and confusion is \[\mathrm{Confusion}=1-\mathrm{PairAcc}. \label{eq:pair95confusion}\tag{30}\] High pair accuracy indicates that KathaBench-25K distinguishes transition meaning rather than relying only on object or setting overlap.

0.0.9 Evaluation Metrics↩︎

This section defines the metrics used for reporting KathaBench-25K results. Section 0.0.8 defines the dataset and question inventory, and Section 0.0.7 defines answer matching, valid-set construction, ambiguity filtering, and judge aggregation. All recoverability values are computed on the filtered valid set. Unless otherwise stated, reported table values are expressed as percentage points.

0.0.9.1 Recoverability

Let the released scored QA dimensions be \[\begin{align} \mathcal{K}_{\mathrm{QA}} = \{& \mathrm{action}, \mathrm{causal}, \mathrm{emotional},\\ & \mathrm{consequence}, \mathrm{temporal}, \mathrm{moral} \}. \end{align} \label{eq:metric95kqa}\tag{31}\] These correspond to action_visibility, causal, emotional, consequence, temporal_order, and moral in the released QA inventory.

For method \(m\) and dimension \(k\), let \(\mathcal{V}_{k,m}\) be the filtered valid set after the text+image check. For evidence condition \(z \in \{S,\hat{X}_m,(S,\hat{X}_m)\}\), dimension recoverability is \[R^{k}_{z,m} = \frac{1}{|\mathcal{V}_{k,m}|} \sum_{q \in \mathcal{V}_{k,m}} \mathbb{I}\!\left[ \mathrm{match}\!\left(J(q,z),\mathcal{A}(q)\right)=1 \right]. \label{eq:metric95subtype95recoverability}\tag{32}\] Here, \(S\) is the source story, \(\hat{X}_m\) is the storyboard generated by method \(m\), \(J(q,z)\) is the judge answer, and \(\mathcal{A}(q)\) is the frozen accepted-answer set for question \(q\).

The evidence aliases are \[\begin{align} R^k_{\mathrm{text},m} &= R^k_{S,m},\\ R^k_{\mathrm{image},m} &= R^k_{\hat{X}_m,m},\\ R^k_{\mathrm{text+image},m} &= R^k_{(S,\hat{X}_m),m}. \end{align} \label{eq:metric95evidence95aliases}\tag{33}\]

Overall recoverability is the equal-weighted average across the six scored QA dimensions: \[R_{z,m} = \frac{1}{|\mathcal{K}_{\mathrm{QA}}|} \sum_{k \in \mathcal{K}_{\mathrm{QA}}} R^k_{z,m}. \label{eq:metric95overall95recoverability}\tag{34}\] Equal weighting prevents high-frequency question types from dominating the score. Causal, emotional, consequence, and moral-target gaps are the main latent-transition diagnostics: \[\mathcal{K}_{\mathrm{gap}} = \{ \mathrm{causal}, \mathrm{emotional}, \mathrm{consequence}, \mathrm{moral} \}. \label{eq:metric95kgap}\tag{35}\] Action visibility and temporal order remain part of overall recoverability, but are reported separately because they measure local depiction and ordering rather than latent transition meaning.

0.0.9.2 Semantic Trajectory Gap

For method \(m\), the Semantic Trajectory Gap is \[\mathrm{STG}_{m} = R_{\mathrm{text},m} - R_{\mathrm{image},m}. \label{eq:metric95stg}\tag{36}\] STG measures the amount of source-supported transition meaning that is recoverable from text but not recoverable from the generated images. The percentage-point version used in tables is \[\mathrm{STG}^{\mathrm{pp}}_{m} = 100 \left( R_{\mathrm{text},m} - R_{\mathrm{image},m} \right). \label{eq:metric95stg95percent}\tag{37}\] Lower STG is better. \(\mathrm{STG}=0\) means text-only and image-only recoverability are equal on the filtered valid set. \(\mathrm{STG}>0\) means image-only recoverability is lower than text-only recoverability. \(\mathrm{STG}<0\) is not interpreted as improved semantic preservation; it is flagged for inspection because it can indicate unstable judging, contradiction, or an invalid item before filtering.

Dimension-specific gaps are \[\mathrm{STG}_{k,m} = R^{k}_{\mathrm{text},m} - R^{k}_{\mathrm{image},m}, \qquad k \in \mathcal{K}_{\mathrm{gap}}. \label{eq:metric95subtype95gap}\tag{38}\] A higher \(\mathrm{STG}_{k,m}\) means greater image-side loss for dimension \(k\). Table 31 summarizes how STG values are interpreted for KathaBench-25K.

4pt

0.0.9.3 Moral-Target Recoverability

Let \(\mathcal{M}\) be the 12-label moral-target set. For story \(i\), let \(m_i \in \mathcal{M}\) be the benchmark-specified moral target, and let \(\hat{m}_{i,z}\) be the predicted label under evidence condition \(z\). Moral-target recoverability is \[\mathrm{MTR}_{z} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\!\left[ \hat{m}_{i,z}=m_i \right]. \label{eq:metric95mtr}\tag{39}\] The moral-target gap is \[\mathrm{Gap}_{\mathrm{moral}} = \mathrm{MTR}_{\mathrm{text}} - \mathrm{MTR}_{\mathrm{image}}. \label{eq:metric95moral95gap}\tag{40}\] Moral labels are benchmark-specified semantic targets, not claims of objective moral truth.

For a uniform random predictor over the 12 moral labels, \[\mathrm{Chance}_{\mathrm{MTR}} = \frac{1}{|\mathcal{M}|} = \frac{1}{12}. \label{eq:metric95mtr95chance}\tag{41}\] For any non-uniform subset, the majority-label baseline is \[\mathrm{Majority}_{\mathrm{MTR}} = \max_{m \in \mathcal{M}} \frac{N_m}{N}. \label{eq:metric95mtr95majority}\tag{42}\]

0.0.9.4 Ambiguity Rate

Ambiguity is defined by the text+image condition. For method \(m\), the ambiguity indicator is \[b_m(q) = \mathbb{I}\!\left[ \mathrm{match}\!\left(J(q,(S,\hat{X}_m)),\mathcal{A}(q)\right)=0 \right]. \label{eq:metric95ambiguity95indicator}\tag{43}\] Thus, \(b_m(q)=1\) means the intended answer is not recoverable even when the judge receives both the source story and the generated storyboard. The ambiguity rate is \[\rho_{\mathrm{amb},m} = \frac{1}{|Q|} \sum_{q\in Q} b_m(q). \label{eq:metric95ambiguity95rate}\tag{44}\] Questions with \(b_m(q)=1\) are excluded from filtered STG and counted only in the ambiguity audit. They are not counted as image-side semantic loss.

0.0.9.5 Human–Judge Calibration

Human–judge calibration compares VLM judge outputs with aggregated human judgments on the KathaBench-25K human-gold subsets. For classification-style answers, agreement is \[\mathrm{Acc}_{\mathrm{human}} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\!\left[ v_i=h_i \right], \label{eq:metric95human95acc}\tag{45}\] where \(v_i\) is the VLM or ensemble answer and \(h_i\) is the aggregated human answer.

For ranked or continuous recoverability scores, we report Spearman rank correlation: \[\rho_{\mathrm{sp}} = \mathrm{corr} \left( \mathrm{rank}(v), \mathrm{rank}(h) \right). \label{eq:metric95spearman}\tag{46}\] For pairwise comparisons, agreement is \[\mathrm{PairAgr} = \frac{1}{|\mathcal{P}|} \sum_{(i,j)\in\mathcal{P}} \mathbb{I}\!\left[ \mathrm{sign}(v_i-v_j) = \mathrm{sign}(h_i-h_j) \right]. \label{eq:metric95pairwise95agreement}\tag{47}\] When judges provide confidence scores, expected calibration error is \[\mathrm{ECE} = \sum_{b=1}^{B} \frac{|B_b|}{N} \left| \mathrm{acc}(B_b) - \mathrm{conf}(B_b) \right|. \label{eq:metric95ece}\tag{48}\] Here, \(B_b\) is confidence bin \(b\), \(\mathrm{acc}(B_b)\) is empirical accuracy in that bin, and \(\mathrm{conf}(B_b)\) is mean confidence in that bin.

0.0.9.6 Pairwise Effect Size

For paired human-preference or signed-rank comparisons, we report Wilcoxon signed-rank \(p\)-values with Bonferroni correction when multiple comparisons are tested. We also report rank-biserial correlation: \[r_{\mathrm{rb}} = \frac{W^{+}-W^{-}}{W^{+}+W^{-}}, \label{eq:metric95rank95biserial}\tag{49}\] where \(W^{+}\) and \(W^{-}\) are the positive and negative signed-rank sums. Positive \(r_{\mathrm{rb}}\) means the first condition is preferred; negative \(r_{\mathrm{rb}}\) means the second condition is preferred; values closer to zero indicate a smaller paired effect.

0.0.9.7 Release Validation Status

The released KathaBench-25K JSONL records include a validation_status field. The value human_adjudicator_accepted marks records included in the human-gold or strict human-gold subsets. The value pre_release_reviewed marks records that passed the full three-reviewer release audit but are not part of the adjudicated gold subsets.

0.0.9.8 Metric Summary

Table 32 summarizes the KathaBench-25K reporting metrics and release fields. Recoverability, STG, moral-target recoverability, ambiguity, calibration, and effect-size metrics are computed using the filtered valid-set definitions above unless a table explicitly states otherwise.

4pt

0.0.10 Judge Protocols and Prompts↩︎

This section specifies the judge protocol used for KathaBench-25K recoverability evaluation. The protocol enforces evidence isolation, fixed prompt templates, structured outputs, deterministic decoding where supported, accepted-answer matching, and conservative judge aggregation. Formal aggregation and calibration metrics are defined in Sections 0.0.7 and 0.0.9. The scored QA dimensions are action_visibility, causal, emotional, consequence, temporal_order, and moral. The transition field intention is retained only for annotation and planning; it is not a separate scored QA dimension.

0.0.10.1 Judge Setup

The KathaBench-25K judge pool contains six VLM judges: Gemini 2.5 Flash, Claude Haiku 4.5, Qwen2.5-VL-7B, Qwen2.5-VL-3B, SmolVLM2-2.2B, and SmolVLM2-500M. This pool combines API-hosted and open-weight judges with different capacities and model families. Claude Haiku 4.5 is used as a fast API-hosted judge with the model identifier claude-haiku-4-5-20251001. All judges receive the same evidence restrictions, question text, dimension instructions, output schemas, and decoding settings where supported. Results are reported per judge and as an ensemble to reduce dependence on any single model.

0.0.10.2 Evidence Isolation

Each judge receives one restricted evidence packet per KathaBench-25K question. The answer is never shown to the judge. The image-only condition excludes the source story, generation prompts, captions, scene descriptions, moral labels, transition annotations, and metadata. Table 33 defines the evidence packet for each condition.

4pt

0.0.10.3 Prompt Templates

The prompt templates are fixed before KathaBench-25K evaluation. The judge receives the recoverability question and the corresponding evidence packet defined in Table 33.

0.0.10.3.0.1 Image-only prompt.

You are given only a sequence of generated storyboard images. You do not have access to the original story, generation prompts, captions, scene descriptions, transition annotations, moral labels, or metadata. Answer the question using only what is recoverable from the images. If the answer is not recoverable, return Unclear. Return only the required structured output.

0.0.10.3.0.2 Text-only prompt.

You are given only the source story text. You do not have access to generated images, image prompts, captions, model outputs, or metadata. Answer the question using only the source story. If the story does not contain enough information, return Unclear. Return only the required structured output.

0.0.10.3.0.3 Text+image prompt.

You are given the source story and the generated storyboard images. Identify the answer intended by the source story, then determine whether the image sequence supports, omits, contradicts, or leaves ambiguous that answer. If the intended answer is not clear even with both modalities, return Ambiguous. Return only the required structured output.

0.0.10.4 Scored QA Dimension Instructions

Each KathaBench-25K question is paired with a dimension instruction. The instruction identifies the type of recoverable meaning being evaluated but does not reveal the answer. Table 34 lists the instruction used for each scored QA dimension.

4pt

0.0.10.5 Output Schema

All KathaBench-25K judges return structured outputs. This avoids ambiguity from long free-form explanations and enables deterministic matching against the frozen accepted-answer sets.

0.0.10.5.0.1 Multiple-choice or label questions.

{
	"answer": "<label>",
	"confidence": "<low|medium|high>"
}

0.0.10.5.0.2 Short-answer recoverability questions.

{
	"answer": "<short phrase>",
	"evidence_status": "<recoverable|unclear|omitted|contradicted>",
	"confidence": "<low|medium|high>"
}

0.0.10.5.0.3 Text+image ambiguity checks.

{
	"source_answer": "<short phrase>",
	"image_support": "<supported|omitted|contradicted|ambiguous>",
	"final_answer": "<short phrase or label>",
	"confidence": "<low|medium|high>"
}

The answer or final_answer field is normalized before matching. Let \(a(q)\) be the normalized judge answer for question \(q\), and let \(\mathcal{A}(q)\) be the frozen accepted-answer set. The binary match is computed as \[\begin{align} M(q) = \mathbb{I}\!\left[ a(q) \in \mathcal{A}(q) \right]. \end{align} \label{eq:judge95output95match}\tag{50}\]

Responses marked as uncertain or unsupported are scored as incorrect unless uncertainty is explicitly part of the accepted-answer set: \[\begin{align} M(q)=0 \quad \text{if} \quad s(q) \in \{& \texttt{Unclear}, \texttt{Ambiguous}, \texttt{unclear},\\ & \texttt{omitted}, \texttt{contradicted} \} \end{align} \label{eq:judge95unclear95scoring}\tag{51}\] where \(s(q)\) is the returned evidence status or image-support status.

0.0.10.6 Answer Normalization and Matching

Before scoring, KathaBench-25K judge answers are lowercased, stripped of punctuation, normalized for whitespace, and mapped to canonical labels when applicable. Multiple-choice and moral-target answers use exact label matching after normalization. Short answers are matched against the accepted-answer set \(\mathcal{A}(q)\), which contains validated paraphrases of the canonical answer. Answers that are unrelated, underspecified, contradictory, or unsupported by the evidence packet are mapped to an unmatched category and scored as incorrect.

0.0.10.7 Judge Ensemble

For each KathaBench-25K question and evidence condition, normalized judge outputs are mapped to either an accepted answer or an unmatched category. The ensemble prediction is the majority vote over judge outputs. Ties are scored as incorrect. This conservative rule prevents recoverability scores from being inflated by judge disagreement. The formal ensemble equation is given in Section 0.0.7.

0.0.10.8 Reproducibility Settings

All KathaBench-25K judge evaluations use fixed question order, image order, answer options, prompt templates, output schemas, and accepted-answer sets. Deterministic decoding is used whenever supported by the judge. For multi-image inputs, storyboard frames are provided in chronological order with frame indices only; frame indices preserve ordering but do not include captions, semantic labels, or transition descriptions. Table 35 summarizes the reproducibility controls used during judging.

4pt

0.0.11 Human Evaluation and Calibration↩︎

This section describes the human evaluation used to calibrate KathaTrace on KathaBench-25K. Human judgments are used to test whether image-only recoverability, STG, moral-target recovery, and Semantic Compass repair trends align with human interpretation under controlled evidence conditions.

0.0.11.1 Human Evaluation Setup

Human evaluation is conducted on the strict 400-story KathaBench-25K human-gold subset, with 50 stories from each of the eight narrative categories. Each item receives at least three independent human judgments per evidence condition. Evaluators follow the same evidence restrictions used for VLM judges. In the image-only condition, they see only generated storyboard images and do not see the source story, prompts, captions, transition annotations, moral labels, method names, or model names.

The human study contains four task types: image-only recoverability QA, text-only recoverability QA, text+image ambiguity checking, and pairwise preference over anonymized storyboards. For Semantic Compass, participants compare the original storyboard with the repaired storyboard in randomized order. Story order, question order, answer-option order, and left–right placement are randomized. Participants may select Unclear when the answer is not recoverable, preventing forced guessing from inflating recoverability. Responses are excluded when participants fail attention checks, complete tasks unrealistically quickly, or give inconsistent answers on repeated controls. Table 36 summarizes the KathaBench-25K human-evaluation design and reporting statistics.

4pt

0.0.11.2 Human Moral-target Recoverability

Human moral-target recoverability measures whether the benchmark-specified moral-semantic target can be recovered from the available evidence. Let \(y_s\) be the intended moral target for story \(s\), and let \(h^{(1)}_{i,s}\) be the top-1 moral label selected by rater \(i\). Top-1 human moral-target recoverability is \[\mathrm{HMR@1} = \frac{1}{|\mathcal{D}|} \sum_{(i,s)\in \mathcal{D}} \mathbb{I}\!\left[ h^{(1)}_{i,s}=y_s \right], \label{eq:hmr95top1}\tag{52}\] where \(\mathcal{D}\) is the set of human moral-label judgments. Let \(H^{(3)}_{i,s}\) be the top-3 label set selected by rater \(i\). Top-3 human moral-target recoverability is \[\mathrm{HMR@3} = \frac{1}{|\mathcal{D}|} \sum_{(i,s)\in \mathcal{D}} \mathbb{I}\!\left[ y_s \in H^{(3)}_{i,s} \right]. \label{eq:hmr95top3}\tag{53}\]

As reported in Table 36, HMR@1 is 46.5%, while HMR@3 is 90.0%. This gap shows that exact top-1 moral recovery is strict: several visually plausible interpretations may be semantically close to the intended target. We therefore report both HMR@1 and HMR@3. Moral labels are benchmark-specified semantic targets, not claims of objective moral truth.

0.0.11.3 Human Rating Scores

Human raters score each KathaBench-25K storyboard on three 1–5 subjective dimensions: Narrative Coherence Score (NCS), Moral Alignment Score (MAS), and Emotional Expressiveness Score (EES). For rating dimension \(d \in \{\mathrm{NCS},\mathrm{MAS},\mathrm{EES}\}\), the mean score is \[\mu_d = \frac{1}{|\mathcal{R}_d|} \sum_{(i,s)\in \mathcal{R}_d} r^{d}_{i,s}, \label{eq:human95rating95mean}\tag{54}\] where \(r^{d}_{i,s}\) is the rating assigned by rater \(i\) to story \(s\).

The observed rating scores are \[\begin{align} \mathrm{NCS} &= 3.69 \pm 0.91,\\ \mathrm{MAS} &= 3.25 \pm 1.16,\\ \mathrm{EES} &= 3.18 \pm 1.12. \end{align} \label{eq:human95ratings}\tag{55}\] These values are also summarized in Table 36. NCS is higher than MAS and EES, showing that visual coherence can be judged favorably even when moral-target or emotional recoverability is weaker. These ratings are used as descriptive human-study signals, not as dataset-label reliability evidence.

0.0.11.4 Ordinal Rating Agreement Diagnostics

Ordinal 1–5 ratings are reported only as subjective human-study diagnostics for KathaBench-25K. We do not use ordinal Krippendorff \(\alpha\) as evidence for dataset-label reliability. The reason is statistical rather than semantic: NCS, MAS, EES, and preference ratings have restricted score ranges and high concentration around positive scores. Under such range compression, chance-corrected ordinal agreement can approach zero even when raters show similar practical trends.

In our audit, ordinal Krippendorff \(\alpha\) values are near zero for NCS, MAS, EES, and pairwise preference. This does not contradict the categorical agreement results. Categorical dataset labels are evaluated using Fleiss’ \(\kappa\) and adjudication, while ordinal ratings are treated as descriptive perception scores. Therefore, the near-zero ordinal \(\alpha\) is interpreted as a limitation of chance-corrected agreement under compressed ordinal ratings, not as evidence that the benchmark labels are unreliable.

0.0.11.5 Human Recoverability and Human STG

Human recoverability uses the same accepted-answer matching and evidence isolation as VLM judging. Let \(H_i(q,z)\) be the answer from human participant \(i\) for question \(q\) under evidence condition \(z\in\{\mathrm{text},\mathrm{image},\mathrm{text{+}image}\}\). Binary correctness is \[M_i(q,z) = \mathbb{I}\!\left[ \mathrm{match}\!\left(H_i(q,z),\mathcal{A}(q)\right)=1 \right]. \label{eq:human95binary95match}\tag{56}\]

For each scored QA dimension \(k\), human recoverability is \[R^{k,\mathrm{human}}_{z} = \frac{1}{|Q^{\mathrm{valid}}_k|} \sum_{q \in Q^{\mathrm{valid}}_k} \frac{1}{n_q} \sum_{i=1}^{n_q} M_i(q,z), \label{eq:human95subtype95recoverability}\tag{57}\] where \(Q^{\mathrm{valid}}_k\) is the ambiguity-filtered question set and \(n_q\) is the number of raters for question \(q\). Overall human recoverability is \[R^{\mathrm{human}}_{z} = \frac{1}{|\mathcal{K}_{\mathrm{QA}}|} \sum_{k \in \mathcal{K}_{\mathrm{QA}}} R^{k,\mathrm{human}}_{z}. \label{eq:human95overall95recoverability}\tag{58}\] The human Semantic Trajectory Gap is \[\mathrm{STG}^{\mathrm{human}} = R^{\mathrm{human}}_{\mathrm{text}} - R^{\mathrm{human}}_{\mathrm{image}}. \label{eq:human95stg}\tag{59}\]

Table 37 reports human recoverability and human STG on the strict 400-story KathaBench-25K human-gold subset.

8pt

Human STG is positive, confirming that some source-supported transition meaning remains easier to recover from text than from generated images. This supports the central recoverability diagnosis without treating either human or VLM judgments as objective truth.

0.0.11.6 Human–VLM Calibration

To test whether automated recoverability scores align with human judgments on KathaBench-25K, we compare VLM outputs with majority human answers. Let \(h_q\) be the majority human answer for question \(q\), and let \(v_q\) be the VLM or VLM-ensemble answer. Human–VLM agreement is \[\mathrm{Acc}_{\mathrm{human}} = \frac{1}{|Q|} \sum_{q \in Q} \mathbb{I}[v_q = h_q]. \label{eq:human95vlm95acc}\tag{60}\] For continuous recoverability scores, Spearman rank correlation is \[\rho_{\mathrm{sp}} = \mathrm{corr} \left( \mathrm{rank}(R^{\mathrm{VLM}}_{\mathrm{image}}), \mathrm{rank}(R^{\mathrm{human}}_{\mathrm{image}}) \right). \label{eq:human95vlm95spearman}\tag{61}\] When judges provide confidence scores, expected calibration error is \[\mathrm{ECE} = \sum_{b=1}^{B} \frac{|B_b|}{N} \left| \mathrm{acc}(B_b) - \mathrm{conf}(B_b) \right|, \label{eq:human95ece}\tag{62}\] where \(B_b\) is confidence bin \(b\), \(\mathrm{acc}(B_b)\) is empirical accuracy in that bin, and \(\mathrm{conf}(B_b)\) is mean confidence in that bin.

Table 38 reports the calibration summary for the VLM ensemble against aggregated human judgments.

4pt

The calibrated VLM ensemble achieves 77.1% agreement with majority human labels, Spearman correlation \(\rho_{\mathrm{sp}}=0.74\), and \(\mathrm{ECE}=0.081\). These results show substantial but imperfect alignment. KathaTrace therefore uses VLMs as scalable calibrated proxies, not as final arbiters of narrative meaning.

0.0.11.7 Preference Scale and Statistical Testing

For pairwise semantic preference and bridge-repair preference, participants use a five-point ordinal scale: \[p_{ij} \in \{-2,-1,0,+1,+2\}. \label{eq:human95preference95scale}\tag{63}\] Positive scores indicate preference for the lower-STG, KathaTrace-guided, or Semantic Compass repaired storyboard. Negative scores indicate preference for the baseline or unrepaired storyboard. A score of zero indicates no preference or comparable recoverability.

Because the preference scale is ordinal, pairwise comparisons use a one-sided Wilcoxon signed-rank test: \[H_0: \mathrm{median}(p_{ij}) = 0, \qquad H_1: \mathrm{median}(p_{ij}) > 0. \label{eq:wilcoxon95hypothesis}\tag{64}\] When multiple criteria are tested, Bonferroni correction is applied: \[\alpha_{\mathrm{corr}} = \frac{\alpha}{C}, \label{eq:bonferroni}\tag{65}\] where \(C\) is the number of tested criteria.

Rank-biserial correlation is reported as the effect size: \[r_{\mathrm{rb}} = \frac{W^{+}-W^{-}}{W^{+}+W^{-}}, \label{eq:rank95biserial}\tag{66}\] where \(W^{+}\) and \(W^{-}\) are the positive and negative signed-rank sums.

0.0.12 Implementation and Reproducibility↩︎

This section summarizes the implementation, evaluation stack, planner–generator configurations, fairness controls, seed policy, compute setup, reproducibility artifacts, and practical evaluation cost used for the proposed KathaBench-25K benchmark. KathaTrace is an evaluation and diagnostic framework rather than a trainable storyboard generator. Reproducibility therefore depends on fixed data splits, fixed transition annotations, fixed recoverability questions, fixed accepted-answer sets, fixed evidence conditions, deterministic judge prompts, archived judge outputs, valid-subset accounting, and held-out evaluation questions.

0.0.12.1 Evaluation Stack

KathaTrace is implemented as a modular evaluation pipeline for KathaBench-25K. Given a source narrative, generated storyboard, transition annotations, recoverability questions, and accepted-answer sets, the pipeline constructs text-only, image-only, and text+image evidence packets. VLM judges answer fixed questions under each evidence condition, and the scoring module computes text-side recoverability, image-side recoverability, Semantic Trajectory Gap (STG), latent dimension gaps, moral-target recoverability, ambiguity flags, valid-subset statistics, and human-calibration summaries. Table 39 summarizes the components of this evaluation stack and clarifies that none of them trains a new generator.

4pt

The stack is generator-agnostic: any method can be evaluated if it produces an ordered image sequence for the same source narrative. KathaTrace does not require access to generator internals.

0.0.12.2 Planner–Generator Configurations

We evaluate each storyboard system as a planner–generator pipeline. The planner converts the source story into a generator-facing representation, and the generator renders the ordered storyboard images. All pipelines are evaluated on the same KathaBench-25K splits, transition annotations, recoverability questions, accepted-answer sets, evidence conditions, judge prompts, ambiguity-filtering protocol, and seed schedule where supported. Table 40 lists the evaluated planner–generator interfaces and the fairness treatment applied to each configuration.

3.5pt

Closed systems are reported separately because their internal prompt rewriting, safety filtering, planning, hardware, and model-version details are not fully controllable. The oracle planner is evaluated only on a calibrated subset and is used as an upper bound, not as a directly comparable deployed method.

0.0.12.3 Fair Comparison Controls

Storyboard systems differ in their supported input formats. To avoid penalizing a method for interface mismatch, each planner–generator pipeline is evaluated under its strongest compatible setting while preserving the same KathaBench-25K source narratives, scene order, transition annotations, recoverability questions, accepted-answer sets, evidence conditions, judge prompts, and ambiguity-filtering protocol.

If a method fails to produce a complete storyboard, the example is counted as invalid for that method. Text-side recoverability is computed on the same valid subset used for the method’s image-side evaluation, so STG reflects the evaluated examples. Common-valid-subset results are computed separately for stricter cross-method comparison. Table 41 summarizes the fairness controls used for all reported planner–generator comparisons.

4pt

0.0.12.4 Semantic Compass Selection Protocol

This subsection details how Semantic Compass selects candidates, freezes validation-time settings, separates validation from held-out evaluation, and controls for sequence-length effects. Semantic Compass is a post-generation reranking and bridge-repair layer; it does not train a new generator.

KathaBench-25K uses three primary splits: train (3,500 stories), validation (750 stories), and held-out test (750 stories). The held-out test split is further subdivided into public-test (500) and hidden-test (250). Candidate count, score weights \(\lambda_r,\lambda_t,\lambda_p\), transition-coverage threshold \(\delta\), copy threshold \(\gamma\), and repair threshold are selected only on the validation split and frozen before test evaluation. In the reported setup, Semantic Compass uses the three open-generator outputs produced by generation seeds \(\{42,123,2026\}\) as the fixed candidate pool, so \(N_c=3\).

Final Semantic Compass reporting uses held-out recoverability questions and held-out judge prompts, disjoint from validation-time selection. This separation prevents test STG from being optimized by the same questions or prompts used for reranking or repair. Because bridge repair changes sequence length, we compare against length-controlled alternatives: random bridge insertion, non-semantic extra-frame insertion, caption-only bridge insertion, candidate reranking alone, and bridge repair alone. Table 42 summarizes the selection quantities, where they are chosen, and how they are used in held-out reporting.

4pt

0.0.12.5 Seeds, Runs, and Determinism

All deterministic preprocessing, split construction, question ordering, bootstrap sampling, and candidate ordering use master seed 42. For stochastic open-generator outputs, we use three generation seeds, \(\{42,123,2026\}\), and report story-level means with 95% bootstrap confidence intervals computed using 10,000 bootstrap resamples. VLM judging uses deterministic decoding and fixed output schemas; therefore each judge is run once per evidence packet, and raw judge outputs are archived for reproducibility. Open-generator and open-VLM experiments are run on NVIDIA A100 80GB GPUs. Closed systems are reported as API-based reference baselines with model name, access date, and available version metadata, since their internal hardware, prompt rewriting, and model versions are not fully controllable. Table 43 reports the run and seed configuration used in KathaBench-25K.

4pt

0.0.12.6 Compute Environment

KathaTrace does not require training a large model. Its main compute cost comes from storyboard generation and VLM judging. Cached judge outputs are reused for recoverability scoring, ambiguity analysis, latent dimension-gap computation, valid-subset accounting, and calibration. Table 44 summarizes the compute environment, seed settings, and execution modes used for KathaBench-25K experiments.

4pt

The number of judge calls scales as: \[C_{\mathrm{judge}} = N_{\mathrm{stories}} \cdot N_{\mathrm{methods}} \cdot N_{\mathrm{questions}} \cdot N_{\mathrm{conditions}} \cdot N_{\mathrm{judges}}, \label{eq:impl95judge95cost}\tag{67}\] where \(N_{\mathrm{conditions}}=3\) for text-only, image-only, and text+image evaluation. For the reported benchmark configuration, KathaBench-25K contains 5,000 narratives, 25,000 structured scenes, 20,000 transition annotations, 28,712 recoverability questions, 10,000 contrastive variants, and six scored QA dimensions. The split is 3,500 train, 750 validation, and 750 held-out test examples. Main planner–generator tables are reported on the held-out evaluation split with method-specific valid-subset accounting.

0.0.12.7 Reproducibility Checklist

The held-out test split contains 750 stories and is subdivided into public-test (500) and hidden-test (250). References to test (750) denote the aggregate held-out test set. Table 45 lists the artifacts required to audit the KathaBench-25K benchmark, evaluation protocol, Semantic Compass selection procedure, and reported metrics.

4pt

For deterministic reporting, we store generated storyboard paths, split identifiers, scene decompositions, transition annotations, valid-subset masks, judge responses, normalized answers, match decisions, ambiguity flags, repair decisions, seed values, candidate order, and final metric tables. Each reported result can therefore be traced to the corresponding story, planner, generator, seed, question, evidence condition, judge, accepted answer set, and repair configuration.

0.0.12.8 Practical Evaluation Cost

KathaTrace deliberately trades lightweight caption-style scoring for diagnostic recoverability evaluation. Its evaluation cost comes from testing each recoverability question under fixed evidence conditions, normalizing structured judge outputs, matching responses against frozen accepted-answer sets, and aggregating transition-level metrics. This design is more expensive than caption or object-overlap metrics, but it is necessary for measuring whether source-supported transition meaning remains recoverable from generated storyboards.

The released KathaBench-25K benchmark contains 5,000 stories, 25,000 annotated scenes, 20,000 adjacent transitions, and 28,712 recoverability questions. Evaluating each question under the three evidence conditions—text-only, image-only, and text+image—requires 86,136 judge calls for a single judge. The cost scales linearly with the number of judges; for example, a three-judge ensemble requires 258,408 judge calls. Storyboard generation is evaluated separately from judge calls and requires 25,000 generated panels for the full benchmark, 5,000 panels for human_gold_1k.jsonl, and 2,000 panels for strict_gold_400.jsonl.

To support different compute budgets, we release two evaluation subsets in addition to the full benchmark. Table 46 reports the number of stories, panels, QA pairs, and judge calls for the strict, human-gold, and full KathaBench-25K evaluation settings. The human_gold_1k.jsonl subset provides a medium-scale evaluation setting, while strict_gold_400.jsonl provides a rapid high-confidence setting for implementation checks, method debugging, and low-budget reproducibility. For this low-budget setting, we recommend reporting the same evidence conditions, judge model, valid-subset accounting, and bootstrap confidence intervals used in the full benchmark. The strict subset is not a replacement for full-benchmark reporting, but it provides a fixed smoke test for verifying implementation correctness before running the complete evaluation.

3.5pt

Since transition annotations, recoverability questions, accepted-answer sets, and evaluation metadata are precomputed, evaluating a new method only requires generating storyboards and running the fixed protocol. We recommend reporting the evaluation subset, number of generated panels, judge calls, judges, evidence conditions, and confidence intervals together with STG.

0.0.12.8.0.1 Leaderboard evaluation cost.

For leaderboard-style use, the held-out test split is divided into a public-test subset and a hidden-test subset. The public-test subset supports open comparison and debugging, while the hidden-test subset supports leaderboard-style evaluation without exposing final test examples. We report leaderboard cost in generated panels and judge calls rather than wall-clock time, because wall-clock runtime depends on the submitted generator, batching strategy, API latency, and hardware. Once a method has produced the required storyboard panels, the benchmark-side evaluation cost is fixed by the number of stories, QA pairs, evidence conditions, and judges.

3.5pt

Table 47 separates generator-side cost from benchmark-side evaluation cost. Generator-side cost is the number of storyboard panels a submitted method must produce. Benchmark-side cost is the number of judge calls required after the panels are generated. For one judge, the judge-call count is approximately three times the number of QA pairs because KathaTrace evaluates text-only, image-only, and text+image evidence conditions. For multi-judge evaluation, the values in Table 47 scale linearly with the number of judges.

0.0.12.9 Community Adoption and Reuse

KathaBench-25K is designed so that a new storyboard generator can be evaluated without changing the benchmark. For example, a researcher proposing a new StoryDiffusion-style, FLUX-based, or consistency-aware storyboard method can take the same source stories from annotations.jsonl, generate five ordered panels for each story, and run the fixed KathaTrace judge packets. The evaluator does not need access to the generator’s model weights, hidden prompts, training data, or intermediate plans. It only needs the generated storyboard images, the released story identifier, the fixed recoverability questions, and the frozen accepted-answer sets.

This makes comparison practical and diagnostic. If a new method improves character consistency but still omits the key rescue action in a reciprocity story, KathaTrace will show a high consequence or moral-target gap rather than only reporting that the images look coherent. If a method preserves the main characters in The Monkey and the Crocodile but fails to communicate the crocodile’s hidden betrayal, the image-only causal or moral-target recoverability will remain low. If a model adds visually pleasing extra frames but does not repair the missing transition, the length-control protocol separates that from true semantic improvement. Thus, future users can diagnose whether a method improves visual quality, entity consistency, causal recovery, emotional recovery, consequence recovery, or moral-target recovery.

The release also supports different evaluation budgets. A full evaluation uses 5,000 stories, 25,000 panels, 28,712 recoverability questions, and 86,136 judge calls for one judge across the three evidence conditions. A medium-cost evaluation can use human_gold_1k.jsonl, with 1,000 stories and approximately 17,226 one-judge calls. A rapid smoke test can use strict_gold_400.jsonl, with 400 stories and approximately 6,891 one-judge calls. These subsets let researchers debug their pipeline before running the full benchmark, while still using the same evidence isolation, accepted-answer matching, ambiguity filtering, and STG computation.

The reusable artifacts are intended to make results auditable rather than one-off. The release includes fixed train/validation/test splits, source narratives, five-scene decompositions, adjacent-scene transition annotations, recoverability questions, accepted-answer sets, contrastive variants, human-validation metadata, ambiguity flags, and evaluation subsets. A future paper can therefore report not only an aggregate STG score, but also causal, emotional, consequence, and moral-target gaps, ambiguity rate, valid-subset size, and common-valid STG. This allows the community to compare new visual narrative generators under the same protocol as the field improves, making KathaBench-25K a reusable diagnostic benchmark rather than only a static story-prompt collection.

0.0.13 Human Perception Study↩︎

This section reports the human perception study used to validate whether KathaTrace-guided outputs on KathaBench-25K are perceived as preserving narrative meaning better than matched baselines. The study is separate from the dataset-construction review in Sec. 0.0.6 and the human–VLM calibration study in Sec. 0.0.11. Here, participants compare anonymized storyboards and rate whether causal, emotional, consequence-bearing, moral-target, and overall transition meaning is visually recoverable. Human judgments are used as perception evidence, not as objective narrative-truth labels.

0.0.13.1 Study Protocol

Participants completed image-only preference and recoverability tasks on KathaBench-25K storyboards. In the primary condition, they saw generated storyboard images only. They did not see the source story, generation prompts, captions, transition annotations, moral labels, method names, or model names. This prevents source-text leakage and tests whether the intended transition meaning is visually recoverable from the storyboard itself. Table 48 summarizes the study tasks, evidence shown to participants, and the purpose of each task.

For pairwise trials, participants compared two anonymized storyboards generated for the same KathaBench-25K source story. For bridge-repair trials, participants compared an original storyboard with a Semantic Compass repaired storyboard in randomized left–right order. Story order, question order, answer-option order, and left–right placement were randomized. Participants could select Unclear or no preference when the available evidence did not support a decision.

4pt

0.0.13.2 Preference Scale

Pairwise and bridge-repair trials use the five-point ordinal preference scale in Eq. 68 . Positive scores favor the lower-STG, KathaTrace-guided, or Semantic Compass repaired storyboard. Negative scores favor the baseline or unrepaired storyboard. A score of zero indicates no preference or comparable recoverability. Table 49 gives the interpretation of each score used in the KathaBench-25K perception study.

\[p_{ij} \in \{-2,-1,0,+1,+2\}. \label{eq:human95preference95scale95final}\tag{68}\]

5pt

0.0.13.3 Perception Criteria

The KathaBench-25K perception study uses 15 criteria. These criteria are broader than the released scored QA inventory and are used only for human preference analysis. Intention-related criteria are included because participants may naturally infer character goals from images; however, intention remains an annotation and planning field, not a released scored QA dimension. The scored QA inventory remains action_visibility, causal, emotional, consequence, temporal_order, and moral. Table 50 lists the 15 perception-study questions shown to participants.

4pt

0.0.13.4 Preference Aggregation and Statistical Testing

For criterion \(c\), the mean preference score is \[\mu_c = \frac{1}{N_c} \sum_{(i,j)\in D_c} p_{ij}, \label{eq:human95preference95mean95final}\tag{69}\] where \(D_c\) is the set of participant judgments for criterion \(c\). We report mean preference with 95% confidence intervals using participant-level bootstrap resampling.

Because preference scores are ordinal, significance is tested using a one-sided Wilcoxon signed-rank test: \[H_0:\mathrm{median}(p_{ij})=0, \qquad H_1:\mathrm{median}(p_{ij})>0. \label{eq:human95wilcoxon95final}\tag{70}\] A positive median indicates preference for the lower-STG, KathaTrace-guided, or repaired storyboard. For multiple criteria, we apply Bonferroni correction: \[\alpha_{\mathrm{corr}} = \frac{\alpha}{C}, \label{eq:human95bonferroni95final}\tag{71}\] where \(C\) is the number of tested criteria. Rank-biserial correlation is reported as the ordinal effect size: \[r_{\mathrm{rb}} = \frac{W^+ - W^-}{W^+ + W^-}, \label{eq:human95rank95biserial95final}\tag{72}\] where \(W^+\) and \(W^-\) are the positive and negative signed-rank sums.

0.0.13.5 Human Preference Results

Figures 5–10 summarize the KathaBench-25K perception-study results. Scores above zero indicate preference for the lower-STG, KathaTrace-guided, or repaired storyboard. The distributional plots in Figs. 5 and 6 show that preferences generally shift above the no-preference baseline. The grouped and ranked summaries in Figs. 7–9 show that the strongest preferences occur for overall meaning preservation, visual coherence, moral-target support, and transition recoverability. Figure 10 provides an additional robustness view across participant groups.

0.0.13.6 Human–VLM Preference Calibration

Human–VLM preference calibration tests whether automated image-only recoverability follows human preference trends on KathaBench-25K. Let \(h_q\) be the majority human answer or preference for item \(q\), and let \(v_q\) be the VLM or VLM-ensemble answer. Human–VLM agreement is \[\mathrm{Acc}_{\mathrm{human}} = \frac{1}{|Q|} \sum_{q\in Q} \mathbb{I}[v_q=h_q]. \label{eq:human95vlm95accuracy95final}\tag{73}\] For continuous recoverability scores, Spearman rank correlation is \[\rho_{\mathrm{sp}} = \mathrm{corr} \left( \mathrm{rank}(R^{\mathrm{VLM}}_{\mathrm{image}}), \mathrm{rank}(R^{\mathrm{human}}_{\mathrm{image}}) \right). \label{eq:human95vlm95spearman95final}\tag{74}\] For pairwise comparisons, agreement is \[\mathrm{PairAgr} = \frac{1}{|\mathcal{P}|} \sum_{(a,b)\in\mathcal{P}} \mathbb{I}\!\left[ \mathrm{sign}(s^{\mathrm{VLM}}_a-s^{\mathrm{VLM}}_b) = \mathrm{sign}(s^{\mathrm{human}}_a-s^{\mathrm{human}}_b) \right]. \label{eq:human95vlm95pairagr95final}\tag{75}\] Figure 11 visualizes the relationship between human image-only recoverability and VLM image-only recoverability.

0.0.13.7 Inter-Rater Agreement for Perception Ratings

The perception study includes subjective ordinal ratings and pairwise preferences. We report their agreement only as diagnostics because these ratings are not dataset-construction labels. For categorical QA labels, we use Fleiss’ \(\kappa\): \[\kappa = \frac{\bar{P}-\bar{P}_e}{1-\bar{P}_e}, \label{eq:fleiss95kappa95final}\tag{76}\] where \(\bar{P}\) is observed agreement and \(\bar{P}_e\) is chance agreement. For ordinal preference scores, we report Krippendorff’s \(\alpha\): \[\alpha = 1-\frac{D_o}{D_e}, \label{eq:krippendorff95alpha95final}\tag{77}\] where \(D_o\) is observed disagreement and \(D_e\) is disagreement expected by chance.

Near-zero ordinal \(\alpha\) values are interpreted cautiously. The perception ratings are compressed around positive scores and contain many nearby ratings rather than wide disagreement. In such restricted ordinal ranges, chance-corrected agreement can be near zero even when aggregate preference trends are stable. Therefore, ordinal \(\alpha\) is not used as evidence for KathaBench-25K dataset-label reliability. Dataset-label reliability is reported separately using trained-annotator categorical agreement and adjudication.

0.0.14 Additional Baseline and Control Results↩︎

This section reports supplementary KathaBench-25K baseline and control results. The goal is to test whether KathaTrace measures transition-level recoverability rather than only visual quality, object overlap, generic QA faithfulness, or sequence length. All reported STG values use the same evidence-isolated judge protocol, frozen accepted-answer sets, ambiguity filtering, and valid-subset accounting defined in Secs. 0.0.7–0.0.10.

0.0.14.1 Extended Planner–Generator Leaderboard

Table 51 reports additional KathaBench-25K planner–generator settings beyond the main paper. Closed systems are included only as reference baselines because their internal prompt rewriting, safety filtering, hardware, and version behavior are not fully controllable. Semantic Compass is reported as a post-generation reranking and single-bridge repair setting applied to Gemma-ST + FLUX candidates; it is not a new standalone generator.

2.6pt

Visual: human visual-quality rating. Text/Image Rec.: recoverability under text-only/image-only evidence. Moral-target Rec.: recovery of the benchmark-specified moral-semantic target.

0.0.14.2 Planner and Generator Ablations

Table 52 separates planning effects from rendering effects on KathaBench-25K. With FLUX fixed, transition-aware planning improves image-only recoverability and reduces STG. With Gemma-ST fixed, stronger renderers improve visual quality, but do not eliminate semantic trajectory loss. This supports the claim that STG is not reducible to image quality alone.

3.0pt

Img.: image-only recoverability. Loc.: dimension-level localization. Human Align.: agreement or rank correlation with human judgments. Fail: invalid generation rate.

0.0.14.3 Evaluator-Baseline Analysis

Table 53 tests whether KathaTrace on KathaBench-25K reduces to generic VQA, object-overlap scoring, or scene-level faithfulness evaluation. Object-centric and generic QA baselines capture visible entities and broad plausibility, but provide weaker transition localization and lower human alignment. Full KathaTrace performs best because it combines transition annotations, dimension-specific questions, frozen accepted answers, contrastive variants, and ambiguity filtering.

3.2pt

0.0.14.4 Length-Controlled Semantic Compass Controls

Bridge-scene repair changes sequence length. Table 54 therefore compares Semantic Compass with length-controlled alternatives on KathaBench-25K using Gemma-ST + FLUX. Random and non-semantic extra frames give only small improvements. Candidate reranking improves recoverability without changing sequence length. Full Semantic Compass gives the strongest improvement by combining validation-selected reranking with transition-localized bridge repair.

3.0pt

0.0.14.5 Latent-Dimension Gap Analysis

Table 55 reports text-to-image gaps for the main latent dimensions in KathaBench-25K. Lower values indicate less loss of recoverable meaning from text to generated images. Action visibility and temporal order are excluded from this table because they are local-visibility and ordering-control dimensions. Intention is excluded because it is an annotation and planning field, not a released scored QA dimension.

4.0pt

Across methods, causal gaps are smaller than consequence and moral-target gaps. This indicates that adjacent causal events are easier to depict than delayed outcomes or abstract story meanings. Semantic Compass reduces all four latent gaps, with the largest absolute gains on consequence and moral-target recoverability.

0.0.14.6 Valid-Subset and Failure Handling

Some generators fail on different KathaBench-25K examples. We therefore report explicit validity accounting. Table 56 uses the test split. Total is the number of requested stories, Valid is the number with complete usable storyboards, Invalid rate counts empty outputs, corrupted images, missing frames, or incomplete storyboards, Raw STG is computed before validity filtering, Filtered STG is computed on each method-valid subset, and Common-valid STG is computed on the intersection of examples valid for all compared methods.

3.2pt

The common-valid column preserves the same qualitative ranking as the method-valid filtered STG column. This indicates that the observed improvements are not explained by invalid-generation filtering. Representative failure cases include missing causal bridge actions, under-specified moral decision points, unrecoverable emotional transitions, and delayed consequences that are visually plausible but semantically incomplete.

0.0.14.7 Scientific Insights Revealed by KathaTrace

KathaTrace reveals three findings that are not captured by visual-quality or generic story-QA evaluation alone. First, visual coherence and semantic-transition recoverability are separable: generated storyboards may preserve characters, setting, and local plausibility while losing the causal, emotional, consequence-bearing, or moral-target link that explains why one scene follows another. Second, this gap is not explained only by renderer quality or generic QA. Transition-specific questions, contrastive variants, ambiguity filtering, and dimension-level scoring improve failure localization and human alignment over object-level, scene-level, and generic multi-frame QA baselines. Third, the gap is actionable: localized low-recoverability transitions can guide candidate reranking and bridge-scene repair, reducing STG without treating Semantic Compass as a new generator.

Together, these results identify semantic trajectory collapse as a distinct failure mode in visual narrative generation. The benchmark therefore contributes not only a dataset and metric, but also an empirical diagnostic: current story generators can produce visually plausible sequences while still failing to preserve recoverable transition meaning.

0.0.15 Additional Ablation Analysis↩︎

This section reports ablations that isolate the main design choices in KathaTrace on the proposed KathaBench-25K benchmark. Unless stated otherwise, all ablations use the same held-out split, generated storyboards, evidence conditions, judge prompts, ambiguity filtering, valid-subset accounting, and frozen accepted-answer sets. The goal is not to introduce new baselines, but to show which protocol components are necessary for reliable transition-level diagnosis.

0.0.15.1 Component Ablation

Table 57 removes one KathaTrace component at a time on KathaBench-25K. Dimension localization measures whether the evaluator identifies the failing semantic dimension. Human correlation measures alignment with human recoverability judgments. Ambiguity leakage is the percentage of defective or underspecified items that would remain in scoring without the corresponding control.

4pt

The largest failure mode in Table 57 is removing text+image filtering: ambiguous or defective items enter the score and make STG less interpretable. Full KathaTrace gives the strongest localization and the lowest ambiguity leakage.

0.0.15.2 Evidence-Condition Ablation

KathaTrace uses three evidence conditions on KathaBench-25K. Text-only estimates source-side recoverability, image-only measures visual recoverability, and text+image identifies ambiguous or contradictory items. Table 58 shows that all three conditions are needed for a clean STG interpretation.

4pt

Image-only evaluation can rank KathaBench-25K storyboards, but it cannot distinguish generator failure from question ambiguity. Table 58 shows that text+image filtering supplies that control, while text-only scoring is needed to interpret STG as text-to-image semantic loss.

0.0.15.3 Question-Construction Ablation

Table 59 compares question-construction strategies on KathaBench-25K. Object-level and scene-level questions are easier but less diagnostic. Transition-derived dimension-specific questions better identify semantic trajectory collapse because they target action visibility, causality, emotional shift, consequence, temporal order, and moral-target recovery.

3.5pt

The full dimension-specific construction in Table 59 gives the highest localization and human correlation. Intention is not included as a scored QA dimension. It remains an annotation and planning field because it is harder to validate against a single canonical answer.

0.0.15.4 Answer-Matching Ablation

KathaTrace uses frozen accepted-answer sets for open-ended KathaBench-25K questions. Table 60 shows why: exact matching rejects valid paraphrases, while keyword and embedding matching accept vague or object-only answers. Accepted-answer sets provide deterministic scoring while allowing validated paraphrases.

3.5pt

The accepted-answer-set rule in Table 60 is used because it avoids nondeterministic semantic matching while still handling validated paraphrases. This is important for KathaBench-25K, where many correct answers are short paraphrases of the same transition meaning.

0.0.15.5 Judge-Aggregation Ablation

Table 61 compares individual judges with ensemble aggregation on KathaBench-25K. The ensemble improves agreement and calibration, so the main tables use ensemble recoverability.

4pt

The ensemble in Table 61 is not treated as ground truth. It is used as a calibrated proxy whose behavior is checked against human judgments.

0.0.15.6 Semantic Compass Scoring-Term Ablation

Semantic Compass uses three scoring terms: image-only recoverability, transition coverage, and a copy penalty. Table 62 ablates these terms on KathaBench-25K using Gemma-ST + FLUX candidates.

3.2pt

Table 62 shows that recoverability and transition coverage improve semantic selection, while the copy penalty prevents repeated-frame shortcuts. The full score gives the best overall trade-off.

0.0.15.7 Held-Out Judge and Held-Out Question Ablation

Table 63 checks whether Semantic Compass gains persist when final KathaBench-25K evaluation uses held-out judge prompts and held-out questions. Scores are slightly lower under the strictest setting, but the improvement remains.

5pt

The strictest setting in Table 63 reduces the risk that improvements are caused by prompt-specific or question-specific overfitting.

0.0.15.8 Moral-Target Granularity Ablation

Table 64 compares moral-target label granularities for KathaBench-25K. Binary and coarse labels are easier but less diagnostic. Open-ended moral answers are expressive but less reliable. The final 12-label taxonomy provides a practical balance between specificity, chance level, and agreement.

4pt

The 12-label setting in Table 64 is used because it preserves diagnostic specificity while keeping agreement and chance level interpretable. Moral labels are benchmark-specified semantic targets, not objective moral truths.

0.0.15.9 Action-Dimension Ablation

Action visibility is included in full KathaBench-25K recoverability, but the main latent-gap analysis focuses on causal, emotional, consequence, and moral-target gaps. Action is more visually local, while the latent dimensions test harder narrative meaning. Table 65 reports how the choice of dimension set affects image recoverability, average STG, and localization.

4pt

Table 65 explains the reporting choice: action visibility remains part of full recoverability, while the main latent-gap table emphasizes causal, emotional, consequence, and moral-target loss.

0.0.16 Additional Qualitative Analysis↩︎

This section provides qualitative support for the main KathaTrace results on KathaBench-25K. The examples show how visually plausible storyboards can still lose recoverable transition meaning. We use three kinds of appendix evidence: dataset examples, same-story multi-model comparisons, and KathaTrace diagnostic/repair cases. These figures are illustrative and are not used as the sole basis for model ranking.

0.0.16.1 Dataset Examples

Figs. 12, 13, and 14 show representative visual-only KathaBench-25K dataset examples. Each row contains a fixed source story, a short intended meaning, and five target scenes. These examples clarify the kind of transition meaning KathaTrace expects to be recoverable from generated storyboards. Table 66 summarizes the story sources and intended meanings used in these qualitative examples.

3.0pt

0.0.16.2 Qualitative Example Selection

Qualitative examples are selected from the held-out KathaBench-25K split using fixed criteria rather than manual visual preference. We compute STG, dimension-level gaps, validity masks, ambiguity flags, and Semantic Compass repair deltas before selecting examples. Invalid generations and examples marked ambiguous under the text+image condition are not used as evidence of semantic trajectory collapse. Table 67 reports the fixed selection groups used for the appendix examples.

3.0pt

0.0.16.3 Same-Story Multi-Model Comparison

For same-story comparisons, the KathaBench-25K source story, intended meaning, scene plan, number of frames, and recoverability questions are held fixed. Only the generator changes. This controls for story difficulty and makes semantic trajectory failures comparable across models. We use SDXL, FLUX, StoryDiffusion, ConSistory, GPT-4o + GPT-image-1, and Nano Banana as generator families. Closed or API-based systems are reported as references because internal prompt rewriting, safety filtering, model versioning, and decoding details are not fully controllable.

KathaTrace does not score only whether the images look good. It tests whether the intended transition meaning can be recovered from the generated images alone. This separates three failure modes: rendering failure, consistency failure, and semantic trajectory failure. The third case is central to KathaTrace: images may look coherent while omitting the causal, emotional, consequence-bearing, or moral-target transition. Table 68 defines the fields used for same-story comparisons, and Table 69 summarizes the expected qualitative diagnostic patterns.

3.0pt

3.0pt

0.0.16.4 KathaTrace Diagnostic and Repair Examples

Figs. 15–19 show KathaBench-25K diagnostic examples covering action, causality, source-side intention, emotion, and consequence transitions. Each figure contains the source story, intended meaning, a ground-truth storyboard, a generated failure case, and a KathaTrace diagnosis or repair. The bottom strip compares generic visual/story benchmarks with KathaTrace and human judgment. These examples support the claim that high local coherence can coexist with poor recoverability of transition meaning.

Table 70 summarizes the diagnostic coverage of the appendix examples. Intention-like cases are included only as qualitative transition-field examples; they are not part of the released scored QA dimensions.

3.0pt

Fig. 20 shows that visually plausible storyboard outputs can differ in how well they preserve the deception-to-refusal transition. The ground-truth storyboard makes the deceptive doctor setup and refusal explicit; weaker generator rows show character drift, implicit deception, or unclear refusal; stronger rows improve scene following but still require image-only recoverability checking. KathaTrace is used here as an evaluator of recoverable narrative meaning, not as a generator.

0.0.17 Hard Cases for KathaTrace↩︎

KathaTrace measures whether source-supported transition meaning is recoverable from generated images. It does not assume that every narrative meaning is objective, universal, or visually obvious. Some stories are hard because the key meaning depends on symbolic action, hidden intention, dialogue-like strategy, or delayed reciprocity across non-adjacent scenes.

The examples below define boundary conditions for the protocol. Red dashed boxes mark the visual evidence needed for the intended meaning to remain recoverable. These cases motivate reporting ambiguity rates, strict-gold filtering, and human calibration alongside STG.

0.0.17.1 Symbolic Moral Sacrifice: King Shibi and the Dove

Fig. 21 shows a hard case where the central meaning is symbolic rather than merely sequential. Scenes 1–2 can be visually recovered as a rescue: the dove seeks protection and the king shelters it. The harder transition occurs in Scenes 3–4, where the hawk’s demand and the king’s self-sacrifice establish the moral target. If a storyboard only shows a king, a dove, and a hawk, the meaning may collapse into “rescue” or “kindness” rather than sacrifice, duty, or justice.

The final divine revelation clarifies the moral frame, but image-only recovery may still require cultural or symbolic knowledge. KathaTrace should therefore treat this example as high-ambiguity unless the storyboard explicitly depicts the weighing scene and the dignity of the sacrifice.

0.0.17.2 Hidden Intention and Verbal Strategy: The Monkey and the Crocodile

Fig. 22 shows a hard case where the key transition is mental and dialogic. Scene 1 establishes apparent friendship, but Scenes 2–3 reveal that the crocodile’s invitation hides a plan to harm the monkey. A visually coherent storyboard may show the monkey riding the crocodile while failing to communicate deception or danger.

The intended meaning depends on two recoverable facts: the crocodile’s friendliness is false, and the monkey survives through quick verbal strategy. Without speech cues, facial cues, or explicit threat evidence, the image-only interpretation may become “friendship,” “travel,” or “river adventure.” This makes the example a boundary case for intention recoverability.

0.0.17.3 Long-Range Reciprocity: The Elephants and the Mice

Fig. 23 shows a hard case where the intended meaning is distributed across the full sequence. Scene 1 introduces harm to the mouse village, Scene 2 shows the mice pleading for mercy, and Scenes 3–5 reverse the power relation when the mice rescue the trapped elephants. If a storyboard preserves only the final rescue, the viewer may recover “small animals help large animals,” but not the fuller reciprocity meaning.

This is a boundary case for adjacent-transition scoring. KathaTrace can test whether the early mercy and later rescue are visually present, but the final moral interpretation requires linking distant events. Such examples show why long-range consequence and reciprocity stories should be interpreted with ambiguity rates and human agreement, not STG alone.

0.0.17.4 Extensibility Beyond Storyboards

KathaTrace is generator-agnostic and format-agnostic. It requires only three inputs: (i) an ordered sequence of narrative units, (ii) source-supported transition targets, and (iii) recoverability questions evaluated under controlled evidence conditions. This makes the protocol directly applicable to comics and illustrated narratives, and partially applicable to video through sampled shots or keyframes. Fig. 24 summarizes these cross-format extensions.

Comics are the closest extension because panels already provide discrete narrative units. A comic page can be evaluated by replacing storyboard scenes with comic panels and applying the same text-only, image-only, and text+image checks to adjacent panel transitions. The core question remains unchanged: whether the transition meaning is recoverable from the visual sequence alone.

Figs. 25 and 26 show two illustrative comic-style narrative examples. These are cross-format pilot examples rather than released KathaBench-25K benchmark records, and they are included to demonstrate that the same failure-localization and bridge-repair logic transfers naturally to comic panels. In both cases, the generated failure rows remain visually plausible and locally coherent, but omit a meaning-bearing transition; the KathaTrace-guided rows insert or restore the missing bridge step and recover the intended target.

Illustrated books are a similarly natural extension. The protocol can evaluate a sequence of illustrated pages or page-level scene units using the same recoverability conditions, provided that the target transition can be grounded in the source story and tested with fixed accepted answers. The main difference from storyboards is layout granularity: a page may contain one dominant scene or multiple sub-events, so the narrative unit must be defined consistently before evaluation.

Video is a natural but harder extension. A lightweight version can evaluate keyframes or shot-level summaries with the same recoverability protocol. A full video benchmark would additionally require shot segmentation, temporal alignment, motion cues, and careful handling of audio or dialogue. We therefore treat video as future work rather than a fully validated setting in this paper.

More broadly, the protocol can be extended to myths, historical anecdotes, moral tales, procedural stories, school or social narratives, and other ordered visual story forms whenever the intended transition meaning can be grounded in the source and tested with fixed accepted answers. The requirement is not genre-specific; it is whether the transition target remains recoverable from ordered visual evidence.

0.0.18 Robustness Analysis↩︎

This section tests whether the main KathaBench-25K Semantic Trajectory Gap (STG) conclusions remain interpretable under common validity threats. The controls address judge dependence, prompt sensitivity, categorical-label reliability, ordinal-rating instability, moral-target-only effects, source-side leakage, and ambiguity filtering. Held-out Semantic Compass evaluation and length-controlled repair controls are reported in Secs. 0.0.15.7 and 0.0.14.4.

For method \(m\) and question \(q\), ambiguity is defined using the text+image condition. Let \(J(q,(S,\hat{X}_m))\) be the judge answer when both the source story and generated storyboard are available, and let \(\mathcal{A}(q)\) be the frozen accepted-answer set. The ambiguity indicator is \[b_m(q)= \mathbb{I}\!\left[ \mathrm{match}\!\left(J(q,(S,\hat{X}_m)),\mathcal{A}(q)\right)=0 \right]. \label{eq:robustness95ambiguity95indicator}\tag{78}\] The ambiguity rate is \[\rho_{\mathrm{amb},m} = \frac{1}{|Q|} \sum_{q\in Q} b_m(q). \label{eq:robustness95ambiguity95rate}\tag{79}\] Thus, a question is counted as ambiguous or defective for method \(m\) when the intended answer is not recoverable even with both text and image evidence. Such questions are excluded from filtered STG so that unclear items are not counted as image-side semantic loss.

Table 71 summarizes the main robustness controls for STG interpretation.

3.5pt

These controls separate dataset-label reliability from subjective perception ratings. Categorical moral-target labels are evaluated with Fleiss’ \(\kappa\), consensus review, and adjudication. Ordinal 1–5 ratings are used only as human-study diagnostics because their restricted score ranges can produce near-zero chance-corrected agreement even when aggregate trends are stable.

The strongest threat to image-only evaluation is source leakage. If image-only packets contained source stories, prompts, captions, labels, or metadata, image recoverability would be inflated. The leakage scan in Table 71 finds no detected source-side fields. This does not mean every visual inference is unambiguous; it means the image-only condition does not receive explicit source-side information.

3.2pt

Recoverability and STG are recomputed on the remaining valid questions after each filtering step, so image recoverability need not change monotonically across rows.

3.5pt

3.2pt

Table 72 shows why the protocol uses all three evidence conditions. Image-only scoring tests whether generated images communicate the transition. Text-only scoring estimates the source-side recoverability ceiling. Text+image scoring removes questions that remain defective, contradicted, or ambiguous even with full evidence. Human alignment improves from \(0.58\) under image-only-only scoring to \(0.71\) under the final ambiguity-filtered protocol.

Together, these controls support a scoped conclusion: the measured STG is not primarily explained by one judge, prompt wording, moral-target-only questions, detected source leakage, or unfiltered ambiguity. The remaining gap is therefore interpreted as a conservative estimate of text-to-image loss in recoverable transition meaning.

0.0.18.1 Visual-Budget and Length-Control Robustness

KathaBench-25K uses a fixed five-scene annotation schema with four adjacent transitions per story. This annotation schema is fixed even when a storyboard system displays a different number of panels. We call the displayed number of panels the visual budget. Changing the visual budget is therefore a robustness control, not a new dataset version.

Bridge repair changes the visual budget by inserting at most one transition-localized panel between two adjacent scenes. To test whether Semantic Compass gains come from semantic repair rather than simply adding frames, we compare it with length-controlled alternatives: random extra-frame insertion, non-semantic extra-frame insertion, caption-only bridge insertion, candidate reranking only, and bridge repair only. Table 73 defines the visual-budget settings, and Table 74 reports the corresponding length-control results.

Table 74 shows that adding a frame alone gives only small gains: random and non-semantic extra frames reduce STG by \(0.6\) and \(1.1\) points. Transition-localized repair gives a larger reduction, and the full Semantic Compass setting gives the strongest result, reducing STG from \(28.4\) to \(21.4\). This supports the claim that the improvement comes from transition-aware selection and repair, not only from a larger visual budget.

References↩︎