1 Introduction↩︎

A score on a temporal video benchmark is meant to certify that a model understands time. But “temporal” hides two questions. The task question: does a question even require watching motion across frames, and in order? The channel question: when it does, does the model recover the order from the pixels, or off the positional encoding? Video understanding works typically use frame shuffle sensitivity [1], [2] and the accuracy a model gains from the full video, which answer only the task question.

Video VLMs receive temporal information through several channels at once, the pixels (motion across frames), in-context timestamps, and the rotary positional encoding (RoPE [3]). A benchmark score aggregates all of them, with weights that depend on the input and the model. Shuffle sensitivity tells us the aggregate temporal signal is nonzero, but not the weighting over which channels carry it.

This gap matters, because two models with similar scores can resolve temporal order by opposite channels. We feed a clip’s frames in reverse but keep RoPE pointing forward, so the pixels indicate one direction and the positions the other (Figure 2). Molmo2 answers the forward event it was never shown, reading order off the positions. Qwen3-VL answers the reversed event it actually sees, reading order from the pixels. Both models gain about \(+0.20\) in accuracy from the full video over a single frame (Table 2), so a multi-frame gain audit rates them the same. Only the conflict reveals the split, and the benchmark score is blind to it. We call Molmo position-dominant, its answer following the positional indices, and Qwen visual-sequence-dominant, which biases towards ordered visual content.

We make this concrete with an intervention protocol that perturbs the three channels independently and in conflict, across three video VLM families (Molmo2 [4], Qwen3-VL [5], PerceptionLM [6]) on two benchmarks (TempCompass [7], TVBench [8]). An information ladder bounds how much of a score is temporal, the task question, and supplies a motion-dependent slice. A reversal test answers the channel question: on TVBench it reads relabel-free as the accuracy a model loses when the pixels are reversed but the positions held forward, and on TempCompass a paired-ground-truth design scores the same split directly. The dissociation holds across both benchmarks and at two model scales. Molmo’s accuracy is essentially unchanged by reversal, while Qwen’s collapses on every temporal task.

A natural worry is that the conflict construction itself manufactures the split. Activation patching shows the channels are separable inside the network and localizes where reversed frame damage can and cannot be undone. Corrected content installed at the first LM layer fully recovers Molmo and PerceptionLM, while Qwen leaves a small residual that traces to midlayer injection of visual features (DeepStack). I.e. the split is a real internal property, not an artifact of the protocol.

Our contributions are an audit, evaluation protocol with the evidence that it matters, and a mechanistic validation:

• A benchmark audit. A temporal score conflates two questions, and the standard diagnostics see only one. Aggregate score mixes whether a task needs motion (the task question) with which channel resolves order when it does (the channel question). We bound the task question with an information ladder (blank pixels, single frame, progressively increasing frames) and we show that frame shuffle sensitivity and multi-frame gain, the field’s two validity checks, answer only the first (Sec. 4.1, 4.2).

• A reversal screen for the channel question. reversal-drop, the accuracy lost when the pixels are reversed but RoPE is held forward, separates a visual-sequence-dominant model from a position-dominant one. It needs no new annotations and drops onto any existing temporal benchmark. Tasks with a deterministic reverse label, such as moving direction, give a paired check for free (Sec. 3.5, 4.3, 5.3).

• Evidence that the protocol matters: two models with similar scores resolve order by opposite channels. Molmo2 reads order off RoPE and Qwen3-VL off the pixels, shown by the conflict protocol and the reversal-drop across two benchmarks at two scales, with paired ground truth on TempCompass. Neither the aggregate score nor a multi-frame gain audit distinguishes them (Section 4.3).

• Mechanistic validation: the split is a real internal property, not an artifact of the conflict. Activation patching separates the channels inside the network and locates the architecture-specific recovery boundary, the first LM layer versus the DeepStack injection depth, so the behavioral split is mechanistic rather than a construction of the conflict protocol (Section 4.5).

2 Related Work↩︎

Auditing what benchmarks measure. A recurring finding in vision evaluation is that aggregate accuracy can measure a capability a model does not have, because models exploit shortcuts correlated with the label rather than the intended signal [9], [10]. Visual question answering models answered from language priors until benchmarks were rebalanced to break the correlation [11], [12], image classifiers leaned on texture rather than shape [13], and many action recognition tasks proved solvable from a single frame or were left largely unchanged when frame order was shuffled [1], [2]. Two recurring diagnostics are single-frame sufficiency and shuffle-invariance. We use them as instruments and build on-top by asking: on the items that do depend on frame order, which input channel a model uses to resolve it?

Temporal video-language evaluation. Recent audits show that aggregate video scores overstate temporal understanding, because many questions are answerable from language or static appearance. TempCompass [7] introduces diverse temporal aspects and conflict-style videos, TVBench [8] rebuilds video VQA to resist single frame and frame-shuffle shortcuts, and VBenchComp [14] sorts questions into language-answerable, shuffle-robust, and order-dependent. Their separators, single frame sufficiency and shuffle-invariance, are blackbox tests that apply to any model, and we adopt them to bound how much of a score is order dependent. These audits identify which questions need temporal order. We hold the items fixed and ask which channel a model uses. The difference is mechanical: whole-clip reversal moves pixels and positions together and measures sensitivity, while our conflict reverses the pixels but holds the RoPE positions forward, isolating the channel. This requires access to model internals, so our test is limited to open models, but it reveals what a score cannot, that two models with similar scores resolve order by opposite channels, the positional index versus the visual sequence.

Positional encodings for time. Timestamp prompting [15] and absolute-time MRoPE [5] introduce positional temporal mechanisms into video VLMs. We show that such encodings can override the pixels during conflict, and that how much a model relies on them depends on architecture.

Mechanistic localization. Concurrent circuit level work localizes temporal processing inside the VLM. CircuitProbe [16] tracks the layerwise emergence of language aligned object and action concepts, finds them decodable in mid-to-late layers, and reports models behaving closer to bag-of-frames evidence accumulation [2]. That work localizes where decoded features mature. We localize where reversed-frame damage can be repaired, the first LM layer versus the DeepStack injection depth, and tie that boundary to the behavioral channel split. The two are complementary, pairing object localization with a decomposition of the temporal input pathways.

Closest to our interventions, recent prior work [17] finds that causal VideoLMs are largely insensitive to perturbing video positional encodings, but degrade sharply when frames are reversed while retaining their original position IDs. It attributes this to an order-sensitive causal-attention pathway rather than to positional encodings. We reach a compatible conclusion for causal models, but show the behavior is not universal. Molmo2, with bidirectional attention over visual tokens, is strongly position-dominant and follows the encoding when the visual sequence is reversed. Where [17] gives a mechanistic account of the causal regime and derives efficiency strategies, we contribute an evaluation diagnostic, the label-free reversal-drop, validated with paired reverse ground truth.

3 Methods↩︎

3.1 Models↩︎

We study 3 video VLM families spanning 2 attention architectures and 2 positional encoding schemes.

Qwen3-VL [5] (4B, 8B): Has Causal attention throughout. Vision tokens use 3D MRoPE with temporal, height, and width axes; the temporal index is replicated as the base for the H/W axes, so temporal ordering is encoded redundantly across all three axes. DeepStack [18] integration injects intermediate ViT features into LLM layers 0–2.
Molmo2 [4] (4B, 8B, O-7B): Has 1D sequential RoPE with bidirectional attention for vision tokens (token_type_ids) and causal attention for text. Inter-frame information flow during prefill is constrained only by RoPE, not by sequential attention masking.
PerceptionLM [6] (1B, 3B): Uses 1D sequential RoPE, fully causal attention, no text timestamp format.
These three families span the two architectural variables we manipulate: bidirectional versus causal LM attention (Molmo2 versus Qwen3-VL and PerceptionLM) and 1D sequential versus 3D MRoPE (Molmo2/PerceptionLM versus Qwen3-VL). Behavioral interventions run across all seven models. Linear probes for Qwen3-VL and Molmo2 only.

3.2 Benchmarks↩︎

TempCompass [7]. 410 videos, \(n=1580\) multiple choice questions across five aspects (action, attribute_change, direction, order, speed). Two aspects (action, speed) are not diagnostic of temporal order and are excluded from per aspect analysis. We treat this as an audit result (Section 4.1) using the remaining 3 (attribute_change, direction, order).
TVBench [8]: 2205 multiple choice questions across nine categories. We use macro-averaged accuracy. Its temporal tasks (scene_transition, moving_direction, action_sequence, action_localization, egocentric_sequence) are relevant to our observations. Unlike TempCompass, TVBench has no paired _reverse videos, so reversal is applied directly to the clips. We exploit the notion that these benchmarks are designed to resist single frame and frame-shuffle shortcuts to understand which pathways video VLMs use.

3.3 Information Ladder↩︎

To quantify how much of a benchmark score requires integrating multiple frames, we evaluate each question at three information levels. 1. blank pixels (blind: answer priors and option leakage only), 2. a single repeated frame with its timestamp (single-frame: adds static appearance), and 3. the full video. The multi-frame gain, accuracy(full) minus accuracy(single-frame), isolates the accuracy that requires integrating across frames. It measures dependence on multiple frames, not on their order. For order we use the order-gain, accuracy(full) minus accuracy(shuffled frames) (Section 3.4), which is the shuffle-invariance criterion of TVBench [8] and VBenchComp [14]. The paired standard error is about \(0.03\); we treat a gain below \(0.05\) as zero. Single frame conditions keep the frame’s timestamp, because a lone frame with its timestamp stripped is out of distribution for some models. An item is motion-dependent for a model when it is answered correctly from the full video but not from a single frame; we use this slice in Section 4.3.

3.4 Intervention Protocol↩︎

We manipulate 3 input channels independently and in combination (Figure 1). Table 1 lists key conditions used. Ref. Supp. 14 for complete results across models and conditions.
Visual order. Frames are presented in one of: normal, single-frame (frame 0 repeated), shuffled (fixed per video), reversed, or blank (zero-pixel frames). Post Conv3d embedding shuffle yields equivalent results, ruling out Conv3d as an independent temporal channel (Supp. 8).
Timestamps. Timestamps precede each frame’s vision tokens. Manipulations: present, absent, mismatched (shuffled independent of the visual shuffle), or reversed.
Positional encoding. RoPE position indices for vision tokens are set to: native (match input order), corrected (follow original frame indices regardless of input order. Under shuffle, slot \(j\) receives the position of frame \(\pi(j)\)), or shuffled/reversed (independently permuted or reversed). We additionally include correct-pixel, wrong-RoPE conditions (rop-shuf; rop-rev for Qwen only) to isolate RoPE effects with correct visual content. For Qwen3-VL, axis specific multimodal RoPE (MRoPE) corrections (T only, H/W only, all) confirm recovery as corrections applied to each axis (Supp. 9). DeepStack ablations confirm it contributes no independent temporal signal beyond RoPE (Supp. 10). We additionally warp the inter-frame RoPE spacing by a factor \(\Delta \in \{0.25,\dots,8\}\) (identity at \(\Delta{=}1\)), holding pixels, order, and timestamps fixed, to test whether the model reads playback rate from positional spacing. Qwen3-VL’s separate MRoPE temporal axis is the clean lever (Molmo2’s 1D RoPE entangles temporal and spatial spacing).

3.5 Conflict and Reversal Conditions↩︎

We construct conditions where channels indicate different orderings: forward visual + reversed timestamps (conf-ts), consistent reversal of visual and timestamps (rev-mvts), visual permutation A + timestamps permutation B (conf-ts2), and reversed visual + corrected RoPE (rev-corr).

Identifying the channel. To determine which channel a model uses to resolve temporal order, we place the visual and positional channels in conflict. Under rev-corr the frames are reversed while the RoPE positions are held at the original forward order, so the pixels indicate one temporal direction and the positions the opposite. A model that reads order from positions answers the forward event and its accuracy is preserved; a model that reads order from the pixels answers the reversed event and its accuracy collapses. We score this two ways. On TempCompass we use the paired ground truth (below). On TVBench, which has no paired reverses, we read it relabel-free as the reversal-drop, accuracy(base) minus accuracy(rev-corr): near zero for a model that reads positions, large for one that reads pixels. For moving_direction the reversed answer is deterministic, since time-reversal flips both motion axes, giving a paired check without manual relabeling.

Reversal protocol with paired ground truth. TempCompass contains 95 paired videos ({id} and {id}_reverse) with independently adjusted ground-truth answers for attribute_change (144 pairs) and direction (157 pairs). On the original videos under rev-corr, we score each prediction against both the original-content GT (E) and the reversed-content GT (F), distinguishing “model interprets video as forward time” from “model interprets video as reverse time.” The per-aspect reversal accuracies in the supplementary report all paired items (original and _reverse), and Supp. 17 reports the original-video dissociation in which rev-corr matches E.
Mirror control and coverage. The paired _reverse videos under rev-corr instantiate the complementary conflict: the pixels show the forward event while RoPE encodes the reversed order. When predictions are scored against the forward content, the family-level pattern reverses: Molmo now fails while Qwen succeeds. This confirms that the original dissociation reflects channel use rather than generic sensitivity to reversal (Supp. 17). We run the reversal conflict on all models on TempCompass and on Molmo-4B/8B and Qwen-4B/8B on TVBench. For Qwen-4B/8B, we also evaluate the motion-dependent slice. Here the channel split must hold if it reflects genuinely temporal evidence rather than statically answerable items. Timestamp-sensitivity partition. To expose question-level timestamp use hidden by aggregate accuracy, we partition questions separately for each model. \(S_\mathrm{ts}^{(m)}\) contains questions answered correctly under base but incorrectly when timestamps are removed (base-nts); \(S_\mathrm{neither}^{(m)}\) contains questions answered correctly in both conditions. We then test whether reversing timestamps selectively affects \(S_\mathrm{ts}^{(m)}\). This is a descriptive, model-specific partition of observed timestamp use, not a calibrated estimate of reliance or an objective annotation of timestamp necessity. Full results are in Supp. 12.

3.6 Activation Patching↩︎

We use residual-stream activation patching [19], [20] to localize where corrected order content is sufficient to recover behavior. For each item we run two forward passes on the same model. First a base pass with original frame order and a target pass (shuf unless noted). On the base pass, residual-stream activations at vision token positions are captured via a pre-hook at the input of each LM block. On the target pass, the captured activations are installed at the matching vision token positions at the selected LM layers, leaving text token positions and RoPE indices untouched (positional encoding enters at each attention block’s Q/K projections). Conditions are scored with the same MCQ accuracy procedure used for base and shuf.
Span alignment. Patching uses the original-frame correspondence: the token at slot \(j\) of a shuf run contains frame \(\pi(j)\), so we install the captured base activation from slot \(\pi(j)\) at slot \(j\). Each patched vision token at every patched layer therefore carries the activation it would have had in the unshuffled run.
Patch extents. We test five extents: act_layer0 (layer 0 only), act_early (first half), act_late (second half), act_all (every layer), and act_ds (LM layers \(\{0,1,2\}\), the DeepStack injection depth in Qwen3-VL); per-model layer-half boundaries are in Supp. 7. Pre-hooks fire at the input of each LM block; in Qwen3-VL, DeepStack adds intermediate ViT features at vision-token positions inside each of layers 0–2’s forward passes (after the input pre-hook), so act_layer0 does not prevent re-injection of shuffled features at layers 1–2 and act_ds is the depth at which all DeepStack injections have fired. A complementary output-patching variant (post-hook on the layer output) used in Section 5.1 and full perlayer specifications are in Supp. 7. Linear probes on the residual stream corroborate this footprint, a slot-index direction transfers across positional configurations and frame-identity readout tracks behavioral recovery (full method & results in Supp. 13).

4 Results↩︎

We ask two questions of a temporal score, how much of it requires watching motion across frames in order (the task question), and, on the part that does, which channel a model uses to resolve temporal order (the channel question). Section 4.1 bounds the task question, Section 4.2 shows the standard diagnostics cannot answer the channel question, and Section 4.3 answers it.

Reading effect sizes. We report accuracy differences in percentage points and read them against two references. Chance is \(0.315\) on TVBench (macro-averaged) and varies with the TempCompass aspect (2 – 4 options; e.g.\({\approx}0.32\) for the 3 – 4 option speed aspect). Run to run variance is small, estimates of the same condition differ by at most \(0.007\) points (e.g.Molmo2 8B base \(0.730\) vs.\(0.737\)). We therefore treat gaps below \(1\) point as noise, \(1\)–\(5\) points as modest, and larger gaps as substantial.

4.1 How Much of a Temporal Score Is Temporal?↩︎

How much of an aggregate temporal benchmark score requires more than a single frame? and how much requires frame order? The information ladder (Sec. 3.3) answers the first through the multi-frame gain, accuracy(full video) – accuracy(single frame) (Table 2). For the second we read the order-gain, accuracy(full video) – accuracy(shuffled frames), the shuffle-invariance criterion that TVBench [8] and VBenchComp [14] use to separate order dependent from static questions. Both are established diagnostics that we build on. Multi-frame gain measures dependence on multiple frames, not on their order, so a positive multi-frame gain can co-occur with order invariance.

Both families have comparable multi-frame gain, about \(+0.20\) overall, so both need several frames. Order-gain is what separates the aspects. It concentrates on attribute_change (\(+0.35\) to \(+0.37\)) and order (\(+0.32\) to \(+0.37\)), is modest on direction, and is near zero on action and speed. Action is near ceiling but barely uses the pixels: its blind accuracy (blank frames) is already about \(0.5\) against chance \(0.29\), so the score reflects single-frame recognition and answer-prior leakage. Speed is the clean separator. It has a clear multi-frame gain (\(+0.11\)) but near-zero order-gain, so it needs several frames yet not their order, an aspect-level instance of VBenchComp’s semantic (shuffle-robust) category and consistent with the rate-free reading of Section 4.4.

Order-gain also removes a confound. The order aspect looks weakly motion-dependent at face value because its single frame baseline uses frame 0, which leaks which event comes first. The shuffled baseline does not leak this, so the order-gain measures order dependence directly. A simpler check further confirms this, we grow the number of sampled frames \(k\) we show the model from \(1\) to the full video. Accuracy on motion-dependent items climbs monotonically vs static control which stays flat, it doesnt matter if the initial frame is a middle frame or first frame (Supp. 18). In summary, our experiments show our motion-dependent subset is a valid subset to use throughout the rest of our study.

4.2 Shuffle Sensitivity cannot identify the Channel↩︎

The standard temporal diagnostics answer the task question and are blind to the channel question. Disrupting RoPE alone while leaving the pixels intact (rop-shuf) costs Molmo about as much as a full pixel shuffle (within \(2\) points) and \(14\)–\(18\) points below baseline, so RoPE is its primary order carrier. The same manipulation costs Qwen only \(3\) to \(7\) points, consistent with its 3D MRoPE encoding order redundantly across axes (Supp. 9). The two models have similar shuffle sensitivity but rely on different channels. Moreover both models have high multi-frame gain on the same temporal tasks, so both genuinely need the frames, yet they resolve order by opposite channels.

4.3 The Channel Dissociation: Position-Dominant vs Visual-Sequence-Dominant↩︎

We place the pixels and the positions in direct conflict. Under rev-corr the frames are reversed while RoPE is held at the original forward order, so a model that reads order from positions answers the forward event (its accuracy is preserved against forward ground truth) and a model that reads order from the visual order answers the reversed event (its accuracy collapses).

TVBench. Dissociation across temporal tasks. On TVBench, where there are no paired reverses, we use the reversal-drop, the accuracy lost from base to rev-corr which doesn’t require paired reverse labels. The two families separate cleanly (Table 3). Molmo’s accuracy is nearly invariant to reversing the pixels (mean temporal-task drop \(+0.04\), largest \(+0.14\)), ie. biased towards RoPE. Qwen’s accuracy collapses on every temporal task (drops of \(+0.24\) to \(+0.55\) on scene transition, moving direction, action sequence, action localization, and egocentric sequence), ie. biased towards frame order. Both models reach high baseline accuracy on these tasks, so both use the frames’ content. They differ only in how they bind temporal order.

TempCompass: paired ground truth. The paired-ground-truth design quantifies the same split (Table 4). The two families start from matched base scores on attribute_change (near \(0.95\) to \(0.97\) for both), then diverge under rev-corr: Molmo is preserved while Qwen collapses. Scoring rev-corr against the original-content (\(E\)) and reversed-content (\(F\)) answers, \(F-E<0\) means the model read reversed content as forward-time. Molmo shows a near-total override on attribute_change (\(F-E\) around \(-0.95\) at 4B/8B, \(-0.82\) at O-7B): it answers the forward event whenever RoPE encodes forward order. Qwen shows the same direction at far smaller magnitude and weakening with scale (\(-0.23\)/\(-0.14\) at 4B/8B). The mechanism is direct: on moving_direction, correcting RoPE is inert for Qwen (rev-corr matches the uncorrected reverse), but for Molmo it restores the forward answer (rev-corr matches baseline).

The override holds on the genuinely temporal items. A natural worry is that Molmo’s override rides items a single frame already answers. It does not. Restricting to the motion-dependent slice (Section 4.1), Molmo’s override is as strong or stronger (\(F-E\) near \(-1.0\) on attribute_change), while Qwen’s override is confined to the static items and vanishes where motion is required, at both 4B and 8B (Table 4). The mirror conflict (forward pixels with reversed RoPE) flips the pattern, with Molmo now failing and Qwen succeeding, confirming that Molmo reads RoPE and Qwen reads the pixels in both directions.

Two models with similar benchmark scores (TempCompass) resolve the temporal items by opposite channels, and no aggregate score, or multi-frame gain audit, distinguishes them. Molmo is position-dominant and Qwen is visual-sequence-dominant. Qwen’s override also weakens with scale (\(F-E\) from \(-0.234\) to \(-0.135\), and the motion-dependent slice from \(+0.014\) to \(+0.237\), at 4B to 8B), so the larger Qwen is not as position dominant; Molmo’s override is saturated at both scales.

4.4 Channel Weighting Is Contingent↩︎

The channel a model uses also shifts with the question, so no single weighting describes a model. Timestamps are the clearest case. Removing them while the pixels stay intact (base-nts vs.base) costs little on average, but the aggregate hides per-question reliance: on the subset a model answers only when timestamps are present, reversing the timestamps is decisive, while on the rest it changes almost nothing (Table 12, Supp. 12). Timestamp reliance falls with scale, and Molmo relies on timestamps less than Qwen. The partition is directional by construction rather than a calibrated reliance estimate. RoPE order is carried in its continuous spacing too, not only the frame permutation, and playback rate is not read from positions at all (Supp. 9).

4.5 Validation: The Dissociation Is Real Inside the Network↩︎

To test whether the behavioral split reflects distinct internal processing rather than only an artifact of the conflict input, we use activation patching (Section 3.6). We record clean order vision-token states from a normally ordered base run, insert them at selected layers of a shuffled shuf run, and measure how much accuracy is recovered. Comparing recovery across patch locations reveals where the effects of shuffled order can be repaired inside each model.

Recovery tracks architecture. Input-space RoPE correction (shuf-corr) recovers Molmo to near baseline but leaves a residual in the causal models, Qwen and PLM (Table 5). This is what attention masking predicts: under causal attention, K/V vectors commit to the shuffled neighbor context during prefill and later RoPE correction cannot re-organize them, whereas bidirectional attention lets Molmo reintegrate against corrected positions (shuf-corr alone mostly recovers it). Patching tests this directly, installing corrected content at layer 0 should fully recover a model if causal attention is the only obstacle.

Layer-0 recovery, and the DeepStack exception. Layer-0 patching fully restores accuracy in Molmo (bidirectional) and in PLM (causal but without DeepStack). Qwen alone leaves a small residual (about \(1.5\) points), which closes exactly when we patch LM layers \(0\) to \(2\), the depth at which Qwen’s DeepStack injection adds intermediate ViT features inside the forward pass after the layer-0 hook fires (Table 5). DeepStack is not an independent temporal channel, ablating it leaves the input-space recovery gap unchanged. It is a patching artifact, and the more general lesson is that a pre-hook at the input of layer \(N\) does not isolate layer \(N\) when features are injected inside its forward pass. TVBench replicates the pattern (Supp. 19). Linear probes [21] corroborate correlationally, a slot-index direction transfers across positional configurations and frame-identity readout tracks behavioral recovery (Supp. 13).

5 Discussion↩︎

5.1 Order is position-dominant or visual-sequence-dominant, and it depends on the architecture↩︎

Read together, the results give a specific account of how temporal order enters these models. Both models are temporal in the usual sense. They need to see several frames, and a multi-frame gain audit confirms it. But they bind the temporal order by opposite channels. Molmo2 is position-dominant. The model reads order off the positional encoding rather than from the order of the visual states, so reversing the pixels while holding RoPE forward leaves its answer essentially unchanged. This is an ordered, position-indexed counterpart to the bag-of-frames behavior reported in prior video-language analyses [2], [16]. The model is not order-invariant, since shuffling still hurts, but the order it uses is symbolic rather than visual. Qwen3-VL is visual-sequence-dominant. The model follows the order of the visual states and reverses its answer when the pixels reverse.

Neither mode is universal. It is a regime a model occupies to a greater or lesser extent. The reversal conflict separates Molmo2 and Qwen3-VL across both benchmarks and at two scales, and PerceptionLM fills the middle. With 1D positional encoding like Molmo2 but causal attention like Qwen3-VL, PLM is an intermediate, partial overrider (\(F-E\) \(\approx-0.5\) on attribute_change, still position-leaning on the motion-dependent slice), between Molmo2’s near-total override and Qwen3-VL’s pixel reading. The ordering Molmo2 \(>\) PLM \(>\) Qwen3-VL is consistent with both 1D positional encoding and bidirectional attention pushing a model toward reading order off positions, so neither factor alone accounts for the split. The point for evaluation is that Molmo2 and Qwen3-VL look identical to a multi-frame gain audit. Only the channel decomposition reveals the split, and a benchmark score, which rewards both modes equally, cannot.

The positional read is distributed across the network rather than set once at the input. Reversing RoPE while keeping the pixels correct reproduces the order cost, and installing correct early layer content at the deeper layers does not undo it (Ref. Supp. 11). This is an architecture specific property. I.e. Temporal channel usage varies by model, and because the two channels fail on different inputs, a temporal score should expose rather than hide this.

5.2 Limitations↩︎

The reversal conflict is an OOD diagnostic and does not estimate natural failure rates. Architecture attribution is limited by available open models, PerceptionLM rules out attention type or positional encoding alone, but the bidirectional result rests on Molmo2 and the families differ in training. The timestamp partition is model-specific, and the experiments cover two short-video VQA benchmarks. Future work may extend to longer video datasets.

5.3 Recommendations for temporal evaluation↩︎

The central recommendation is to separate \(3\) questions a single temporal score collapses. The first two make up the task question, is the task temporal, and the third is the channel question, which pathway a model uses to resolve order. 1. whether a task requires multiple frames, 2. whether it requires their order, and 3. which input pathway determines that order. Multi-frame gain addresses the first, shuffle sensitivity the second, and the reversal conflict the third.

• Report the decomposition, not aggregate accuracy. Give multi-frame gain and shuffle sensitivity, which separate tasks that need several frames from those that need them ordered. A positive multi-frame gain does not imply order matters: speed needs multiple frames yet is order-invariant (Section 4.1).

• Use a reversal conflict to screen for pathway differences. The reversal-drop, the accuracy lost when the visual sequence is reversed while positions stay forward, distinguishes visual-sequence-dominant from position-dominant behavior, which the other two diagnostics cannot.

• Do not compare mechanism from the score. Aggregate accuracy supports task-level comparison but not whether two models resolve order through the same pathway.

Scope of the reversal-drop. The test requires independent control of frame order and positional indices. Its label-free form is a screen, not a classifier. A drop may reflect following the reversed event or confusion under conflict. Paired reverse labels, or tasks with deterministic reverse labels such as moving_direction, distinguish the two.

5.4 Conclusion↩︎

Temporal scores conflate multi-frame dependence, order dependence, and the pathway used to determine order. Across two benchmarks, similarly scoring Molmo2 and Qwen3-VL follow different cues under reversal conflict. Reporting multi-frame gain, shuffle sensitivity, and reversal conflict separately exposes failure modes hidden by a single aggregate score.

Supplemental/Appendix↩︎

6 Reproducibility↩︎

6.0.0.1 Models.

We evaluate seven models from three families, using their official released checkpoints: Qwen3-VL [5] (4B, 8B), Molmo2 [4] (4B, 8B, O-7B), and PerceptionLM [6] (1B, 3B). Behavioral interventions run on all seven models; linear probes on Qwen3-VL and Molmo2 only.

6.0.0.2 Benchmarks.

TempCompass [7] (410 videos; \(n=1580\) questions, \(1564\) on Molmo2 O-7B) and TVBench [8] (\(n=2205\) questions, nine categories), both multiple-choice. Per-aspect TempCompass analysis uses attribute_change, direction, and order. Action and speed are excluded as non-diagnostic (Section 4.1).

6.0.0.3 Interventions and the paired-GT split.

All conditions are defined in Table 1 (main text) and Table 17 (supplementary). The paired-ground-truth reversal split comprises 95 paired videos (144 attribute_change and 157 direction pairs) and is reconstructed from TempCompass by the relabeling procedure of Section 3.5.

6.0.0.4 Compute, Seeds and statistics.

Single seed runs use seed 42. Conditions base, shuf, rop-shuf, and shuf-corr are reported as multi-seed averages where indicated. all other conditions are single seed (seed 42). Bootstrap confidence intervals use 1000 resamples. All experiments were run using a single consumer NVIDIA RTX 4090 GPU.

6.0.0.5 Artifacts.

The paired-GT split is reconstructible from TempCompass via the outlined procedure.

7 Full Activation Patching Profiles↩︎

8 Conv3d Patch Embedding: Pre- vs.Post-Embedding Shuffle↩︎

Qwen3-VL’s vision encoder uses a 3D Conv patch embedding (kernel \([2, 16, 16]\)) that pairs two consecutive raw frames into one temporal unit, potentially encoding local motion prior to any attention. This raises the question of whether shuffling frame order before the Conv3d embedding (as in our main conditions shuf/shuf-corr) captures the same temporal disruption as shuffling after embedding (emb-shuf, embed-then-shuffle).

8.0.0.1 Procedure.

Condition emb-shuf runs a base forward pass, captures the final ViT output embeddings and DeepStack tensors via forward hooks, then shuffles those embeddings by the same permutation used in shuf and injects them into a second generate pass that bypasses the ViT entirely. This isolates the effect of frame-order disruption at the LLM level without recomputing the ViT; any difference between emb-shuf and shuf (pixel-level shuffle, pre-embedding) would indicate that Conv3d motion signal contributes independently. Two recovery variants build on emb-shuf: emb-shuf-ts additionally remaps text timestamps to match the shuffle permutation; emb-shuf-corr additionally corrects temporal RoPE positions.

8.0.0.2 Results.

The gap between emb-shuf and shuf is \(+0.006\)–\(+0.008\), within the range of run-to-run variability. We treat emb-shuf \(\approx\) shuf as confirmation that our pixel-level shuffle conditions (shuf/shuf-corr) are equivalent to post-embedding shuffle for the purpose of temporal disruption, and that the temporal motion captured by Conv3d does not constitute an independent recoverable signal after shuffling. Within this experiment, correcting text timestamps on shuffled embeddings (emb-shuf-ts) recovers \(+0.060\)/\(+0.086\) across model sizes, larger than RoPE correction alone (emb-shuf-corr: \(+0.026\)/\(+0.012\)), consistent with the timestamp-channel dominance reported in Section 4.4.

The residual between act_layer0 patching and full baseline recovery in Qwen3-VL (\(-0.016/-0.015\), \(\approx\)​1 pp) is caused by DeepStack: the pre-hook fires at the input of LM layer 0, but DeepStack adds shuffled intermediate ViT features inside each of layers 0–2’s forward passes, re-contaminating after each hook. A targeted act_ds condition (patch layers {0,1,2} only, exactly the DeepStack injection depth) recovers the full residual at both Qwen scales (act_ds \(-\) act_layer0 \(= +0.016/+0.014\) at 4B/8B; act_early \(-\) act_ds \(\approx 0.000\) at both; \(n=1580\), seed 42), confirming that layers 0–2 are the complete and sole source and that patching beyond layer 2 adds nothing. The present experiment confirms that Conv3d is not the source of this residual (emb-shuf \(\approx\) shuf), isolating DeepStack as the explanation.

A full-scale replication (4B, \(n=1580\)) confirms emb-shuf \(\approx\) shuf (\(-0.003\), consistent with the prior \(+0.008\) within run-to-run variation) and extends the experiment with condition act_layer0_e1: layer-0 activation patching applied on top of emb-shuf inputs. The gap act_layer0_e1 \(-\) act_layer0 \(= -0.003\) (\(0.691\) vs.\(0.694\)). patching layer 0 with Q0 hidden states gives the same recovery regardless of whether the ViT processed shuffled pixels (Q3) or correct pixels (E1). This directly isolates the residual to sequence ordering and DeepStack injection, independent of ViT pixel-encoding quality.

9 MRoPE Axis-Specific RoPE Corrections↩︎

Qwen3-VL uses 3D MRoPE with three axes: temporal (T), height (H), and width (W). All three axes share a per-frame base \(t_i\); spatial offsets \(h\) and \(w\) are added within the frame. Under shuffle, the token at slot \(j\) (containing frame \(\pi(j)\)) receives slot \(j\)’s temporal base \(t_j\) across all three axes. shuf-corr reassigns the temporal base to \(t_{\pi(j)}\) across all axes. shuf-corr-T and shuf-corr-HW correct only the T axis or only the H/W axes respectively, allowing the contribution of each axis to be read from the accuracy increment.

9.0.0.1 Results.

Recovery increments are monotone in the number of corrected axes (\(+0.012 < +0.036 < +0.053\) for 4B). The H/W axes contribute more than the T axis alone (\(+0.036\) vs \(+0.012\)), but neither reaches shuf-corr alone; all three together are needed for full (partial) recovery. This is consistent with M-RoPE’s design: \(t_i\) is the base for all three axes, so correcting any subset partially restores the per-frame temporal identity signal.

9.0.0.2 RoPE inter-frame spacing warp.

Holding pixels and frame order fixed, we scale the inter-frame RoPE gap by a factor \(\Delta\) (identity at \(\Delta{=}1\), reproduced token-for-token; timestamps stripped) to test whether the model reads playback rate from positional spacing. Order and attribute_change degrade under large stretch while speed and direction are unmoved at every \(\Delta\) (Table 9, Qwen3-VL 4B; the pattern replicates on Qwen3-VL 8B and Molmo2 4B/8B). The largest-\(\Delta\) degradation partly reflects positions pushed toward the edge of the trained RoPE range; a spacing-only control isolating gap size from absolute magnitude is left to future work.

10 DeepStack Ablation↩︎

Qwen3-VL injects intermediate ViT features (from layers 8, 16, 24) into LLM layers 0–2 via residual addition at visual token positions (DeepStack). To test whether DeepStack carries an independent temporal signal not present in the main ViT output pathway, we ablate it (D-conditions: DeepStack replaced with identity) and correct it (Qc-conditions: inverse permutation applied to re-align shuffled intermediate features).

10.0.0.1 Interpretation.

DeepStack contributes a small positive effect at baseline (\(-0.011/-0.009\) when ablated) and a similarly small effect under shuffle. Correcting DeepStack routing (shuf-dsc vs.shuf: \(+0.014/+0.026\)) does not close the shuffle gap. The input-space residual gap remains large without DeepStack: shuf-corr\(-\)base is \(-0.099/-0.115\) with DeepStack, whereas shuf-corr-nds\(-\)base-nds is \(-0.097/-0.085\) without it. Thus, removing DeepStack changes the residual by only \(+0.002\) at 4B and \(+0.030\) at 8B and does not eliminate it. This indicates that DeepStack is not the independent temporal information channel responsible for the input-space shuf-corr gap, the result is instead consistent with the gap arising primarily from causal processing in the LLM.

10.0.0.2 Two residuals are not the same.

This experiment concerns the input-space shuf-corr residual (\(\approx 10\) pp) and shows DeepStack does not explain it. Separately, in Section 4.5, the much smaller \(\approx 1.5\) pp act_layer0 patching residual on Qwen3-VL is closed by act_ds: during activation patching, DeepStack reinjects shuffled intermediate ViT features at vision-token positions after the layer-0 input pre-hook fires, contaminating the patched run at layers 0–2. The two findings are compatible: DeepStack is not an independent temporal signal in normal forward operation, and is also the source of the small patching residual specifically because of when its injections fire relative to our pre-hook. Full per-condition TVBench numbers for DeepStack-off conditions are included in Appendix 19. In recovery terms, input-space RoPE correction (shuf-corr) closes only \(29\)–\(33\%\) of the shuffle-to-baseline gap in Qwen, whereas activation patching closes \(96\)–\(100\%\). The remainder is causally-mixed content that positional correction alone cannot reorganize. In Molmo both reach the same ceiling, consistent with bidirectional attention leaving no content-side contamination.

11 RoPE Depth Separation: Content vs.Positional Channels↩︎

The activation-patching experiments in Appendix 7 establish that visual content order (via DeepStack) commits at LM layers 0–2. We ask where the RoPE positional effect accumulates by constructing a coherent conflict: correct visual content with fully reversed MRoPE positions, then patching early layers to see whether the RoPE damage is upstream or downstream of the DeepStack window.

11.0.0.1 Conditions.

• rop-rev: correct pixels, all three MRoPE axes deterministically reversed (\([T{-}1,\ldots,0]\) temporal, reversed H/W). Since pixels are correct, ViT outputs and DeepStack intermediate features are identical to base; the conflict is purely between correct visual content and reversed positional encoding.

• act-ds-rop-rev: rop-rev base with pre-hook input patching at LM layers 0–2. Because rop-rev and base share identical pixels their layer-0 inputs are identical; the layer-0 pre-hook is a no-op. Effective correction is layers 1–2 only.

• act-out-ds-rop-rev: rop-rev base with post-hook output patching at LM layers 0–2, replacing each layer’s output with Q0’s output (captured_act_q0[i+1]). Installs the exact Q0 hidden state entering layer 3; reversed RoPE runs only at layers 3–35.

Both input and output-patching are null results, localizing the RoPE accuracy damage entirely to layers 3–35. This is consistent with DeepStack content injections dominating the residual stream at layers 0–2 and absorbing any positional distortion there; beyond layer 2 no further content injections occur and reversed-RoPE attention accumulates unchecked. The contrast with act_ds (same patch depth, shuf base, full recovery) confirms the asymmetry: content-ordering errors are remediable at layers 0–2, positional-encoding errors are not.

12 Timestamp-Sensitivity Partition: Per-Aspect Breakdowns↩︎

Section 4.4 summarizes the timestamp-sensitivity partition. Table 12 gives the pooled results, and Table 13 the per-aspect breakdowns for aspects where \(|S_\mathrm{ts}| \geq 20\). Aspects below this threshold are suppressed; bootstrap CIs use 1000 resamples.

Several patterns are worth noting beyond the pooled results. First, the action aspect contributes negligibly to \(S_\mathrm{ts}\) across all four models (0–5 questions), because action accuracy is near ceiling regardless of timestamp presence; the aspect is not informative for this partition. Second, the order aspect enters \(S_\mathrm{ts}\) only for Molmo2 8B (\(n=32\), Acc\(|S_\mathrm{ts}=0.531\)), with a \(\Delta\) of \(+0.434\) — among the largest single-aspect gaps in the table. Molmo2 8B specifically appears to use timestamps to judge event order, and reversed timestamps are strongly misleading on those questions. Third, the \(S_\mathrm{ts}\) partitions are model-specific by construction: Qwen’s timestamp-reliant questions are concentrated in direction and speed, while Molmo 4B’s include attribute_change. The four models do not agree on which questions require timestamps, consistent with timestamp-dependence being a property of how each model uses the channel.

13 Full Probe Results↩︎

13.0.0.1 Probe method.

We train multi-class logistic regression probes on residual-stream activations mean-pooled over each frame’s vision span, at LM layers \(L_0, L_4, L_8, \ldots\), targeting two quantities that coincide under unshuffled inputs and dissociate under shuffle: the original-frame index (frame target) and the slot index (slot target). Cross-condition transfer trains on shuf or shuf-corr and evaluates on base to test whether a readout direction is condition-specific or general. Splits and bootstrap CIs follow the conventions in Section 4.5. Probes are applied to Qwen3-VL and Molmo2 only. A complementary probe trained on only the frame-delimiter (first) token is reported in App. ¿sec:app:first-token?.

Frame-Identity and Slot Readout Under Disruption↩︎

The base frame-identity probe peaks at strong accuracy in both architectures: 0.83/0.81 in Qwen 4B/8B at L20/L35; 0.42/0.46/0.80 in Molmo 4B/8B/O-7B at L20/L20/L12 (Table 14). Frame identity is linearly recoverable from baseline mean-pooled activations through most of the network, peaking in middle-to-late layers.

Under shuf, frame identity collapses to chance in both architectures: \(\le 0.02\) across all sizes. Under shuf-corr, frame identity is partially recovered, substantially more in Molmo than Qwen: gap closed is 74%/64%/30% (Molmo 4B/8B/O-7B) vs.%/20% (Qwen 4B/8B). Bidirectional LM attention permits more representational recovery than causal LM attention does, paralleling the behavioral recovery pattern of Section 4.5.

Under shuf, the slot-identity probe trained on base activations remains at near-baseline accuracy in both architectures (Qwen \(0.800/0.785\); Molmo \(0.402/0.451/0.783\)). Under shuf-corr, the slot readout collapses to chance in Molmo (\(0.040/0.042/0.045\)) and near-chance in Qwen (\(0.216/0.204\)). Correcting RoPE to follow original frame indices destroys the slot-organized readout direction; Molmo’s tighter RoPE anchoring makes this collapse more complete than in Qwen, where causal sequential ordering provides residual slot signal.

13.0.0.2 RoPE disruption with correct pixels damages the readout.

The rop-shuf and rop-rev rows of Table 14 isolate the contribution of RoPE to the readout: visual content is intact but positional encoding is shuffled or reversed. Frame-probe accuracy drops substantially in Qwen (base\(-\)rop-shuf \(= 0.629/0.610\) at 4B/8B; base\(-\)rop-rev \(= 0.651/0.617\)) and to chance in Molmo (rop-shuf frame-probe \(0.044/0.047/0.048\) across 4B/8B/O-7B). The base-trained probe reads off a representation that depends on RoPE matching the position the model expects; when RoPE is wrong, the readout direction does not recover frame identity even with correct visual content. This is the representational counterpart to the behavioral rop-shuf effect in Section 4.3.

Cross-Condition Probe Transfer↩︎

First-token delimiter probe↩︎

The main probe analysis (Section 4.5) uses mean-pooled activations across all tokens in each frame’s vision span (frame delimiter plus visual patches). Here we report a complementary probe trained on the activation at only the first token of each frame’s span — the frame delimiter token.

13.0.0.3 Motivation.

Delimiter tokens sit at the boundary between frames in the input sequence and receive the frame’s sequential position index via RoPE. Their activations therefore primarily reflect positional-embedding lookup rather than visual content aggregated across patches. Probing them serves as a sanity check: if our RoPE injection is working correctly, the delimiter probe should show near-perfect slot accuracy under native RoPE (the delimiter token’s position encodes its slot) and near-perfect frame accuracy under corrected RoPE (the delimiter token’s position encodes the original frame index). Failure of these checks would indicate the RoPE manipulation is not reaching the model’s activations.

13.0.0.4 Results.

At layer 0, the delimiter probe under base achieves near-perfect slot (and frame) accuracy (\(\approx 0.988\)), as expected: the first layer’s delimiter activation is dominated by the positional embedding at that slot. Under rop-shuf (correct pixels, shuffled RoPE), delimiter accuracy at L0 drops to \(\approx 0.011\) for the slot direction, confirming RoPE injection changes what the delimiter encodes. Under shuf-corr (shuffled pixels, corrected RoPE), delimiter accuracy at L0 is \(0.749\) for the frame direction and \(\approx 0.000\) for the slot direction, consistent with the corrected position encoding the original frame index rather than the input slot.

These checks confirm the RoPE injections are operating as intended at the input to the first LM layer. The delimiter probe is not used for the main probe analysis because delimiter activations are dominated by positional embedding lookup and do not reflect the distributed visual representation of the frame; mean pooled activations across the full vision span are a more informative measurement of how frame identity is encoded after visual processing.

14 Full Condition Reference↩︎

Table 17 extends Table 1 with all supplementary conditions used in appendix analyses and referenced in the main text but not listed in the main table.

15 PerceptionLM Patching Results↩︎

PerceptionLM (PLM) uses a LLaMA3 backbone with a per-frame independent ViT (no cross-frame attention, no DeepStack injections) and 1D RoPE. This makes it structurally distinct from both Molmo2 (bidirectional LM attention) and Qwen3-VL (causal LM + DeepStack), providing a third data point that separates the DeepStack and LM attention type mechanisms.

15.0.0.1 Behavioral conditions.

shuf-corr leaves a recovery gap of \(-0.076/-0.115\) (1B/3B), comparable to Qwen’s \(-0.099/-0.115\), confirming that RoPE correction cannot undo the damage from causal LM processing of shuffled frame embeddings. This matches the pattern for all causal-LM models. The reversal damage (\(-0.190/-0.253\)) shows PLM does track temporal order (from positions in part, see below), despite having no timestamp channel in its prompts.

15.0.0.2 Channel dissociation (rev-corr).

The reverse-plus-corrected-RoPE condition (rev-corr, reversed pixels with positions held forward) places PLM on the dominance axis of Section 4.3, reported per scale in Table 19 on attribute_change (the diagnostic paired aspect), original videos only. On the original videos, accuracy falls from base by \(+0.167/+0.236\) (1B/3B). Scored against the forward (\(E\)) and reversed (\(F\)) ground truth on the \(n=141\) informative pairs (original GT \(\neq\) reversed GT), \(F-E = -0.525/-0.489\) (motion-dependent slice \(-0.545/-0.487\)): under conflict PLM predominantly answers the forward event (at 1B a fraction \(E=0.72\) answer forward and \(F=0.19\) reversed, the rest a distractor; 3B is \(0.71/0.22\)), so it is a partial, position-leaning overrider. This is weaker than Molmo2 (\(F-E\) near \(-0.95\)) and more position-leaning than Qwen3-VL (\(-0.23\), and \(+0.01/+0.24\) on the motion slice), placing PLM between the two and consistent with both positional encoding and attention type contributing (Section 5.1). The plain rev row in Table 18 (reversed pixels, native RoPE) is a different condition and does not isolate the channel.

15.0.0.3 Activation patching.

act_layer0 \(=\) base exactly at both scales, in contrast to Qwen’s \(-0.015/-0.016\) residual. PLM has no DeepStack injections, so the layer-0 pre-hook captures exactly the point at which all visual features enter the LM; no subsequent injection can re-introduce shuffled order information. act_early \(=\) act_all \(=\) base, confirming no further gain from patching deeper layers once the correct per-frame features are in place.

act_late is notably below baseline, especially at 3B (\(-0.143\), nearly at the shuffled-input floor). With 14 early layers processing shuffled frame embeddings causally before the late-patch window begins, early-layer contamination is largely committed and cannot be undone by late patching. The depth-dependence of this penalty (1B: \(-0.055\); 3B: \(-0.143\)) confirms that deeper causal LMs accumulate order information more pervasively.

16 Full TempCompass Results↩︎

Tables 20–23 report TempCompass accuracy for Molmo2 and Qwen3-VL across all behavioral conditions referenced in the main text. PerceptionLM behavioral results are in Table 18 (Appendix 15). MRoPE axis-specific corrections (shuf-corr-T, shuf-corr-HW, shuf-corr-ds) and DeepStack-off conditions (base-nds, shuf-nds, shuf-corr-nds) for Qwen3-VL are tabulated separately in Appendices 9 and 10 respectively. base, shuf, rop-shuf, and shuf-corr are multi-seed averaged where indicated; other conditions are single-seed (seed 42). \(n=1580\) on 4B/8B; \(n=1564\) on O-7B. Conditions follow the naming in Tables 1 and 17.

17 Architecture Dissociation Under Reversal↩︎

Under rev-corr (reversed pixels, RoPE corrected to forward order) on the original videos, the two families resolve the pixel-vs-RoPE conflict through opposite channels. Table 24 scores against the forward ground truth: Molmo’s accuracy is preserved (rev-corr \(\approx\) base), so it reads order from RoPE, while Qwen’s collapses, so it reads order from the pixels. The mirror column applies the opposite conflict (forward pixels, reversed RoPE; the paired _reverse videos under rev-corr) graded against the forward content shown: the pattern flips, Molmo now fails and Qwen succeeds, confirming that Molmo follows RoPE and Qwen follows the pixels in both conflict directions. (rev-corr here is original-video accuracy and equals the main-text E, Table 25, up to the few non-informative pairs.) TempCompass permutes the answer options between {id} and {id}_reverse, so the reversed GT is matched by option text rather than letter, and we restrict to the informative pairs whose original and reversed GT differ (a few pairs per aspect whose answer is unchanged by reversal are excluded).

On TVBench (no paired _reverse videos, so rev-corr is applied directly to the clips), the same dissociation appears across temporal tasks (main-text Table 3): Molmo’s accuracy is essentially unchanged by reversal at both scales (mean temporal-task drop \(+0.04\)), while Qwen’s collapses on every temporal task at both scales (Qwen-8B overall \(0.558 \rightarrow 0.333\)).

Restricting the TempCompass reversal to the motion-dependent slice (correct from the full video, wrong from a single frame) confirms the override is not an artifact of statically answerable items (Table 26). Molmo’s override is as strong or stronger on the slice, while Qwen’s (4B and 8B, attribute_change) is confined to the static rest and vanishes where motion is required. At both scales the motion-dependent slice sits at or above zero while the static rest stays strongly negative, with the two CIs non-overlapping at 8B. Qwen direction is noisier and does not dissociate at either scale, so we do not claim it. The paired bootstrap CIs are Qwen attribute_change motion-dependent slice 4B \(+0.01\) \([-0.21,+0.23]\) and 8B \(+0.24\) \([+0.01,+0.45]\), against a strongly negative static rest (4B \(-0.49\), 8B \(-0.57\)).

Tables 25 and 26 give the full per-aspect breakdown that main-text Table 4 consolidates. Table 25 reports the original-content (\(E\)) and reversed-content (\(F\)) accuracies for both diagnostic aspects, and Table 26 splits \(F-E\) into the motion-dependent slice and the static rest.

18 Dose-Response: Multi-Frame Gain Is Not a Single-Frame Artifact↩︎

Multi-frame gain compares the full video to a single repeated frame, and that frame was frame 0. On sequence tasks frame 0 can leak which event comes first, which would inflate single-frame accuracy and understate the gain, or make the gain depend on which frame we picked. To rule this out we measure accuracy as a function of the number of frames the model sees, \(K\in\{1,2,4,8,\text{full}\}\), sampling \(K\) evenly-spaced frames with timestamps kept. Crucially, the \(K{=}1\) point uses a middle frame, not frame 0, so it is a single-frame baseline independent of the frame-0 one. We run Molmo2-4B and Qwen3-VL-4B (greedy) on all nine TVBench tasks and pool the five temporal tasks into a model-specific motion-dependent slice (correct from the full video, wrong from frame 0) and the static remainder (Table 27).

The motion-dependent items climb steeply and monotonically with \(K\) (Molmo \(0.18\!\to\!1.0\), Qwen \(0.41\!\to\!1.0\)) while the static items stay flat at \(0.5\) to \(0.6\). Three points follow. First, the climb is not a frame-0 leak: at \(K{=}1\) the frame is a middle frame, independent of the frame-0 baseline, and accuracy is still near the floor there, so the gain is caused by adding frames, the definition of cross-frame dependence. Second, a smooth monotonic curve with a flat static control is far harder to attribute to a baseline artifact than a two-point gap. Third, the signature replicates across a bidirectional (Molmo) and a causal (Qwen) model, so it is a property of the tasks, not one model’s quirk. Both families are genuinely motion-dependent (the task question), which is the premise the channel dissociation (the channel question) builds on.

19 TVBench Results↩︎

Tables 28–31 report TVBench accuracy for all models and conditions. Macro-average is computed with equal weight per category. Chance baselines vary by task (0.500 for 2-choice tasks, 0.333 for 3-choice, 0.250 for 4-choice); the reported macro-average chance level is 0.315.

19.0.0.1 TVBench notes.

The main patterns from TempCompass replicate: shuf-corr \(\approx\) base (Molmo), shuf-corr \(\not\approx\) base (Qwen), shuf-mvts \(>\) shuf (timestamps help when moved with frames). On moving_direction, the main-run shuf-corr collapse below shuf (0.177 vs 0.220 at 4B; 0.086 vs 0.091 at 8B) is a sampling artifact: the main TVBench run draws independent permutations per condition, so shuf and shuf-corr compare different shuffles at \(n=232\) (SE \(\approx 0.033\)); the observed differences are within 1 SE. Shared-permutation targeted runs confirm shuf-corr \(>\) shuf at both scales (\(+0.051\) at 4B; \(-0.026\) at 8B, within 1 SE). Activation patching (act_ds) fully recovers moving_direction at both scales (4B: act_ds \(= 0.487 \approx\) q0 \(= 0.461\), \(\Delta = +0.026\); 8B: act_ds \(= 0.388 \approx\) q0 \(= 0.397\), \(\Delta = -0.009\); both within noise). act_early \(=\) act_ds at both scales, confirming saturation at the DeepStack injection depth (layers 0–2). Act_layer0 residuals (\(-0.121\)/\(-0.099\) at 4B/8B) are substantially larger than on TempCompass overall (\(-0.019\)/\(-0.007\)), consistent with motion-direction answers being more directly dependent on visual content order. Conv3d baking does not contribute at either scale (e1 \(\approx\) shuf: \(+0.013\) at 4B, \(-0.009\) at 8B). One unexplained observation remains: base-nts \(>\) base by \(+0.077\) on moving_direction at 4B.

References↩︎