1 Introduction↩︎

Vision–language models are increasingly expected to answer questions over long surveillance, industrial, and operational videos. The difficulty is not only the number of frames, but the need to preserve temporally ordered evidence for future questions whose content is unknown at ingest time. Text captions are useful but lossy: a caption written before the question is asked may omit precisely the detail needed later. Storing the model’s KV cache offers a richer alternative because it preserves internal visual activations rather than a fixed natural-language summary.

The mechanism is straightforward for one retrieved chunk. During transformer attention, each token produces query, key, and value projections, and the keys and values of past tokens can be stored for later attention. Rotary position embedding (RoPE) encodes relative order through rotations in the attention space [1]. In multimodal models such as Qwen2-VL and Qwen2.5-VL, visual tokens are not merely arranged as a text-like sequence; their positions are represented through multimodal RoPE coordinates that combine temporal and spatial axes [2], [3]. This three-axis structure is central to video because tokens in the same frame should share a temporal coordinate even when they occupy different image patches.

Existing work provides important ingredients but does not fully solve the composition problem addressed here. Text-side KV reuse systems such as KVLink and CacheBlend precompute and combine cached document states, and CacheBlend shows that selective recomputation can repair quality loss when independently encoded text chunks are fused [4], [5]. Video-side systems such as ReKV, StreamMem, StreamKV, HERMES, and DSCache store, retrieve, compress, or hierarchically manage video KV caches for streaming and long-video settings [6]–[10]. Separately, VideoRoPE and related positional-encoding studies show that video RoPE requires explicit spatio-temporal structure [11]. These lines of work motivate ChronoStitch but leave open a specific question: how should independently prefilled visual KV chunks be composed under a model-native three-axis mRoPE, and how much cross-chunk attention must be repaired to recover temporal reasoning?

ChronoStitch addresses this question as a two-part, training-free procedure. First, it directly rotates stored post-rotary keys from local chunk coordinates to global video coordinates, preserving the model’s time–height–width position geometry. Second, it selectively recomputes only the most affected visual tokens from later chunks while allowing them to attend to earlier chunks. The method is evaluated on controlled order-sensitivity probes and the temporal split of TempCompass, a benchmark designed to test temporal perception in video LLMs [12]. The central finding is that positions are necessary but not sufficient: mathematically correct re-basing reconstructs the layer-0 positional geometry, but temporal question answering improves only after selective cross-attention repair.

2 Methodology↩︎

Let a long video be divided into chunks \(C_1, C_2, \ldots, C_N\). Each chunk is independently passed through the vision–language model and stored as a KV cache containing layer-wise keys and values, token coordinates, and layer-0 visual embeddings. The retained embeddings add modest storage overhead, but they avoid re-running the vision tower when selected tokens are repaired. At query time, a retrieval step may return multiple chunks. The objective is to assemble their stored caches so the question can attend over them as if the chunks had originally been prefilled as one continuous video. In ordinary transformer attention, hidden states are projected into queries, keys, and values as follows. \[Q = hW_Q, \quad K = hW_K, \quad V = hW_V, \quad \mathrm{Attn}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^{\top}}{\sqrt{d}}\right)V .\]

For one-dimensional text, RoPE rotates token features according to a scalar position. In Qwen-style multimodal RoPE, a visual token instead has a coordinate \(p = (p^t, p^h, p^w)\), corresponding to time, height, and width. Frequencies are assigned to one of these axes, and the rotary angle for each frequency depends only on the associated coordinate. This can be summarized as follows, where \(\sigma\) selects the axis assigned to frequency \(i\). \[\alpha_i(p) = p_{\sigma(i)}\,\theta_i, \qquad f(x,p) = R\big(\alpha(p)\big)\,x, \qquad p = (p^t, p^h, p^w).\]

The composition problem appears because each stored chunk was originally encoded at local coordinates beginning near zero. Naively concatenating \(\mathrm{KV}_{C_1}\), \(\mathrm{KV}_{C_2}\), and \(\mathrm{KV}_{C_3}\) therefore makes unrelated chunks share overlapping rotary phases. ChronoStitch first fixes this by assigning every chunk a non-overlapping global coordinate interval and applying a delta rotation to the stored post-rotary keys. Because planar rotations compose additively, the key that was stored at local coordinate \(p\) can be moved to target coordinate \(p'\) without recovering the original pre-rotary key \(W_k x\) and without retraining the model. Values are not rotated. \[\Delta_i = \big(p'_{\sigma(i)} - p_{\sigma(i)}\big)\theta_i, \qquad R(\Delta)\,f(W_k x, p) = f(W_k x, p').\]

This re-basing must be three-axis rather than scalar. If visual tokens are flattened into consecutive scalar positions, two different patches from the same frame receive different scalar time indices even though their true temporal coordinate is identical. The resulting attention phases falsely encode spatial patch order as temporal displacement. Three-axis re-basing avoids this by shifting time, height, and width coordinates according to the model’s own visual-token layout, matching the geometry of a joint prefill at layer 0.

However, positional correction alone cannot reproduce a joint prefill at deeper layers. In a joint prefill, a token in a later chunk can attend to earlier chunks while its hidden state is being formed. In independent prefill, that causal cross-chunk interaction never occurred. The residual content gap at layer \(\ell\) can be written as follows. \[\Gamma^{\ell} = \mathbb{E}_t \left\lVert h_t^{\ell,\mathrm{joint}} - h_t^{\ell,\mathrm{indep}} \right\rVert_2 .\]

ChronoStitch repairs this gap by recomputing only a selected subset of later-chunk visual tokens. Candidate tokens are ranked using an oracle-free proxy: after a shallow recomputation against the composed cache, tokens whose layer-1 keys move the most are considered most affected by missing cross-chunk attention. The score is defined below. \[\delta_t = \left\lVert k_t^{1,\mathrm{recomp}} - k_t^{1,\mathrm{stored}} \right\rVert_2 .\]

The top \(\rho\) fraction of later-chunk visual tokens is recomputed through all transformer layers while attending causally to the full composed cache. Their newly computed keys and values are scattered back into the stored cache; all other tokens remain unchanged. In the four-chunk TempCompass setting used in our experiments, repairing \(\rho=0.35\) of the later-chunk visual tokens corresponds to refreshing approximately 25% of the total composed visual cache. This makes the procedure training-free and query-time efficient: most visual memory is reused, the vision tower is skipped, and only the selected fraction is refreshed. The dominant cost terms compare as follows, where \(V\) is the vision-tower cost, \(T\) is the composed token length, \(L\) is the number of layers, \(c_a\) and \(c_f\) are attention and feed-forward constants, \(c_r\) is the re-basing constant, and \(\rho\) is the repair fraction. \[C_{\mathrm{joint}} \approx V + L\big(c_a T^2 + c_f T\big), \qquad C_{\mathrm{Chrono}} \approx L c_r T + 2\big(c_a |C| T + c_f |C|\big) + L\big(c_a \rho T^2 + c_f \rho T\big).\]

The theoretical speedup is therefore close to \(1/\rho\) for the dominant attention term, with additional gains because the vision tower is not rerun. In implementation, the procedure can be summarized as: retrieve stored chunks, shift their three-axis coordinates to a global timeline, delta-rotate stored keys, concatenate the cache, rank later-chunk visual tokens by the key-deviation proxy, recompute the selected tokens with attention over the composed cache, and return the repaired KV memory for answering the user query.

3 Results and Discussion↩︎

Experiments were conducted with mlx-community/Qwen2.5-VL-3B-Instruct-4bit using MLX on Apple M-series hardware. The model has 36 layers, head dimension 128, 16 query heads, 2 key/value heads, and mRoPE frequency sections of 16 temporal, 24 height, and 24 width frequencies. Video chunks contained three frames, and the stored prefixes were vision-only. Five policies were compared: full joint prefill, ChronoStitch, three-axis re-basing only, one-dimensional scalar re-basing, and naive concatenation. The joint prefill is not a deployable cached-memory method; it is used as an upper bound because it processes the relevant chunks together from scratch.

Table 1 confirms the mechanism. At layer 0, where keys depend only on token embeddings and positional phases, three-axis re-basing has a relative error of \(3.6\times10^{-8}\), effectively matching the joint prefill. Scalar re-basing is not merely less precise; it is geometrically wrong for video tokens and performs worse than the naive baseline in this diagnostic. The deeper-layer decline in cosine similarity is expected because hidden states are no longer position-only: they also reflect attention history. The independently stored chunks did not observe earlier chunks during their original computation, so no rotation of keys alone can recreate the missing content.

The controlled order-swap probe isolates the cross-attention gap. Even with correct positions, the re-based cache fails to encode order because the later chunk was originally memorized without access to the earlier chunk. Repairing 15% of later-chunk tokens improves the result, and repairing approximately 35% recovers the joint ceiling on this probe. This supports the paper’s main negative result: positions are necessary, but not sufficient. Temporal coherence also requires a limited amount of cross-chunk content repair.

On the real temporal benchmark, ChronoStitch is the best non-oracle method. The difference between naive, Scalar1D, and position-only re-basing is small at the question-answering level, even though the representational diagnostic strongly favors three-axis re-basing. This is an important interpretation: three-axis re-basing provides the correct positional substrate, but the measurable downstream improvement on the tested 3B reader comes primarily from selective repair. The method recovers roughly one-third of the naive-to-joint gap overall and about half of the gap on event ordering, while attribute-change questions show smaller gains because many such questions can be answered from local evidence rather than long-range temporal interaction.

The efficiency result is central to the practical value of the method. Full joint prefill is the most faithful way to answer a query because the model reprocesses the relevant chunks together. However, it pays the vision-tower and full-attention cost for every query. ChronoStitch reuses the existing visual memory, applies a cheap elementwise key rotation, and recomputes only a selected fraction of tokens. This gives a measured 3.3\(\times\) query-time speedup while recovering a substantial portion of the temporal-ordering benefit of the joint ceiling.

The study also has limitations. The reader model is relatively small, and the joint ceiling on TempCompass is only 63.9%, which compresses the observable margin. The repair fraction was chosen from a small control, and the optimal \(\rho\) may vary with model scale, chunk length, or video horizon. Efficiency is reported as wall-clock latency on one hardware configuration; a hardware-independent FLOP and memory analysis would strengthen the claim. Finally, while the scalar layout is mathematically and representationally defective, the present 3B-model benchmark does not yet show a large downstream QA gap between scalar and three-axis re-basing without repair.

4 Conclusion↩︎

ChronoStitch shows that independently stored visual KV memories can be composed for long-horizon temporal reasoning, but only if two distinct errors are addressed. The first is a positional error: independently cached chunks carry overlapping local mRoPE phases. This is corrected by a training-free three-axis delta rotation applied directly to stored post-rotary keys. The second is a content error: later chunks were encoded without attending to earlier chunks. This is addressed by selectively recomputing high-deviation visual tokens while allowing them to attend over the composed cache. The method is therefore not simply a cache concatenation scheme or a position-reindexing trick; it is a two-stage memory-composition procedure that restores both temporal geometry and a limited amount of missing cross-chunk attention. Experiments demonstrate exact layer-0 key reconstruction, expose the insufficiency of positions alone, improve TempCompass temporal reasoning over all non-oracle baselines, and reduce query-time cost by 3.3\(\times\) compared with full joint re-prefilling. These results support ChronoStitch as a practical step toward coherent, query-time video memory for long-duration visual AI systems.

References↩︎