1 Introduction↩︎

Many vision tasks are inherently temporal: information unfolds across frames, providing cues that single images cannot capture. Applications such as action recognition, video object detection, and articulated pose estimation rely on temporal context to handle occlusions, motion blur, and rapid appearance changes. Modern single-stage video detectors process multiple frames jointly and report consistent accuracy gains over their single-frame counterparts. However, it remains unclear to what extent these models actually exploit information across time. Do they learn meaningful temporal representations, or do they primarily base their predictions on a single highly informative frame? Answering this question requires both (i) an architecture with explicit, controllable temporal operators whose contribution can be isolated, and (ii) a diagnostic methodology that can expose single-frame dependence hidden behind headline accuracy metrics.

To this end, we make two complementary contributions: YOLO-3D, a lightweight spatiotemporal model built on YOLOv8 [1], whose temporal operators can be isolated and stress-tested, and a model-agnostic diagnostic framework that applies controlled perturbations—occlusion schedules, temporal shuffling, redundancy injection, and mid-sequence resolution degradation—to directly measure temporal integration.

Throughout this work, we distinguish between stacked-frame 2D models, which extend the YOLO family by treating multiple frames as extended input channels and applying purely spatial 2D convolutions without temporal locality or order, and 3D spatiotemporal models, which use convolutions that explicitly span the time dimension via local temporal kernels, thereby enforcing temporal continuity. This distinction is central to our analysis: the proposed diagnostics determine whether explicit temporal operators translate into meaningful sequence integration, while the ablation study identifies which architectural decisions enable that integration.

A key insight from our ablation is that temporal reasoning in the neck and head is bottlenecked by the backbone’s temporal downsampling schedule. Conventional 3D backbones aggressively reduce the temporal dimension, often collapsing it to \(D{=}1\) before features reach the neck, rendering any downstream temporal attention ineffective on short sequences. Our temporal-preserving backbone addresses this by maintaining uniform temporal depth across mid-to-late pyramid levels; on UCF101-24 this single design choice yields the largest accuracy gains (\(+3.7\) pp mAP@50 at 32 frames averaged across scales), outweighing more complex fusion or squeeze modules and establishing that preserving temporal information matters more than sophisticated temporal processing.

Complementing the architectural study, our diagnostic framework reveals behavioral differences that standard mAP metrics obscure. Applied across surgical scene analysis (kidney transplantation) and articulated animal pose estimation (dairy-cow skeleton), it exposes clear failure modes: stacked-2D models collapse catastrophically when the target frame is removed, whereas even the vanilla 3D variant retains predictive capability by integrating earlier frames, while conversely exhibiting heightened sensitivity to mid-sequence spatial degradation, confirming genuine rather than incidental temporal reliance. The same probes show that temporal reasoning provides limited benefit in appearance-dominated surgical scenes but substantially improves robustness under occlusion and pose ambiguity in motion-dependent pose estimation.

Our contributions are twofold.

1. A modular spatiotemporal architecture for real-time video detection We build on YOLOv8 with (a) a temporal-preserving 3D backbone that retains non-degenerate temporal depth at all FPN levels, (b) Spatio-Temporal Attention Fusion (STAF) with factored linear attention for learned cross-scale fusion, and (c) Adaptive Temporal Focus (ATF) for key-frame-aware temporal squeeze. The architecture maintains real-time performance (Fig. 1) across nano, small, and medium scales.

2. A model-agnostic diagnostic framework for temporal reasoning We propose a reusable perturbation suite that systematically stress-tests whether video detectors rely on temporal information or collapse to single-frame cues, enabling principled model selection beyond headline mAP.

We validate these contributions through (i) a comprehensive architectural ablation on UCF101-24 (10 configurations \(\times\) 3 model scales \(\times\) 2 sequence lengths \(=\) 60 experiments) that factorizes the design space into backbone type, neck fusion, and temporal squeeze strategy; and (ii) diagnostic evaluation on the two real-world domains above. The division of labor is deliberate: the perturbation diagnostics isolate the backbone-level temporal contrast (vanilla 3D vs.stacked-2D) on the real-world datasets, while the attention-fusion and temporal-focus modules are evaluated through the controlled UCF101-24 ablation.

2 Related Work↩︎

2.0.0.1 Video Models

Early deep video architectures extended 2D CNNs into the temporal domain using 3D convolutions. C3D [2] demonstrated that spatiotemporal kernels can jointly capture appearance and motion, while I3D [3] leveraged 2D pretrained networks for effective transfer learning to action recognition. Subsequent developments explored efficiency–accuracy trade-offs: 3D ResNets [4], factorized R(2+1)D [5], dual-rate SlowFast networks [6], and X3D [7], which systematically scales network dimensions. Transformer-based video models further advanced temporal modeling through attention mechanisms. TimeSformer [8], ViViT [9], and Video Swin [10] achieve strong accuracy in classification tasks, but at high computational cost. While recent self-supervised approaches such as VideoMAEv1-v2 [11], [12] and VideoMAEv2 [12] improve data efficiency, yet still prioritize classification and operate at reduced input sizes. Collectively, these works demonstrate powerful temporal reasoning but remain prohibitive for real-time deployment in detection/pose tasks.

2.0.0.2 Temporal Extensions of Single-Stage Detectors

The YOLO family has evolved toward real-time efficiency [13], [14], but remains fundamentally frame-based. Several works introduce temporal context: YOWO [15] couples a 2D and a 3D CNN branch with a YOLO-based detection head for real-time spatiotemporal action localization; YOWOv2 [16] replaces YOWO’s single-level anchor-based design with a multi-level FPN for the 2D backbone branch and anchor-free heads, and introduces a channel encoder that fuses the upsampled 3D spatiotemporal features with each 2D FPN level through channel concatenation followed by a channel self-attention mechanism, improving action accuracy at real-time speed; YOLOV [17] aggregates short-range temporal features with modest overhead; and frame-stacking strategies such as Temporal-YOLO [18], Temporal-YOLOv8 [19], and TYOLOv8 [20] encode consecutive frames as extended input channels. These methods improve accuracy in specific settings, but they differ from our work in two important respects. First, they generally treat the backbone’s temporal downsampling schedule as inherited from the classification literature, whereas we identify aggressive temporal striding as the dominant bottleneck and show that a simple temporal-preserving strategy outweighs more complex neck or head modifications. Second, none of them offers a principled evaluation of whether their temporal extensions genuinely integrate sequence information or merely benefit from augmented input bandwidth. Beyond the YOLO family, other real-time single-stage detectors inject temporal context through explicit motion streams: Zhang et al. [21] jointly learn action localization and optical-flow estimation end-to-end with multi-scale appearance–motion fusion, while ACDnet [22] augments an SSD detector with flow-guided feature approximation and memory aggregation from 3 frames, sustaining real-time speed and retaining decent accuracy. Efficiency-driven VOD approaches such as MaskVD [23] and FAIM [24] reuse features or masks across frames, reducing compute without directly assessing temporal dependence.

2.0.0.3 Perturbation and Occlusion Analyses

Perturbation-based explanations offer insights into spatiotemporal feature usage. Adaptive occlusion-sensitivity analysis for video CNNs [25] identifies influential regions over time, while classical occlusion [26] and its generative extensions [27] reveal feature dependence in static images. Broader structured perturbation work [28]–[30] highlights model vulnerabilities under masking. These methods provide post-hoc interpretability of individual predictions but do not offer a systematic way to quantify a detector’s reliance on temporal context as a whole. Our diagnostic framework closes this gap: rather than explaining which spatial or temporal regions matter for a single sample, it systematically stress-tests whether the architecture integrates information across time, exposing failure modes that headline mAP scores obscure.

2.0.0.4 Summary

Prior video models, YOLO-style temporal extensions, and perturbation-based analyses advance temporal modeling or interpretation individually, but none combines (i) an architectural analysis that pinpoints the backbone’s temporal downsampling as the key design lever with (ii) a controlled diagnostic protocol that verifies whether temporal operators translate into genuine sequence integration. Our work addresses both gaps jointly: YOLO-3D provides a modular architecture whose components can be isolated and ablated, while the proposed perturbation suite offers a reusable evaluation instrument applicable to any video detector.

3 YOLO-3D↩︎

We build on the YOLOv8 [1] design, lifting its key spatial operations to lightweight 3D counterparts. The design preserves YOLOv8’s two-part structure (backbone + Neck/head) while enabling temporal feature extraction with minimal changes to the original architecture. The resulting system, YOLO-3D, is organised around three axes of design: the 3D backbone and its temporal downsampling schedule (Sec. 3.1), the neck fusion strategy that aggregates features across pyramid levels (Sec. 3.2.2), and the temporal squeeze that collapses the 3D feature volume to a 2D map for the detection head (Sec. 3.2.3). Along each axis we provide a simple, parameter-free vanilla option as well as a learned, attention-driven alternative, enabling systematic ablation of component contributions (Sec. 5.2).

3.0.0.1 Base YOLOv8 Overview

YOLOv8 consists of a CSPDarknet [14], [31]-inspired backbone with C2f (Cross Stage Partial with two convolutions) blocks and an SPPF (Spatial Pyramid Pooling-Fast) layer, followed by a unified head that performs FPN/PAN-style multi-scale fusion, i.e., top-down and bottom-up aggregation with up/down-sampling and concatenation paths [32], [33], and an anchor-free, decoupled prediction head. All components prioritize real-time efficiency.

3.1 3D Backbone and Vanilla Spatiotemporal Extension↩︎

We generalize the YOLOv8 backbone to the temporal domain, as shown in Fig. 2, by replacing 2D convolutions with Conv3d layers and extending C2f into C2f3d, which preserves its split-transform-merge structure while learning short-range motion cues. The SPPF3d module applies multi-kernel pooling in space and time to capture broader temporal context. Successive stride-2 Conv3d layers progressively reduce the temporal depth, yielding a multi-scale feature pyramid \(\{P_3, P_4, P_5\}\) where deeper levels may have a temporal depth as low as \(D{=}1\).

The neck and squeeze stages mirror the FPN/PAN organization of YOLOv8, with all operations lifted to 3D. Cross-scale fusion introduces mismatched temporal depths due to stride differences; we address this with DynamicConcat, which inspects the temporal dimensions of two input feature maps and upsamples the shorter one via nearest-neighbor interpolation along the time axis before concatenation. This prevents fusion failures caused by unequal temporal lengths and ensures consistent cross-scale aggregation. After FPN/PAN fusion, the 3D feature maps are collapsed to 2D by applying AdaptiveAvgPool3d with a target depth of \(1\), producing a uniform temporal average at each spatial location. While simple and parameter-free, this strategy treats all frames equally and discards temporal ordering—an acceptable trade-off when the detection target coincides with the dominant frame in the clip, but a potential limitation when precise temporal localization is required. However, for almost all sequences shorter than 16 frames, the temporal depth is only 1-2. Together, the 3D backbone with DynamicConcat and AdaptiveAvgPool3d constitute the vanilla YOLO-3D variant; the detection head remains unchanged from YOLOv8.

3.2 Temporal Preserving and Attention Fusion Extension↩︎

We replace both the concatenation-based fusion and the average-pool temporal reduction with learned, attention-driven modules. At the same time, the most important factor is changing some of the backbone temporal strides, in order to allow more temporal information to flow to the neck, while the detection head remains unchanged.

3.2.1 Temporal Resolution Preservation in the Backbone↩︎

Conventional 3D CNN backbones inherit spatial downsampling schedules from their 2D counterparts, applying strided spatiotemporal convolutions at mid-to-late stages (e.g., conv3, conv4, conv5) with joint temporal and spatial strides [4]. While such temporal downsampling reduces computational cost and expands the temporal receptive field, it rapidly collapses the temporal dimension to a point where downstream temporal reasoning modules, such as attention-based fusion or learned temporal pooling, operate on degenerate single-frame features and are effectively reduced to no-ops.

3.2.1.1 Quantifying the problem

Table 1 traces the temporal depth at each pyramid level under both strategies for the range of clip lengths commonly used in action detection benchmarks. With full temporal downsampling (stride \(2{\times}2{\times}2\) at all five backbone convolutions), the temporal dimension is divided by \(2^5 = 32\) from input to P5. At a common training length (8-frame clip), all three pyramid levels collapse to \(D{=}1\), rendering any temporal fusion or attention in the neck entirely inactive. Even at 16 frames, only P3 retains \(D{=}2\), while P4 and P5 remain at \(D{=}1\). Across the five sequence lengths examined (\(T \in \{8, 12, 16, 24, 32\}\)), P5 never exceeds \(D{=}1\) under the baseline strategy, and P4 reaches \(D{>}1\) only for \(T{\geq}24\). In practice, this means that the spatiotemporal attention modules in the neck, which are designed to exploit temporal context for more discriminative feature fusion, receive at most a single temporal slice and collapse to purely spatial operations.

3pt

3.2.1.2 Prior evidence for temporal preservation

Empirical evidence from several recent architectures corroborates the importance of preserving temporal resolution. Chen et al. [34] demonstrate that removing temporal pooling from I3D yields a \(+1.1\%\) gain on Kinetics-400 and a more pronounced \(+6\%\) gain on Something-Something V2 [35]—a benchmark that explicitly requires fine-grained temporal reasoning, bringing I3D to parity with temporally-aware models such as TAM [36]. The Fast pathway of SlowFast [6] deliberately omits all temporal downsampling layers throughout the network hierarchy, maintaining the full temporal resolution of feature tensors up to the final global pooling stage; the authors note that temporal downsampling would be detrimental precisely because fine temporal detail is the pathway’s primary representational objective. Aligned with this, the DiST architecture [37] explicitly avoids temporal downsampling within its lightweight temporal encoder, preserving temporal detail while delegating spatial compression to a separate branch.

3.2.1.3 Our approach

Motivated by both the quantitative analysis above and the prior evidence, our backbone adopts a temporal-preserving downsampling strategy. We retain temporal stride 2 in the two stem convolutions (layers 0 and 1), which reduce the temporal dimension from \(T\) to \(T/4\), a manageable compression that keeps early-stage activations within memory budgets. At the three subsequent strided convolutions (layers 3, 5, and 7, which feed P3, P4, and P5 respectively), we replace the \(3{\times}3{\times}3\) kernels with \(1{\times}3{\times}3\) kernels and the \((2,2,2)\) strides with \((1,2,2)\), applying spatial downsampling while leaving the temporal dimension intact. As shown in the right half of Table 1, this ensures that all pyramid levels carry the same temporal depth (\(T/4\)) throughout the FPN, enabling the spatiotemporal modules in the neck to operate on non-degenerate temporal sequences at every scale.

3.2.1.4 Cost analysis

Temporal preservation trades parameters for activation memory at essentially unchanged latency. Replacing the \(3{\times}3{\times}3\) kernels with \(1{\times}3{\times}3\) at three backbone layers reduces total model parameters (by up to \(14.1\%\) when the neck is otherwise unchanged), while retaining \(D{>}1\) through the neck raises per-sample activation memory by \(30\)–\(46\%\) depending on clip length—the primary cost, and the only one that may require modest batch-size adjustment on memory-constrained hardware. This overhead is constant in relative terms across the \(n\)/\(s\)/\(m\) scales, making the trade-off predictable when scaling the architecture.  8 gives the full per-dimension breakdown and measurements (Table 7).

3.2.2 Spatio-Temporal Attention Fusion↩︎

We replace DynamicConcat with SpatioTemporalAttentionFusion (STAF), an attention-based cross-scale fusion module.

Given two feature maps \(\mathbf{x}_1 \in \mathbb{R}^{B \times C_1 \times D_1 \times H_1 \times W_1}\) and \(\mathbf{x}_2 \in \mathbb{R}^{B \times C_2 \times D_2 \times H_2 \times W_2}\) from adjacent FPN levels, STAF proceeds in five stages, shown in Fig. 3:

1. Projection Each input is mapped to a shared embedding dimension \(d_e = \min(C_1, C_2)\) via a \(1{\times}1{\times}1\) Conv3d–BatchNorm–SiLU block. 2. Alignment Spatial and temporal dimensions are matched via trilinear interpolation to the element-wise maximum of each axis. 3. Factored spatiotemporal attention The element-wise sum of the two projections is processed by a factored attention block that decomposes the 3D volume into: A) Temporal attention across \(D\) for each spatial location, and B) Spatial attention across \(H{\times}W\) for each temporal slice, each implemented as kernel-linearised multi-head attention Katharopoulos2020Transformers? with \(\mathrm{ELU}{+}1\) feature maps, yielding \(O(N\,d)\) complexity instead of \(O(N^2)\). When \(D{=}1\), temporal attention is skipped entirely. A position-wise feed-forward network (two linear layers with a GELU non-linearity and \(2{\times}\) expansion) follows the two attention stages. The design draws on factored attention strategies employed in efficient vision transformers [39], [40] and the channel/spatial self-attention modules of YOWOv2 [16]. 4. Channel gating A squeeze-and-excitation gate [38] modulates the attended features channel-wise. 5. Gated residual and output projection The attended features are blended into the fused representation via a learnable scalar \(\gamma\) (bounded by sigmoid), followed by a \(1{\times}1{\times}1\) projection to the target output channels.

Numerical stability under mixed-precision training is ensured by performing all attention arithmetic in float32 with \(N\)-aware key scaling and a clamped normalizer, following best practices for linear attention in low-precision regimes [39]. Per-stage learnable residual scales bound the contribution of each attention and FFN branch.

3.2.2.1 Lite variant

For latency-sensitive deployments we also provide SpatioTemporalAttentionFusionLite, which preserves the project-align-fuse structure but replaces the factored self-attention with lightweight multiplicative gating. Both inputs are projected by \(1{\times}1{\times}1\) convolutions, aligned by nearest-neighbor interpolation, summed, and passed through a BatchNorm-SiLU block; the result is then refined by two sequential gates:

• A temporal gate applies adaptive average pooling along the depth axis followed by a \(1{\times}1{\times}1\) convolution and sigmoid activation, producing per-channel temporal importance weights.

• A spatial gate applies a \(1{\times}1{\times}1\) convolution to a single-channel map and sigmoid activation, highlighting salient spatial regions.

Both gates modulate the fused features by direct element-wise multiplication. Unlike STAF, the Lite variant uses neither the factored attention block nor a learnable scalar residual blend (\(\gamma\)): the gated features are passed straight to the output projection. It thereby incurs roughly \(40\%\) fewer parameters than the full STAF while retaining the learned cross-scale fusion and channel/spatial reweighting. Both variants accept their input channel counts automatically from the model parser, ensuring compatibility with all width/depth scaling factors.

3.2.3 Adaptive Temporal Focus↩︎

The average-pool depth squeeze (performed by AdaptiveAvgPool3d) before the prediction head is replaced by AdaptiveTemporalFocus (ATF), which converts the 3D feature map to 2D (\(B{\times}C{\times}D{\times}H{\times}W \to B{\times}C{\times}H{\times}W\)) while preserving temporal awareness. ATF supports three operating modes:

a) Explicit anchor (last or middle): the feature slice at the designated key-frame index is taken as the primary detection signal. b) Learned selection (learn): a lightweight \(1{\times}1{\times}1\) convolution followed by softmax over the temporal axis produces per-channel frame-attention weights, yielding a soft-selected anchor without requiring prior knowledge of the key-frame position.

In all modes, a gated context summary is computed in parallel: a sigmoid-activated \(1{\times}1{\times}1\) convolution produces per-frame importance weights, which are used to form a weighted temporal average. The final output blends the anchor with the context via a learnable mixing coefficient \(\alpha\) (initialized to \(0.1\)): \[\mathbf{y} = \mathbf{f}_{\mathrm{anchor}} + \alpha \cdot \mathbf{f}_{\mathrm{context}}\,, \label{eq:atf}\tag{1}\] where \(\mathbf{f}_{\mathrm{anchor}}\) is the selected (hard or soft) key-frame and \(\mathbf{f}_{\mathrm{context}}\) is the gated summary. This two-path design provides a stronger inductive bias than a pure attention-weighted average: when the key-frame position is known, the explicit anchor achieves near-perfect reconstruction of the target frame, while the context path supplies complementary motion cues. When the position is unknown, the learned mode degrades gracefully to a slightly-informed average that can be refined end-to-end by the detection loss.

3.2.3.1 Discussion

Compared to the vanilla extension, the enhanced modules introduce a modest parameter overhead (governed by the shared embedding dimension \(d_e\) for STAF and a single \(C{\times}C\) convolution for ATF) but offer three practical advantages: (i) cross-scale fusion is learned rather than heuristic, letting the network weight the importance of each feature level; (ii) temporal aggregation preserves frame ordering and can focus on the detection-relevant key-frame; and (iii) both modules are scale-agnostic, their channel dimensions are derived automatically from the model’s width multiplier, enabling a single configuration file to serve all model sizes (nano through extra-large). The loss functions and the anchor-free detection formulation remain unchanged from YOLOv8.

4 Diagnostic Framework: TemporalLens↩︎

One of our goals is to determine whether a video detector genuinely leverages temporal context or effectively reduces to single-frame inference. We adopt an occlusion-based perturbation strategy in which systematically masking portions of the input sequence reveals their contribution to model predictions. While occlusion directly probes input-segment influence [26], naive black or gray masking introduces distribution shift [28]; generative inpainting mitigates this but incurs substantial overhead [27]. Following evidence that simple occlusion remains competitive with more sophisticated explainability methods [29], [30], we replace masked regions with the dataset-wide RGB mean \(\boldsymbol{\mu}\), mitigating distribution shift at negligible cost.

Since exhaustive masking across all \(2^T\) temporal combinations is infeasible, we design a suite of seven structured perturbations, each targeting a complementary aspect of temporal reasoning: early-frame reliance, temporal continuity, frame-order sensitivity, redundancy dependence, and robustness to spatial degradation—factors identified as essential to temporal analysis in prior video understanding research [6], [41]. Table 2 summarizes each probe and the behavioural signature it is designed to elicit; the formal masking operators, the generalization to an arbitrary target-frame index \(\tau\), and the per-probe expectations are specified in 9.

4.0.0.1 Quantifying sensitivity: the \(\Delta_X\) metric

To compare the influence of each perturbation across models and tasks, we define the diagnostic sensitivity metric \(\Delta_X\) as the difference between perturbed and baseline performance: \[\Delta_X = \text{mAP}_X - \text{mAP}_{\text{baseline}},\] where \(\text{mAP}_{\text{baseline}}\) is performance on the unperturbed full sequence and \(\text{mAP}_X\) is performance under perturbation \(X\). A negative \(\Delta_X\) indicates degradation, with larger magnitudes reflecting stronger reliance on the probed aspect (e.g., \(\Delta_{\text{HL}} \ll 0\) implies heavy last-frame dependence), while \(\Delta_X \approx 0\) suggests robustness or irrelevance of that property. The metric enables direct comparison of architectural sensitivities: if \(|\Delta_{\text{HL}}^{\text{2D}}| \gg |\Delta_{\text{HL}}^{\text{3D}}|\), the stacked-2D model exhibits greater last-frame reliance than its 3D counterpart. We report \(\Delta_X\) alongside absolute mAP in all ablation tables to aid interpretation.

5 Experiments and Results↩︎

To quantify the degree to which models exploit temporal information rather than relying on a single frame, we measure performance degradation under controlled perturbation protocols and compare two architectural variants:

• YOLOv8-2D (stacked temporal): standard YOLOv8 applied to \((3T,H,W)\) channel-concatenated input, which treats sequences as extended channels and applies 2D convolutions over \((3T)\) channels, suggested by [19], [20] for the YOLOv8 model.

• YOLO-3D Vanilla (temporal backbone / ours): YOLOv8 model with a 3D-convolutional backbone operating on \((T,3,H,W)\) input, a 3D-adjusted neck, and task-specific head. Task heads and losses remain unchanged from YOLOv8 to isolate the effect of spatiotemporal feature extraction.

For fairness, each protocol is exported in two serialized formats: stacked TIFFs \((3T,H,W)\) for the 2D model (with channel order \(R_1,G_1,B_1,\dots,R_T,G_T,B_T\)) and unstacked NPYs \((3,T,H,W)\) for the 3D model, ensuring bitwise-identical conditions across architectures. Our primary diagnostic experiments target two domain-specific, real-world datasets in surgery and veterinary surveillance:

1. Operating Room (Object and Action Detection): Surgical scenes from kidney transplantation with seven action classes (skin incision/stitching, kidney pre/post-reperfusion, surgical bowl, artery forceps, and opened surgical site). The dataset consists of 5,492 training and 1,372 validation samples. Each sample is a 12-frame sequence with a 4-frame skip (\(\approx 2\) seconds at 30 fps). Challenges include occlusions, variable illumination, and non-rigid motion.

2. Dairy-Cow Pose Skeleton (Pose Estimation): Farm surveillance videos depicting adult dairy cows (Bos taurus) for articulated keypoint detection and pose estimation, a task critical for lameness diagnosis. The dataset contains 1,484 training and 366 validation samples. Each sample is a 20-frame sequence with a skip of 2 frames (\(\approx 2\) seconds at 30 fps). Robustness to occlusion is important.

Both datasets are annotated with ground-truth bounding boxes (and 23 keypoints for the dairy cow pose estimation task) of the last frame in each sequence. This design makes Hide-Last and Last-Only particularly diagnostic for last-frame reliance. Results are reported following COCO’s evaluation protocol [42], using mean Average Precision (mAP) at Intersection over Union (IoU) or Object Keypoint Similarity (OKS) thresholds. Specifically, we report mAP averaged over thresholds from 0.50 to 0.95 in increments of 0.05 (mAP@50:95).

5.0.0.1 Implementation details

For each dataset the 2D-stacked and 3D variants are trained with identical schedules, augmentation, optimizer settings, and hardware, so that performance differences reflect architecture rather than training setup. The two experimental regimes do differ from each other: the real-world datasets are trained to convergence (100 epochs, \(960{\times}960\)), whereas the 60-run UCF101-24 ablation uses a shorter, lower-resolution budget (20 epochs, \(320{\times}320\)). Full optimizer hyperparameters, the augmentation pipeline, and hardware are given in 10.

5.1 Results↩︎

We report model complexity (computational efficiency, including floating-point operations (FLOPs), parameter counts), baseline performance, and diagnostic ablations across both tasks. Together, these results reveal trade-offs between computational cost, raw accuracy, and robustness to temporal perturbations.

5.1.0.1 Computational Cost

As shown in Fig. 1, the nano and small variants do not exhibit a proportional change in inference latency, yet compute cost and parameter count rise sharply, especially at higher resolutions. We conduct all diagnostic experiments at nano scale for three reasons. First, the diagnostic framework is designed to compare architectural behaviours — specifically, how temporal information flows through 2D stacked versus 3D convolutional pipelines — rather than to maximise absolute accuracy; nano scale isolates this architectural contrast with the fewest confounding capacity effects. Second, at nano scale the 3D and stacked-2D models have comparable parameter counts (Table 3), ensuring that observed differences in perturbation sensitivity reflect temporal integration strategy rather than raw model capacity. Third, both real-world datasets are moderately sized (\(\sim5.5k\) and \(\sim1.5k\) training samples), making larger model scales prone to overfitting and less representative of practical deployment scenarios in specialised domains. We note that the UCF101-24 ablation (Sec. 5.2) already demonstrates that the performance hierarchy among configurations is consistent across nano, small, and medium scales (Fig. 5), supporting the generalisability of conclusions drawn at nano scale.

3.5pt

3pt

5.1.0.2 Baseline mean Average Precision

Table 3 shows that both stacked-2D and 3D models achieve high overall accuracy on both datasets. The 3D model provides a modest but consistent improvement in Operating Room mAP@50:95 (0.868 vs.) and Dairy-Cow pose OKS–mAP@50:95 (0.960 vs.), indicating better use of temporal cues for fine-grained localization.

5.1.0.3 Diagnostic insights

Table 4 reveals how the two architectures rely on temporal information in markedly different ways:

Operating Room Both models achieve near-maximal performance with full sequences. When early frames are occluded (FP-\(p\)), accuracy declines gradually for both. Under every-second occlusion (ES), however, the 2D model drops sharply—falling below its FP-80% result despite having more visible frames—indicating a strong bias toward the latter part of the sequence. The 3D model remains closer to its FP-50% performance, suggesting more distributed temporal use. Hide-Last (HL) reveals the clearest contrast: the 2D model nearly collapses (0.088), while the 3D model retains moderate accuracy (0.344), demonstrating its ability to recover information from earlier frames. Control (CL, last-only) and shuffle-except-last maintain high performance for both models, confirming that last-frame cues dominate this short 2-second setting. Full shuffle, frame replacement (FR), and resolution degradation (RD) have limited impact, implying that static appearance within this window is largely sufficient and temporal variation is minimal.

Dairy-Cow Pose Temporal reasoning plays a much larger role here. Under increasing first-frame occlusion, both models degrade, but the 3D model remains more resilient at high occlusion (0.432 vs. at FP-80%). Hide-Last (HL) makes the architectural gap starkest: the 2D model drops heavily (0.246), whereas the 3D model preserves high accuracy (0.880), reconstructing pose from temporal history. Full temporal shuffle (TS) reduces performance for both models, confirming sensitivity to temporal order. Shuffle-except-last again demonstrates the 2D model’s last-frame reliance, with accuracy recovering as long as the final frame remains intact. In the Control (CL, last-only) condition, the 2D model collapses entirely (0.0), while the 3D architecture maintains moderate accuracy (0.390), indicating reliance on broader temporal context. Finally, the 3D model is notably more sensitive to mid-sequence resolution degradation (RD-\(q\)), dropping to 0.372 compared to the 2D model’s near-invariance (0.943). This aligns with its reliance on motion continuity and mid-sequence detail, whereas the 2D model depends primarily on the last, uncorrupted frame.

Resolution Degradation Diagnosis (Dairy-Cow Pose) Figure 4 summarizes OKS-mAP@50:95 under mid-sequence resolution degradation. As degradation length and factor increase, performance drops monotonically for both models, but the 3D model collapses from 0.990 to 0.372 when the middle \(60\%\) of frames are downsampled by ×8 and upsampled back. This is expected: the dairy-cow skeleton requires fine, mid-sequence detail (joints/limbs become blocky), and the 3D model suffers most when they are corrupted. In contrast, the stacked-2D model, which leans on the last frame, remains comparatively stable when that last frame is unaltered. Sequence-aware models benefit from temporal context but are more sensitive to widespread mid-sequence degradation; the diagnosis cleanly exposes this trade-off.

Takeaway The diagnostic framework reveals that the 2D stacked model succeeds when last-frame cues dominate, as in the Operating Room detection task, but offers little genuine temporal reasoning. The 3D backbone, while computationally heavier, demonstrates meaningful sequence integration: it maintains substantially higher accuracy even when the final frame is hidden, and it shows sensitivity to frame order and continuity, particularly in the dairy cow skeleton pose estimation task, where motion cues are essential. Still, the 3D model is not immune to degradation; its performance drops faster under heavy early frame occlusion and resolution reduction, highlighting its reliance on temporal continuity. Overall, the nano-scale 3D model strikes a practical balance, retaining stronger temporal modeling capacity than stacked 2D without the prohibitive cost of larger 3D variants.

5.2 Architectural Ablation on UCF101-24↩︎

Our model contribution targets real-world challenges on domain-specific video datasets. By building on Ultralytics YOLOv8, we enable support for a variety of computer vision tasks and benefit from future ecosystem improvements. While our main focus remains on demonstrating performance in real-world medical and agricultural applications, we provide reproducible public benchmarks without prohibitive overhead by adopting UCF101-24 [43]: a subset containing 24 action classes from 3,207 videos. We utilize the dataset prepared by [15], which comprises \({\sim}338\)k training clips and \({\sim}137\)k test clips extracted from the annotated frames of the \({\sim}3.2\)k videos in the dataset.

For reference, YOWO [15] reports frame-mAP@50 of 0.610 (2D), 0.705 (3D), 0.730 (2D+3D), and 0.790 (2D+3D+CFAM), while YOWOv2 [16] achieves 0.780 (nano), 0.805 (tiny), and 0.831 (medium) for 16-frame sequences, and 0.794 (nano), 0.830 (tiny), and 0.837 (medium) for 32-frame sequences.

We use UCF101-24 to conduct a systematic ablation study that isolates the contribution of each architectural component introduced in Sec. 3. The ablation factorizes the design space into three axes: backbone type, neck fusion module, and temporal squeeze strategy, yielding 10 architectural configurations evaluated at three model scales.

5.2.1 Ablation Design↩︎

Table 5 specifies the 10 configurations (A–J), each isolating one architectural change relative to a neighboring configuration. Crossed with three model scales (nano, small, medium), this yields 30 experiments. Each experiment is further evaluated at two sequence lengths (16 and 32 frames), producing 60 data points in total.

4pt

The three axes are:

• Backbone: baseline (temporal stride 2 at all five convolutions, Sec. 3.1), temporal (temporal-preserving with stride \((1,2,2)\) at layers 3, 5, 7, Sec. 3.2.1), or temporal2 (partial preservation at layers 3 and 7 only, with layer 5 retaining stride \((2,2,2)\)).

• Neck fusion: DynamicConcat (Sec. 3.1), STAFLite (lightweight gating variant, Sec. 3.2.2), or STAF (full factored spatiotemporal attention, Sec. 3.2.2).

• Temporal squeeze: AvgPoolDepthSqueeze (parameter-free adaptive average pooling to \(D{=}1\), Sec. 3.1), ATF(learn) (learned soft attention over temporal axis, Sec. 3.2.3), or ATF(last) (explicit last-frame anchor, Sec. 3.2.3).

5.2.2 Overall Performance↩︎

Table 6 summarises the results for the medium-scale model, which provides the clearest separation between configurations.

4pt

The top-performing configuration across both sequence lengths is F (temporal preserving backbone + DynamicConcat + AvgPool), reaching 83.5% mAP@50 and 51.1% mAP@50:95 at 32 frames. Configurations D and E, which add STAFLite, STAF fusion and ATF squeeze, are competitive at 82.5–82.6% mAP@50 but do not surpass the simpler F configuration.

Figure 5 presents the full results across all three model scales, revealing that the performance hierarchy remains consistent from nano through medium, with temporal preserving-backbone configurations (D, E, F, I, J) consistently outperforming excessive temporal downsampled baseline-backbone configurations (A, B, C), particularly at 32 frames.

5.2.3 Component-wise Ablation↩︎

We isolate the contribution of each proposed module by comparing pairs of configurations that differ in exactly one component, averaging across the three model scales. Figure 6 visualises these component-wise deltas.

5.2.3.1 Effect of the temporal-preserving backbone

The temporal backbone is the single most impactful modification. Comparing A\(\to\)F (same DynamicConcat + AvgPool neck, only the backbone changes) yields \(+1.8/+1.2\) pp mAP@50/50:95? at 16 frames and \(+3.7/+2.6\) pp at 32 frames. The benefit scales with sequence length, confirming the design hypothesis: under the baseline backbone’s aggressive temporal striding (\(\div32\)), 16-frame clips reach the neck with \(D{=}1\) at most pyramid levels, rendering temporal reasoning moot, whereas the temporal-preserving backbone (stride \((1,2,2)\) at layers 3, 5, 7) retains \(D{=}4\) for 16-frame and \(D{=}8\) for 32-frame input. Holding the full proposed neck constant (C\(\to\)D, STAFLite + ATF) gives smaller but consistent gains (\(+0.8/+0.9\) pp at 16f, \(+2.0/+1.6\) pp at 32f), indicating the backbone improvement is partly independent of the neck design.

5.2.3.2 Temporal2 vs.full temporal backbone.

The temporal2 variant (preserving temporal depth at layers 3 and 7 only, while layer 5 retains stride \((2,2,2)\)) consistently underperforms the full temporal backbone: at 32 frames and medium scale, D\(^\dagger\) trails D by 1.7 pp mAP@50 and E\(^\dagger\) trails E by 2.2 pp. Allowing temporal collapse at even one intermediate stage loses meaningful temporal context, confirming that consistent preservation across all deep layers is necessary.

5.2.3.3 Effect of attention-based fusion (STAFLite).

Replacing DynamicConcat with STAFLite while holding other components constant yields mixed results: \(-1.1\) pp mAP@50 at 16f and \(-0.6\) pp at 32f on the baseline backbone (A\(\to\)B), and \(-2.7/-2.4\) pp on the temporal backbone (F\(\to\)I). STAFLite does not improve over simple DynamicConcat in either setting at this training budget (20 epochs), but serves as a prerequisite for the full STAF variant.

5.2.3.4 Effect of full spatio-temporal attention (STAF).

Upgrading from STAFLite to the full STAF module (D\(\to\)E, temporal backbone) gives \(+1.0/+0.5\) pp mAP@50/50:95? at 16 frames and \(+0.8/+1.0\) pp at 32 frames. The factored self-attention provides a small but consistent improvement over the lite variant, particularly for mAP@50:95 at 32 frames, suggesting it improves localisation quality when given richer temporal input.

5.2.3.5 Effect of adaptive temporal focus (ATF).

Replacing AvgPool squeeze with ATF(learn) on the baseline backbone (B\(\to\)C) costs \(-0.4/-0.6\) pp at 16 frames but recovers \(+0.7/+0.3\) pp at 32 frames: ATF helps only where P3 retains \(D{=}2\) since at 16 frames the other pyramid levels collapse to \(D{=}1\) and the learned temporal weighting has no signal to operate on. Comparing the two ATF modes (D\(\to\)J, temporal backbone), learned soft attention and last-frame anchoring perform comparably—\(+0.7\) pp mAP@50 for ATF(last) at 16f, \(-0.2\) pp at 32f—with the anchor’s slight edge on shorter clips possibly reflecting less benefit from a soft attention query.

5.2.4 Effect of Sequence Length↩︎

Longer input clips (32 vs. frames) improve performance for nearly all configurations, but the magnitude varies by architecture. Figure 7 quantifies this gain per configuration.

• Baseline backbone (A): minimal gain (\(+0.2\) pp mAP@50 averaged across scales), since temporal information is already discarded by the backbone.

• Temporal backbone configs (D, E, F): substantial gains of \(+2.0\) to \(+3.0\) pp mAP@50, confirming that the temporal backbone is necessary to exploit longer sequences.

• Configuration I (temporal + STAFLite + AvgPool) shows the largest 32-frame benefit at \(+2.5\) pp averaged, with the medium scale alone gaining \(+4.8\) pp—suggesting the simple AvgPool squeeze becomes more effective when averaging over richer temporal features.

An experiment-level view of the same trend—temporal-backbone configurations clustering above the \(32\text{f}{=}16\text{f}\) diagonal, most clearly at larger model scales—is provided in Appendix 11 (Fig. 8).

6 Discussion↩︎

Across both the real-world diagnostics and the UCF101-24 ablation, a single theme recurs: temporal modelling is valuable precisely, and only, where single-frame cues cannot resolve the underlying dynamics. The stacked-2D model reaches near-ceiling accuracy on the Operating Room task at far lower compute, confirming that last-frame appearance suffices in static or slow-changing scenes; the 3D model’s advantage materialises in the motion-dependent dairy-cow setting, where it reconstructs predictions from earlier frames once the target frame is removed and the stacked-2D baseline collapses. Temporal convolutions are therefore not universally superior, and the diagnostic suite is what makes this distinction visible, it separates genuine sequence integration from incidental accuracy that raw mAP cannot tell apart.

The same diagnostics expose the cost of that integration. The 3D backbone’s heightened sensitivity to mid-sequence resolution degradation (RD-\(p\)-\(q\)) is not an independent weakness but the direct corollary of its reliance on coherent motion cues: a model that genuinely reads across frames is necessarily more exposed when those frames are corrupted than one that leans on a single clean target frame. Read together, the robustness gains under occlusion and the losses under spatial degradation describe one and the same property from two directions. The tolerance both architectures show to frame redundancy (FR-\(k\)) is consistent with this picture, with reduced temporal novelty acting as a mild regulariser rather than a meaningful perturbation in these short clips.

The ablation and the diagnostics play complementary roles rather than redundant ones. The ablation identifies what architectural decision matters most—preserving non-degenerate temporal depth through the backbone, without which downstream fusion and focus modules operate on collapsed single-frame features and the benefit of longer clips disappears. The diagnostics verify that this decision translates into temporal reasoning rather than a accidental capacity effect. A notable consequence is that, under the training budget explored here, the simplest downstream configuration is sufficient once temporal information is preserved upstream: the bottleneck is information availability in the backbone, not the sophistication of the temporal processing that follows. The learned attention modules still contribute consistent, if smaller, localisation gains when given a sufficient temporal signal, leaving open whether longer schedules would shift the balance further in their favour—a question we return to in Sec. 6.1.

These observations carry two practical implications. As an evaluation instrument, the perturbation suite exposes robustness gaps that raw accuracy conceals when single-frame cues dominate, making it suited to deployment-critical model assessment. As a design guide, it reframes the choice between single-image, stacked-2D, and 3D architectures as one to be settled by diagnostic evidence against task requirements, weighing temporal robustness against compute and latency, rather than by mAP alone.

6.1 Limitations and Future Directions↩︎

While our 3D variant demonstrates genuine temporal reasoning, it introduces notable computational overhead, especially at larger scales and longer sequence lengths. Future work should explore optimizing the backbone (e.g., a lighter alternative to the C2f3d layer) further while preserving temporal information. Additionally, systematic studies on varying sequence length and sampling rates would further clarify how temporal integration scales across applications. Broader evaluations on diverse public benchmarks would further test generalizability; e.g., TITAN-Human Action benchmark [44], which targets multi-person spatial–temporal action detection in urban driving scenes where temporal cues are critical, is a natural next target for applying both YOLO-3D and the TemporalLens diagnostics. Extensions to volumetric modalities such as MRI could also highlight new application domains and strengthen claims of generalizability.

Our diagnostic framework is designed to be model-agnostic in the sense that it applies to any detector that receives a single temporal input tensor and produces predictions from it. This covers the dominant design pattern in single-stage video detection, including the stacked-frame 2D approach proposed by Corsel et al. [18], van Leeuwen et al. [19] and Van Lier et al. [20], which serves as our stacked 2D baseline. However, dual-branch architectures such as YOWO [15], [16] fall outside this scope: their 2D branch receives the target frame directly as a separate input, so perturbations applied to the 3D branch (e.g., Hide-Last, Control-Last) leave the 2D pathway—and its access to the target frame—intact, rendering the perturbation semantics ill-defined without further architectural adaptation. Extending the diagnostic protocol to such multi-input designs, for instance, by jointly perturbing both branches or by selectively masking the 2D target-frame input, is a promising direction for future work.

Training compute and schedule asymmetry The two experimental regimes use deliberately different budgets. The UCF101-24 ablation spans 60 training runs (10 configurations × 3 scales × 2 clip lengths), which makes long schedules prohibitively expensive, so we cap it at 20 epochs, sufficient to establish the relative ordering of configurations that our analysis targets. The two smaller real-world datasets are inexpensive enough to train to convergence (100 epochs), where even at the nano scale and under the stricter mAP@50:95 criterion, both models reach high accuracy (Table 3), indicating the comparison there is not training-limited. Because the diagnostic suite probes temporal integration rather than augmentation strength or absolute accuracy, this asymmetry does not confound the architectural contrast it is designed to measure. The 20-epoch cap does, however, leave open whether longer schedules would let the learned fusion and focus modules (STAF, ATF) overtake the simpler configuration F. Compute-efficient training is a natural enabler: the anti-forgetting sampling of Xie et al. [45], for instance, reports more than 1.4× YOLO training speedups with no loss in accuracy by skipping already-learned images each epoch. Such methods would make both longer ablation schedules and the extension of YOLO-3D to larger, higher-resolution datasets, where the cost of 3D convolutions is most acute, substantially more tractable.

7 Conclusion↩︎

We presented YOLO-3D, a family of modular spatiotemporal detectors built on YOLOv8 for real-time video detection, and a model-agnostic diagnostic framework that probes whether video detectors genuinely reason over time or collapse to single-frame cues.

A systematic ablation on UCF101-24 (60 experiments spanning 10 configurations, 3 model scales, and 2 sequence lengths) established a clear hierarchy of design importance: the temporal-preserving backbone, which retains non-degenerate temporal depth at mid-to-late feature pyramid levels, is the single most impactful component, yielding up to \(+3.7\) pp mAP@50 and scaling its benefit with clip length. Notably, the simplest downstream modules (concatenation-based fusion and average pooling) suffice to achieve the best overall accuracy when temporal information is properly preserved in the backbone, suggesting that preserving temporal signal matters more than sophisticated temporal processing, at least on the chosen dataset and the training budgets explored here.

The diagnostic framework, applied across surgical scene analysis and articulated animal pose estimation, confirmed that these architectural gains correspond to genuine temporal integration: 3D models maintain predictive capability when the target frame is removed and exhibit sensitivity to frame order and mid-sequence degradation, whereas stacked-frame 2D models collapse under the same conditions. These behavioural differences are invisible to standard mAP evaluation, underscoring the practical value of controlled perturbation-based diagnostics for model selection and deployment.

Together, the architectural study and the diagnostic protocol provide complementary tools for the design and evaluation of temporally aware single-stage detectors.

Acknowledgements↩︎

This research was supported by the Federal Ministry of Research, Technology and Space (BMFTR) through projects K3VR [grant: 13N16388] and Kiara [grant: 16SV9036]. Thanks to Daniia Vergazova for her assistance with software development.

Data Availability↩︎

The UCF101-24 benchmark dataset used in the architectural ablation study is publicly available through its original release. The two domain-specific datasets — surgical scenes from kidney transplantation and dairy-cow pose estimation from veterinary farm surveillance — contain confidential data collected under institutional agreements and cannot be made publicly available due to privacy and contractual restrictions. Requests for access to these datasets may be directed to the corresponding author and will be considered subject to the approval of the data-providing institutions. The full source code for the YOLO-3D model family and the diagnostic perturbation framework (TemporalLens) is publicly released at our GitLab repository to enable the research community to reproduce the methodology and apply it to their own datasets.

CRediT authorship contribution statement↩︎

Karam Tomotaki-Dawoud: Conceptualization of this study, Methodology, Writing - Original Draft.Anna Hilsmann: Writing - Review Editing. Peter Eisert: Writing - Review Editing. Sebastian Bosse: Supervision, Writing - Review Editing

Declaration of generative AI and AI-assisted technologies in the manuscript preparation process↩︎

During the preparation of this work the authors used Claude (Anthropic) to assist with editing, language clarity and grammar improvement of this manuscript.

Declaration of competing interest↩︎

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

8 Temporal-Preserving Backbone: Cost Analysis↩︎

The temporal-preservation strategy involves a favourable trade-off across three distinct cost dimensions (Table 7).

First, switching from \(3{\times}3{\times}3\) to \(1{\times}3{\times}3\) kernels at three backbone layers reduces the per-filter parameter count by \(67\%\) at each affected layer, yielding an \(18\%\) reduction in backbone parameters. This translates to a \(14.1\%\) reduction in total model parameters when the neck is otherwise unchanged (config F), and a \(6.4\%\) reduction once the new fusion and squeeze modules are added, since those do not fully offset the kernel savings.

Second, because feature tensors retain \(D{>}1\) through the neck, activation memory per sample increases by \(30\)–\(46\%\) depending on clip length. This is the primary cost and may require modest batch-size adjustments on memory-constrained hardware.

Third, forward-pass latency is essentially unchanged at \(T{=}8\), where the extra temporal dimension is small, and increases at longer clips, scaling with the additional volume processed by the C2f3d bottleneck blocks.

Crucially, the activation-memory overhead is constant in relative terms across model scales (\(n\), \(s\), \(m\)), making the trade-off predictable when scaling the architecture.

4pt

9 TemporalLens Perturbation Specifications↩︎

This appendix gives the formal masking operators for the seven perturbations summarized in Table 2, together with their per-probe diagnostic expectations and the generalization to an arbitrary target-frame index. Throughout, \(\mathbf{x} \in \mathbb{R}^{C \times T \times H \times W}\) denotes the input clip and \(\boldsymbol{\mu}\) the dataset-wide RGB mean frame.

9.0.0.1 (1) First-\(p\)-Fraction (FP-\(p\))

Definition: replace the first \(\lceil pT\rceil\) frames with \(\boldsymbol{\mu}\): \(\mathbf{x}[:,\, 0:\lceil pT\rceil] = \boldsymbol{\mu}\), with \(p \in \{0.20, 0.50, 0.80\}\). Expectation: if early temporal context contributes meaningfully, performance should decline with increasing \(p\), though the trajectory depends on where the occlusion boundary falls relative to the labeled target frame.

9.0.0.2 (2) Every-Second Occlusion (ES-\(s\))

Definition: replace alternating frames with \(\boldsymbol{\mu}\), starting at offset \(s \in \{0, 1\}\): \(\mathbf{x}[:,\, s::2] = \boldsymbol{\mu}\). Expectation: disrupting temporal continuity should hurt models that integrate motion. Unlike contiguously occluding the first 50% of frames, this interleaved pattern reduces redundancy and may yield comparatively better performance.

9.0.0.3 (3) Hide-Last (HL)

Definition: replace only the final frame with \(\boldsymbol{\mu}\): \(\mathbf{x}[:,\, T{-}1] = \boldsymbol{\mu}\). Expectation: severe degradation signals heavy last-frame reliance; 3D convolutional models may retain partial accuracy by leveraging earlier frames.

9.0.0.4 (4) Control: Last-Only (CL)

Definition: replace all frames except the last with \(\boldsymbol{\mu}\): \(\mathbf{x}[:,\, :T{-}1] = \boldsymbol{\mu}\). Expectation: establishes an upper bound on performance achievable without temporal information, serving as a baseline for quantifying last-frame dependence.

9.0.0.5 (5) Temporal Shuffle (TS / TS-Except-Last)

Definition: randomly permute all frames using a fixed seed: \(\mathbf{x}[:,\, t] \leftarrow \mathbf{x}[:,\, \pi(t)]\). In the TS-Except-Last variant the final frame remains in its original position. Expectation: full shuffling should strongly degrade models that exploit sequential structure. TS-Except-Last isolates temporal order relative to the target frame: if accuracy remains high the model is target-frame dominated; if it drops, the model relies on ordering.

9.0.0.6 (6) Frame Replacement (FR-\(k\))

Definition: for indices \(i \in \{s, s{+}k, s{+}2k, \dots\}\) (default \(s=2\), \(k=3\)), duplicate the preceding frame: \(\mathbf{x}[:,\, i] \leftarrow \mathbf{x}[:,\, i{-}1]\). Expectation: reduces temporal novelty via redundant frames; motion-sensitive models should degrade more than appearance-driven ones.

9.0.0.7 (7) Resolution Degradation (RD-\(p\)-\(q\))

Definition: downsample the middle \(p\%\) of frames by factor \(q\) via average pooling, then upsample to the original resolution by nearest-neighbor interpolation. Expectation: probes robustness to spatial-detail loss. Models that exploit temporal dynamics may compensate for degraded spatial information, particularly when the target frame is among those degraded.

9.0.0.8 Generalization to arbitrary target frames

Probes (CL) and (HL) extend naturally to an arbitrary labeled target-frame index \(\tau \in \{0, \dots, T{-}1\}\):

• (C*) Hide-Target (HT-\(\tau\)): \(\mathbf{x}[:,\, \tau,\, :,\, :] = \boldsymbol{\mu}\)

• (D*) Target-Only (TO-\(\tau\)): \(\mathbf{x}[:,\, t \neq \tau,\, :,\, :] = \boldsymbol{\mu}\)

In this work we set \(\tau = T{-}1\) (last frame), consistent with the datasets’ labeling convention.

10 Implementation Details↩︎

For a given dataset, the 2D-stacked and 3D variants are trained with identical schedules, augmentation, optimizer settings, and hardware, so that performance differences reflect architecture rather than training setup; settings differ between the real-world datasets and the UCF101-24 ablation, as detailed below. All models use the Ultralytics framework with a fixed random seed (0) and SGD (learning rate 0.01, momentum 0.937, weight decay 0.0005), with the remaining optimizer hyperparameters following the Ultralytics YOLOv8 defaults.

For the two real-world datasets, we train for 100 epochs at \(960{\times}960\) input and batch size 8, without data augmentation to keep the stacked-2D versus 3D comparison clean. For the UCF101-24 ablation, we train for 20 epochs at \(320{\times}320\) and batch size 8, adopting the augmentation pipeline of YOWOv2 [16] applied consistently to every frame of a clip: random scaling (range [0.8, 1.2], \(p=0.5\)), random crop (jitter 0.2, \(p=0.5\)), horizontal flip (\(p=0.5\)), and HSV jitter (hue 0.1, saturation 1.5, exposure 1.5, \(p=0.1\)). All latency and compute figures (Fig. 1) are measured in FP32 on a single NVIDIA RTX 4090.

11 Experiment-Level View of Sequence-Length Gains↩︎

Figure 8 complements the per-configuration summary of Sec. 5.2 (Fig. 7) with an experiment-level scatter of 16- vs.-frame accuracy across all 30 runs, making the per-experiment spread visible behind the averaged gains.

References↩︎