1 Introduction↩︎

The rapid advancement of generative video models has resulted in AI-generated videos that are increasingly more realistic [1]–[5]. It has already been shown that humans cannot differentiate reliably anymore between real and synthetic media [6]. Although there are multiple interesting use cases for AI-generated media, for example in the creative sector, the dangers of this technology cannot be underestimated. One possible danger is the generation of media content that can be perceived as being harmful or insulting [7]. AI-generated media can also be used to mislead the public, fraud people, and sway public opinion [8]. The fact that exposure to AI-generated content can even distort human memory, amplifies these societal dangers even more [9]. These risks necessitate the development of reliable AI-generated video detectors that can be deployed in real-world scenarios.

Research on this topic was originally focused on the detection of deepfakes (AI-generated videos of human faces)[10], [11]. These methods often focus on subtle facial inconsistencies, limiting their applicability outside of this specific domain. Only recently, with the upcoming of diffusion-based generative video models, has the direction also shifted towards the detection of more general AI-generated videos. These detectors rely on a range of techniques that can broadly be divided into three main categories, with some methods combining multiple strategies. A common approach involves training a machine learning model on a large dataset of videos [12]–[18]. Such methods often enhance their inputs using derived representations of the original videos, such as optical flow maps [13], reconstruction errors obtained from pre-trained diffusion models [18], or modifications to specific frequency bands [14]. A second category of detectors integrates large language models (LLMs) into their frameworks, using them not only for detection but also to enhance the explainability of the predictions [19]–[21]. Lastly, a third set of techniques look at inter-frame differences to detect AI-generated videos based on unnatural motion [15], [22]–[24]. These approaches either train models to recognize such motion irregularities [15], [23] or detect them using training-free methods [22], [24]. Inter-frame differences are mostly calculated in the feature space of a pre-trained feature encoder, like DINOv2 [25] or XCLIP [26].

We focus on this third category of motion-based detectors. In particular, we evaluate four recent methods: D3 (ICCV 2025) [22], ReStraV (NeurIPS 2025) [23], Over-Coherence (WACV 2026) [24] and NSG-VD (NeurIPS 2025) [15]. Reported results for three of these approaches indicate near-perfect performance on their evaluation datasets (D3: 97.72 AUC, ReStraV: 98.81 mAP, NSG-VD: 96.14 AUC). However, we identify significant sampling and preprocessing biases underlying these methods, which they might inadvertently exploit. We systematically analyze the extent of these biases and evaluate how detection performance changes once they are removed. Our findings indicate that a substantial portion of the reported performance of these detectors stems from exploiting such biases, rather than capturing motion-related differences between AI-generated and real videos.

Furthermore, our analysis reveals that the high accuracy of these detectors also depends on motion biases embedded in recent evaluation datasets. An analysis of five recent datasets [12], [14], [21], [24], [27] suggests a noticeable imbalance in motion patterns: in four of the five datasets, AI-generated videos tend to exhibit limited inter-frame movement, while real videos more often contain dynamic motion and scene transitions. As generative models rapidly evolve, such biased datasets fail to accurately represent the current capabilities of AI video generation. When we re-evaluate the detectors on temporally (re-)balanced datasets, their performance degrades significantly. We also demonstrate that this over-reliance on motion biases exposes a critical vulnerability. By applying targeted spatial augmentations that cause minimal perceptual degradation, accuracy is drastically reduced.

To test if these problems are specific to motion-based detectors, we also evaluate a non-motion-based detector (WaveRep [14]) on the same five datasets used in our main evaluation. In contrast to the motion-based models evaluated, WaveRep maintains high accuracy accros all datasets, suggesting that feature-based detectors may offer a more robust and promising path forward for generalizable AI-generated video detection. Finally, based on the learned insights, we discuss practical implications for the construction of evaluation datasets and the design of more reliable detection benchmarks.

2 Related Work↩︎

AI video detectors↩︎

Detecting AI-generated videos has become a crucial task given the rapid advancement of generative video models. [13] train a spatial domain detector and an optical flow detector and combine their outputs on a per-frame basis. [14] propose a data augmentation strategy by replacing specific frequency bands of frames of fake videos with real ones using wavelet decomposition, after which a DINOv2 model [25] is trained on this augmented dataset. Other approaches train spatio-temporal models. For instance, [18] train a CNN+LSTM architecture on sequences of video frames. [17] leverage a pre-trained video feature encoder and train a transformer model on the extracted features. Similarly, [16] use a pre-trained image feature encoder together with a transformer architecture and introduce an attention-diversity loss. [12] train a Mamba-style architecture on a large-scale dataset of real and AI-generated videos.

A second set of detectors use Multimodal Large Language Models (MLLM’s) as (part of) their method to detect AI-generated videos. [19] train a dynamic fusion head to combine feature outputs of an LLM branch and a trainable spatio-temporal branch, while [20] and [21] use reinforcement learning to improve reasoning capabilities.

Lastly, multiple recent papers leverage differences between feature embeddings of subsequent frames (or direct frame differences) as a cue to detect AI-generated videos. [22] take the standard deviation of the second order differences of feature embeddings of subsequent frames as a metric for detection. [24] follow a similar approach by computing the cosine similarity between subsequent frame embeddings. Based on the perceptual straightening hypothesis, [23] train an MLP on the distances and angles between embeddings of subsequent frames. Lastly, [15] combine the results of the score function of a pre-trained diffusion model and inter-frame differences in a single metric that is compared against a reference set of real videos. These four methods are the ones that will be evaluated in this paper.

Biases and shortcut learning↩︎

Research about dataset biases and possible shortcut learning as a result has been conducted way before the upcoming of modern generative video models. Early work on images has repeatedly shown that models can achieve strong performance by exploiting dataset biases rather than learning the intended task. For example, [28] showed that a widely used dataset for tampered image detection contains a bias related to JPEG compression: untampered images were saved at different JPEG quality factors than tampered images. They observed significant performance drops when these biases were removed. Similarly, [29] uncovered a bias in a kinship verification dataset, where face crops of related individuals were consistently extracted from the same original image. As a result, paired images shared low-level characteristics unrelated to kinship, enabling even a simple chrominance distance metric to rival state-of-the-art methods in detecting if two people are related. Later, [30] questioned the high accuracies of an algorithm that detects if someone is a criminal based on an image of their face. They identified multiple biases in the evaluation dataset, including differences in camera model and image format (JPEG vs PNG) between the two classes. More recently, [31] showed that a classifier trained for camera identification relied heavily on scene content and color distributions, since images from the same camera were often captured under similar conditions, rather than true sensor-specific fingerprints. These examples highlight a long known issue in machine learning: models tend to look for the easiest way to complete a task. If not enough care is taken in constructing unbiased datasets, a model might exploit biases not related to the task at hand, also known as shortcut learning, leading to misleadingly high performance that does not generalize beyond the biased data.

Unfortunately, these types of biases are also present in the field of AI-generated media detection. Several studies have identified differences in JPEG compression within commonly used detection datasets: real images are frequently JPEG-encoded, whereas AI-generated images are not [32]–[34]. Convincing evidence is provided that these compression biases can be picked up by detectors, leading to high performance that does not reflect true generalization capabilities. Additionally, [32] and [33] also evaluate the impact of mismatches in image resolution between real and AI-generated images on detection performance. Because most detection models resize inputs to a fixed resolution, differences in the original image size introduce implicit upsampling or downsampling artifacts. When such resolution inconsistencies exist, detectors may rely on these artifacts rather than meaningful semantic cues, thereby introducing another source of bias that can significantly impact evaluation results. Lastly, differences in semantic content between AI-generated images and real images can create another bias detectors might rely on. [35] address this by proposing a data generation pipeline that minimizes such biases. Their approach generates images through self-conditioned reconstructions of real ones, producing samples that closely match the original semantic content. All previous studies focus on shortcut learning in AI-generated image detection. To the best of our knowledge, we are the first to investigate biases and shortcut learning in AI-generated video datasets and detectors.

3 Sampling biases↩︎

In this section, we examine sampling and preprocessing biases in motion-based AI-generated video detectors. Specifically, we identified such biases in three out of the four previously mentioned detectors (D3 [22], ReStraV [23] and NSG-VD [15]). We use the official codebases provided by the authors of the papers, as well as the same datasets and train/test splits. If changes are made to the code, these are explicitly mentioned.

3.1 D3↩︎

[22] propose a technique to detect AI-generated videos by taking the standard deviation of the second order differences of feature encodings of adjacent frames. They argue that this metric is generally lower for AI-generated videos due to the difficulty of generators to simulate second-order dynamics correctly. Their main evaluation is done on the GenVideo [12] dataset: AI-generated videos are sourced from 10 different generators, while real videos are collected from the MSR-VTT [36] (Microsoft Research Video to Text) dataset. Surprisingly, the real videos in the GenVideo dataset are saved at 3 frames per second (fps), while the real videos from the original MSR-VTT dataset are between 25 and 30 fps. [22] sample frames at 8 fps over a fixed duration, which introduces a major sampling bias: real videos at 3 fps are upsampled to 8 fps, resulting in frames being duplicated (up to three consecutive duplicate frames per video). This artificially introduces peaks in second order differences, which D3 can exploit as their detection signal. Table ¿tbl:tab:d3? compares D3’s performance on the original GenVideo dataset (real videos at 3 fps) with its performance on the version of GenVideo where real videos are replaced by their original 25–30 fps versions. We follow the original codebase and limit the number of videos per class to 1000. We observe an average drop in AUC of 0.1736, confirming the substantial effect this bias has on the performance of D3. We note that this observation has been made concurrently and independently by [37].

3.2 ReStraV↩︎

[23] base their AI-generated video detector on the temporal straightening hypothesis [38], which states that the human visual system transforms complex motion into a straighter internal representation. They use DINOv2 [25] to extract features from each frame and train an MLP on the first 7 distances and 6 angles (and the mean, variance, minimum and maximum of all distances and angles) between feature encodings of adjacent frames, resulting in 21 features. They claim that real videos follow a smoother path in the embedding space of DINOv2, a pattern that the MLP is able to identify. Their main evaluation is done on the VidProM [27] dataset, where 24 frames are sampled over a 2-second interval from each video. A bias is introduced in the way videos are handled that are less than 2 seconds long or have a lower fps than 12. For videos with fps lower than 12, frames are duplicated to reach this rate (similar to D3). Additionally, if a video is shorter than 2 seconds, the final frame is repeated until the required number of frames is obtained. We calculated one or both of these conditions to be true in 18715 out of 24997 AI-generated videos from the test split of the VidProM dataset used by ReStraV, while only 2 out of 24999 real videos trigger this behaviour.

This sampling bias influences the 21 features that the MLP is trained on. The most noticeable change is that the minimum distance value will be 0 if duplicate frames are present. We hypothesize that this is a pattern that can easily be exploited by the MLP to learn its decision boundary. To test this hypothesis, we simply use this one feature as our prediction value (without using the MLP) and compare the results to the original results in Table ¿tbl:tab:restrav95single95feature?. We observe that for the generative models where sampling biases occur (Cog, MS, st2v, t2vz and vc2), performance is close to identical to the original ReStraV model, with an average AUC drop over the entire test split of the VidProM dataset of only 0.0535. Despite these major sampling biases, the original ReStraV model still achieves an AUC of 0.9355 on the unbiased Opensora videos. This indicates that even without sampling biases, there still exists a signal that the model can pick up to differentiate between AI-generated videos and real videos. As we will later show in Section 4, a large part of this performance can be attributed to motion biases in the dataset.

In addition to these sampling biases, we observed a fundamental difference between the results and the theoretical assumptions of ReStraV [23]. Their claim that real videos have a smoother trajectory in the representation space of a pre-trained vision encoder contradicts the conclusions of other papers in the field that use similar approaches. [22], for example, report that "real videos exhibit more chaotic speed variations", while "generated videos tend to show very flat patterns". Accordingly, [24] come to the conclusion that "unnaturally high temporal coherence is a key indicator of generated text-to-video content". To analyze this discrepancy, we recreated part of Figure 5 in the ReStrav paper by plotting the distribution of the mean, minimum and maximum distance values of the unbiased subset of the VidProM test set, which can be seen in Figure 1. To our surprise, we get almost completely opposite distributions than those that are presented in the original paper, with AI-generated videos having lower distances on average than real videos. These results actually align with the conclusions of [22] and [24], and with our motion bias analysis in Section 4. While this substantial difference doesn’t automatically invalidate ReStraV’s findings, it does raise significant concerns about the validity of its theoretical foundation.

3.3 NSG-VD↩︎

[15] propose NSG-VD, an AI-generated video detection framework based on Normalized Spatiotemporal Gradients (NSG). This metric combines the output of the score function of a pretrained diffusion model together with inter-frame pixel-wise differences and is based on the physics theory of probability flow conservation: \[\label{eq:nsgvd} \mathbf{g}(\mathbf{x}, t) \approx \frac{\mathbf{s}_\theta(\mathbf{x}_t)}{\mathbf{s}_\theta(\mathbf{x}_t) \cdot \frac{\mathbf{x}_{t+\Delta t} - \mathbf{x}_t}{\Delta t} + \lambda} \cite{NSGVD}\tag{1}\] They use Maximum Mean Discrepancy (MMD) to measure these NSG features against the NSG features of a reference set of real videos. They train the kernel used by MMD on a train set of real videos from the Kinetics-400 dataset [39] and AI-generated videos from either SEINE or Pika, sourced from the GenVideo train set [12]. They perform their main evaluation on the GenVideo validation set, similar to D3 [22].

Sampling is performed by extracting eight frames uniformly spaced across the entire video sequence. Consequently, this sampling method establishes a correlation between the total video length and the temporal distance between consecutive sampled frames. In the GenVideo validation set, real videos have an average duration of 15.52 seconds, while AI-generated videos have an average duration of 2.98 seconds. This means that the sampling method of NSG-VD results in an average time between frames of 1.94 seconds for real videos compared to 0.3725 seconds for AI-generated videos; a more than 5-fold difference. Also, in the official codebase, the division by \(\Delta t\) in Equation 1 is not performed, mathematically treating the temporal gap between adjacent frames as identical for all videos.

A second bias is introduced in the way the score function behaves relative to the resolution of the original video. [15] first extract 8 frames from each H.264-encoded video at their native resolution and save them in JPEG format. These frames are subsequently loaded and resized to 256x256 pixels to serve as input for the diffusion model. This pipeline introduces a resolution-dependent distortion: when high-resolution videos are significantly downscaled, the JPEG and H.264 quantization blocks are surpressed. Conversely, when a video’s native resolution is already close to 256x256, minimal scaling occurs, and these grid artifacts remain intact. All real videos in the GenVideo dataset have a resolution of 298x224, close to the input resolution of the diffusion model. AI-generated videos in the GenVideo dataset have resolutions ranging from 512x320 (Lavie) to 1920x1080 (Sora). Because the MMD kernel compares these scores against a reference set of real videos, it may inadvertently detect these resolution-linked JPEG/H.264 signatures rather than the underlying generative quality of the video.

To analyze the effect of this resolution bias, we evaluate three preprocessing variations: (1) JPEG compression before downscaling (original NSG-VD), (2) No JPEG compression and (3) JPEG compression after downscaling (at 298×224 resolution with a quality factor of 95). For pipeline (1) and (3), we visualize the average FFT transform of the output of the score function of NSG-VD for the real reference set, the real test set and three different models from the GenVideo dataset in Figure 2. Compression artifacts can be identified by specific spectral peaks on the horizontal and vertical axes in the FFT domain. Clear differences can be seen in the locations of these peaks using the original pipeline (1) of NSG-VD between AI-generated videos and the real reference set. This is a direct consequence of the differences in resolution between both classes. However, when JPEG compression is applied after downscaling (3), the spectral peaks become indistinguishable between real and generated videos. Table ¿tbl:tab:nsgvd? shows a substantial performance drop of 0.2794 AUC under this setting. This indicates that NSG-VD largely relies on resolution-related artifacts present in the preprocessing pipeline, rather than learning true discriminative features between real and AI-generated videos.

4 Motion biases↩︎

We perform our motion bias analysis on five recent AI-generated video datasets: GenVideo [12], GenBuster++ [21], WaveRep [14], Over-Coherence [24] and VidProM [27]. We evaluate the impact of these biases for three motion-based detectors: D3 [22], ReStraV [23] and Over-Coherence [24].

4.1 Evaluation setup↩︎

For the GenVideo dataset, we use the high-fps versions of real videos, as explained in Section 3.1. We also remove the WildScrape subset due to a lack of quality samples. For the VidProM dataset, we use the test split as used in the ReStraV codebase. To ensure a fair comparison, we use the same frame sampling pipeline for each dataset and detector. It is important to note that this pipeline differs from the individual pipelines employed for each detector. Although this might result in different absolute evaluation metrics, we argue that it is essential for a controlled evaluation of motion-related biases. Each video is sampled at 8 frames per second over a 2-second interval. Videos that have a frame rate lower than 8 or that are shorter than 2 seconds are discarded. The original and discarded number of videos can be found in Appendix 6.1. Frames are saved in the lossless PNG format. Each frame is resized so that its shortest side is 224 pixels, followed by a center crop to a resolution of 224×224. All images are converted to RGB and normalized depending on the normalization used in the original codebases (ImageNet [40] normalization for D3, CLIP [41] normalization for Over-Coherence and standard [0, 1] normalization for ReStraV).

For D3, we use the original implementation without modification. For ReStraV, because it was initially trained with a biased sampling strategy (as discussed in Section 3.2), we retrain the MLP using our preprocessing pipeline as described above. We keep the same train split as in the original setup, but exclude unsatisfactory videos as previously outlined in our evaluation setup. For Over-Coherence, the method relies on a threshold T to select either the minimum or maximum cosine similarity between consecutive frame embeddings. The original paper reports using T=0.99, while the official implementation uses T=0.995. In our experiments, we find that T=1.0 (always selecting the minimum cosine similarity) gives the best performance under our preprocessing pipeline. Given that [24] report only a 0.01 AUC drop when only using the minimum cosine similarity, we consider this adjustment justified for our setting.

4.2 Dataset motion biases↩︎

We hypothesize that the aforementioned datasets contain real videos that, in general, contain more erratic motion and scene transitions than AI-generated videos. To demonstrate this bias, we use a very simple metric: the standard deviation of mean pixel values over all frames of a video. We plot the distributions of this metric for each dataset in Figure 3. As can be seen, for each dataset, the distribution of this metric is not equal. In GenVideo, WaveRep, Over-Coherence, and VidProM, AI-generated videos tend to have more stable motion than real videos. The only exception is GenBuster++, where AI-generated videos display a standard deviation similar to that of real videos. The GenBuster++ dataset is specifically designed to have a strong alignment with real-world scenarios [21]. They achieve this by extracting prompts from real videos and using those to generate AI-videos. They then give all videos (real and generated) blindly to a group of experts and only keep the generated videos that were identified as real by the experts. This pipeline explains our results and motivates the fact that the motion differences in the other four datasets are not inherent to AI-generated videos, but rather a bias stemming from unaligned data and possibly outdated models. To analyze the effect this dataset imbalance has on the results of the aforementioned models, we apply a re-balancing strategy. We partition all videos into 100 bins based on the previously explained standard deviation metric. Within each bin, we randomly undersample the majority class to match the minority class. This results in a filtered dataset in which real and AI-generated videos share a similar motion distribution.

4.3 Evaluation of detectors↩︎

We evaluate the models on the original datasets and the re-balanced datasets. We compute the AUC and report the results in Table 1. The first notable finding is that all three detectors perform better than random on the four biased datasets, but drop to random-level performance on the GenBuster++ dataset, which does not contain the aforementioned motion bias. This result alone strongly indicates that the signals these detectors rely on are not inherent properties of AI-generated videos, but are instead a reflection of motion imbalances present in their evaluation datasets. Remarkably, our custom motion metric achieves second or close-to-second performance on all datasets. While we do not claim to have developed a new competitive detector, the fact that such a simple metric can rival state-of-the-art detectors implies that these detectors are largely using the same trivial motion distribution signal rather than features that are genuinely discriminative of AI-generated videos.

This conclusion is further supported by the rebalancing experiments. On the rebalanced datasets (GenBuster++ excluded), all detectors experience a performance drop in AUC (D3: -0.0970, ReStraV: -0.0650, Over-Coherence: -0.1127). Additionally, to show the brittleness of only relying on inter-frame encoding differences for detection, we apply a Gaussian blur of 7x7 to a single random frame in all AI-generated videos. This augmentation results in two consecutive encodings being further away from each other, while having a minimally noticeable impact on the original video. As can be seen in Table 1, this simple attack causes extreme AUC drops accross all datasets (excluding GenBuster++) for all detectors evaluated (D3: -0.3795, ReStraV: -0.2723, Over-Coherence: -0.5693), either collapsing to random predictions or even dropping below an AUC of 0.5. We show the results for this specific augmentation, but suspect a wide range of augmentation attacks can achieve a similar effect. For detectors to be useful in real-world scenarios, simple augmentations like the one we evaluated should not have this big of an impact.

5 Conclusions↩︎

Frequency-based detectors↩︎

The above results raise the question of whether the observed limitations are inherent to video detection, or specific to motion-based approaches. To answer this question, we evaluate an additional detector on the five datasets used in our main evaluation. We chose the WaveRep detector [14], a recent approach that trains a DinoV2 model using an augmentation strategy that replaces specific low-frequency bands in AI-generated videos with real counterparts to force the model to focus on mid- to high-frequency artifacts. It is important to note that WaveRep does frame-by-frame prediction and averages those predictions, thus inherently not exploiting motion artifacts. WaveRep achieves consistently strong performance across all evaluated datasets, as can be seen in Table 2. While a robustness analysis of frequency-based detectors is beyond the scope of this work, these results suggest that the limitations we identified may be specific to motion-based approaches rather than inherent to AI-generated video detection more broadly. We hope these results motivate future work to investigate what signals frequency-based detectors exploit, and whether they are similarly vulnerable to targeted adversarial attacks and distribution shifts.

Implications for future datasets and detectors↩︎

The societal need for reliable AI-generated media detectors necessitates the development of detectors that work under ever evolving data distributions. We showed in this paper that the high accuracies reported in state-of-the-art AI-generated video detectors do not translate well to real applicability outside their evaluation setup. Our findings point to several practical considerations for future datasets, detectors and evaluation setups. Firstly, great care should be taken that real and AI-generated videos have similar characteristics (duration, fps, resolution...). If there is a substantial difference in these characteristics, they should be explicitly mentioned and correctly handled. As we showed, if differences in fps and resolution are not accounted for, detectors can pick up these biases instead of detecting real discriminative features between AI-generated and real videos. Next, the artifacts that a detector relies on should be representative of the current state of generative models, rather than tied to specific biases in evaluation datasets. As generative models rapidly improve, signals that were once discriminative, such as limited motion in AI-generated videos, may quickly become obsolete, as we have shown in this paper.

6 Technical appendices and supplementary material↩︎

6.1 Dataset overview↩︎

We report the number of videos in the five datasets we evaluated in Table 3. We also report the number of videos that remain after unsatisfactory videos are filtered by our preprocessing pipeline.

6.2 Rebalanced datasets↩︎

We extend Figure 3 of the main paper with the distributions and number of videos after rebalancing in Figure 4.

6.3 Score function outputs of NSG-VD↩︎

We extend Figure 2 of Section 3.3 with the output of the score functions (both FFT and spatial domain) for all three preprocessing pipelines we evaluated (Figure 5).

6.4 Dataset licenses↩︎

• GenVideo

  • License: CC BY-NC 4.0

  • https://modelscope.cn/datasets/cccnju/Gen-Video

• VidProM

  • License: CC BY-NC 4.0

  • https://huggingface.co/datasets/WenhaoWang/VidProM

• Over-Coherence

  • License: Unknown

  • https://github.com/FujitsuResearch/training-free-detection-of-text-to-video-generations-via-over-coherence

• WaveRep

  • License: Unknown

  • https://github.com/grip-unina/WaveRep-SyntheticVideoDetection

• GenBuster++

  • License: MIT License

  • https://huggingface.co/datasets/l8cv/GenBuster-plusplus

References↩︎