1 Introduction↩︎

Recent Multimodal Large Language Models (MLLMs) [1]–[6] achieve strong results on standard action recognition benchmarks, but closer analysis shows that much of this performance can arise from spurious correlations rather than genuine temporal reasoning [7]–[11]. Instead of verifying the motion that defines an activity, models often rely on contextual cues such as background or object presence. This aligns with evidence that MLLMs display an agreeableness or affirmative bias, tending to endorse a premise rather than negate it [12]–[14]. In action recognition, this bias appears as predicting that an action is occurring whenever context is suggestive, even if the defining motion is missing, leading to false positives and potential failures in settings like surveillance, autonomous systems, and sports analytics. We formalize this challenge as action denial: the ability of a model to determine that an action is not occurring when its defining motion is absent, despite compelling contextual evidence (1, left). Robust video understanding therefore requires models to confirm actions through their defining motion and to reliably deny them when that motion is not observed.

Despite the scale and diversity of benchmarks such as Kinetics [15], UCF101 [16], HMDB51 [17], and ActivityNet [18], they primarily contain positive examples where the labeled action truly occurs and its defining motion is visible. They rarely include hard counterfactual negatives in which relevant objects, actors, or scenes are present but the action does not take place. This reinforces object-action correlations and allows models to succeed without explicitly verifying motion. Since models are neither trained nor evaluated on contextually similar yet motion-absent clips, current benchmarks offer limited insight into whether a system can separate genuine action evidence from misleading context and reliably deny an action when its defining motion is missing.

To address this gap and directly evaluate action denial, we introduce UCF101-ActionDenial (UCF101-AD), a dataset of 11,283 videos built on UCF101 [16] action categories, with paired positive and negative classes for each action. Each negative clip preserves the typical context of its positive counterpart while ensuring the target action does not occur, and includes realistic motion that often co-occurs with the action. For instance, in “Not Basketball Dunk,” players dribble on a court with a ball present, but no dunk happens. This controlled design isolates defining motion as the key discriminative signal and provides a focused benchmark for evaluating action denial.

Our experiments reveal that the capacity for denial, a fundamental reasoning skill, is exceptionally challenging for multimodal action models and general-purpose MLLMs. We observe that several state-of-the-art architectures struggle to differentiate these negatives, most achieving performance below 50%, often confusing with the positive action (1, right). This failure highlights that models are learning spurious correlations rather than the true causal relationship between interaction, motion and target action. They lack a reliable mechanism to verify if the defining interaction is present.

We hypothesize that models can more reliably decide whether an action is occurring, including its absence, if they are guided to reason about how contextual evidence (persons, objects, location) supports the defining motion. Existing debiasing methods that weaken correlations with background or object cues [11], [19]–[21] operate mainly in positive-action regimes and reward correct labels, not explicit denial when motion is missing. Robust denial instead calls for a structured representation that organizes scene components and checks whether the preconditions of an action are actually instantiated. Prior work on causal structure and spatio-temporal scene graphs shows that modeling dependencies among entities, relations, and events can improve robustness [22]–[24], but is typically used for prediction or explanation. In contrast, we ask whether an explicit structure can help models decide when to deny a target action despite strong contextual cues.

Motivated by this, we propose CausalAct, a causal framework that organizes scene components and their dependencies with a focus on verification. Building on graph-based prompting and structured scene descriptions that improve multimodal video reasoning [25]–[27], we express the causal graph in natural language and ask the models to check whether the necessary conditions for the target action hold. We find that this improves action-denial performance for models with strong language backbones, indicating that the language module must be able to parse such graph-style prompts. We further finetune smaller MLLMs on auxiliary graph-related VQA tasks that teach graph concepts without exposing target action labels, avoiding label leakage, and observe similar gains, showing that causal graphs can aid denial once the model learns to interpret them. Our contributions are:

• We introduce UCF101-AD, a benchmark for action denial reasoning where the scene context and motion cues remain intact but the target action is absent, comprising 11,283 video samples.

• We evaluate 20 state-of-the-art MLLMs, demonstrating that their strong action recognition capabilities fail to translate to robust action denial when misleading contextual cues are present.

• We propose CausalAct, a graph-based finetuning framework that represents scene structure as natural language graphs and trains models through graph-centric VQA tasks to improve reasoning about action absence.

2 Related Work↩︎

Video Action Recognition Benchmarks: Action recognition datasets span trimmed clips [15]–[17], [28], [29], untrimmed videos [18], [30], [31], and domain-specific settings such as sports, instructional content, and driving [32]–[37], as well as egocentric views [38], [39], multi-label action understanding [40]–[44], and skeleton/depth/RGB-based benchmarks [45]–[47]. Most of these focus on recognizing positive actions; negatives are usually implicit, e.g., background or unlabeled frames [30], [31], or are used to probe related failure modes such as similar same-scene actions, missing context, or failed executions [48]–[50]. [51] addresses related co-occurrence ambiguities in still images. However, there remains a gap to test denial of a target action in videos when familiar context, objects, or motion cues are present but the defining action is absent.

Causal Reasoning: Work on causal reasoning for multimodal models includes synthetic benchmarks with ground-truth graphs for probing interventions and temporal dependencies [22], [52], [53], and large-scale annotated datasets for real-world event-level reasoning [23], [24]. Graph-based approaches use scene graphs or structured prompts to organize visual content for multimodal reasoning [25]–[27], [54]–[56], while other efforts ask whether LLMs can internalize abstract causal structures and use them for robust inference [57]–[61]. Building on these, our work decomposes videos into structured scene components and models their relations, specifically evaluating whether MLLMs can decide if a target action truly occurs and deny it when the proposal conflicts with the causal evidence.

Multimodal Large Language Models (MLLMs): Video-language research has progressed from retrieval and contrastive pretraining [62]–[66] to temporal action models [1]–[5], and more recently to LLM-centric vision-language systems [6], [67]–[69] and instruction-tuned video MLLMs [70]–[73]. Emerging “thinking” models add explicit reasoning [74]–[78], but many are tuned in ways that induce agreeableness or sycophancy [12]–[14], [79]–[81]. While these works study bias and reasoning in isolation, evaluation of causal reasoning for action denial in videos remains absent, leading to a dual failure: models are biased by misleading context and prone to affirming false positives. Our work provides a first systematic assessment of this denial capacity and a method to guide MLLMs toward robust, non-sycophantic action understanding.

3 Benchmarking Action Denial↩︎

To evaluate a model’s ability to deny actions under strong contextual cues, we propose UCF101-ActionDenial (UCF101-AD), a benchmark derived from UCF101 action classes [16]. The dataset contains both Action-Denial and Action-Presence clips. Action-Denial videos preserve scenes, objects, and actors associated with a target class, but the defining motion of that action does not occur. These include cases where no defining motion is present or where a different motion occurs despite similar context. This design creates visually plausible clips where context suggests an action while the action itself does not take place, directly testing a model’s ability to recognize action absence. Action-Presence videos contain the defining motion of the target action and serve as a reference for standard action recognition. An overview of the benchmark is shown in 2 with 1 showing comparison with prior benchmarks. While prior works primarily treat negatives as localization distractors, failed attempts, or contextually ambiguous clips, UCF101-AD is explicitly designed around action denial: negatives are constructed to remove the defining motion while preserving plausibly misleading context, and are paired with positive counterparts to support structured reasoning about absence.

3pt

3.1 UCF101-AD Construction↩︎

Video Curation: We construct the negative Action-Denial videos by collecting clips corresponding to each UCF101 action class where the defining action does not occur while preserving similar contextual cues such as scene, objects, or actor appearance. These clips resemble the typical setting of the target action but lack the characteristic motion that defines it. Here, the defining motion refers to temporally extended activities whose recognition requires modeling motion dynamics over time. Accordingly, we focus on actions characterized by such temporal patterns rather than atomic states such as sitting, standing, or listening. Moreover, the presence of an action requires the full execution of its characteristic motion pattern; partial or incomplete movements are not considered sufficient evidence. For instance, slightly moving a golf club does not constitute a golf swing unless the complete swing motion is performed. Based on how the absence of the target motion manifests, we divide the Action-Denial videos into two types:

1. Context only: Videos preserve the salient objects and environment associated with the target action, but no defining interaction takes place. For example, “Not Playing Piano” shows a person near a piano without actually playing it.

2. Same context, different motion: Videos retain the same setting and object interactions as the target action, but the actor performs other plausible actions while never executing the defining motion. For example, “Not Basketball Dunk” shows players dribbling, passing, or shooting on a basketball court, yet no dunk occurs.

Together, these two negative categories disentangle contextual cues from motion evidence, requiring models to verify the defining motion rather than rely on correlated context. In contrast, the positive Action-Presence videos contain clips where the defining motion occurs, serving as a reference to verify that models correctly recognize actions when they are actually performed. All videos are sourced from YouTube and temporally segmented into short clips. Following UCF101 [16], clips originating from the same source video are assigned entirely to either the training or test split to avoid information leakage.

QA Generation. For each video, we generate a question asking which description best matches the content of the video, along with several candidate options in a multiple-choice format. We employ this fixed multiple-choice-question based VQA setup to enable objective evaluation by avoiding the variability of free-form generation [82], [83]. For Action-Denial videos, the answer choices include a diverse set of candidate actions, including the main distractor corresponding to the original UCF101 target action, additional actions sampled from other categories, and a “None” option. This setup introduces strong visual and linguistic cues toward the target action, allowing us to evaluate whether models can correctly deny the action despite these cues. For Action-Presence videos, the correct answer is the target action, and the remaining options are closely related actions plus the same “None” option, forcing the model to distinguish the specific motion rather than rely on coarse context.

Quality control. We employ a fully human verification pipeline to ensure dataset reliability. Annotators first collect candidate videos through keyword searches targeting relevant context without the target action for the Action-Denial set. Each clip is manually verified to ensure sufficient contextual cues and the complete absence of the corresponding action, including partial executions. Low-quality or irrelevant clips are discarded. For the Action-Presence set, only clips containing a clear instance of the action are retained. All clips are cross-validated by annotators, removing about 20% of candidates. No MLLMs were used in this process to avoid model-induced bias. After this process, UCF101-AD contains 11,283 clips in total: the Action-Denial split includes 7,059 negative clips for training and 3,549 for testing, while the Action-Presence split consists of a test-only set of 675 clips with no training clips. In QA generation for every distractor choice, the original UCF101 label is rewritten as a clear descriptive sentence to reduce ambiguity; for example, “HulaHoop” becomes “A person or group of people is spinning a hula hoop.”

3.2 Benchmarked Models↩︎

We evaluate 20 representative state-of-the-art open-source MLLMs on the test splits of UCF101-AD to assess their capacity for action denial. All of our chosen models are trained on very large (public and/or proprietary) datasets.

General-Purpose Models. The evaluated models span major families including Ovis [69], [84], Intern [6], Qwen [68], and VideoLLaMA [70], alongside other key architectures [71]–[73], [85]; differing in training scale and alignment strategies. We also include the closed-source model GPT-4o mini [86].

Reasoning Models. To examine whether explicit multi-step inference improves action denial, we additionally evaluate recent reasoning-oriented MLLMs, including Ovis 2.5 [69], Video R1 [77], ARC-Hunyuan [75], Kimi-VL [78] and Lumian-VLR [76]. These models generate intermediate reasoning steps before producing a final prediction, enabling more systematic verification.

3.3 Implementation Detail, Evaluation Protocol and Metric↩︎

All models are evaluated using their official implementation and released weights. In the multiple-choice (MCQ) setting, the order of the answer options is randomized, and the ground-truth answer is assigned to a random position among them. Each question contains 11 options in total. which ensures that the prompt is reliably processed by all MLLMs while limiting the effectiveness of random guessing to \(<10\%\).

We evaluate all models in a visual question answering (VQA) setup and report several accuracy metrics. For negative clips, we measure Type 1 and Type 2 accuracy and their size-weighted average, Overall Action-Denial (Overall-AD). For positive clips, we report Action-Presence accuracy. We also provide an Overall accuracy as the size-weighted average of Overall-AD and Action-Presence. To assess a model’s balanced ability to both recognize and deny actions, independent of class sizes, we report the Harmonic Mean (HM) of Overall-AD and Action-Presence.

3.4 Results↩︎

2 summarizes zero-shot performance of all models on UCF101-AD. Across models, Action-Presence accuracy is generally high (often above 90%), confirming that current MLLMs recognize actions reliably when the motion is present. In contrast, Action-Denial accuracy is much lower: even the strongest model, VideoLLaMA3-7B, reaches only 51.5% Overall-AD on negatives, and most models fall in the 20-35% range. Type 1 negatives, which preserve the canonical objects and environments but do not contain the motion, are harder for most models, suggesting that they heavily over-index on context and object presence rather than verifying the critical motion. Type 2 negatives, which keep the same setting but replace the target motion with other plausible activities, yield slightly higher scores for most models, yet performance remains far from robust, implying that models often fail to distinguish fine-grained motion patterns that disambiguate the target action from related activities. As a result, the Overall and Harmonic Mean between Overall-AD and Action-Presence accuracy remain modest (maximum 58.3 and 67.0), showing that strong action recognition alone does not translate into robust action denial under strong contextual cues.

Human evaluation of action denial. To assess whether the action-denial examples are reliably verifiable by humans, we conducted a human evaluation on UCF101-AD with 36 participants. Each participant viewed videos from five randomly sampled denial classes and judged whether the target action was absent. Participants achieved 86.6% Overall-AD, indicating that humans can reliably identify action absence even under misleading contextual cues.

3.5 Analysis↩︎

Effect of Progressive Reduction of Contextual Confusion. To better understand denial failures, we evaluate two modified MCQ setups that progressively reduce contextual ambiguity. In Explicit Denial, the generic “None” option is replaced with a statement that the scene may contain cues for the action but the action itself is not occurring, nudging models toward verification rather than context matching. In Primary Distractor Removed, the main competing action label is replaced with a random distractor, removing the strongest alternative. As shown in 3, accuracy increases from the Standard Overall-AD MCQ to Explicit Denial and improves further when the primary distractor is removed, often nearing performance on positive actions. This suggests that models can deny actions when ambiguity is reduced, but still struggle in realistic settings where strong contextual cues and plausible distractors remain present. However, in the real-world we cannot simply remove competing actions or simplify the label space, so models must learn to deny actions even when multiple plausible interpretations remain. Details of setups are in supplementary.

Correlating shortcut learning and sycophancy. To test whether shortcut learning and sycophancy are independent failure modes or manifestations of a single underlying bias, we construct an additional Binary Yes/No setup that directly measures sycophancy (e.g., “Is a person or group of people applying makeup to their eyes?”). We then compute a per-video Pearson correlation between shortcut learning and sycophancy, each measured as an error rate: the MCQ error rate is the frequency of failing to select the correct “None” option in the Standard MCQ, and the sycophancy rate is the frequency of incorrectly answering “yes” in the binary task. As shown in 4, all models exhibit a positive correlation between these two errors, with a mean correlation of 0.388 and \(p < 0.01\) for every model, indicating at least a medium association by Cohen’s conventions [87] for most models. This relationship is slightly stronger on Type 2 clips (same context, different motion; mean \(r = 0.409\)) than on Type 1 clips (context only; mean \(r = 0.367\)), and most models follow this pattern. Moreover, for every model we observe \(P(\text{yes} \mid \text{MCQ wrong}) > P(\text{yes} \mid \text{MCQ correct})\) and \(P(\text{yes} \mid \text{primary distractor}) > P(\text{yes} \mid \text{random distractor})\), usually by \(0.2\)-\(0.6\), showing that sycophantic answers concentrate precisely on cases where the model has already latched onto the misleading action hypothesis. This provides evidence that shortcut learning and sycophancy are tightly correlated: models infer actions from contextual cues rather than verifying the defining motion.

General vs. Thinking models. Our analysis of the negative videos in the UCF101-ActionDenial dataset reveals a counterintuitive pattern: reasoning models generally perform worse than their standard counterparts (except Ovis2.5). Instead of improving robustness, their chains-of-thought often over-index on visual context, confidently selecting the most plausible distractor action or even unrelated options. In many cases they also struggle to properly interpret the denial instruction itself, instead hallucinating descriptions of actions that are not present. This suggests that current “thinking” models are poorly calibrated for negative constraints: tending to explain what might be happening rather than justify that nothing is.

4 Improving Action Denial through Causal Graph↩︎

UCF101-AD reveals a fundamental limitation in current MLLMs: they often equate contextual cues with the occurrence of an action. These models operate under a correlational paradigm, implicitly modeling the presence of context with the existence of action and thus lack the capacity for denial. To address this limitation, we move from context-based reasoning to causal reasoning. We treat an action as the outcome of a structured combination of factors including participants, environment, interactions, and most importantly, motion dynamics. We therefore introduce CausalAct, which organizes these elements into a causal structure and encourages the model to verify the required evidence before predicting an action, rather than relying on contextual shortcuts.

4.1 CausalAct↩︎

Causal Framework. We model the evidence required for recognizing an action using a directed acyclic graph (DAG) that organizes scene elements into contextual, relational, and dynamic components. Inspired by structured video representations such as ActionGenome [24], CausalAct decomposes an action scene into the following variables: contextual nodes for persons (P), objects (O), and location (L); relational nodes capturing spatial relations (S) and interactions (I) between entities; a dynamic node representing motion (M); and an action node (A) representing the final activity label (5). The dependencies between these variables follow a causal hierarchy: \[S \leftarrow f_S(P,O,L), \quad I \leftarrow f_I(P,O), \quad M \leftarrow f_M(I), \quad A \leftarrow f_A(I,M),\] where each \(f(\cdot)\) denotes an abstract mapping from the video elements that may influence the corresponding variable.

Intuitively, contextual elements (persons, objects, and environment) determine the spatial configuration of the scene, which enables potential interactions. These interactions generate motion patterns, and only when the appropriate motion is observed should the action be inferred. This structure explicitly prevents shortcut reasoning where models predict actions directly from context (\(A\leftarrow L\) or \(A\leftarrow O\)). Instead, the model must verify the presence of interaction and motion before confirming the action. The graph also accommodates different action types: for person-centric actions, reasoning may proceed directly from motion to action, whereas object-centric actions require both interaction and motion as prerequisites.

Operationalizing CausalAct. To operationalize the proposed causal structure within MLLMs, we translate CausalAct into a structured reasoning framework using the CausalAct Prompt, a natural language instantiation of the graph. The prompt describes the causal variables and their dependencies, guiding the model to examine contextual elements (persons, objects, and environment), relational evidence (spatial relations and interactions), and motion dynamics before predicting the action. Rather than acting as a strict rule-based module, CausalAct provides visual-grounding guidance that encourages the model to organize the observed evidence through the causal graph rather than shortcutting from scene/object context. By explicitly structuring this reasoning process, it encourages the model to verify whether the required causal chain is present and to deny the action when critical evidence, particularly the defining motion, is absent. However, models with weak vision-language grounding may struggle to reliably follow this structured reasoning.

To address this, we introduce an auxiliary finetuning stage that teaches models the dependencies encoded in CausalAct. For each video, we construct a graph over the variables \((P,O,L,S,I,M,A)\) and automatically generate graph-based question-answer tasks probing node identities, edge relationships, causal paths, and property consistency. These questions are derived directly from the graph structure and combined with synthetic distractors, forcing the model to reason over the causal dependencies rather than memorizing action labels. Importantly, the tasks avoid exposing the ground-truth action to prevent label leakage. Learning these graph reasoning tasks encourages models to internalize the causal relationships between context, interaction, motion, and action, resulting in improved ability to correctly deny actions in Action-Denial scenarios. Details of finetuning are provided in Supplementary.

4.2 Evaluating CausalAct↩︎

Effect of CausalAct in Action Denial. We first evaluate the graph in a zero-shot setting (CausalAct-0). As shown in 6, models with stronger language reasoning (Ovis2.5, Qwen2.5-VL) leverage it to better reject false positives, whereas VideoLLaMA3 often fails to follow the causal instructions and can even degrade, indicating that prompting alone is insufficient under weak vision-language alignment.

We then finetune the smaller models on graph-based reasoning questions derived from CausalAct, which substantially improves denial accuracy on the UCF101-AD Action-Denial test set and shifts behavior from contextual shortcutting to denying actions when the defining motion is absent. Although trained only on abstract graph-structure questions, not negative labels, the models acquire a transferable skill of checking the causal chain before asserting an action, as illustrated in 7.

r6.1cm

Generalizability to other datasets: To evaluate whether CausalAct-0 transfers beyond our benchmark, we apply it to Ovis2.5-9B on additional action datasets by replacing the ground-truth action with “None of the choices...”. As shown in 3, CausalAct-0 improves denial accuracy across all evaluated datasets, including UCF101 [16], K400 [15], HMDB51 [17], and SSv2 [28]. We further evaluate on fine-grained motion datasets, Diving48 [33] and FineGym99 [34], where verifying an action often requires subtle temporal motion, interaction, or ordering cues. These gains suggest that causal reasoning helps the model verify whether the defining motion or outcome of a candidate action truly occurs, improving action denial across diverse dataset distributions.

4.3 Ablation Studies↩︎

Does graph structure matter? To test whether performance gains come from the specific causal structure or just any arbitrary graph, we ablate CausalAct by comparing it to a Pruned Graph (only P, O, I, A nodes) and, a Random Graph (all nodes, but shuffled edges). 8 shows that in zero-shot setting (CausalAct-0), models show limited sensitivity to these variants, suggesting that without additional training, they struggle to fully understand the graph’s structure. After finetuning, however, accuracy drops in most cases when the causal dependencies are disrupted, indicating that models have learned to utilize the full structure. These trends suggest that the prompt only serves as an interface for expressing the causal graph to the MLLM. Finetuning with structured representations helps models internalize and use the intended causal graph for reasoning, while cases with minimal drops hint that they weakly rely on the graph or do not parse it.

Is the object node necessary for person-only actions? To test this, we evaluate a reduced graph that omits object and interaction nodes (mapping only Person, Location, Spatial Relation, Motion, and Action) on a subset of 25 purely body and location-based negatives (e.g., Not Baby Crawling, Not Cliff Diving). As shown in 9, zero-shot accuracy with both the full and reduced graphs follows the same trend and matches or exceeds the baseline, indicating that CausalAct’s gains largely stem from organizing causal relations among core scene elements, even when no explicit object reasoning is required.

Is it just language tuning? We ask whether graph conceptualization finetuning simply teaches new text patterns or requires deeper vision-language alignment. We compare finetuning the full model vs. finetuning only the LLM and projector, freezing the vision encoder. We observed that restricting updates to the visual stack consistently hurts performance, with accuracy dropping by roughly 15% on average. This systematic degradation indicates that our method is not simply encouraging prompt-following. To effectively ground the causal graph, the vision encoder must also be updated so that visual evidence about the presence or absence of the action is utilized by the MLLM.

5 Conclusion↩︎

We introduce UCF101-AD, a benchmark targeting a critical failure of modern MLLMs: denying false actions under strong contextual cues. Our experiments on 20 MLLMs show that models excel at positive action recognition but perform poorly on our Action-Denial test set (most scoring \(<50\%\)), revealing a dual bias: a perceptual reliance on static shortcuts and a linguistic tendency toward agreeableness. To guide models toward causal reasoning, we propose CausalAct and accompanying graph-conceptualization finetuning tasks, which improve their capacity for denial. We intend UCF101-AD to serve as a tool for advancing causal reasoning and teaching models to deny.

Acknowledgements↩︎

We gratefully acknowledge Vibhav Vineet [Microsoft Research, USA], Shahzad Ahmad [Norwegian University of Science and Technology, Norway], and Sukalpa Chanda [Østfold University College, Norway] for their valuable support and contributions to this work.

Supplementary Material of
Learning to Deny: Action Denial in Multimodal Large Language Models

This supplementary material provides additional details that complement the main paper.

• 6 and 7 presents additional implementation details and analysis.

• 8 provides additional details on dataset curation, evaluation setups, and model prompts.

• 9 elaborates on the CausalAct framework, including the full prompt structure, graph-based finetuning questions.

• 10 lists the semantic descriptions used to construct textual distractors for multiple-choice evaluation and the full set of UCF101-AD Action-Denial classes.

• 11 and 12 discusses future scope, ethical considerations and dataset use.

6 Additional Implementation Details↩︎

We evaluate 20 diverse MLLMs, spanning both standard architectures and five reasoning-oriented models. Four of these reasoning models are directly derived from standard counterparts: Ovis2.5-9B with reasoning mode enabled; Kimi-VL-A3B-Thinking, the reasoning counterpart to Kimi-VL-A3B-Instruct [78]; and Video-R1-7B [77] and Lumian-VLR-7B-Thinking [76], both built on Qwen2.5-VL. In the zero-shot setting, most models are evaluated on a single NVIDIA RTX A6000 (48GB), while a small number of larger models require a single A100 (80GB).

For fine-tuning with the train set based on CausalAct, we use a single A100 (80GB) GPU for the 2B/3B variants of Ovis2.5, Qwen2.5-VL, and VideoLLaMA3. The training setups share several common aspects: DeepSpeed-based single-GPU training, bfloat16 precision, gradient accumulation over 16 steps, cosine learning-rate scheduling, and gradient checkpointing.

7 Additional Analysis↩︎

4 shows the detailed scores of the discussed setups.

Effect on number of choices: We further analyze a subset of the higher-performing models under a reduced 4-choice setting, while keeping both the primary distractor and the “None” option. As shown in 5, performance fluctuates when moving from 11 choices to 4 choices: several models show modest gains, whereas others decline. Importantly, despite the reduced answer space, nearly all evaluated models still remain below 50% accuracy on Overall-AD. These results suggest that the difficulty is not primarily driven by the number of answer choices. Instead, the central challenge lies in action denial itself: models struggle to reject the primary distractor when strong contextual cues make the target action appear plausible.

Effect of scale: As shown in 6, increasing model scale generally improves performance on the standard Overall-AD setting, but the strength of this effect differs across model families. The Ovis2.5 series exhibits a relatively modest gain. Qwen2.5-VL-Instruct shows progressively larger variants yielding increasingly larger gains. VideoLLaMA3 displays the largest jump among the compared families. Taken together, these results suggest that model scale is helpful for action denial, but only when the underlying architecture is able to translate added capacity into better rejection of contextually plausible yet absent actions.

Computational cost of CausalAct: To quantify the additional computation introduced by CausalAct, we report GPU inference time in minutes on UCF101-AD while keeping the model, dataset, hardware, frame sampling, output length, and decoding settings fixed, changing only the input prompt. As shown in [tbl:tab:computation], CausalAct-0 increases runtime due to its longer structured prompt, but the overhead remains moderate for most models, and GPU memory usage stays comparable to the baseline. Furthermore, Ovis2.5-9B-Thinking requires 639 minutes to achieve 40.4% Overall-AD, whereas standard Ovis2.5-9B with CausalAct-0 reaches a comparable denial accuracy in only 109 minutes. This suggests that CausalAct-0 provides a practical way to encourage reasoning without introducing substantial computational overhead.

8 UCF101-AD Benchmark: Detailed Prompt and Evaluation Setups↩︎

For Action-Denial clips, we began by defining negative classes corresponding to the UCF101 actions and querying YouTube with keywords designed to retrieve videos that preserve similar contextual cues, including people, objects, and locations, while excluding the defining motion of the target action. The retrieved videos were then manually trimmed into shorter clips, with an average duration of \(\sim\)​7 seconds. The overall curation procedure is illustrated in 10. To prevent information leakage, clips originating from the same long source video were grouped together and assigned entirely to either the training or test split, following the UCF101 protocol.

Examples of the Action-Denial MCQ evaluation and analysis setups, along with the question prompt, are shown in 11. Notably, in the Action-Denial setting, all distractor options other than the primary distractor are randomly sampled from unrelated action classes. By contrast, for UCF101 [16] and our Action-Presence videos, distractors are selected to be semantically similar to the target action; for example, playing guitar may be paired with playing sitar. Despite this stronger distractor similarity in the positive setting, models still perform well on those videos. This indicates that the primary difficulty lies not in action recognition itself, but in correctly rejecting an action when contextual evidence strongly implies its presence.

9 The CausalAct Framework↩︎

9.1 Causal Prompting↩︎

We translate CausalAct into a structured natural language prompt. This prompt serves as a comprehensive reasoning instruction for the model. It begins by defining each component (node) of the graph. It articulates the causal dependencies, explaining how each component can be influenced by others and how their interplay ultimately shapes the perceived target action. To reinforce this logic, the prompt includes examples demonstrating both positive and negative scenarios, effectively teaching the model the rules of the causal system. This graph-based prompting aligns with recent efforts to structure inputs for Large Language Models [25], [26], [54]. As illustrated in 12, this detailed causal prompt is prepended to the user’s task query, guiding the model to evaluate the video through this causal lens. The complete prompt structure is detailed below.

9.2 Detailed CausalAct Prompt↩︎

Your primary task is to act as a causal reasoning engine for video understanding. You will be given a set of observed components from a video. Your goal is to use the provided causal graph structure to logically determine the final action and select the most accurate descriptive caption from a list.

Component Definitions

First, understand the meaning of each component (node) in our causal graph. These are the building blocks we use to describe any event in a video.

• P (Persons): This identifies the person or group of people present in the video. For example, ‘a man’, ‘a woman’, ‘two children’.

• O (Objects): This lists the main inanimate objects that are present in the video. For example, ‘a guitar’, ‘a basketball’, ‘a chair’.

• L (Location): This describes the environment or setting where the event takes place. For example, ‘a park’, ‘a kitchen’, ‘an office’.

• S (Spatial Relation): This describes the positioning of persons and objects relative to each other and the location. It’s a snapshot of where things are. For example, ‘a person is standing next to a table’.

• I (Interaction): This describes the sustained physical contact or manipulation between a person and an object over a duration. The presence of an interaction is a crucial clue. For example, ‘a person’s hands are on a keyboard’ or ‘a child’s feet are on the bicycle pedals’. If there is no contact, the interaction is ‘None’.

• M (Motion): This describes the specific movements and body motions of the persons. It is the dynamic part of the video. For example, ‘fingers pressing keys’, ‘legs pedaling’, ‘arms swinging’.

• A (Action): This is the high-level activity that you must infer. It is the final conclusion based on all the other components. For example, ‘typing’, ‘biking’, ‘tennis swing’. There might be no relevant action happening at all.

The Causal Graph: How Components are Related

The components above are not independent. They influence each other in a specific order, which we can represent as a Directed Acyclic Graph (DAG). Think of it as a flowchart where one thing leads to another.

The directed edges (shown with right arrows \(\rightarrow\) or \u2192 character) show the direction of influence. For example, \(P \rightarrow I\) means that the presence of a Person (P) is a prerequisite for an Interaction (I) to occur.

The directed edges of the graph are:

\(P \rightarrow I\)

\(O \rightarrow I\)

\(P \rightarrow S\)

\(O \rightarrow S\)

\(L \rightarrow S\)

\(I \rightarrow M\)

\(M \rightarrow A\)

\(I \rightarrow A\)

Here are the causal relationships (the directed edges) explained in detail:

• \(P \rightarrow I\) and \(O \rightarrow I\) (How Interaction is formed): For an Interaction (I) to happen, there must be both a Person (P) to perform the interaction and an Object (O) to be interacted with. The interaction is the direct result of the person engaging with the object.

• \(P \rightarrow S\), \(O \rightarrow S\), and \(L \rightarrow S\) (How Spatial Relation is formed): The Spatial Relation (S) is determined by where the Persons (P) and Objects (O) are located within the Location (L).

• \(I \rightarrow M\) (Interaction causes Motion): The specific way a person Interacts (I) with an object over time is what causes the Motion (M) we see. For example, the interaction of ‘hands on a guitar’ may cause the motion of ‘fingers strumming strings’.

• \(I \rightarrow A\) and \(M \rightarrow A\) (How the final Action is determined): The final Action (A) is a direct result of both the Interaction (I) and the Motion (M). The Interaction tells you what is being engaged, and the Motion tells you how it is being engaged. Together, they define the action.

Illustrative Examples

To make this perfectly clear, let’s walk through two examples.

Example 1: Interaction is PRESENT

Imagine a video where a person is playing a guitar.

• P (Persons): ‘a person’

• O (Objects): ‘a guitar’

• L (Location): ‘a room’

• Causal Analysis:

  1. The ‘person’ (P) and ‘guitar’ (O) exist in the ‘room’ (L).

  2. Their Spatial Relation (S) is: ‘The person is sitting and holding the guitar’.

  3. Because the person is holding the guitar, a direct Interaction (I) occurs: ‘The person’s hands are on the guitar’s neck and strings’.

  4. This interaction may lead to a specific Motion (M): ‘Fingers are pressing on the frets and the other hand is strumming the strings’.

  5. Therefore, the combination of the Interaction (I) (‘hands on guitar’) and the Motion (M) (‘strumming’) leads to the undeniable conclusion for Action (A): ‘playing guitar’.

Example 2: Interaction is ABSENT

Now, imagine a video of the same person and guitar, but the person is not playing it.

• P (Persons): ‘a person’

• O (Objects): ‘a guitar’

• L (Location): ‘a room’

• Causal Analysis:

  1. The ‘person’ (P) and ‘guitar’ (O) still exist in the ‘room’ (L).

  2. However, the Spatial Relation (S) is different: ‘The person is sitting on a chair, and the guitar is leaning against the wall next to them’.

  3. Because the person is not touching the guitar, the Interaction (I) is: ‘None’.

  4. Since there is no interaction, no relevant Motion (M) is generated. The observed motion might be ‘The person is sitting still’.

  5. Without a direct Interaction (I) or a related Motion (M) involving the guitar, the Action (A) cannot be ‘playing guitar’.

Your Task

You will now be given a new video. The goal is to select the single best caption from a list.

Infer the components internally. Do not print them.

• P (Persons): Think what person or group of people is present in the video.

• O (Objects): Think what main inanimate objects are present in the video.

• L (Location): Think about the environment or setting where the event takes place in.

• S (Spatial Relation): Think about the positioning of persons and objects relative to each other and the location.

• I (Interaction): Carefully observe and think about any sustained interaction (physical contact or manipulation) between a person and an object over a duration. There might be no interaction at all.

• M (Motion): Carefully observe and think about the specific movements and body motions of the persons.

• A (Action): ??? Unknown, you must infer this.

Use the causal relationships (directed edges) and the component definitions from above to decide the Action. Reason with these components in your head. Do not output your reasoning.

From the list of options below, select the caption that most accurately describes the action happening in the video. There is also a choice if none of the other captions apply. Review the video and captions carefully. Only one choice is correct. Reply with the choice number only.

Output format: reply with the choice number only.

9.3 Adapting models for CausalAct↩︎

We propose an auxiliary finetuning stage. The objective of this stage is not to teach action recognition, but to instil the model with an axiomatic understanding of the CausalAct graph’s properties. We construct a set of question-answer pairs grounded in the CausalAct structure based on our training videos. This dataset teaches the model the principles of causal reasoning specific to our graph, analogous to methods that teach causal axioms [61].

The dataset comprises 14 question types across four main categories:

9.3.0.1 I. Graph Structure Queries:

This category assesses the model’s foundational knowledge of the Directed Acyclic Graph (DAG) topology, including component existence and valid connectivity.

1. Node Count Query: Asks for the total number of nodes in the graph. Since the graph structure is fixed with seven components (\(P, O, L, S, I, M, A\)), this serves as a static verification that the model has internalized the complete set of causal variables.

2. Single Node Membership Check (Binary): Poses a yes/no question regarding the existence of a specific node (e.g., “Is V (Viewpoint and Camera) a node of this graph?”). It tests the distinction between valid causal nodes and distractor attributes like lighting or audio.

3. Multiple Node Selection: Presents a mixed list of valid nodes and distractors, requiring the model to select all legitimate components. This evaluates the ability to filter the full set of graph nodes from irrelevant noise in a multiple-choice format.

4. Edge Count Query: Asks for the total number of directed edges in the graph. This verifies knowledge of the fixed causal skeleton, which contains exactly eight directed connections.

5. Single Edge Validity Check (Binary): A verification task where the model must accept valid directed edges (e.g., \(P \to I\)) and reject non-existent or reversed edges. This tests specific knowledge of pairwise causal links.

6. Multiple Edge Selection: A multiple-choice task requiring the identification of all valid edges from a mixed list of real and synthetic connections. This probes edge-level discrimination and directionality.

9.3.0.2 II. Node Relationship Questions:

These tasks examine local graph connectivity, focusing on immediate parent-child relationships and the cardinality of direct causes and effects.

7. Parent/Child Listing: Selects a specific node and asks for an exhaustive list of its immediate parents or children. This tests the ability to retrieve the full adjacency list for a given component (e.g., identifying that \(I\) is caused by both \(P\) and \(O\)).

8. Relationship Cardinality Check: Asks for the count of parents or children for a specific node (e.g., “How many parents does M have?”). This requires the model to summarize incoming or outgoing connections without necessarily listing them.

9. Relationship Validity Check (Binary): Presents a specific pair of nodes and a relationship type (parent or child) for verification. This probes fine-grained understanding of immediate adjacency (e.g., confirming \(A\) is a child of \(M\)).

9.3.0.3 III. Path Discovery Tasks:

This category evaluates global graph reasoning, requiring the model to trace multi-hop connections across the DAG.

10. Full Path Enumeration: Selects start and end nodes and asks for every valid path connecting them (e.g., finding paths from \(P\) to \(A\)). The model must traverse the graph recursively to identify all valid causal routes, such as \(P \to I \to M \to A\) and \(P \to I \to A\).

11. Specific Path Validity Check: Presents a concrete sequence of nodes and asks if it constitutes a valid path. Negative samples are generated by stitching together unconnected nodes or creating cycles, testing the model’s ability to validate trajectory consistency.

9.3.0.4 IV. Property Consistency Checks:

These questions integrate semantic video content with the graph structure, testing causal logic applied to specific textual descriptions.

12. Distractor Property Identification: One component’s description is replaced with a property from a conflicting action class (e.g., swapping the “interaction” from a typing video with one from a biking video). The model must identify which node is semantically inconsistent with the rest of the causal chain.

13. Causal Antecedent Identification: Samples a causal link and asks what earlier property is required given a later observation (e.g., “If we observe [Motion Description], what must be present earlier?”). This tests the understanding that downstream effects logically imply specific upstream causes.

14. Reverse Causality Verification: Asks if an earlier property can be influenced by a later property (e.g., “Can [Interaction] be influenced by [Action]?”). As the graph is acyclic, the answer is invariably negative; this task enforces the strict temporal and causal directionality of the model’s reasoning.

Each question is randomly sampled to ensure balanced coverage of all reasoning patterns. By finetuning smaller models on this graph reasoning set, we hypothesize they will be better equipped to process and utilize the graph-structured prompts for the downstream task of action denial.

As shown in 8, CausalAct improves performance on both Type 1 and Type 2 Action-Denial videos across. The gains are consistently larger on Type 2 negatives, where the context remains similar but a different motion is performed, suggesting that graph-based knowledge is particularly effective at helping models distinguish the target action from closely related non-target motions. Improvements on Type 1 also indicate better resistance to context-only cues when the defining motion is absent. Overall, these per-type improvements show that CausalAct strengthens action-denial reasoning.

10 UCF101-AD Classes and Semantic Descriptions↩︎

To mitigate ambiguity inherent in short class names, each original UCF101 [16] category is rewritten as a clear, descriptive sentence to serve as a robust distractor in the MCQ evaluation.

1. ApplyEyeMakeup: A person or group of people is applying makeup to their eyes.

2. ApplyLipstick: A person or group of people is applying lipstick to their lips.

3. Archery: A person or group of people is shooting arrows with a bow.

4. BabyCrawling: A person or group of people is crawling on the floor.

5. BalanceBeam: A person or group of people is flipping on a balance beam.

6. BandMarching: A person or group of people is marching in a band.

7. BaseballPitch: A person or group of people is pitching a baseball.

8. Basketball: A person or group of people is playing basketball shooting at the basket.

9. BasketballDunk: A person or group of people is successfully dunking a basketball through the basket.

10. BenchPress: A person or group of people is performing a bench press.

11. Biking: A person or group of people is riding a bicycle causing it to move forward.

12. Billiards: A person or group of people is playing billiards hitting shot with stick.

13. BlowDryHair: A person or group of people is blow drying their hair.

14. BlowingCandles: A person or group of people is blowing out candles.

15. BodyWeightSquats: A person or group of people is doing body weight squats.

16. Bowling: A person or group of people is bowling a ball down an alley knocking down the pins.

17. BoxingPunchingBag: A person or group of people is punching a boxing bag vigorously with force.

18. BoxingSpeedBag: A person or group of people is hitting a speed bag.

19. BreastStroke: A person or group of people is swimming breaststroke.

20. BrushingTeeth: A person or group of people is brushing their teeth.

21. CleanAndJerk: A person or group of people is lifting a barbell with a clean and jerk.

22. CliffDiving: A person or group of people is diving off a cliff into water.

23. CricketBowling: A person or group of people is bowling a cricket ball releasing it from the hand.

24. CricketShot: A person or group of people is batting in cricket hitting the ball.

25. CuttingInKitchen: A person or group of people is cutting ingredients in a kitchen.

26. Diving: A person or group of people is diving into water.

27. Drumming: A person or group of people is playing drums.

28. Fencing: A person or group of people is engaged in the sport of fencing, using specialized swords, and following formal fencing rules and techniques.

29. FieldHockeyPenalty: A person or group of people is taking a field hockey penalty shot and scoring successfully.

30. FloorGymnastics: A person or group of people is performing floor gymnastics.

31. FrisbeeCatch: A person or group of people is catching a frisbee.

32. FrontCrawl: A person or group of people is swimming front crawl.

33. GolfSwing: A person or group of people is swinging a golf club.

34. Haircut: A person or group of people is cutting hair.

35. Hammering: A person or group of people is hammering a nail.

36. HammerThrow: A person or group of people is throwing a hammer in athletics.

37. HandstandPushups: A person or group of people is doing handstand push ups.

38. HandstandWalking: A person or group of people is walking on hands in a handstand.

39. HeadMassage: A person or group of people is giving or receiving a head massage.

40. HighJump: A person or group of people is performing a high jump over a bar.

41. HorseRace: A person or group of people is riding in a horse race.

42. HorseRiding: A person or group of people is riding a horse.

43. HulaHoop: A person or group of people is spinning a hula hoop.

44. IceDancing: A person or group of people is performing ice dancing.

45. JavelinThrow: A person or group of people is throwing a javelin.

46. JugglingBalls: A person or group of people is juggling balls.

47. JumpingJack: A person or group of people is doing jumping jacks.

48. JumpRope: A person or group of people is skipping with a jump rope.

49. Kayaking: A person or group of people is paddling a kayak.

50. Knitting: A person or group of people is knitting with needles.

51. LongJump: A person or group of people is performing a long jump.

52. Lunges: A person or group of people is doing lunges.

53. MilitaryParade: A person or group of people is marching in a military parade.

54. Mixing: A person or group of people is mixing ingredients.

55. MoppingFloor: A person or group of people is mopping the floor.

56. Nunchucks: A person or group of people is swinging nunchucks.

57. ParallelBars: A person or group of people is performing on parallel bars.

58. PizzaTossing: A person or group of people is tossing pizza dough.

59. PlayingCello: A person or group of people is playing a cello.

60. PlayingDaf: A person or group of people is playing a daf drum.

61. PlayingDhol: A person or group of people is playing a dhol drum.

62. PlayingFlute: A person or group of people is playing a flute.

63. PlayingGuitar: A person or group of people is playing a guitar.

64. PlayingPiano: A person or group of people is playing a piano.

65. PlayingSitar: A person or group of people is playing a sitar.

66. PlayingTabla: A person or group of people is playing a tabla.

67. PlayingViolin: A person or group of people is playing a violin.

68. PoleVault: A person or group of people is pole vaulting.

69. PommelHorse: A person or group of people is performing on a pommel horse.

70. PullUps: A person or group of people is doing pull ups.

71. Punch: A person or group of people is throwing punches in combat.

72. PushUps: A person or group of people is doing push ups.

73. Rafting: A person or group of people is rafting on white water.

74. RockClimbingIndoor: A person or group of people is climbing indoor rock walls.

75. RopeClimbing: A person or group of people is climbing a rope.

76. Rowing: A person or group of people is rowing a boat.

77. SalsaSpin: A person or group of people is spinning while dancing salsa.

78. ShavingBeard: A person or group of people is shaving a beard.

79. Shotput: A person or group of people is putting a shot.

80. SkateBoarding: A person or group of people is riding a skateboard and moving forward.

81. Skiing: A person or group of people is skiing and progressing downhill.

82. Skijet: A person or group of people is riding a jet ski and moving forward.

83. SkyDiving: A person or group of people is skydiving, free-falling straight downwards risking hitting the ground if there was no parachute.

84. SoccerJuggling: A person or group of people is juggling a soccer ball.

85. SoccerPenalty: A person or group of people is taking a soccer penalty kick.

86. StillRings: A person or group of people is performing on still rings.

87. SumoWrestling: A person or group of people is wrestling in sumo style.

88. Surfing: A person or group of people is surfing on waves.

89. Swing: A person or group of people is swinging on a playground swing.

90. TableTennisShot: A person or group of people is playing a table tennis shot.

91. TaiChi: A person or group of people is practicing tai chi performing martial forms with striking and defensive gestures.

92. TennisSwing: A person or group of people is swinging a tennis racket and hitting the ball.

93. ThrowDiscus: A person or group of people is throwing a discus.

94. TrampolineJumping: A person or group of people is jumping on a trampoline.

95. Typing: A person or group of people is typing on a keyboard.

96. UnevenBars: A person or group of people is performing on uneven bars.

97. VolleyballSpiking: A person or group of people is spiking a volleyball causing it to hit the ground.

98. WalkingWithDog: A person or group of people is walking with a dog and moving forward.

99. WallPushups: A person or group of people is doing push ups against a wall.

100. WritingOnBoard: A person or group of people is writing on a board.

101. YoYo: A person or group of people is playing with a yoyo.

11 Future Scope↩︎

While CausalAct improves action denial in the current benchmark setting, there remains room to extend this problem to more challenging scenarios. Future work can study action denial in longer and more complex videos, where relevant evidence may be sparse, delayed, or distributed across time, and move beyond closed-set evaluation toward open-world settings with unseen actions and greater semantic diversity. A key challenge in such settings is scalability: complex real-world videos may contain many objects, making it impractical to include all visible entities in the causal graph or prompt. Future variants could therefore incorporate actor-centric filtering, focusing only on objects that are spatially close to, manipulated by, or otherwise interacting with the actor before constructing the denial prompt. It would also be interesting to examine how action denial generalizes to embodied or interactive environments. More broadly, future work can scale the benchmark to more diverse domains and investigate how causal reasoning can be adapted to support robust video understanding in these broader settings.

12 Ethics and Dataset Use↩︎

The dataset is constructed from publicly available YouTube videos released under Creative Commons licenses. The released dataset contains trimmed clips rather than full original videos, and is intended solely for research on action denial/recognition. The dataset is not designed for identity recognition, face analysis, or sensitive attribute inference, and no identity-related labels are separately collected. Given the use of publicly available material, we consider the ethical risk of this dataset to be limited.

References↩︎