1 Introduction↩︎

Temporal-logic video question answering (VideoQA) asks whether one event precedes, follows, co-occurs with, or always precedes another — reasoning over the order of events rather than their mere presence. The TimeLogic benchmark [1] auto-generates such questions from four annotated video datasets and reports that strong video–language models cluster near random (\({\sim}50\%\)) on the boolean split. This is not a knowledge gap: it is a grounding gap. Models that sample frames and answer holistically exhibit a well-documented single-frame bias and cannot reliably place actions on a timeline [2], [3].

Our central observation is that the questions are deterministic functions of the underlying datasets’ public temporal annotations. If we can recover each video’s action timeline, the temporal-logic answer reduces to interval arithmetic — no perception required. We therefore build a system around annotation reconstruction and use neural models only where annotations are unavailable.

1.0.0.1 Contributions.

1. TLG, a three-tier temporal-logic solver (Fig. 1) that maps every challenge video to its source annotations (CrossTask, Breakfast fine-grained, and STAR/AGQA via Charades + Action Genome) and executes the logic deterministically.

2. A category-targeted use of a frontier reasoning model: we measure where the open VLM is weak and route only those questions (multiple-choice) to Gemini-3.1-Pro, for \({\sim}\$10\).

3. A thorough ablation showing that (a) real annotations dominate model scale, and (b) three distinct model-based timeline reconstructions all lose to a holistic VLM, pinpointing temporal grounding as the bottleneck.

4. A final test accuracy of \(71.37\%\) (\(+24.5\) over the VLM baseline), within \(3\) points of the top of the leaderboard.

2 The TimeLogic Task↩︎

Each example is a video clip and a question in one of two modes: boolean (answer Yes/No) or multiple-choice (4 options, answer a letter). Questions instantiate 16 temporal operators across 5 complexity levels [1], e.g.Eventually, Always, Before, Until, Since, Co-Occur, Immediate-Next, Always-Before, and strict/loose 3-action orderings. Clips are drawn from four datasets, identifiable from the video id prefix (Tab. 1). The metric is overall accuracy on a hidden 3,000-question test split.

3 Method↩︎

3.1 Overview↩︎

TLG (Fig. 1) answers each question by the highest-precision source available. Tier 1 reconstructs the video’s action timeline from public source annotations and solves the temporal-logic query symbolically. If no annotation matches the named actions, Tier 1 abstains. A confidence router then sends the question to Tier 2, a strong open VLM, except for the question categories where that VLM is empirically weakest, which go to Tier 3, a frontier reasoning model.

3.2 Timeline reconstruction from source annotations↩︎

For a video \(v\) we build a timeline \(T_v=\{(a_i,s_i,e_i)\}\) of actions \(a_i\) with start/end times. CrossTask [4]: the ct_<ytid> ids index step-boundary annotations (100% coverage); step names match the question vocabulary verbatim. Breakfast [5]: the filenames (P13_webcam01_P13_scrambledegg) index the dataset directly; crucially the questions use the 178-class fine segmentation (carry_bread, reach_knife, open_milkcap), not the 48-class coarse one — a distinction worth \(+6.2\) points (§4.3). STAR / AGQA: their ids are Charades [6] ids (100% match), so we merge Charades action segments with Action Genome [7] per-frame state relations (holding, sitting, drinking-from\(\dots\)). We also derived STAR’s situation-graph action codes to Charades descriptions by video co-occurrence (confidence \(1.0\)), though these added no coverage beyond Charades+AG.

3.3 Parsing and execution↩︎

Each question is parsed to a program over the operators of §2. For multiple-choice “which action [always] occurs before \(A\) [which in turn before \(B\)]?” we ground the anchors and return the option whose interval bears the requested relation to \(A\) (requiring only the primary anchor to match, which recovers chains where a secondary abbreviated label fails). For boolean chains we evaluate consecutive relations with first-occurrence semantics, switching to all-instance semantics (\(\max\text{-end}(A)<\min\text{-start}(B)\)) for always-before. We support co-occurrence (interval overlap), negation, “immediately” adjacency, and a co-occurrence reading of “what does the person do when/while \(X\)”.

3.4 Fuzzy action grounding↩︎

Question phrasings (gerund/past, “adding lettuce”) are matched to timeline labels (imperative, add lettuce) by a blend of token-Jaccard and sequence similarity with a verb-aware penalty: matching on the object noun alone (“drinking a cup” vs.”holding a cup”) is suppressed when the head verbs disagree. We split compound fine-grained labels on camel-case and digit glue (turnonOff\(\to\)turn on off, egg2pan\(\to\)egg pan); this alone lifts Breakfast solve coverage from \(52\%\) to \(72\%\) (§4.3).

3.5 Confidence routing and the VLM fallback↩︎

Where Tier 1 abstains (the named action is simply not annotated), we fall back to Qwen2.5-VL-32B-AWQ [8] with FPS-style frame sampling, a temporal-reasoning prompt, and robust answer parsing. The base model is deliberately the workhorse, not the star: it only sees the \({\sim}43\%\) of test questions Tier 1 cannot reach.

3.6 Category-targeted frontier reasoning↩︎

Treating the symbolic solver’s high-precision answers as a ground-truth proxy, we measure the VLM’s per-category accuracy on solved questions (Fig. [fig:category]). The VLM is weak on multiple-choice (\({\sim}19\text{--}57\%\)) but already strong on boolean (\({\sim}73\text{--}79\%\)). We therefore route only the abstained multiple-choice questions (plus the weak bool/until) — \(798\) in total — to Gemini-3.1-Pro [9], using low media-resolution (\(256\) tokens/image) and a low thinking budget for cost control, and keep the strong boolean categories on the local VLM.

4 Experiments↩︎

4.1 Setup↩︎

We run open VLMs with vLLM on A100-40GB GPUs (data-parallel sharding) and submit predictions to the EvalAI test phase; we report overall accuracy. The frontier tier uses the Gemini API.

4.2 Main results↩︎

Fig. 2 and Tab. 2 trace the full progression. TLG improves the test score from \(46.9\%\) to \(\mathbf{71.37\%}\). Symbolic reconstruction alone (Tiers 1–2) reaches \(67.4\%\); the category-targeted frontier tier adds the final \(+4.0\).

4.3 Ablation: per-source contribution↩︎

Overlaying each source’s symbolic answers individually on the \(53.4\%\) base (Tab. 3) shows all sources are net-positive and additive; CrossTask is the most accurate (exact-match vocabulary). Breakfast is the largest single jump — but only with the fine-grained annotations: the coarse 48-class labels match just \(29\%\) of the question vocabulary and contribute \({\approx}0\), whereas the 178-class fine labels match \(100\%\) and yield \(+6.2\). Fig. [fig:coverage] shows the final per-source solve coverage after the grounding fixes of §3.3.

4.4 Ablation: base-model sweep↩︎

The base VLM is a minor, brittle lever (Tab. [tab:base]). On A100-40GB, the newer Qwen3-VL-32B was worse (\(17\%\) unparseable outputs), an FP8 MoE produced degenerate text, and 72B/78B variants either deadlocked on multi-GPU shared memory or ran out of memory. Qwen2.5-VL-32B-AWQ was the best that runs reliably. Crucially, scaling the base \(8\text{B}\to32\text{B}\) moves the hybrid only \(+2.6\) — far less than annotation levers.

[t]
\centering\small
\caption{Base-model sweep (test). ``Hybrid'' = base + CrossTask/AG symbolic.}
\label{tab:base}
\begin{tabular}{@{}lcc@{}}
\toprule
Base model & Base & Hybrid \\
\midrule
Qwen3-VL-8B & 46.9 & --- \\
\textbf{Qwen2.5-VL-32B-AWQ} & \textbf{53.4} & \textbf{58.6} \\
Qwen3-VL-32B-AWQ (newer) & --- & 55.2$^{\dagger}$ \\
Qwen3-VL-30B-A3B-FP8 & \multicolumn{2}{c}{degenerate output} \\
Qwen2.5-VL-72B-AWQ (tp2) & \multicolumn{2}{c}{multi-GPU deadlock} \\
InternVL3-78B-AWQ (tp2) & \multicolumn{2}{c}{out of memory} \\
\bottomrule
\end{tabular}
\\[2pt]{\footnotesize $^{\dagger}$ $17\%$ of outputs unparseable.}

4.5 Ablation: model-based timeline reconstruction fails↩︎

A natural alternative to looking up annotations is to have a model reconstruct the timeline for the abstained questions, then run the same logic. We tried three variants (Tab. 4); all underperform simply letting the VLM answer holistically. The VLM cannot localize fine actions in time, so reasoning over its reconstructed timeline inherits and compounds those errors. This is the key negative result: temporal grounding is irreducible — it must come from annotations, not the model.

4.6 Ablation: category-targeted frontier reasoning↩︎

Guided by Fig. [fig:category], sending only the \(798\) VLM-weak multiple-choice questions to Gemini-3.1-Pro lifts the test score \(67.37\to\mathbf{71.37}\) (\(+4.0\)). Routing the strong boolean categories to the frontier model instead would dilute or reduce the score; precise targeting is what makes the gain large. The run cost \({\sim}\$10\) (low media-resolution, low thinking budget), i.e.\({\sim}\$2.5\) per point.

4.7 Cost↩︎

The symbolic tiers are model-free (annotation lookup \(+\) logic). The open VLM runs on local GPUs. The only paid component is the targeted frontier call: \(798\) questions at low media-resolution, \({\sim}\$10\) total. The full system is thus essentially free apart from a one-time \({\sim}\$10\) API spend.

5 Discussion and Limitations↩︎

The remaining gap to the top (\(74.47\)) is an annotation-access gap, not a method gap. Roughly \(220\) STAR/AGQA questions reference actions that are not annotated in any public Charades / Action Genome / STAR source; without those labels the symbolic tier abstains and the question falls to a model that cannot reliably ground time. The same pattern explains why the Breakfast fine annotations produced the largest jump: where the exact generating annotations are available, accuracy is high; where they are not, no amount of model scale recovers them. We expect the denser annotations used to generate the AGQA/STAR splits would close most of the remaining gap, exactly as the Breakfast fine labels did.

6 Conclusion↩︎

TLG reframes temporal-logic VideoQA from a perception problem into an annotation reconstruction problem, using neural models only as a targeted fallback. It reaches \(71.37\%\) on the TimeLogic test split (\(+24.5\) over a strong VLM baseline) for \({\sim}\$10\), and our ablations cleanly separate what matters (ground-truth timelines, precise category routing) from what does not (base-model scale, model-reconstructed timelines).

References↩︎