1 Introduction↩︎

Try the three tasks in Figure 1. Count the separated regions of grassland in the aerial photo on the left. In the middle, trace the rope from the green “S” through all crossings to the far end, reading off each labeled point that you pass. On the right, count every beige fabric patch on the linen whose silhouette matches the one shown on the kraft-paper template. None can be solved at a glance. Each requires several seconds of iterative, hypothesis-driven inspection: fixate, predict, return. The visual system executes this loop effortlessly, and the conscious mind barely notices it. Decades of research in psychophysics, cognitive science, and computer vision have argued that this iterative looking, known as active observation, is not an ornament on top of perception but, for a wide range of tasks, an essential part of perception itself [1]–[7].

Multimodal large language models (MLLMs) have advanced rapidly, yet it remains unclear whether their benchmark gains reflect progress in this capability. From the GPT-4 family [8] onward, frontier models such as Claude Fable 5 [9], GPT-5.5 [10], and Gemini 3.1 Pro [11] have achieved strong results on benchmarks commonly used to distinguish among frontier models. Performance on multimodal benchmarks such as MMMU-Pro [12] and CharXiv [13] is nearing saturation, leaving little separation among frontier models.

Yet benchmark saturation does not mean that vision-language understanding is solved. Applications in robotics [14], design [15], manufacturing [16], [17], computer use [18], spatial understanding [19], and scientific discovery [20], [21] require models to revisit visual evidence as they reason, a capability current MLLMs do not reliably exhibit. Existing evaluations leave this requirement largely unmeasured. Public leaderboards remain dominated by captioning, single-image visual question answering, and multiple-choice tasks that can often be answered from a static description of the image. Few benchmarks assess whether models can revisit visual evidence to form, test, and refine hypotheses as reasoning unfolds.

Current MLLMs appear to be passive perceivers: the image is encoded once as a fixed sequence of visual tokens, with no explicit perception–action loop. Yet this need not preclude active observation. During autoregressive reasoning, the model may shift attention among visual tokens as hypotheses evolve, letting earlier visual findings guide later steps [22]. The question is therefore behavioral rather than architectural:

We answer the question with ActiveVision, a benchmark that makes iterative visual perception measurable. It consists of 17 tasks spanning three cognitive demands of active vision: Distributed Scanning for exhaustive coverage of many local signals, Sequential Traversal for ordered stepping along a connected structure, and Visual Attribute Transfer for fine-grained comparison across regions. Across these tasks, the underlying configurations resist concise language description, so a model cannot easily summarize an image once in language and answer from that summary alone; it must keep returning to the pixels as its reasoning unfolds.

To create this structural complexity and render it realistically, each task follows a two-stage construction pipeline. First, a procedural scaffold specifies the underlying geometry exactly, including arbitrary positions, shapes, and curves. Second, GPT-image-2 [23] re-renders the scaffold as a photorealistic image while preserving positions, counts, labels, and topology (§3.3). Voronoi regions become aerial fields and rivers, arrow chains become stones linked by footprints, and tangled loops become ropes on driftwood. The resulting images introduce realistic textures and visual complexity absent from sparse, cartoon-like renderings.

Every frontier model we evaluate fails on the overwhelming majority of the 85 released items. We test each model at every reasoning-effort tier exposed by its API. Across all settings, the best result is 10.6%, achieved by GPT-5.5 xhigh, which still scores zero on 11 of the 17 tasks. Claude Fable 5, despite topping most reasoning and coding leaderboards, solves just 3.5%. Three human participants, by contrast, average 96.1%, roughly nine times the best model’s accuracy. Increasing reasoning effort barely narrows this gap.

Tool use narrows but does not close the gap. We evaluate Codex and two Claude Code agents, which write and run their own vision code and reach only 24.7–50.6% accuracy. Gains concentrate where tasks admit reliable code-based solutions, but traversal remains difficult. On realistic imagery, models often miss tool failures. Tool use therefore shifts the active-observation bottleneck to verification in the agent loop.

In summary, we (i) introduce ActiveVision, a controlled benchmark that isolates active observation through 17 diverse tasks spanning three cognitive axes (§3); (ii) systematically evaluate frontier MLLMs across their full reasoning-effort range and, as a tool-use test, autonomous coding agents built on them, finding a large and consistent human–model gap that neither additional reasoning nor tool use closes (§4); and (iii) trace the residual failures to a perceptual bottleneck unresolved by additional reasoning or tooling, positioning active vision as a distinct capability that current models lack but real-world applications require (§5).

2 Related Work↩︎

ActiveVision draws on three lines of work: a long-running argument from cognitive science and computer vision that perception is intrinsically active, evidence that MLLMs struggle with basic visual perception, and existing benchmarks constructed to stress vision rather than language. We elaborate on each below.

2.1 Active Vision in Cognitive Science and Computer Vision↩︎

Eye Movements and Vision [7] showed that the same painting elicits different scanpaths under different task prompts, establishing that gaze is allocated by the task rather than the image alone. Later works [1], [2] formalized the consequence in computer vision: inverse problems such as shape from shading, structure from motion, and optical flow are ill-posed for a passive observer but become well-posed once the observer can actively control its sensors as part of inference.

Animate vision [3] reframed the same argument in terms of computational economy, using gaze-targeted local computation rather than global computation. The sensorimotor account [6] advanced the strongest version of this position, arguing that seeing is the mastery of how visual input changes under one’s own movements. Eye-tracking of everyday tasks supports this account beyond the laboratory [4], [5]. The shared prediction is sharp: without iterative sensor redirection, a vision system will fail on perceptual problems that humans solve routinely. ActiveVision operationalizes that prediction for multimodal large language models.

2.2 Vision-Centric Benchmarks for MLLMs↩︎

A first line of work set out to isolate visual perception from linguistic priors. Early diagnoses found frontier models leaning on memorized priors and overlooking queried visual details [22], [24], and benchmarks such as MMVP [25], MMStar [26], and CV-Bench [27] were built to exclude items solvable from text alone and reward reading the image itself, alongside a parallel push on vision-centric model design [28]–[32]. These benchmarks are now largely saturated, indicating that current MLLMs can often read the image when linguistic shortcuts are controlled, while leaving their remaining perceptual limitations unresolved.

A second line instead targets the human–model gap directly. BLINK [33] collects tasks a person answers in the blink of an eye, BlindTest [34] poses questions so simple that a sighted human never misses them, ArtQA [35] asks painting questions that require looking, and BabyVision [36], ChildBench [37], and KidGym [38] draw on tasks an infant or child can already solve. What these share is a framing by human-model gap alone: they establish that models fall short of people without isolating which capability is missing; we recognize active vision as a major part of that missing capability, one that deserves to be measured on its own terms.

A third line is application-driven: examples include OSWorld [18] for real-world computer-use tasks and BenchCAD [17] for parametric CAD generation. These usefully track practical deployment value, but each task bundles coding, reasoning, and visual perception together, so a failure cannot be attributed to perception. ActiveVision isolates exactly this capability. It targets active observation, the iterative looking loop that prior benchmarks do not explicitly name. Its tasks resist shortcuts from any single language description and are rendered photorealistically to resemble real-world inputs encountered in downstream applications.

3 ActiveVision↩︎

ActiveVision collects tasks where a single one-shot glance is unlikely to suffice and the answer benefits from continuously returning to the pixels. §3.1 grounds the design in three elemental operations of human vision, following the definitions of three task families that instantiate them in Figure 2. §3.2 states the design principle every generator follows: discriminative visual state whose information content exceeds what one language description can losslessly carry. §3.3 introduces each of the 17 tasks and describes the photorealistic rendering pipeline that turns each procedural scaffold into the natural-looking image models actually see.

3.1 Task Families: Grounding the Cognitive Foundations↩︎

ActiveVision is built around three elemental operations. Specifically, exhaustive enumeration beyond the subitizing range [39], [40], curve tracing along contours [41], [42], and fine-grained comparison under visual working memory limits [43], [44]. Psychophysics and neurophysiology characterize these operations as serial, attention-demanding routines and are carried out actively through repeated shifts of gaze [45], [46]. We instantiate them below as three task families, each targeting a distinct dimension of active observation.

Distributed Scanning. The image contains many spatially distributed local signals, including dots, strokes, regions, and graph faces. Each must be found and accumulated. Difficulty scales with the number of signals and how evenly they cover the canvas. The characteristic failure has two forms. In partial coverage, the model counts only five or six of ten items and stops before completing a full scan. In faulty individuation, it fails to recognize each signal as a discrete element or distinguish adjacent, similar signals. Items may then be merged, split, or confused with the background before counting begins.

Sequential Traversal. The image encodes a connected structure, such as an arrow chain, a tangled curve through colored regions, or a winding tube. The model must follow it step by step while maintaining its current position, direction, and running tally. Difficulty scales with path length, crossing density, and the visual similarity of decoys to the correct next step. The characteristic failure is gestalt interpolation. The model guesses the endpoint from the start without traversing the intermediate steps.

Visual Attribute Transfer. Fine-grained comparison across regions. The model extracts a visual property from a reference region. This may be length, curvature, thickness, color arrangement, dot pattern, or orientation. It then matches or compares that property against candidates elsewhere in the image. Difficulty scales with the subtlety of the attribute distinction and the number of candidates. The characteristic failure is prior substitution. Instead of measuring both regions, the model applies a learned linguistic prior.

3.2 Task Design Principle: Forcing Iterative Perception↩︎

ActiveVision is built around a single design principle: every task instance carries discriminative visual state whose information content exceeds what a single language description can losslessly carry. An observer that compresses the image once (e.g., “six red circles in the top-left, three blue squares in the bottom-right”) and then reasons over that summary will, by construction, lose the information the answer depends on; solving the task requires keeping the image itself in the loop. The principle is realized via three properties as follows.

We realize this principle through three design properties:

• Arbitrary positions. Items are placed at continuous, sampled coordinates rather than on a grid or at named anchors. Twenty dots scattered over the canvas carry 190 pairwise spatial relations and twenty real-valued coordinate pairs, far beyond what any single linguistic summary can capture.

• Arbitrary shapes. Region boundaries, contours, blob silhouettes, and tile-motif outlines are synthesized fresh for every instance rather than drawn from a fixed library of named shapes. Each is either a closed contour whose radius is modulated by a random number of Fourier harmonics with sampled amplitudes and phases or a periodic spline through jittered ring waypoints. Consequently, the silhouette space is continuous and high-variance, and no two instances repeat a shape. Each boundary is defined by its exact polyline rather than by a named shape such as a triangle.

• Arbitrary traces. The routes the tasks ask the observer to follow, including paths, arrow chains, tangled loops, and the connecting curve through color zones, are smooth random splines through sampled control points, with dozens of meaningful inflection points that no single description preserves.

Together, these properties make iterative perception the only natural solution path. Each instance carries more visual state than a concise description can preserve, so the tasks remain readily tractable for human observers who can revisit the image yet consistently difficult for a one-pass observer to solve reliably.

3.3 Task Instantiation: From Synthetic Scaffolds to Real-world Images↩︎

We instantiate this design with 17 task generators grouped by the three families above. Each produces a synthetic scaffold, question, and ground-truth answer from a deterministic seed, making instances reproducible. Answers span broad ranges with flat per-task distributions, preventing the modal answer or task identity from serving as a reliable shortcut. Table ¿tbl:tab:tasks? summarizes the resulting task set.

6pt 0pt

These tasks are not merely abstract puzzles. Each isolates a perceptual operation that people use in everyday life and across practical settings. For example, Region Counting resembles counting countries on a political map or zones in a thermal scan, while Signal Difference Spotting mirrors comparing ECGs or reviewing seismic traces. Other tasks capture topological reasoning, path tracing, and attribute matching relevant to wiring inspection, floor-plan analysis, microscopy, manufacturing inspection, and satellite change detection.

We place these operations in realistic settings via a two-stage pipeline (Figure 3). Each task generator first produces a procedural Matplotlib scaffold with full ground truth attached. To make the images noisy and realistic rather than cartoon-clean, we re-render every scaffold with GPT-image-2 using a task-specific prompt that maps the procedural primitives to a real-world setting. Only the rendered image is shown to the model.

The pipeline serves three purposes. First, it removes the cartoon-input confound: the perceptual difficulty seen by models is dominated by the task’s discriminative structure rather than by an unfamiliar rendering style. Second, the photorealistic regime is where classical computer-vision baselines degrade sharply, in line with their known brittleness on scanned documents, ultrasound speckle, low-resolution phone snapshots, and similar real-world inputs, so the same images that test MLLMs also bound what a tool-using script can recover. Third, the imagery resembles what downstream applications actually face, making the active-vision diagnosis externally meaningful rather than an artifact of toy renderings.

4 Benchmark Evaluation↩︎

We first report the headline accuracy of frontier MLLMs on ActiveVision (§4.1), then dive deeper with two ablation studies on the effect of reasoning effort (§4.2) and of agentic tool use (§4.4).

From each of the 17 generators, we sample a fixed evaluation split of \(N=5\) instances, yielding \(85\) items in total. Every instance passes through the photorealistic rendering pipeline described in §3.3. We use exact-match accuracy as the primary metric. Unless stated otherwise, evaluations use pure CoT with no external tools or code execution. Each item is presented as a single user message containing the question verbatim from the released manifest, followed by the image. No system prompt is used. The model is instructed to wrap its final answer in <answer> tags.

We score the last such block by exact match after normalizing case, whitespace, and separators. The human baseline comprises three participants, each of whom completed the full 85-item split unaided through a self-paced web interface and was scored by the same criterion. Table ¿tbl:tab:main? reports frontier models at their highest reasoning effort, while Figure 4 reports the full reasoning-effort range for GPT-5.5 [10], Claude Fable 5 [47], Claude Opus 4.8 [48], Gemini 3.1 Pro [11], and Gemini 3.5 Flash [48].

2.4pt

4.1 Main Results: Every Frontier Model Fails Most Tasks↩︎

Table ¿tbl:tab:main? reports per-task accuracy for six frontier models at their highest effort settings alongside a three-participant human baseline. The highest-scoring model, GPT-5.5, solves only 10.6% (9 of 85); even Claude Fable 5, despite its strong performance on reasoning and coding leaderboards, solves just 3.5%. On 11 of 17 tasks, GPT-5.5 scores zero. The human mean is 96.1%, roughly nine times the accuracy of the best model. The six models have only weakly overlapping success sets. No item is solved by all six, suggesting that the gap is not specific to a single model. As a shortcut check, a GPT-5.5 question-only control with the image omitted solves only 2 of 85 items, for an accuracy of 2.4%. This result matches the run with both image and question at the same none effort, indicating that prompt priors alone do not explain the reported performance.

4.2 Does More Reasoning Help?↩︎

The gap is not a deliberation deficit. Figure 4 plots accuracy against API cost per item for every pure-CoT run across reasoning-effort tiers. Every model remains in a narrow low-accuracy band, far below the human baseline. Scaling GPT-5.5’s reasoning effort from none to xhigh increases its per-item cost nearly a hundredfold but improves accuracy only from 2.4% to 10.6%. Task-level failures persist across tiers. The other frontier models show the same pattern. Spending \(31\times\) more per item on Fable 5 does not make it more accurate, and differences between effort tiers are within sampling noise on 85 items. The models do not run out of reasoning steps; they fail to pull the right visual evidence out of the image.

4.3 What Do Models Do When They Fail?↩︎

Pooling the 18 pure-CoT runs of the evaluation sweep, we examine the response behavior of each task family: what the models count, how far they trace, and when they declare a difference.

Counting falls behind as scenes grow crowded. Figure 5 plots predicted count against ground-truth count on the eight counting tasks, with a least-squares fit per model. All fitted slopes fall well below the perfect-counting diagonal, and the fits drop further below it as the true count grows. The models are conservative counters: the more there is to count, the larger the share left uncounted, as if the model glimpses the image rather than scanning it exhaustively.

Tracing loses its way at the very start. Figure [fig:walk-survival] shows, for the three ordered-walk tasks, the share of walks whose first \(k\) steps are all correct. Survival collapses within the first one or two moves, and not a single walk in the pool is completed exactly. The models cannot reliably trace along a structure in the image; failure is not gradual drift late in a long walk but the loss of the walk’s frame at its very beginning.

“Same” becomes the safe default in visual comparison. Table 1 reports per-run error rates on the two panel-naming difference tasks: the share of real differences missed and the share of identical panels falsely flagged. Miss rates are high across the board while false alarms stay low, and the highlighted runs answer “none” on nearly every item, missing all real differences. Unable to resolve the fine-grained comparison, the models fall back on “same” because it is the safe answer, a response bias rather than an act of perception.

2pt

4.4 Can Active Vision Be Substituted by Tool Use?↩︎

Our main result shows that frontier models lack active observation. We next ask whether computation can substitute for it. Code offers a strong substitution channel: brightness thresholding can segment regions, while connected-component analysis can return their count, bypassing direct visual inspection. We test this by running three autonomous coding agents per item: Codex (GPT-5.5), Claude Code (Opus 4.8), and Claude Code (Fable 5). All operate at xhigh effort in fresh sandboxes containing only the image and question.

4.4.1 How Far Does Substitution Carry?↩︎

Agentic tools improve performance substantially but do not close the human gap (Figure 6 (a)). The Fable 5 agent solves 43 of 85 items (50.6%), Codex 32 (37.6%), and the Opus 4.8 agent 21 (24.7%), compared with 10.6% for the best tool-free model. The gains are highly task dependent (Figure 6 (b)). All three agents perform best on Visual Attribute Transfer (43–66%), where template cropping and measurement admit clean algorithmic reductions. In contrast, Sequential Traversal remains difficult: Codex solves 1/25 items, Claude Code with Opus 4.8 solves 3/25, and Claude Code with Fable 5 solves 10/25. Failure is not limited to traversal; all three agents score zero on Tangled Loop Counting. Overall, tool use helps most when perception can be reduced to a reliable computation. When no such reduction is available, the gap persists.

4.4.2 Why Does Substitution Stop There?↩︎

Figure 7 shows the two recurring failure modes. First, tools are not robust on realistic textures (Figure 7, top). On Bounded Face Counting, the agent’s color mask dissolves into the cobblestone texture, so nodes and edges are mis-detected and its Euler-formula count comes out wrong (12 vs.a truth of 10); on Contour Difference Spotting, shadows and paper creases bake into the regenerated silhouettes and a badly scaled crop distorts the true contours, netting out to one wrong “same.”

Second, weak perception cannot catch the tool’s errors (Figure 7, bottom). On Color Zone Sequencing, the agent’s tracer jumps onto a crossing strand and halts early, and the agent audits neither error, submitting the truncated route; on Maze Path Tracing, the binarized mask reads a shadowed hedge as corridor and fuses two openings, and the agent trusts the flood-fill’s incorrect path. In both cases, a glance at the overlay it had just produced would falsify the answer. In short, a model that cannot see cannot tell when its code is wrong.

4.4.3 How Costly Is Substitution?↩︎

Substitution is also expensive and slow. Table ¿tbl:tab:cost? reports the per-item cost of the three tracks: each item costs an agent $2.74–$7.63 of compute and 12–15 minutes of wall-clock time, spent across dozens of tool calls and tens of thousands of output tokens, about 25 times longer than the half minute a human needs, plus compute a human never spends. Even the most expensive agent, Fable 5, solves only about half the benchmark; the human participants answer 96.1% with no tools and no code.

5 Discussion↩︎

5.1 Interpreting the Agentic Evaluation↩︎

Tool-use gains and remaining bottlenecks. ActiveVision evaluates models as pure visual observers. The three agentic tool use runs (§4.4) show why this distinction matters. Classical computer vision primitives such as findContours, Canny, and template matching work best on clean inputs. On ActiveVision’s photorealistic images, they fragment contours, merge objects, and fail under variations in style. A human can look back at the image and recognize that two masks belong to the same shape or that a dashed route continues beyond where the tracer stopped, while the script may return a confident but incorrect answer. Agentic tools therefore depend on active perception for robustness rather than replacing it. This challenge is greater in real deployments, where medical scans, satellite images, phone snapshots, and scanned documents contain more varied and less predictable noise than fixed pipelines can anticipate.

Scope of the agentic evaluation. The agentic ablation instead measures tool-orchestrated visual problem solving: decomposing a visual question into checkable operations, writing extraction code that works on noisy imagery, and recognizing when an output requires visual verification. In effect, the agent offloads iterative visual reasoning to code and reduces verification to simpler visual checks, a lower bar than solving the task through vision alone. Even so, the strongest agent remains far less accurate and efficient than an unaided human. Because ActiveVision is organized around cognitive axes rather than real-world scenarios, fixed pipelines transfer unevenly across tasks. Figure 6 (b) shows this split. The attribute transfer tasks are largely solved, while the traversal tasks remain near zero for two of the three agents, and all agents fail Tangled Loop Counting. We therefore report the agentic evaluation as an ablation rather than a standalone track and leave a dedicated benchmark for agentic visual problem solving to future work.

5.2 Limitations and Broader Implications↩︎

Limitations. Our images are synthetic even with photorealistic re-rendering: GPT-image-2 produces realistic-looking outputs from a controlled prompt rather than samples of the natural-image distribution. We accept this in exchange for exact ground truth and controlled task structure. External validity therefore rests on the elemental visual operations the tasks measure, including scanning, tracing, and comparing, rather than on the renderings themselves (§3.1). Furthermore, our tasks are built so that no short language description carries the answer; as models get better at describing images, this property can erode, and future versions of the benchmark would need to tighten the tasks to restore it.

Outlook. ActiveVision reveals and quantifies a wide gap between human and machine perception that current vision-language benchmarks do not surface, turning “active vision” from a rhetorical claim into a number that future architectures and training objectives can be held to. The gap matters beyond the benchmark because the three operations it isolates (§3.1) are the documented substance of high-stakes visual work. Exhaustive scanning underlies radiology, cell counting, inventory inspection, and aerial-image search. In chest radiography, about 30% of missed nodules are never fixated at all [49]. Sequential tracing is central to connectomics, where proofreaders follow neurites through electron microscopy volumes [50], as well as to vessel and schematic tracing. Fine-grained comparison underlies latent-print examination, pathology, and industrial quality control [51]–[53]. Eye-tracking across these professions shows that expertise largely is knowing where and how to look [54]. A model that cannot count, trace, or compare reliably in a controlled image has no path to reliability in these settings; closing the gap is a prerequisite for the broader goal that motivates this work: reliable MLLMs and the multimodal agents built on top of them.

6 Conclusion↩︎

We introduce ActiveVision, a benchmark that isolates active observation, the ability to keep returning to an image, forming and checking hypotheses, as reasoning unfolds. Its tasks are built so that no single language description carries the answer, and its generation pipeline preserves the exact underlying geometry while rendering it as a realistic, noisy image. In ActiveVision, humans solve nearly all items in about half a minute each, whereas frontier MLLMs fail most items. Increasing reasoning effort does little to close this gap. Code and tools provide larger but uneven gains, concentrated on tasks amenable to reliable algorithmic reduction; even the strongest agent solves only half of the items while requiring far more time and compute than unaided human participants. Closing this gap is a prerequisite for many real-world applications such as robotics, healthcare, manufacturing, and scientific discovery, and ActiveVision turns the gap into a number that future architectures and training objectives can be held to.

References↩︎