1 Introduction↩︎

Photographs frequently contain visual distractors, objects that compete for attention with the photographer’s intended subject or weaken the overall composition. Modern editing tools have made removal trivial once a distractor is identified. However, the preceding step, deciding which objects to remove, remains entirely manual. In addition, this process demands compositional expertise that photographers may not have, especially when the scene contains multiple objects with complex relationships.

Existing approaches partially automate detection but share a structural limitation: they produce one fixed distractor suggestion per image with no mechanism for subject-conditioned adaptation. Saliency-based methods [1], [2] highlight regions with strong visual contrast. Data-driven predictors [3] learn a generic notion of “distracting” from human removal choices. Interactive cleaners [4] propagate a user click to visually similar objects. None accepts a specification of the intended main subject, so none can adapt when the user’s intent changes: the same object may be a distractor when the woman in the frame is the subject, and an essential compositional anchor when the dog is. Current VLM benchmarks also fail to probe this subject-aware compositional reasoning.

This paper formalizes this problem as subject-aware distractor localization. Given an image and a textual description of a main subject \(M\), the task is to identify which other objects are distractors with respect to \(M\). An object is a distractor if it satisfies at least one of five inclusion factors (perceptual and relational properties driving attentional competition). It must simultaneously trigger none of the three exclusion rules, which enforce contextual retention and override all inclusion factors.

To measure how well current models solve this task, we introduce SADL. This benchmark covers diverse compositional scenarios, including a substantial set of hard negatives (objects that look like distractors but must be retained). We evaluate seven VLMs and four non-VLM baselines under two complementary protocols, where guided classification isolates pure compositional reasoning and open detection measures end-to-end performance under realistic deployment conditions. Evaluation exposes a hierarchy of systematic failure modes, in exclusion calibration, factor detection, and spatial grounding, that binary benchmarks tend to conflate or obscure.

Our key contributions can be summarized as follows:

1. A formal framework for subject-aware distractor localization with five inclusion factors (\(F_1\)–\(F_5\)) and three exclusion rules (\(E_1\)–\(E_3\)), grounded in visual attention theory and expert photography, enabling process-level error decomposition by root cause.

2. SADL, the first real-world benchmark for this task: 1,800 subject-aware cases across 1,000 images, 14,617 annotated candidates, 1,938 hard negatives, and a dual-protocol design isolating compositional reasoning from spatial grounding.

3. Controlled ablation and IAA-stratified analysis exposing four empirically distinct failure modes (exclusion over-triggering, output contamination, factor detection failure, and spatial grounding failure), each with a distinct diagnostic signature and suggested remediation direction.

2 Related Work↩︎

2.0.0.1 Distractor Detection and Scene Editing

Classical saliency models [1], [2], [5] and data-driven predictors [3], [6], [7] produce one fixed distractor map per image with no subject-conditioned adaptation. Editing tools and segmentation models (SAM [8], SAM 2 [9], InstructPix2Pix [10], SimpSON [4]) assume a completed upstream identification step. SADL targets exactly this missing step, formalizing subject-aware identification with multi-factor annotation and hard negatives.

2.0.0.2 Object Grounding and Subject-Aware Reasoning

Referring expression comprehension [11], [12] localizes a specified entity; our task is the inverse, identifying objects that compete with that entity. Open-vocabulary detectors [13] and grounding models [14] lack subject-aware exclusion reasoning; treating all non-subject objects as distractors yields near-random precision. Works that add synthetic distractors to VLM inputs [15]–[17] use distractors as a confound variable, not an identification target; SADL uses real photographs where multi-factor relationships and contextual retention must be jointly reasoned.

2.0.0.3 Diagnostic Benchmarks for MLLMs

MathGLANCE [18] and UReason [19] decouple visual perception from reasoning in MLLMs, exposing capability gaps that aggregate metrics obscure. Our dual-protocol design, GC isolating compositional reasoning and OD measuring end-to-end grounding, applies this diagnostic principle to subject-aware distractor localization.

3 Problem Definition↩︎

Let \(I\) denote an image and \(T\) a textual description specifying a main subject \(M\). Let \(\mathcal{O} = \{o_1, \ldots, o_N\}\) be the set of all detected objects in \(I\) excluding \(M\). An object \(o_i\) is classified as a visual distractor with respect to \(M\) if and only if it satisfies at least one inclusion factor and triggers no exclusion rule: \[\text{Distractor}(o_i \mid M, I) = \Bigl[\bigvee_{k=1}^{5} F_k(o_i)\Bigr] \wedge \Bigl[\bigwedge_{j=1}^{3} \neg E_j(o_i)\Bigr]. \label{eq:distractor}\tag{1}\]

Tables 1 and 2 define the benchmark’s scope. The five inclusion factors, grounded in visual attention theory and expert photography, cover the principal mechanisms by which objects compete for attention relative to \(M\). Any triggered exclusion rule (\(E_1\)–\(E_3\)) overrides all inclusion factors, retaining objects that extend the subject’s identity (\(E_1\)), constitute expected scene fabric (\(E_2\)), or are functionally indispensable to the subject’s activity (\(E_3\)).

3.0.0.1 Factor coverage

The five factors are jointly sufficient: any real-world distractor falls into at least one category. Marginal rates (\(F_1\): 26.1%, \(F_2\): 30.0%, \(F_3\): 13.7%, \(F_4\): 13.8%, \(F_5\): 16.2%) confirm no single factor dominates; a detailed breakdown of factor combinations is provided in the supplementary material (sadl_supp.pdf).

4 SADL Dataset Construction↩︎

4.1 Image Collection↩︎

We collected 1,000 high-resolution (\(\geq\)​1024 px) images from public datasets (COCO [20], Winoground [21], Visual Genome [22], Open Images [23]), stratified across eight scene categories to ensure diverse compositional structures: street/outdoor (170), indoor (160), nature (140), portrait (150), event (130), sports (100), product/still-life (75), and artistic (75). Images with zero distractors (clean portraits, isolated subjects) are intentionally included to penalize models that over-segment indiscriminately.

4.1.0.1 Dynamic subject specification

A single image yields multiple main-subject cases, each defined by a distinct \(T\) specifying a different \(M\). An image of a woman walking a dog yields separate cases for \(M{=}\)“the woman” and \(M{=}\)“the dog”, with the distractor set shifting accordingly (Figure [fig:teaser]). These images yield 1,800 annotated cases (mean 1.80 cases/image): 30.0% single-case, 61.8% two-case, 6.6% three-case, and 1.6% four-or-more-case images.

4.2 Hybrid Annotation Pipeline↩︎

4.2.0.1 Stage 1: Agentic pre-labeling

Qwen3-VL-32B-Instruct [24] generates candidate object labels per image; SAM 3 [25] segments each candidate to obtain masks and bounding boxes. The same model proposes 1–5 main-subject cases per image. Qwen3-VL-32B-Instruct is excluded from evaluation to prevent annotation–evaluation circularity; the gold ground truth reflects majority human vote.

4.2.0.2 Stage 2: Human annotation

A human reviewer checks all labels for vague or incorrect entries and adjusts bounding boxes where necessary. Two annotators independently label each candidate with all five inclusion factors and three exclusion rules. The final binary distractor label is derived from Eq. 1 . An adjudicator with professional photography experience resolves all disagreements; the final GT is the majority vote of the three reviewers. Factor-level recall columns in Section 6.3 measure behavioral adherence to the annotation schema; they characterize output patterns and are not claims about latent model representations.

4.3 Inter-Annotator Agreement↩︎

Overall binary distractor agreement between the two primary raters reaches Cohen’s \(\kappa\) [26] \(= 0.534\) (raw agreement 77.0%). Per-factor Krippendorff’s \(\alpha\) [27] reveals a clear objectivity spectrum: geometric and categorical factors (\(F_2\), \(F_4\), \(F_5\): \(\alpha \geq 0.81\)) achieve substantially higher agreement than perceptual factors (\(F_1\), \(F_3\): \(\alpha \approx 0.57\)). Both raters agree more strongly with the adjudicated gold standard (\(\kappa = 0.674\) and \(0.718\)) than with each other, confirming adjudication resolves systematic biases. Full IAA tables are provided in the supplementary material (sadl_supp.pdf).

4.4 Dataset Statistics↩︎

Figure 1 provides a comprehensive overview of SADL’s composition and diversity across twelve dimensions. The 14,617 annotated candidates are partitioned into three label classes: 7,378 distractors (D, 50.5%), 1,938 hard-negative exclusions (E, 13.3%), and 5,301 non-distractors (N, 36.3%) (panel a).

4.4.0.1 Candidate and factor diversity

The number of distractors per main object follows a right-skewed distribution peaking at 3–4 (panel b), while most images contain one or two main objects (panel c). Coverage is broadly balanced across the five inclusion factors \(F_1\)–\(F_5\) (panel d). Object descriptions are concise, averaging 4.3 words, with 90% comprising 1–6 words (panel e), and model accuracy varies by at most 8 percentage points across length bins, ruling out a systematic length-based difficulty confound. The UpSet plot (panel f) shows that single-factor activations dominate, with \(F_2\) (spatial proximity) and \(F_1\) (visual similarity) as the most frequent sole triggers, while multi-factor co-activations are rarer but present throughout.

4.4.0.2 Exclusion structure and multi-case sensitivity

Among the 1,938 hard negatives, rules \(E_1\) and \(E_3\) account for the majority, with fewer than 5% of objects triggering multiple rules simultaneously (panel g). Distractor yield grows sub-linearly with the number of satisfied inclusion factors (panel h), indicating that co-occurring factors are not independently additive. In multi-case images, 59.8% of case pairs contain at least one candidate that switches distractor status across subjects (panel k), directly quantifying cross-subject compositional sensitivity.

4.4.0.3 Scene coverage

SADL spans eight broad scene categories balanced between indoor and outdoor settings (panels i, l). The top-10 factor combinations (panel j) confirm \(F_2\)-only as the single most common pattern, with \(F_1{\times}F_2\) co-activation as the leading two-factor combination.

5 Evaluation Framework↩︎

5.1 Evaluation Protocols↩︎

5.1.0.1 Guided Classification (GC)

The model receives the image, description \(T\), and a pre-enumerated candidate list \(\mathcal{O}\) as text labels. For each candidate it predicts a three-class label (distractor D, excluded E, or non-distractor N) along with per-factor (\(F_1\)–\(F_5\)) and per-rule (\(E_1\)–\(E_3\)) binary attributions. GC isolates pure compositional reasoning with no localization step.

5.1.0.2 Open Detection (OD)

The model receives only the image and \(T\), freely enumerating distractor candidates as short text labels with optional bounding-box hints. SAM 3 [25] converts each label into a segmentation mask; Hungarian matching at IoU@0.5 against GT masks determines TP, FP, and FN. OD measures end-to-end deployment performance.

The GC–OD gap (\(\Delta\)F1 = OD F1 \(-\) GC F1, negative for all models) decomposes the shortfall into the component attributable to classification (measurable via GC) and the residual attributable to spatial grounding and detection coverage.

5.2 Metrics and Baselines↩︎

5.2.0.1 Metrics

Each candidate is assigned one of three ground-truth labels: distractor (D), hard-negative exclusion (E), or non-distractor (N). We report per-class recall, D-Rec, E-Rec, and N-Rec, to expose class-level sensitivity. Average Recall \(\text{AR} = \tfrac{1}{3}(\text{D-Rec} + \text{E-Rec} + \text{N-Rec})\) is the primary metric; it weights exclusion judgment equally with detection, so a model that ignores exclusion entirely scores at most 33%. DE-GMean \(= \sqrt{\text{D-Rec} \times \text{E-Rec}}\) isolates the D/E trade-off: if either recall collapses to zero, so does the score. GC F1 and OD F1 are distractor-class F1 scores under each protocol; their difference \(\Delta\)F1 (always negative) quantifies the spatial grounding bottleneck. Rankings are stable across weighting variants (\(\rho \geq 0.93\)) and IoU thresholds 0.25–0.75 (\(\rho = 1.0\)).

5.2.0.2 Baselines

Seven VLMs spanning three proprietary models (GPT-4.1 [28], Gemini 3 Pro [29], GPT-4o [30]) and four open-weight models (Qwen3-VL-235B [24], Llama-4-Scout [31], Molmo2-8B [32], Pixtral-12B [33]) are evaluated zero-shot under both protocols. The Random assignment baseline, which randomly assigns D/E/N labels based on the dataset distribution and ignores all factors and rules, is included as a lower bound. Four non-reasoning methods (GroundingDINO [13], U²-Net [34], CLIP-text and CLIP-image [35]) yield binary D/N outputs with no exclusion reasoning and are reported separately in Table 4. A human reference (180 GC cases, blind to gold labels) provides an upper bound.

6 Experiments and Analysis↩︎

6.1 Main Results↩︎

Tables 3 and 4 report GC and OD performance; SAM 3 success rate is included per VLM as a proxy for label-to-mask conversion quality.

6.1.0.1 Overall performance

On AR and DE-GMean in GC settings, proprietary VLMs substantially outperform Random assignment, but still witness a large gap to human performance. This gap is even larger under OD, which is the nearest to real-world deployment conditions. Non-reasoning methods seem to score competitively with VLMs on GC F1, but they are not directly comparable due to the difference in class structure (binary D/N vs.three-class D/E/N). The fairer comparison is to look at Ablation D-Rec in Table 5, where non-reasoning methods score at most 60.8% recall, while VLMs reach up to 88.9% when exclusion rules are removed.

6.1.0.2 Spatial grounding bottleneck

GC F1 and OD F1 rankings diverge across models: Qwen3-VL-235B ranks 4th by GC F1 but rises to 2nd by OD F1, while GPT-4.1 holds the highest GC F1 yet falls to 3rd under OD. The divergence is driven by localization rather than reasoning quality: models producing concise, noun-phrase outputs ground more successfully via SAM 3 and lose less in OD than those with verbose or imprecise descriptions. Most of the F1 shortfall is attributable to label-to-mask conversion failure rather than SAM 3 limitations.

6.2 Exclusion Over-Triggering↩︎

6.2.0.1 E over-triggering

Removing exclusion rules from the prompt recovers nearly all distractors that frontier models previously suppressed, establishing over-triggering as the root cause of their D-Rec deficit (Table 5). Models apply exclusion criteria at scene-level plausibility rather than anchoring them to the named subject. Binary benchmarks cannot surface this failure mode, as excluded candidates are indistinguishable from true negatives without the three-class distinction.

6.2.0.2 Factor detection failure

Pixtral’s errors persist even after exclusion rules are removed, pointing to a different root cause. Many true distractors receive no triggered inclusion factors, meaning the model fails at the perceptual grounding step rather than the exclusion step. Weaker models need factor-level improvements; exclusion recalibration alone is insufficient. Llama’s cross-protocol results further show that E over-triggering is a consistent failure across both GC and OD.

6.3 Process-Level Diagnostics↩︎

6.3.0.1 Factor difficulty is non-uniform and model-stable

Categorical similarity (\(F_4\)) is consistently the easiest factor, likely because category matching aligns with VLM pre-training. Semantic incongruity (\(F_3\)) is the hardest, with systematic under-detection across all models. Spatial proximity (\(F_2\)) produces the most false positives, as models over-project proximity beyond true distractor candidates, particularly in the common \(F_1 \wedge F_2\) co-occurrence.

6.3.0.2 Models fail where annotators agree most

IAA-stratified analysis inverts the usual assumption: model errors concentrate on geometric candidates (\(F_2\), \(F_4\), \(F_5\)), where human annotators agree most strongly, rather than on perceptually ambiguous ones. This points to genuine capability gaps in geometric relational reasoning, not annotation noise, and highlights a direction where current VLMs fall furthest behind human performance.

6.4 Failure Mode Taxonomy↩︎

Four empirically distinct failure modes are identified across models (non-mutually exclusive). For each, we note the diagnostic signature and a suggested remediation direction; these directions are hypotheses motivated by the observed pattern, not validated interventions.

6.4.0.1 \(E\) over-triggering (GPT-4.1, Gemini 3 Pro, GPT-4o, Qwen3-VL-235B)

Redistribution rates of 91–97% confirm that frontier models systematically misapply exclusion rules at scene-level plausibility rather than anchoring them to the named subject. The most promising remediation direction is decoupling factor detection from exclusion judgment in the prompt structure, and explicitly conditioning each exclusion criterion on the named subject \(M\) rather than scene context.

6.4.0.2 Output contamination (Llama-4-Scout, Molmo2-8B, Pixtral-12B)

Open-weight models produce structurally inconsistent outputs where exclusion criteria fire without any triggered inclusion factor, a logical precondition violation that silently suppresses true distractors (contamination rate 55–63%). Correcting these inconsistencies substantially recovers recall for Llama, while frontier models are nearly unaffected. Schema enforcement via structured output formats and adding a factor precondition check to the prompt are candidate directions for future work.

6.4.0.3 Factor detection failure (Pixtral-12B, Molmo2-8B)

Pixtral’s errors persist even after exclusion rules are removed, with \(\geq\)​54.6% of true distractors receiving zero triggered inclusion factors. The failure is at the perceptual grounding step rather than the exclusion step, suggesting that factor-targeted fine-tuning or chain-of-thought grounding prompts may be more effective than exclusion recalibration alone.

6.4.0.4 Spatial grounding failure (all models; worst GPT-4.1, \(\text{FN}_\text{loc} = 96.4\%\))

All VLMs suffer large GC-to-OD drops driven by label-to-mask conversion failure rather than SAM 3 limitations. Models producing concise noun-phrase labels ground more successfully; verbose or ambiguous outputs fail at the grounding step. Adopting noun-phrase output constraints and iterative label-to-mask verification are suggested as practical remediation directions.

Qualitative examples illustrating these failure modes in detail are provided in the supplementary material.

7 Discussion↩︎

The dominant finding, exclusion over-triggering as the causal bottleneck, would be conflated with detection failure under a binary benchmark, where excluded candidates collapse into the negative class. The three-class design is what makes this failure mode observable.

7.0.0.1 Practical implications

The benchmark provides process-level attribution of why current models fail (E-calibration for frontier models, factor detection for open-weight models, and spatial grounding for all), and the rich candidate set with full factor-level attributions supports fine-tuning on structured distractor reasoning. The factor-and-rule schema functions as a subject-aware attribute decomposition [36]: a model that fully understands the five factors can adapt as user intent changes, for instance shifting from “remove all audiences” to “remove only audiences near the subject,” which requires proximity reasoning (\(F_2\)) conditioned on a new \(M\). Dynamic subject specification supports this directly by making every annotated image reusable across different subject specifications without re-annotation.

7.0.0.2 Dataset release and reproducibility

SADL is publicly released at the project page under CC BY 4.0 for all annotations; source images retain their original licenses (COCO, Winoground, Visual Genome, Open Images), all of which permit research use. The release includes per-candidate factor/rule labels, bounding boxes, segmentation masks, subject descriptions, evaluation code, and prompt templates. All VLMs are evaluated zero-shot at fixed model snapshots with documented random seeds. Source images are drawn from publicly available datasets; no personally identifiable data was collected, and all annotation work was performed by compensated, consenting annotators.

7.0.0.3 Limitations

The 1,000-image scale reflects a deliberate quality-first design; diagnostic power derives from annotation density and dual-protocol evaluation rather than image count, and no evaluated model approaches saturation. The five factors were selected through expert consultation and literature review to cover the principal mechanisms by which objects compete for attention, and their joint coverage is validated by distractor-type saturation across the dataset; the schema can be extended by adding new factors, though we believe the current set already covers the large majority of real-world distracting patterns. The candidate universe was seeded by an agentic pre-labeling step; annotators added only 1.46% of candidates not proposed by the model and removed 4.04% as false proposals, confirming near-exhaustiveness. OD detections outside GT are counted as false positives; a small fraction could correspond to valid distractors missed at the proposal stage.

8 Conclusion↩︎

We introduced SADL, the first diagnostic benchmark for subject-aware visual distractor localization. Controlled ablation reveals that frontier VLMs fail primarily at exclusion calibration, misapplying rules at scene-level plausibility rather than anchoring them to the named subject. Beyond reasoning, spatial grounding forms the primary barrier to end-to-end deployment. Open-weight models additionally suffer from output contamination and factor detection failure as distinct, diagnosable pathologies. SADL is publicly released with annotations, evaluation code, and prompt templates, providing a reproducible process-level diagnostic foundation for subject-aware compositional understanding.

References↩︎