1 Introduction↩︎

Composed Image Retrieval (CIR) [1], [2] aims to retrieve a target image using a reference image and a textual modification, enabling users to specify how the reference should be changed rather than describing the target from scratch. Recently, Zero-Shot CIR (ZS-CIR) [3] has emerged as an important paradigm that eliminates the need for triplet annotations (reference image, textual modification, target image) during training, making the task both practical and challenging.

Most existing ZS-CIR approaches design proxy tasks on pretrained vision–language models to avoid triplet supervision. However, these methods mainly focus on learning feature representations that conform to a predefined, non-learnable composition mechanism, leaving the composition process itself unmodeled. Two prevalent paradigms illustrate this limitation. Some methods project image features into pseudo-words and compose them with a frozen text encoder [4]–[8]. Such encoders are pretrained to maximize global image–text semantic alignment, rendering them ill-suited to capturing the localized semantic changes central to CIR. Others rely on simple vector arithmetic for composition [9], [10], which is fundamentally limited in representing the non-linear semantic transformations required in real-world modifications. In essence, these methods tailor representations to a fixed composition rule, whereas CIR calls for a learnable composition mechanism that, by being learned from data, can naturally adapt to diverse, context-dependent modifications. Motivated by human cognitive processes [11], we observe that humans interpret scenes by first directing attention to regions of interest and then using broader contextual cues to refine their understanding. This principle motivates modeling CIR composition as a two-stage procedure, as illustrated in 1 (c): first identifying modification-relevant semantics in the reference image, then integrating them with the textual instruction to form the target representation. Accordingly, we propose Focus-then-Complete (FoCo), a ZS-CIR framework with a learnable composition mechanism. CIR textual modifications typically refer to specific objects, attributes, or background elements, yet such fine-grained semantics are not explicitly reflected in standard image–caption pairs. To expose the model to these diverse semantic aspects, we decompose each caption into local captions describing distinct components and a contextual caption capturing the remaining scene information. This caption decomposition naturally supports the focus–complete process. Leveraging these data, we design two proxy objectives that induce the two capabilities required by FoCo. The first, Text-Anchored Visual Aggregation, selectively gathers visual evidence conditioned on each local caption to obtain a localized visual–semantic representation. The second, Context-Conditioned Semantic Completion, transforms each localized representation with the corresponding contextual caption to recover the full semantics. Together, these objectives allow FoCo to learn where to focus and how to complete. Training these modules, however, is nontrivial. Caption decomposition yields multiple semantic views of an image, each paired with a contextual caption. While these multiple semantic cues are essential for learning the composition mechanism, they also make the optimization considerably more challenging than standard image–text contrastive learning. A standard contrastive objective assumes a single positive caption per image and constructs negatives through coarse instance-level mismatches, which are too weak to supervise such fine-grained distinctions and often lead to unimodal shortcut behavior. To address this, we design a Cross-Instance Contrastive Training scheme that leverages multiple positives and carefully constructed negatives to properly supervise learning from these multiple semantic cues introduced by caption decomposition. This formulation encourages accurate alignment between each local caption and its visual evidence while reducing shortcut behavior.

Our contributions are fourfold: (1) We propose Focus-then-Complete (FoCo), a ZS-CIR framework that models composition as a learnable process aligned with human cognition. (2) We introduce two proxy objectives with a caption decomposition strategy, enabling semantic focusing and context-aware completion without triplet supervision. (3) We develop a cross-instance contrastive scheme that uses multiple positives and cross-instance negatives to improve localized grounding and avoid shortcut learning. (4) FoCo delivers consistent gains across ZS-CIR benchmarks, demonstrating strong effectiveness and robustness.

2 Related Work↩︎

The core challenge of composed image retrieval (CIR) lies in identifying how the modification text should alter the reference image, and encoding this change while preserving the unchanged visual content [12], [13]. Early methods mainly adopt generic fusion mechanisms. TIRG [14] introduces a gated residual connection to modulate visual features using textual input, and MAAF [15] employs residual attention fusion to guide retrieval through cross-modal interactions. To achieve more controllable and semantically guided composition, later approaches design dedicated modules for integrating the reference and the textual modification. CoSMo [16] decomposes textual features into content and style components to support targeted modulation. TMCIR [17] introduces token-level modality balancing to better capture fine-grained user intent. DetailFusion [18] performs hierarchical adaptive fusion to preserve both coarse and detailed cues. PAIR [19] separates coherent and incoherent relations between the image and the text to enable more controlled semantic integration. This line of work [20], [21] consistently highlights the importance of explicitly modeling how multimodal signals interact during composition. However, all these approaches rely on triplet-level supervision, which limits scalability and constrains semantic diversity due to the small size of existing CIR datasets. These limitations have motivated the development of ZS-CIR, which removes this dependency and enables training with large-scale image–caption pairs. A key challenge in this setting is designing proxy tasks that mitigate the distribution mismatch between training inputs (image–caption pairs) and inference inputs (image–text modifications). In this context, we focus on building a learnable composition mechanism that can reliably combine the visual and textual inputs under the ZS-CIR paradigm.

3 Method↩︎

We address Zero-Shot CIR using only paired image–text data. To this end, we propose Focus-then-Complete (FoCo), a self-supervised framework that treats visual–textual composition as a two-stage procedure: it first extracts the visual semantics relevant to the textual instruction, and then transforms the focused representation under textual guidance to yield the target semantics. This two-stage formulation is supported by decomposed caption signals introduced in 3.1, which provide supervision tailored to each stage. Concretely, FoCo consists of text-anchored visual aggregation to gather modification-relevant semantics and context-conditioned semantic completion to transform them into a target representation under textual guidance.

3.1 Local Caption Generation↩︎

Most vision–language models (VLMs) align images and captions primarily at a global level, capturing overall correspondences between text and the visual scene but lacking the ability to model fine-grained linguistic elements and localized visual cues [22]. However, textual modifications in CIR often correspond to localized semantic changes, such as adding an object, changing an action, or modifying the scene. Global captions, which entangle multiple semantic components, are therefore insufficient for supervising fine-grained composition. To address this, we introduce Local Caption Generation, which uses a large language model (LLM), Llama-3.1-8B-Instruct [23], to semantically decompose each global caption \(C_i\) into a set of coherent local captions \(\{c_{i_k}\}_{k=1}^{K}\), where \(K\) denotes the number of local captions for image \(i\). Each local caption describes a distinct semantic component of the scene, capturing fine-grained textual cues that align with localized visual content. As illustrated in 2, these local captions provide the supervision signals for our proxy tasks and correspond to the focus stage of FoCo. Alongside each \(c_{i_k}\), the LLM also generates a contextual caption \(X_{i_k}\), which contains the scene information beyond \(c_{i_k}\) from the original caption. Together, each pair \(\langle c_{i_k}, X_{i_k} \rangle\) covers the full semantics of the scene from different perspectives: \(c_{i_k}\) serves as the focus anchor for visual aggregation, while \(X_{i_k}\) provides the transformation condition for semantic completion. The prompt used for generation is provided in the Appendix.

3.2 Proxy Tasks for Aggregation and Completion↩︎

To equip the model with localized semantic focusing and contextual completion, we design two proxy tasks built upon the generated local captions and their corresponding contextual captions described in 3.1. Each local caption acts as a textual anchor, providing the basis for visual aggregation, while its contextual caption guides subsequent semantic completion, transforming the aggregated visual cue into the target representation. Together, these two tasks implement the overall “focus-then-complete” pipeline.

0.5ex plus 0.2ex minus .1ex -1em Text-Anchored Visual Aggregation. CIR requires reasoning about how specific semantic concepts vary across images, yet global image representations often obscure subtle differences tied to object- or region-level details. We therefore treat each local caption as a textual anchor that conditions the model to dynamically aggregate the relevant visual content. Formally, given an image \(I_i\) with local captions \(\{c_{i_k}\}_{k=1}^K\), this module generates localized visual features conditioned on each anchor: \[v_{i,i_k}^{\text{loc}} = F_{\text{Aggr}}(c_{i_k}, I_i),\] where \(F_{\text{Aggr}}(\cdot)\) is the text-anchored aggregation operator. The image \(I_i\) is represented by a global feature \(v_i^g\) and a set of patch features \(\mathcal{V}_i^p\) extracted from the vision backbone. Each anchor \(c_{i_k}\) is fused with \(v_i^g\) through a lightweight MLP to form a text-guided query: \[q_{i_k} = \mathrm{MLP}(c_{i_k} \odot v_i^g),\] where \(\odot\) denotes element-wise multiplication. Here, \(q_{i_k}\) is conditioned on both textual and global visual context, enabling anchor-specific attention over the visual feature space. The patch features \(\mathcal{V}_i^p\) serve as keys and values in a cross-attention operation: \[\label{eq:focus95attn} v_{i,i_k}^{\text{loc}} = \mathrm{CrossAttn}(q_{i_k}, \mathcal{V}_i^p).\tag{1}\] In practice, it is implemented with a multi-head cross-attention layer. We use \(i\) to index images, \(k\) to index local captions of image \(i\), and \(j \neq i\) to index other images. For clarity, we denote \(v^{\text{loc}}_{i,j_{k'}}\) as the localized feature of image \(I_i\) conditioned on the local caption \(c_{j_{k'}}\) derived from \(C_j\).

0.5ex plus 0.2ex minus .1ex -1em Context-Conditioned Semantic Completion. Building on the aggregation process, this module transforms the localized visual evidence \(v_{i,i_k}^{\text{loc}}\) under the guidance of a contextual caption \(X_{i_k}\) to complete the target semantics.

Formally, we construct a composite input sequence that aligns with the composition logic: the textual context \(X_{i_k}\) serving as a transformation instruction, the localized visual feature \(v_{i,i_k}^{\text{loc}}\) representing the subject to be transformed, and the remaining visual features \(\bar{v}_{i,i_k}\) as the background. We apply modality-specific mapping networks to project these components and concatenate them along the sequence dimension: \[S_{i,k} = [\,f_t(X_{i_k})\,;\, f_v(v_{i,i_k}^{\text{loc}})\,;\, f_v(\bar{v}_{i,i_k})\,],\] where \(\bar{v}_{i,i_k}\) denotes the background representation, obtained by aggregating visual patch features weighted by the inverse of the attention map derived in 1 . This design ensures the dynamic incorporation of unattended regions as visual background during the composition process.

We model the composition as a text-guided transformation process. Specifically, we initialize a learnable query token with the localized visual feature \(v_{i,i_k}^{\text{loc}}\), establishing it as the semantic anchor for transformation. This query token interacts with the sequence \(S_{i,k}\) through multi-layer Transformer blocks \(\mathcal{T}(\cdot)\), dynamically attending to the transformation instructions in \(X_{i_k}\) and the background details in \(\bar{v}_{i,i_k}\) to generate the target representation: \[h_{i,i_k} = \mathcal{T}(q=v_{i,i_k}^{\text{loc}}, kv=S_{i,k}).\] The final output \(h_{i,i_k}\) encapsulates a text-guided transformation of the localized visual anchor. The contextual caption thus acts as a training proxy for the modification text encountered at inference, guiding the transformation of the focused visual subject toward the intended target semantics.

Although the two modules form a coherent composition process, training them effectively remains challenging, as it requires robust cross-modal reasoning across diverse contexts. We therefore adopt a unified cross-instance semantic contrastive framework.

3.3 Cross-Instance Semantic Contrastive Training↩︎

To enable fine-grained semantic grounding and context-aware completion for ZS-CIR, we move beyond standard contrastive learning. We propose a cross-instance semantic contrastive framework with two synergistic objectives: an aggregation loss (\(\mathcal{L}^{\text{aggr}}\)) for grounding local concepts and a completion loss (\(\mathcal{L}^{\text{comp}}\)) for supervising the transformation toward globally consistent target semantics. This framework leverages the decomposed local captions and their contextual captions from 3.1 to construct challenging positive and negative pairs that enforce robust cross-modal representations.

0.5ex plus 0.2ex minus .1ex -1em Aggregation Objective. Given an image \(I_i\), the aggregation operator \(F_{\text{Aggr}}\) produces, for each local caption \(c_{i_k}\), a text-conditioned visual representation \(v^{\text{loc}}_{i,i_k}\) that selectively captures the visual content grounded to that local description. Since each anchor highlights a distinct semantic aspect of the image, the resulting pairs \(\langle v^{\text{loc}}_{i,i_k},\, c_{i_k} \rangle\) naturally form multiple positive samples within the same image. To effectively learn from these multiple positives, we adopt the sigmoid-based contrastive loss from [24], which accommodates multi-positive supervision and provides more stable optimization than softmax-based InfoNCE.

A critical aspect of our framework lies in the construction of negative pairs, which play a key role in preventing shortcut learning. To illustrate this concretely, consider a naive negative sampling strategy where a visual feature is aggregated from image \(I_i\) using an anchor \(c_{j_{k^{\prime}}}\) from another sample. This feature is then contrasted with the anchor \(c_{i_k}\), , \(\langle v_{i,j_{k^{\prime}}}^{\text{loc}}, c_{i_k}\rangle\). In this setup, the aggregation module may degenerate by merely encoding the semantics of the conditioning text \(c_{j_{k^{\prime}}}\), ignoring the visual evidence from \(I_i\). As a result, the model simply distinguishes pairs by comparing the textual similarity between \(c_{j_{k^{\prime}}}\) and \(c_{i_k}\) without genuinely producing the desired text-conditioned visual feature \(v_{i,j_{k^{\prime}}}^{\text{loc}}\).

To avoid this degeneration, we formulate our negative pairs as \(\langle v_{i,j_{k^{\prime}}}^{\text{loc}}, \! c_{j_{k^{\prime}}} \rangle\). These negatives force the model to discern whether \(v_{i,j_{k^{\prime}}}^{\text{loc}}\) is truly supported by the visual evidence in image \(I_i\), rather than simply relying on the anchor text \(c_{j_{k^{\prime}}}\). This ensures meaningful contributions to the contrastive learning objective from both modalities. The aggregation loss for sample \(i\) is defined as: \[\begin{align} \!\! \mathcal{L}^{\text{aggr}}_i = -\Big[ \sum_{k=1}^{K} \phi^{+}\!\big(v_{i,i_k}^{\text{loc}}, c_{i_k}\big) +\sum_{j\neq i} \phi^{-}\!\big(v_{i,j_{k^{\prime}}}^{\text{loc}}, c_{j_{k^{\prime}}}\big) \Big], \label{eq:agg95loss} \end{align}\tag{2}\] where \(k^\prime\) indexes a randomly sampled local caption from the global caption \(C_j\), and \(\phi^{\pm}(\cdot,\cdot)\) denotes the sigmoid-based scoring function [24], defined as \[\phi^{\pm}(u,v) = \log \sigma\!\left(\pm\,\tfrac{\operatorname{sim}(u,v)}{\tau}\right),\] where \(\operatorname{sim}(u,v)\) is the cosine similarity, and \(\tau\) is a temperature scaling factor.

0.5ex plus 0.2ex minus .1ex -1em Completion Objective. As with the aggregation objective in 2 , we extend the cross-instance contrastive principle to the context-guided transformation process. Specifically, the composition module \(F_{\text{Comp}}\) yields a composed representation \(h_{i,i_k}\) by transforming the localized visual feature \(v^{\text{loc}}_{i,i_k}\) conditioned on its contextual caption \(X_{i_k}\). Each \(h_{i,i_k}\) is aligned with the corresponding global visual embedding \(v_i^g\), forming multi-perspective positive pairs that capture context-conditioned semantic completion. For negatives, we construct cross-instance pairs by replacing the text with one from another sample, yielding the contrastive pair \(\langle h_{i,j_{k^{\prime}}}, v_j^g \rangle\). This construction intentionally disrupts semantic coherence by combining a localized visual cue from \(I_i\) with an irrelevant linguistic context. This ensures that both visual and textual modalities contribute effectively to the completion objective, preventing shortcut learning in a manner similar to 2 . The loss is formulated as: \[\begin{align} \!\! \mathcal{L}^{\text{comp}}_i \!= -\big[ \sum_{k=1}^{K} \phi^{+}\!\left(h_{i,i_k}, v_i^g\right) +\sum_{j\neq i} \phi^{-}\!\left(h_{i,j_{k^{\prime}}}, v_j^g\right) \big]. \label{eq:comp95loss} \end{align}\tag{3}\]

0.5ex plus 0.2ex minus .1ex -1em Discussion. Our contrastive design brings two key benefits. (i) Multi-fragment positives provide simultaneous supervision over multiple localized correspondences within a single image–text pair, strengthening localized grounding without intra-pair competition. (ii) Cross-instance negatives couple text from one sample with visual evidence from another, compelling joint reasoning across modalities and suppressing unimodal shortcuts. Together, these mechanisms align the aggregation and completion modules: the model learns to activate the right visual evidence under language control and to validate global semantic coherence under visual grounding, producing representations that are both locally precise and globally consistent for ZS-CIR.

0.5ex plus 0.2ex minus .1ex -1em Overall Training Objective. The two objectives act in synergy: \(\mathcal{L}^{\text{aggr}}\) learns which visual evidence is anchored by each local caption, while \(\mathcal{L}^{\text{comp}}\) enforces consistency between the transformed representation and the full image semantics. The final training objective is the sum of the aggregation and completion losses, computed over all samples in the mini-batch. \[\begin{align} \mathcal{L} = \sum_i \left( \mathcal{L}^{\text{aggr}}_i + \mathcal{L}^{\text{comp}}_i \right), \label{eq:final95loss} \end{align}\tag{4}\] By jointly optimizing grounding and completion, the model learns visual–language representations that are both locally precise and globally coherent.

3.4 Inference with FoCo↩︎

At inference, we follow the “focus-then-complete” principle. The model first grounds the modification text \(T_m\) within the visual context of the reference image \(I_r\), producing a localized feature \(v_r^{\text{loc}} = F_{\text{Aggr}}(T_m, I_r)\) that highlights the regions relevant to the modification. This feature is then passed to the completion module, which transforms it under the modification instruction to produce the composed representation \(h_r = F_{\text{Comp}}(v_r^{\text{loc}}, T_m)\). Finally, \(h_r\) is compared with the global embeddings \({v_j^g}\) of gallery images using cosine similarity to retrieve the target. This focus-then-complete pipeline enables FoCo to interpret modification instructions by first localizing their visual references and then composing the required semantic change.

Note that the default inference pipeline does not involve an LLM, and uses \(T_m\) directly for both aggregation and completion. This design prioritizes inference efficiency and avoids external dependencies at test time. Optionally, an LLM can be further employed to decompose \(T_m\) in line with the training protocol, thereby more fully exploiting the model’s capacity for complex modifications. Detailed results of this variant are provided in Appendix Sec. C.

2pt width=0.9

4 Experiments↩︎

4.1 Settings↩︎

Implementation Details. Following [4], [6], [25], FoCo uses frozen CLIP backbones, including ViT-L/14 [26] and ViT-G/14 [27]. The CLIP image and text encoders remain fixed during training. Two lightweight trainable modules are introduced: the aggregation module \(F_{\text{Aggr}}\) is implemented as a single-layer cross-attention block followed by a feed-forward projection, while the completion module \(F_{\text{Comp}}\) comprises two Transformer blocks (each with 8-head attention and a feed-forward network) for integrating contextual and visual features. We train FoCo on the Conceptual Captions 3M (CC3M) [28] with short captions from [29]. For each caption, we further employ the Llama-3.1-8B-Instruct model [23] to generate three to five pairs of local captions and contextual captions, which provide fine-grained supervision for the two proxy tasks. More details about the training data and additional results with other backbones (e.g., BLIP [30]) are provided in the Appendix Sec. A. Training proceeds in two stages: we first pretrain \(F_{\text{Aggr}}\) for 3 epochs with a learning rate of \(5e^{-4}\), and then jointly optimize \(F_{\text{Aggr}}\) and \(F_{\text{Comp}}\) for 30 epochs under the unified contrastive objective, using a batch size of 512 and learning rates of \(5e^{-5}\) and \(3e^{-5}\), respectively. With ViT-L/14, FoCo trains 38M parameters in 17h and infers in 0.034s/query, using LLM decomposition only as cached preprocessing.

We evaluate our method on four ZS-CIR benchmarks. (1) FashionIQ [31] contains three fashion categories (Dress, Shirt, Toptee), each with human-annotated relative captions; we report Recall@10 and Recall@50 following the Original-Split protocol. (2) CIRR [32] features real-world image pairs with natural language modifications and visually similar distractors; we report Recall@1, 5, 10, and 50 on the official test set. (3) CIRCO [33] is a large-scale benchmark built on the COCO dataset, providing multiple ground-truths per query with a 120K-image gallery; we report mAP@5, 10, 25, and 50. (4) GeneCIS [34] is a benchmark designed to evaluate conditional image similarity in a zero-shot manner, featuring four tasks that require models to focus on or change specific attributes or objects in an image based on textual conditions. We report Recall@1, 2, and 3 for all four task types.

We compare FoCo against a suite of representative ZS-CIR baselines. Pic2Word [3] maps reference images into pseudo-word tokens in the CLIP token space. SEARLE-XL [33] integrates such tokens into GPT-generated captions. LinCIR [6] masks subject entities in captions for efficiency-aware training. Context-I2W [4] focuses on extracting visual cues aligned with the text, while PrediCIR [25] enhances pseudo-tokens by inferring missing visual semantics from the caption. MoA [7] introduces multi-object pseudo-token learning, while HIT [8] learns multiple pseudo-words organized across hierarchical semantic levels. SlerpTAT [9] leverages LoRA-based tuning of the image encoder for additive composition. For fair comparison, we report results on ViT-L/14 for all baselines and further evaluate generalization of FoCo on ViT-G/14. We further compare with recent training-free and LLM-based ZS-CIR methods, including CoTMR, IP-CIR, and PDV, in the Appendix.

1.5pt width=0.85

4.2 Main Results↩︎

FashionIQ. On this benchmark, which focuses on domain-specific attribute edits such as color or material changes, FoCo achieves superior results across most metrics (¿tbl:tab:fiq-only?). This indicates its ability to integrate fine-grained visual attributes into coherent queries. Unlike CIRR, which emphasizes object-level changes, FashionIQ stresses subtle property modifications, and FoCo demonstrates strong generalization across different types of changes.

2pt width=0.95

CIRR. FoCo surpasses all existing ZS-CIR baselines on CIRR, improving R@1 by 3.6 points over SlerpTAT on ViT-L/14 (¿tbl:tab:cirr-circo?). CIRR contains natural images with complex scenes and multiple objects, making localized edits especially challenging. Models that rely on global feature fusion or naive pseudo-token composition often fail to distinguish visually similar distractors. While methods like PrediCIR and HIT [8] attempt to mitigate noise in pseudo-token mapping, our results show that explicit localization leads to more accurate disambiguation.

CIRCO. With a 120K-image gallery, CIRCO introduces significant retrieval noise. FoCo consistently leads in mAP metrics, reflecting the quality of its composed features (¿tbl:tab:cirr-circo?). On ViT-L/14, it achieves 17.5 mAP@5, outperforming SlerpTAT and PrediCIR by 0.5 and 1.8 points, respectively. These results validate the effectiveness of the proposed context-guided completion module in generating robust and precise query representations.

GeneCIS. This benchmark evaluates conditional similarity across four task types that combine focus/change intents with attribute/object granularities. FoCo achieves the highest scores (¿tbl:tab:genecis?), outperforming PrediCIR most notably on Focus (+2.4 Recall@1) and Object (+1.8 Recall@1) tasks, where precise grounding of the condition-specified visual region is most critical. In contrast to pseudo-token concatenation, FoCo explicitly models both what to modify and how to compose, resulting in a more interpretable composition process. Collectively, these results demonstrate that a learnable, two-stage formulation yields superior generalization across diverse modification types compared to static pseudo-token baselines, without requiring triplet supervision.

4.3 Ablation Study and Analysis↩︎

FoCo learns an adaptive composition mechanism within the CLIP semantic space. Specifically, \(F_{\text{Aggr}}\) grounds modification-relevant visual evidence, while \(F_{\text{Comp}}\) performs a dynamic, text-driven semantic transformation. Removing either module leads to an ill-posed proxy task. To evaluate them independently, we follow [3] and replace \(F_{\text{Comp}}\) with linear feature fusion (1b), and use the class-activation-based patch masking strategy from [35] to provide coarse visual evidence to \(F_{\text{Comp}}\) (1c). As shown in ¿tbl:tab:ablation?, the naive global baseline (1a) performs poorly. Using only \(F_{\text{Aggr}}\) (1b) significantly improves accuracy via meaningful grounding, while using only \(F_{\text{Comp}}\) (1c) yields smaller gains due to coarse localized activations. This gap is particularly pronounced in CIRR, which requires precise object-level localization, compared to the attribute-centric FashionIQ. Overall, two proxy tasks decouple the composition process: aggregation determines what to modify, while completion defines how to modify.

2pt

We first ablate our training strategies and architectural choices in ¿tbl:tab:ablation?. For the contrastive objective, replacing our multi-positive loss with InfoNCE (2a) drops performance. Using naïve intra-image negatives (2b, 2c) weakens cross-instance discrimination, and removing the aggregation loss during joint training (2d) confirms \(F_{\text{Aggr}}\) benefits from continued explicit supervision. Regarding architectural designs, removing the text-guided query (4a) weakens localization, as it lacks global visual context and cannot disambiguate expressions whose referents differ across images. Replacing the cross-attention layer with a dot-product attention (4b) also reduces performance, highlighting the need for deeper cross-modal interaction. For the completion module, removing the Transformer blocks (5a) disrupts cross-modal interaction, leading to a performance drop. Removing the \(\bar{v}_{i,i_k}\) from the key/value sequence (5b) also degrades results, confirming that explicitly supplying unattended visual context helps the model preserve semantics that should remain unchanged. Replacing the visual-anchor initialization with a random learnable token (5c) also degrades performance, confirming that the localized visual subject \(v^{\text{loc}}_{i,i_k}\) provides a necessary semantic anchor for extracting modification-relevant information.

The quality and granularity of our text decomposition are crucial. Replacing LLM-decomposed descriptions with rule-based syntactic extraction (3a) causes significant degradation, as rigid grammatical splitting creates isolated fragments and ungrammatical contexts, disrupting semantic coherence. Furthermore, relying on a single local caption per image (3b, i.e., \(K\!=\!1\)) noticeably reduces accuracy. This highlights that multi-view training pairs are essential to capture diverse compositional patterns. To investigate this systematically, we visualize the sensitivity to decomposition granularity (\(K\)) on the large-scale CIRCO dataset in 3. Consistent with 3b, insufficient decomposition leads to the lowest performance. As \(K\) increases, performance steadily improves and peaks at \(K\!=\!4\). Over-decomposition (\(K\!\ge\!6\)) causes performance to decline, as it compels the LLM to generate hallucinated visual details or excessive semantic noise. These results confirm that a dynamic \(K\!\in\![3,5]\) provides the optimal balance between fine-grained supervision and semantic fidelity.

A natural concern is whether the LLM decomposition introduces extra semantic information that drives FoCo’s gains. To test this, we concatenate the decomposed local captions and contextual captions back into complete sentences and append them as additional training pairs to two baselines (denoted “+AD”). As shown in [fig:data95ablation], the additional data brings only marginal or even negative effects: Context-I2W\(^\dagger\) (+AD) improves by at most +0.3, while LinCIR\(^\dagger\) (+AD) actually drops on CIRCO mAP@5 (\(-0.3\)), likely because its text-only training regime is sensitive to the distributional shift introduced by the restructured captions. In contrast, FoCo surpasses all augmented baselines by a large margin, confirming that our gains originate from the proxy-task design rather than extra training data (full results in Appendix).

4.4 Visualization and Analysis↩︎

To understand how FoCo performs modification-guided image retrieval, we visualize the activation maps from the text-anchored visual aggregation module \(F_{\text{Aggr}}\) alongside the retrieval results in 4. Each query includes a reference image and modification text, and the red-highlighted patches indicate the regions most relevant to the modification. On the right, multiple modification texts are applied to the same reference image. This reveals how FoCo dynamically adjusts its focus as the intended change varies. Attribute-level edits (e.g., changing color or texture) result in fine-grained and localized activation patterns, while object-level or relation-level modifications lead to broader spatial activation that covers the relevant entity or configuration. These consistent and meaningful shifts indicate that FoCo does not rely on a fixed composition rule, but actively restructures its visual grounding conditioned on the modification text. \(F_{\text{Comp}}\) then integrates these focused representations with global context to form a coherent final representation. In summary, FoCo localizes the modification-relevant regions and integrates them into a target representation in a controllable manner. More retrieval examples are shown in Figs. 1 and 2 in the Appendix.

A key challenge in ZS-CIR is the inherent domain gap between descriptive training captions and instruction-driven modification texts. We argue that FoCo mitigates this gap through its adaptive composition mechanism, which modulates visual features within the CLIP semantic space according to textual conditions. Leveraging the pretrained CLIP text encoder, FoCo also inherits linguistic robustness across varied input forms. Specifically, \(F_{\text{Aggr}}\) first grounds modification-relevant visual evidence, whether the intent is to add, modify, or remove. \(F_{\text{Comp}}\) then integrates these features with contextual cues to complete the transformation. This is supported by our decomposition strategy, which provides non-overlapping localized and contextual features, while the multi-view training pairs expose the model to varying visual-semantic dependencies ranging from small object preservation to large scene shifts. Moreover, the cross-instance contrastive objective prevents textual shortcuts, forcing the model to discern the necessary visual contribution based on these dependencies. Consequently, as shown in 4, this design enables FoCo to dynamically modulate evidence in response to diverse instructions.

5 Conclusion↩︎

We introduce Focus-then-Complete (FoCo), a framework for zero-shot composed image retrieval that treats visual–textual composition as a learnable procedure. FoCo first isolates visual cues relevant to the textual modification and then completes the target semantics through contextual reasoning. These two stages correspond to two proxy tasks, text-anchored visual aggregation and context-conditioned semantic completion, trained jointly with a cross-instance contrastive objective. This design yields accurate, stable, and interpretable compositional representations without relying on triplet supervision. We believe FoCo offers a clear and effective foundation for learning controllable multimodal composition [36], and we plan to extend it to more complex, open-ended vision–language settings [37], [38].

Acknowledgments↩︎

This work was supported by the National Natural Science Foundation of China under Grants 62576329 and 62336001.

References↩︎