1 Introduction↩︎

Given a picture of a scene with two objects of different colors and shapes, humans can easily determine which color is associated with which shape, an ability often referred to as concept binding [1], [2]. While straightforward for humans, this has proved challenging for modern vision-language embedding systems such as CLIP [3]. These models often behave like bag-of-concepts systems: they reliably recognize individual concepts (e.g., red, square), yet struggle to correctly represent their combinations in multi-object scenes (e.g., distinguishing red square vs. blue square) [4]–[6]. At the same time, within individual modalities of vision and language, object-related information can often be recovered from embeddings using probes [7]. In other words, despite their failures in multi-object settings, these models appear to internally encode object information to some extent. This tension motivates a closer examination of the geometry of CLIP’s binding: if object information is present, how is it represented, and why does it fail to align across modalities?

We begin by analyzing how multi-object scenes are represented in CLIP. We find that scene embeddings exhibit a clear additive structure: they decompose into sums of object representations. This structure explains why uni-modal probes can recover object information—object-level components are explicitly present in the embedding space and can be manipulated directly. However, this geometric structure alone does not explain binding. We show that CLIP’s binding function, which maps concepts to object representations, is high-complexity in the sense that it cannot be captured by a simple MLP mapping. Instead, the mapping from concepts to objects differs across concept combinations, effectively requiring memorization of object identities. This prevents generalization to unseen objects and makes it difficult for the image and text encoders to learn a shared binding mechanism, despite exhibiting uni-modal binding.

We then ask whether this limitation is fundamental. We show that it is not. In controlled transformer-based dual-encoder models trained from scratch on synthetic multi-object data, binding generalization emerges with sufficient data coverage. These models learn low-complexity binding functions that reuse structure across concept combinations and generalize to unseen objects. Finally, we analyze the structure of these generalizing binding functions. We find that models that generalize implement multiplicative interactions between concepts, rather than purely additive composition. This multiplicative structure enables generalization and explains how different modalities can produce aligned object representations for unseen concept combinations.

Together, these results show that binding failures in CLIP arise not from the absence of object structure, but from the complexity of the binding function it learns. Embedding models are capable of generalizable binding, but doing so requires learning a low-complexity, systematic concept-to-object mapping that can be shared across modalities.

2 Related work↩︎

Binding. The binding problem asks how models associate features that belong to the same object [8], [9]. Surveys and empirical studies examine binding limits and emergent symbolic mechanisms [10]–[12]. In autoregressive language models, mechanistic studies have identified position-independent binding IDs that link related tokens [9], [13]. Binding at patch-level of DINO and CLIP has also been studied [14], showing that vision models naturally develop an ability to bind at a patch level. However, such mechanisms operate at the token level and it remains unclear how embedding models that produce a single vector per input could implement analogous structure. In contrast, we study how binding is realized in the geometry of embedding models: we ask how object representations are constructed from concepts, and argue that the complexity of this mapping explains cross-modal binding failures.

Binding in embedding models. CLIP has been observed to exhibit uni-modal binding [7] but fail cross-modally [5], [6]. These failures have been attributed to encoder-level weaknesses such as difficulty with fine-grained details, negations, and spatial reasoning [15]–[18] and over-reliance on concept-level information [19]. Several works propose fine-tuning approaches to improve CLIP’s compositional understanding, including word order sensitivity and attribute-object association [4], [20], [21] and argue of the importance of data [22]. Some works suggest that concept recognition and object recognition (binding) are fundamentally at odds, suggesting a trade-off [23]. In contrast, we show that both concept and object recognition can coexist in the same embedding space, explain how uni-modal binding can succeed via linear probes, and argue that cross-modal binding failures stem from the complexity of how concepts are combined into objects, not from an inherent incompatibility.

Compositional generalization. Research on compositional generalization investigates how models can generalize to novel combinations of known parts. Many approaches attempt to inject this ability into neural networks [24]–[26], while others study conditions under which it can be achieved [27]–[34]. However, this literature largely focuses on single-object settings with factorized concept spaces. For CLIP specifically, generalization of concept recognition to novel combinations has been studied [35], but generalizing binding, object-level recognition, to novel combinations remains difficult [5]. In contrast, we study multi-object scenes and find that generalization of binding is possible and emerges with data scale: at small scales, object recognition fails to generalize (consistent with prior findings), but concept recognition of novel objects remains robust; at larger scales, both concept and object recognition generalize.

Geometry of learned representations. Many works study the shape of learned features. General notions of the Linear Representation Hypothesis mostly report linearity in language models [36], [37]. [38] find evidence of disentanglement in CLIP models, [39] show additive decomposition in CLIP text encoder, and [35], [40] show it in vision encoder. However, these analyses typically characterize geometry in terms of single objects. In contrast, we extend this to multi-object scenes and show that scene embeddings admit a two-level additive decomposition into objects and concepts.

3 Formalisation of binding↩︎

Before analyzing how embedding models represent multi-object scenes, we formalize the problem. We define what it means to recognize concepts, recognize objects, and bind them together. This formalization guides 4 and 5.

We illustrate the setup in Fig. 1. We assume images and captions depict scenes. A scene contains one or more objects. Each object is specified by values of a fixed set of concepts (e.g. color and shape). We follow prior work [33] in defining objects as combinations of concept values (Fig. 1 (a)) and extend this formalisation to scenes containing multiple objects (Fig. 1 (b)).

Models. We study embedding models that encode inputs and queries independently. An input corresponds to a scene, such as an image, while a query specifies the information to retrieve, such as a concept value or an object (e.g. CLIP).

We denote the input encoder by \(f : \mathcal{X}\rightarrow \mathbb{R}^d\) and the query encoder by \(q : \mathcal{Y}\rightarrow \mathbb{R}^d\). Given an input \(\boldsymbol{x}_{\boldsymbol{s}}\) and a query \(\boldsymbol{y}\in \mathcal{Y}\), the matching score is given by cosine similarity, \(s(\boldsymbol{x}_{\boldsymbol{s}}, \boldsymbol{y}) = \frac{ f(\boldsymbol{x}_{\boldsymbol{s}})^\intercal q(\boldsymbol{y})}{\| f(\boldsymbol{x}_{\boldsymbol{s}}) \| \, \| q(\boldsymbol{y}) \|}\). We refer to \((f,q)\) as a model.

For a scene \(\boldsymbol{s}= (\mathbf{o}^{(1)}, \dots, \mathbf{o}^{(m)}) \in \mathcal{S}\), we define the set of values taken by concept \(i\) across all objects in the scene as \(V_i(\boldsymbol{s}) \;:=\; \{ c^{(1)}_i, \dots, c^{(m)}_i \} \subseteq \mathcal{C}_i.\) We also define the set of objects present in the scene as \(O(\boldsymbol{s}) \;:=\; \{ \mathbf{o}^{(1)}, \dots, \mathbf{o}^{(m)} \} \subseteq \mathcal{C}.\)

Binding ability. A model exhibits binding ability when it has (Fig. 1 (c)): (1) concept-level recognition: recognize which concept values are present in a scene, (2) object-level recognition: recognize which objects are present in a scene.

While concept recognition captures the presence of individual concept values, it does not test whether the model can identify which values belong together. This distinction is formalized by object recognition.

Concept recognition and object recognition capture complementary aspects of how a model represents scenes. Binding concerns the ability to support both levels consistently.

The definitions above characterize binding as a behavioral property. We distinguish cross-modal binding, where the model \((f,q)\) directly satisfies concept and object recognition via cosine similarity, from uni-modal binding, where a probe must be trained on top of embeddings within a single modality. Prior work has shown that CLIP exhibits uni-modal binding but fails cross-modally [7]. To understand why, we examine how embeddings are constructed from scenes.

Intuitively, cross-modal binding requires \(B_{\text{img}}\) and \(B_{\text{txt}}\) to produce compatible embeddings for the same scene, including novel ones. By Occam’s razor, among functions consistent with the training scenes, those with simpler, more reusable structure are more likely to generalize identically off-domain [41]: if both binding functions admit a low-complexity, compositional rule, fewer mappings are consistent with the observations, so the two encoders are more likely to converge on the same rule and remain aligned on unseen objects. If they are high-complexity, each modality may instead fit the training domain in a combination-specific way and disagree off-domain, preventing cross-modal alignment.

In the following sections, we first study the geometry of CLIP’s binding functions: how scene embeddings decompose into objects and concepts (§4). We then investigate whether these functions can be approximated using concept indices and study controlled models in which binding generalizes (§5).

4 Geometry of embedding models’s binding↩︎

CLIP embeddings have been shown to encode binding information within each modality, while this information is misaligned across modalities [7]. However, it remains unclear how this binding information is represented in the embedding space and how it becomes accessible to probes. To address this, we examine the geometry of CLIP embeddings and find that scene embeddings decompose into object components, which further decompose into concept components (4.1). We show that these object components are directly responsible for object recognition (4.2), explaining how binding information can be recovered uni-modally. While prior work mostly focused on single-object representations [39], [40], uselis2025does?, we study embeddings in the multi-object setting. Metric definitions are provided in [sec:app:clip95geometry95experiments]. Descriptions of the visual datasets and text compositions are given in [sec:app:dataset95details] and [sec:app:text95encoder95decomposition].

4.1 Embeddings decompose into objects and concepts↩︎

To gain intuition about the geometry of CLIP embeddings, we visualize them in 2. It shows an MDS [42] projection of embeddings for two-object scenes. Two-object scene embeddings lie roughly between the embeddings of the corresponding single-object scenes, suggesting that scene embeddings are approximately sums of their constituent object embeddings. Motivated by this geometric structure, we study how multi-object scenes are represented in CLIP. We consider scenes with two objects. Throughout the section, we will let \(\boldsymbol{s}= (\boldsymbol{o}_1, \boldsymbol{o}_2)\) denote a scene and \(\boldsymbol{x}_{\boldsymbol{s}}\) denote the corresponding input.

We hypothesize that the embedding space exhibits a hierarchical additive structure, where scenes decompose into objects and objects further decompose into concepts:

1. (Level-I) A scene embedding approximately decomposes into a sum of object embeddings.

2. (Level-II) An object embedding approximately decomposes into a sum of concept embeddings.

Under this hypothesis, the embedding of a two-object scene can be written as follows. Writing each object as a tuple of concept values, \(\boldsymbol{o}_1=(c_{1,1},\dots,c_{1,C})\) and \(\boldsymbol{o}_2=(c_{2,1},\dots,c_{2,C})\): \[\begin{align} \label{eq:decomp952obj} \smallf(\boldsymbol{x}_{\boldsymbol{s}}) &\approx \boldsymbol{u}_{\boldsymbol{o}_1} + \boldsymbol{u}_{\boldsymbol{o}_2} \approx \sum_{i=1}^C \boldsymbol{u}_{c_{1,i}} + \sum_{i=1}^C \boldsymbol{u}_{c_{2,i}}. \end{align}\tag{1}\] We estimate object and concept embeddings using the methods summarized in 1. We empirically evaluate both levels of this decomposition below.

Scenes decompose into object components (Level-I). We test the first claim of the Level-I decomposition: that scene embeddings decompose into object-level components. We estimate object embeddings \(\boldsymbol{u}_{\boldsymbol{o}}\) according to Tab. 1. We then reconstruct scene embeddings by summing the corresponding object components. We evaluate the reconstruction using the retrieval score, probing accuracy, and \(R^2\).

4pt

2 shows that CLIP scene embeddings are well approximated by sums of object components (text: \(R^2=0.90/0.92\); image PUG:SPARE CLIP: \(R^2=0.75/0.84\)), supporting the proposed object-level decomposition. The same decomposition also extends to 3-object CLEVR scenes (including with occlusions; [sec:app:three95object]) and to natural images generated by Gemini Nano Banana 2 ([sec:app:natural95images]).

Editing scene embeddings via object replacement. The Level-I decomposition in 1 has a concrete implication. If a scene embedding is well approximated by a sum of object components, then editing a multi-object embedding by removing one object component and inserting another should produce a meaningful counterfactual representation. We examine this through linear operations in embedding space as depicted in 3.

Setup. Let \(\boldsymbol{s}=(\boldsymbol{o}_1,\boldsymbol{o}_2)\) be a two-object scene, and let \(\boldsymbol{o}_1'\) be a counterfactual object obtained by changing one concept value of \(\boldsymbol{o}_1\) (e.g.color). We construct an edited embedding by substituting the object component of \(\boldsymbol{o}_1\) with that of \(\boldsymbol{o}_1'\): \[\small \tilde{\boldsymbol{z}} \;:=\; f(\boldsymbol{x}_{\boldsymbol{s}}) \;-\; \boldsymbol{u}_{\boldsymbol{o}_1} \;+\; \boldsymbol{u}_{\boldsymbol{o}_1'} . \] We evaluate edited embeddings \(\tilde{\boldsymbol{z}}\) with retrieval and probing metrics; full details are in [sec:app:edit]. We report results for text encoders ([sec:app:text95encoder95decomposition]) and for image encoders on CLEVR, CLEVR-2D, and PUG:SPARE ([sec:app:dataset95details]).

Results.

4pt

As shown in 3, replacing an object in a multi-object scene embedding with another object produces embeddings that behave like the intended counterfactual scenes under both probing and retrieval (e.g., CLIP on CLEVR achieves 1.00 retrieval with scene-averaged object embeddings). Notably, object embeddings obtained from single-object scenes can be directly used to edit multi-object scene embeddings (e.g., retrieval accuracy 0.97 for CLEVR and CLEVR-2D with CLIP). This supports the Level-I decomposition hypothesis that scene embeddings are composed of additive object-level components.

Objects decompose into concepts (Level-II). Having established scene-level decomposition, we now analyse whether objects themselves decompose into concepts as in 1 . A purely additive concept-level decomposition corresponds to a bag-of-concepts model, which cannot encode within-object associations required for binding. We therefore quantify how much variance is explained by object-only and concept-only components using \(R^2\).

4pt

Results. From 4, we observe that a bag-of-concepts representation explains a large fraction of embedding variance, up to 84% for the text encoder and 71% for the image encoder. Representing scenes as bags of objects explains roughly 10% more variance. Thus, most variance is captured by concepts, with object representations contributing additional structure beyond a bag-of-concepts.

4.2 Uni-modal binding is explained by object decomposition↩︎

In the previous section, we showed that CLIP embeddings decompose into object and concept components. In this section, we demonstrate that these components are directly responsible for the corresponding object and concept recognition in scene embeddings.

Setup. We perform targeted component removal on the scene embedding and then fit linear probes on the residuals. Using the decomposition in 1 , we define the concept part of a scene as \(\sum_i \boldsymbol{u}_{c_{1,i}} + \sum_i \boldsymbol{u}_{c_{2,i}}\), and the object part as \(\boldsymbol{u}_{\boldsymbol{o}_1} + \boldsymbol{u}_{\boldsymbol{o}_2}\). We then form intervened embeddings by subtracting these components, e.g.\(f(\boldsymbol{x}_{\boldsymbol{s}}) - \sum_i \boldsymbol{u}_{c_{1,i}} - \sum_i \boldsymbol{u}_{c_{2,i}}\) (remove concepts) or \(f(\boldsymbol{x}_{\boldsymbol{s}}) - \boldsymbol{u}_{\boldsymbol{o}_1} - \boldsymbol{u}_{\boldsymbol{o}_2}\) (remove objects), and train probes on the resulting residual embeddings. As a control, we repeat the same removals using permuted-ID concept/object components (details in [sec:app:interventions]).

Results. 5 demonstrates that concept recognition and object recognition in scenes depend on their corresponding components in the scene embeddings. For CLIP-B/32, subtracting concept components nearly eliminates concept decoding (text: 1.00 → 0.06, image: 0.94 → 0.05) while largely preserving object decoding (text: 1.00 → 0.99, image: 0.96 → 0.85). In contrast, subtracting object components collapses both object and concept decoding (text: 0.04/0.05, image: 0.01/0.02). Permuted-ID controls do not produce comparable degradation. Removing object components that are not present in the scene has little effect on decoding accuracy, confirming that the performance drop is specific to the correct object component.

These results explain why prior work observes binding in CLIP embeddings. Object components in a scene embedding encode concept information jointly, allowing probes to recover the correct combinations of concepts.

4pt

5 Binding generalization↩︎

Having established that scene embeddings decompose into objects, explaining uni-modal binding, we next examine why cross-modal binding fails through the complexity and generalization of the underlying binding functions.

5.1 CLIP’s binding function does not generalize↩︎

Inspired by prequential/MDL perspectives that interpret simplicity as fast generalization from limited data [41], [43], we measure how generalization of a binding function (Def. 6) depends on training-set size and MLP capacity (width). Concretely, we study the sample-capacity requirements of a 1-hidden-layer ReLU MLP whose width is varied, trained to approximate a binding map from discrete concept values to scene embeddings. We use MLPs because SGD-trained MLPs favor simple, compressible solutions [44], so if a simple compositional binding rule exists, we expect this approximator family to find it.

Setup. We train an approximator \(g: \mathcal{C}^2 \rightarrow \mathbb{R}^d\) to predict scene embeddings from concept indices (4(a)). For a scene \(\boldsymbol{s}= (\boldsymbol{o}_1, \boldsymbol{o}_2)\) where each object \(\boldsymbol{o}_i = (c_{i1}, c_{i2})\) is specified by concept values, we minimize \[\label{eq:binding95complexity95setup} \min_g \, \sum_{\boldsymbol{s}\in {\mathcal{S}_{\text{train}}}} \| f(\boldsymbol{x}_{\boldsymbol{s}}) - g(\boldsymbol{o}_1, \boldsymbol{o}_2) \|^2 , \tag{2}\] where \(f(\boldsymbol{x}_{\boldsymbol{s}})\) is the CLIP embedding of scene \(\boldsymbol{s}\) and \(\mathcal{S}_{\text{train}}\) is the set of training scenes. We vary (1) MLP hidden layer width \(\{64, 256, 1024, 4096\}\), and (2) the fraction of objects seen during training \(\{10\%, \dots, 90\%\}\). At test time, we evaluate on scenes containing only held-out objects. We measure whether the predicted embeddings support both concept recognition (Def. 3) and object recognition (Def. 4) by using the linear probes trained on the scene embeddings. See details in [sec:app:pretrained95binding95function95details].

Results. 4(b) reports the best accuracy across the MLP configurations. Concept recognition yields \(\geq\) 80% accuracy among all conditons where the fraction of data exceeds 0.3, whereas object recognition is substantially lower, including at large widths and high training coverage. This gap indicates that correctly predicting object-level structure is markedly harder than concept-level structure, suggesting a higher effective complexity of object-level binding under this probe family. Among encoders, the text encoder performs slightly better than the image encoders; CLIP image embeddings outperform DINO but remain far below concept recognition, never exceeding \(20\)% accuracy. The result does not depend on the approximator family: swapping MLPs for XGBoost or Random Forest yields a similar qualitative pattern on CLIP and DINOv2 ([sec:app:approximator95robustness]). These results indicate that CLIP’s binding function is high-complexity: it resists approximation even by high-capacity MLPs and fails to generalize to novel object combinations.

5.2 It is possible to have a generalizable binding model↩︎

The previous section showed that CLIP’s binding function is hard to approximate and does not generalize to unseen concept combinations. This raises a natural question: is this failure inherent, or can embedding models learn binding that does generalize when the data and training signal are controlled? To answer this, we train CLIP-style dual-encoder transformers on synthetic multi-object scenes where we can precisely vary object-space coverage (5).

Setup. We illustrate the process in 5. We use synthetic scenes with up to 2 objects; each object is defined by \(C\) concepts with \(V\) values (\(|O|=V^C\)), and we vary \(C\), \(V\), and the training fraction of seen objects from 0.1 to 0.9. Train scenes draw only from the training split; test scenes are composed entirely of held-out objects. We use two transformer [45] encoders (query and scene); we refer to them as “text” and “vision” by analogy only since both process tokenized sequences, differing only in whether they encode object/concept queries or full scenes. Both encoders output embeddings in \(\mathbb{R}^{512}\) and have \(\approx\)​20M parameters each. We train end-to-end with AdamW [46] on a contrastive retrieval objective [3]; see [sec:app:controlled95models] for full details.

Metrics. We evaluate both concept and object recognition. For concepts, we test against all \(C \cdot V\) values. For objects, testing against all \(V^C\) is infeasible; instead, we use \(\mathrm{Swap}(\boldsymbol{o}_1, \boldsymbol{o}_2)\), objects formed by exchanging concept values between \(\boldsymbol{o}_1\) and \(\boldsymbol{o}_2\), which directly tests binding by ranking correct object identities against negatives.

Results. We observe two key findings (6). First, compositional generalization emerges: object recognition on held-out objects starts near the bag-of-concepts baseline but rises sharply with coverage, reaching near-perfect accuracy despite all test objects being unseen. Second, concepts are easier than objects: concept recognition generalizes early, while object recognition requires substantially more coverage. When the number of possible objects is small, generalization requires observing more objects, consistent with prior findings in low-combinatorial regimes [5]. As the number of possible objects increases, the fraction of objects required for reliable object recognition decreases substantially. For example, when \(|O|=400\), high accuracy is achieved only after observing roughly 50% of objects, whereas for \(|O|\geq 2{,}500\), generalization emerges at around 30% coverage. In the largest setting (\(|O| = 125{,}000\)), increasing coverage from 30% to 40% yields a sharp transition from below-chance to near-perfect object recognition. Notably, this occurs despite the corresponding scene space being extremely large (e.g., \((50^3)^2 \approx 15\) billion two-object scenes for \(C=3, V=50\)).

5.3 Binding functions of generalizing models tend to have low complexity functions↩︎

In 5.1 we operationalized binding complexity via generalization of an MLP approximator: even wide MLPs struggled to predict CLIP embeddings on held-out objects, revealing a high-complexity, non-generalizing binding map. Here we apply the same diagnostic to the controlled models from 5.2 and test the converse prediction: models that do generalize binding should admit a low-complexity (easy-to-approximate) concept-to-object mapping.

Setup. We measure binding complexity exactly as in 5.1: we train MLPs of varying width to predict scene embeddings from discrete concept indices and evaluate on scenes composed of held-out objects, reporting concept vs.object recognition on the predicted embeddings. If binding is low-complexity, small MLPs should generalize; if it is high-complexity (as in CLIP), generalization should require large capacity and still fail. We further test this in the visual domain by training scene encoders from scratch on pixel inputs, including noisy and occluded settings (see [sec:app:from95scratch95vision]).

Results. 7 shows that for controlled models that generalize, the binding map is easy to approximate in the sense that even small nonlinear MLPs achieve strong accuracy. At the same time, as the object space grows (larger \(C\) and \(V\)), the performance gap between the smallest nonlinear MLP and the largest-capacity MLP increases, showing that the binding map does become moderately more complex with scale. These findings are analogous when the scene encoder is trained from scratch directly on pixel inputs ([sec:app:from95scratch95vision]). This contrasts with 5.1, where even high-capacity MLPs fail to generalize object recognition on CLIP embeddings.

5.4 Multiplicative structure best explains binding↩︎

We next ask what functional form the learned binding takes. We compare simple, structured probes that predict scene embeddings from concept values via additive and multiplicative composition. The probe is the bag-of-concepts baseline. The Per-obj. products and Global product probes introduce the simplest possible departure from additivity: product interactions between concept embeddings, which can produce a unique representation for each object combination. In this subsection, we consider scenes of two ordered objects, \(\boldsymbol{s}=(\boldsymbol{o}_1,\boldsymbol{o}_2)\), where each object is described by two concept types (e.g., color and shape), so \(\boldsymbol{o}_i=(c_{i1},c_{i2})\) with \(c_{ik}\in\mathcal{C}_k\) the value of concept type \(k\) in the \(i\)-th object. The parameters we train are concept-value embeddings: \(\boldsymbol{u}_{k,c}\) is indexed by concept type \(k\) and value \(c\in\mathcal{C}_k\), shared across the two object positions. The multiplicative embedding \(\boldsymbol{v}_{i,k,c}\) carries an additional index \(i\in\{1,2\}\) for the object’s position, so its products in Per-obj. products (within an object) and Global product (across both objects) yield a distinct vector for every combination of concept values — the binding signal that pure addition cannot express. Training and testing data are constructed following [sec:app:pretrained95binding95function95details].

Results. 8 shows that all probes do well at concept recognition, but the baseline fails at object recognition. Adding multiplicative interactions recovers binding generalization, with the Global product form performing best, especially in larger object spaces.

Multiplicative structure underlies generalization. To test whether this relationship holds more broadly than on the four considered models, we examine the correlation between a model’s generalization ability and how well its embeddings are approximated by the Global product probe. For each of \({\sim}500\) models trained with varying hyperparameters across all object space sizes, we fit the Global product probe using 50% of the object space and measure both (1) the model’s object recognition accuracy on held-out objects, and (2) the probe’s accuracy at predicting scene embeddings. 9 shows a strong positive correlation: models that generalize well are precisely those whose embeddings are well-approximated by the Global product form. This suggests that generalizing models are internally implementing something akin to multiplicative binding—a simple functional form that composes concept representations systematically. The correlation weakens slightly for the largest object space (\(|O|=125\)k), but the pattern remains clear: the multiplicative form is sufficient to capture how concepts combine, which is why it generalizes to unseen combinations. Applying the same Global product probe to pretrained CLIP and DINOv2 embeddings instead recovers concept recognition but leaves object recognition near zero ([sec:app:mult95clip]), consistent with CLIP’s failure to bind concepts cross-modally.

6 Discussion↩︎

In this work we formalized concept binding by distinguishing concept recognition from object recognition, and then analyzed the geometry of multi-object embeddings through this lens. Scene embeddings show a clear additive object-level structure (supporting uni-modal decodability and direct embedding edits), but the concept-to-object mapping appears to be high-complexity and combination-specific, which blocks generalization and makes cross-modal alignment brittle. In controlled transformer dual-encoders trained from scratch, binding generalization emerges with enough data, and the generalizing mapping is best explained by low-complexity structure with multiplicative interactions between concepts.

Limitations. Our analysis uses synthetic datasets. To our knowledge, no real-world dataset currently provides the controlled, combinatorially complete structure required to study binding in our setting. In addition, our notion of complexity is necessarily relative to a chosen approximator family, since Kolmogorov complexity is not computable in practice.

Acknowledgments↩︎

The authors thank Declan Campbell for helpful discussions and comments on initial drafts of this work, Alexander Rubinstein for insightful discussions, and the anonymous reviewers for their valuable feedback. This work was supported by the Tübingen AI Center. Arnas Uselis and Darina Koishigarina were supported by the International Max Planck Research School for Intelligent Systems (IMPRS-IS). Seong Joon Oh was supported by the Institute for Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS-2019-II190075, Artificial Intelligence Graduate School Program gram(KAIST)).

Impact statement↩︎

This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

*Appendix

6pt

We train controlled models from scratch on synthetic multi-object scenes represented as token sequences. We use two transformer encoders: one for scenes and one for queries (either a single concept value or an entire object). We optimize a retrieval objective that makes matching scene-query pairs more similar than mismatched pairs (akin to CLIP’s contrastive loss [3]). Because both scenes and queries are generated procedurally, we can vary the object-space size \(V^C\), the maximum number of objects per scene \(O_{\max}\), and whether evaluation uses held-out concept combinations. Table 7 summarizes the key parameters used throughout.

6pt

We parameterize the synthetic environment by the number of concepts \(C\), the number of values per concept \(V\), and the maximum number of objects per scene \(O_{\max}\) (max_num_objects). Each object is represented as a \(C\)-tuple of discrete concept values \(\boldsymbol{v}=(v_1,\dots,v_C)\) with \(v_c\in\{0,\dots,V-1\}\), and a scene is an ordered list of \(n\in\{1,\dots,O_{\max}\}\) such objects.

Scene sampling. We draw the scene length \(n \sim \mathrm{Unif}\{1,\dots,O_{\max}\}\) and then sample \(n\) objects independently. By default, each component \(v_c\) is drawn uniformly from \(\{0,\dots,V-1\}\). To implement train/test splits, we instead sample objects from a restricted list allowed_objects.

Tokenization. Table 8 summarizes the sequence format. We map each concept/value pair \((c,v)\) to a token id \(t(c,v)=(c-1)V+v\), yielding \(CV\) concept-value tokens, and add four extra tokens: SOO (start-of-object), EOO (end-of-object), EOS (end-of-scene), and PAD. An object \(\boldsymbol{v}=(v_1,\dots,v_C)\) is encoded as \[[\texttt{SOO},\;t(1,v_1),\dots,t(C,v_C),\;\texttt{EOO}],\] and a scene is the concatenation of its objects followed by EOS, padded to the fixed length \[L_{\max} = (C+2)O_{\max} + 1.\] Scenes are generated on the fly by an “infinite” iterator, so training is naturally described in gradient steps rather than epochs.

6pt

Example. With \(C=2\) and \(V=10\), the object \(\boldsymbol{v}=(3,7)\) (i.e., \(v_1=3\), \(v_2=7\)) is encoded as \([\texttt{SOO},3,17,\texttt{EOO}]\) since \(t(1,3)=3\) and \(t(2,7)=10+7=17\).

Scenes and queries are embedded by two transformer encoders with the same architecture (SceneEncoder) but independent parameters: a scene encoder \(f_{\text{scene}}\) and a probe encoder \(f_{\text{probe}}\). Each maps a padded length-\(L_{\max}\) token sequence to a single embedding via token embeddings, positional encoding, and an \(L\)-layer Transformer encoder with \(H\) attention heads (masking PAD). The scene encoder outputs \(\boldsymbol{s}\in\mathbb{R}^{d_{\text{out}}}\). The probe encoder outputs \(\boldsymbol{q}\in\mathbb{R}^{d_{\text{out}}}\).

For a minibatch of \(B\) scenes and \(T\) queries, let \(\boldsymbol{S}\in\mathbb{R}^{B\times d_{\text{out}}}\) and \(\boldsymbol{Q}\in\mathbb{R}^{T\times d_{\text{out}}}\) collect their embeddings row-wise. We form a logit matrix \(\boldsymbol{Z}\in\mathbb{R}^{B\times T}\). We then optimze the embeddings \(\hat{\boldsymbol{x}}=\boldsymbol{x}/(\|\boldsymbol{x}\|_2+10^{-6})\) and define \(\hat{\boldsymbol{S}}\), \(\hat{\boldsymbol{Q}}\) by row-wise normalization. We use \(\tau=\exp(\tilde{\tau})\) as a learned temperature parameter, akin to CLIP (clamped to \(\tau\le 100\)).

At each training step we sample a batch of scenes and construct a set of query sequences. We form a multi-hot label matrix \(\boldsymbol{Y}\in\{0,1\}^{B\times T}\), where \(\boldsymbol{Y}_{ij}=1\) if query \(j\) matches scene \(i\). We use two query types:

1. Concept retrieval. Concept queries specify a single concept value. We collect the unique concept-value tokens that appear in the batch (excluding SOO/EOO/EOS/PAD), subsample up to num_concept_values_take_max, and tokenize each as a concept query.

2. Object retrieval. Object queries specify a full object. We parse object blocks from the tokenized scenes, take the set of unique objects present in the batch, subsample up to 1024, and tokenize each as an object query. We then augment the query set with hard negatives (random/perturbed objects and attribute swaps).

Concept queries probe whether the model can retrieve scenes based on single attributes, while object queries probe multi-attribute binding.

Given logits \(\boldsymbol{Z}\) and labels \(\boldsymbol{Y}\), we treat multiple positives by normalizing each row to a target distribution \(\boldsymbol{P}_{i\cdot}=\boldsymbol{Y}_{i\cdot}/(\sum_j \boldsymbol{Y}_{ij}+10^{-10})\) and minimizing the cross-entropy \[\mathcal{L} \;=\; -\frac{1}{B}\sum_{i=1}^B \sum_{j=1}^T \boldsymbol{P}_{ij}\log \mathrm{softmax}(\boldsymbol{Z}_{i\cdot})_j .\] Scenes with \(\sum_j \boldsymbol{Y}_{ij}=0\) (none of the sampled queries match) have \(\boldsymbol{P}_{i\cdot}=0\) and contribute zero loss. When both concept and object queries are enabled we sum their losses and backpropagate through both encoders and \(\tilde{\tau}\).

Training uses AdamW over the parameters of both encoders and \(\tilde{\tau}\), with a linear warmup (first \(10\%\) of steps) followed by linear decay [47]. The code can optionally take multiple optimizer steps per sampled minibatch (num_grad_steps). Table 9 lists the default hyperparameters and numerical-stability choices.

12pt

We use three real-image datasets: PUG:SPARE [7] (10) and CLEVR [48] (11). We additionally introduce CLEVR-2D (12), a 2D adaptation of CLEVR in which 3D shapes are replaced by their flat 2D counterparts (cube \(\to\) square, sphere \(\to\) circle, cylinder \(\to\) triangle). Each scene contains two objects; objects vary along two concepts—color (8 values) and shape (3 values)—yielding 576 two-object scenes. Paired captions follow the same template as CLEVR (e.g., “a purple square and a gray circle”).

This section specifies how we compute the retrieval and probing metrics reported for the Level-I/II decomposition experiments (2) and for the intervention experiments ([sec:interventions]). To evaluate whether binding-relevant information is present in embeddings within a modality, we use trained linear probes on frozen embeddings.

Probing metric (concept/object recognition accuracy). Let \(\boldsymbol{z}_{\boldsymbol{s}}=f(\boldsymbol{x}_{\boldsymbol{s}})\in\mathbb{R}^d\) denote an embedding of a scene \(\boldsymbol{s}\) (image or text, depending on the encoder). For each concept \(i\in[C]\), we train a linear probe \(h_i(\boldsymbol{z})=\boldsymbol{W}_i\boldsymbol{z}+\boldsymbol{b}_i\in\mathbb{R}^{|\mathcal{C}_i|}\) that outputs logits over the values of concept \(i\). We also train an object probe \(h_{\text{obj}}(\boldsymbol{z})=\boldsymbol{W}_{\text{obj}}\boldsymbol{z}+\boldsymbol{b}_{\text{obj}}\in\mathbb{R}^{|\mathcal{C}|}\) that outputs logits over the object space (concept tuples). Because scenes contain multiple objects, the target for a probe is generally multi-hot (multiple values/objects are present). We train probes on frozen embeddings by minimizing cross-entropy to the normalized multi-positive target distribution (as in [sec:app:controlled95models]).

At evaluation time, we evaluate using the metrics in Def. 3 and Def. 4. For concept \(i\), let \(h_i(\boldsymbol{z}_{\boldsymbol{s}})_v\) be the logit for value \(v\in\mathcal{C}_i\). We count the scene as correct for concept \(i\) if \[\min_{v\in V_i(\boldsymbol{s})} h_i(\boldsymbol{z}_{\boldsymbol{s}})_v > \max_{v\in \mathcal{C}_i\setminus V_i(\boldsymbol{s})} h_i(\boldsymbol{z}_{\boldsymbol{s}})_v.\] Concept recognition accuracy is the mean of this indicator over scenes and then averaged across concepts. Object recognition accuracy is computed analogously from \(h_{\text{obj}}(\boldsymbol{z}_{\boldsymbol{s}})\), using the set of objects \(O(\boldsymbol{s})\).

Retrieval metric (nearest-neighbor scene retrieval). For a fixed dataset of two-object scenes \(\{\boldsymbol{s}_n\}_{n=1}^N\), we precompute their embeddings \(\boldsymbol{z}_n=f(\boldsymbol{x}_{\boldsymbol{s}_n})\). Given a predicted embedding \(\tilde{\boldsymbol{z}}_n\) (e.g., a reconstruction from object means, or an edited embedding from [sec:interventions]), retrieval is treated as a nearest-neighbor problem in embedding space: \[\hat{n} \;=\; \arg\max_{m\in[N]} \frac{\langle \tilde{\boldsymbol{z}}_n,\boldsymbol{z}_m\rangle}{\|\tilde{\boldsymbol{z}}_n\|_2\|\boldsymbol{z}_m\|_2}.\] Retrieval is correct if the nearest neighbor corresponds to the intended scene. When multiple scenes share the same underlying object tuple (e.g., because of positional variation), we treat any scene with the intended object tuple as correct. We report top-1 accuracy over all evaluated scenes.

Reconstruction metric. For the \(R^2\) score, we measure how much variance is explained by the addition of the two object embeddings. In other words, the score measures how well \(V^2\) vectors can reconstruct all of the \(V^4\) embeddings. The \(R^2\) score is defined as \[R^2 := 1 - \frac{ \sum_{\boldsymbol{x}_{\boldsymbol{s}}} \lVert f(\boldsymbol{x}_{\boldsymbol{s}}) - \boldsymbol{u}_{\boldsymbol{o}_0} - \boldsymbol{u}_{\boldsymbol{o}_1} \rVert_2^2 }{ \sum_{\boldsymbol{x}_{\boldsymbol{s}}} \lVert f(\boldsymbol{x}_{\boldsymbol{s}}) - \boldsymbol{f} \rVert_2^2 }.\]

Setup. We generate a synthetic caption dataset for scenes by specifying a finite concept space \[\mathcal{C} := \mathcal{C}_1 \times \mathcal{C}_2,\] where \(\mathcal{C}_1\) contains adjective tokens (e.g., color/size/material) and \(\mathcal{C}_2\) contains noun tokens (e.g., shape/category). In our experiments we generate \(10{,}000\) two-object scenes plus \(100\) single-object scenes; each scene is a list of objects, and each object is represented by a pair of integer indices \((i_1, i_2)\) selecting tokens from \(\mathcal{C}_1\) and \(\mathcal{C}_2\).

Caption construction. For each scene, we linearize objects into a string of the form “a \(\langle\)attr\(\rangle\) \(\langle\)obj\(\rangle\) and a \(\langle\)attr\(\rangle\) \(\langle\)obj\(\rangle\)”. The following snippet shows the exact procedure used.

concept_space = [
    ["red", "blue", "green", "yellow", "purple", "orange", "pink", "brown", "gray", "black",
     "small", "large", "tiny", "round", "sharp", "rough", "bumpy", "smooth", "matte", "glossy"],
    ["circle", "square", "triangle", "star", "heart", "spiral", "sphere", "cylinder", "cone", "pyramid",
     "car", "cat", "dog", "man", "woman", "tree", "flower", "rock", "bird", "fish"],
] 

captions_dataset = []

for scene in scene_dataset:
    scene_caption = "a"
    for obj_i, obj in enumerate(scene):
        for concept_i, concept in enumerate(obj):
            scene_caption += f" {concept_space[concept_i][concept]}"
        if obj_i != len(scene) - 1:
            scene_caption += " and a"
    captions_dataset.append(scene_caption)

This section provides experimental details for the binding function approximation experiments on pre-trained models (5.1). Our aim is to determine whether CLIP’s binding function can be approximated by a learnable function, rather than to find the most efficient approximator. To ensure that any failure to generalize reflects genuine complexity of the binding function rather than insufficient model capacity, we deliberately over-parameterize the approximator and sweep over a wide range of configurations.

MLP architectures. We use ReLU MLPs that take as input a concatenation of one-hot encodings of concept indices for both objects in the scene (i.e., \(2 \times C \times V\) binary features, where \(C\) is the number of concepts and \(V\) is the number of values per concept). We experiment with 1-layer MLPs with hidden widths \(\in \{64, 128, 256, 512, 1024, 2048, 4096\}\);

The output dimension matches the embedding dimension \(d\) of the encoder (e.g., \(d=512\) for CLIP ViT-B/32). Results reported in the main text correspond to the best performance achieved across all architectures for each training fraction, ensuring that we report an upper bound on what the approximator family can achieve.

Training. We use the Adam [49] optimizer with learning rates in \(\{10^{-2}, 10^{-3}, 10^{-4}\}\), batch size 4096, for 10,000 steps. We select the best configuration based on test object accuracy. We split the \(V^C\) possible objects into train/test sets, varying training coverage over \(\{10\%, 20\%, \ldots, 90\%\}\). Test scenes contain only held-out objects, i.e., objects whose concept configurations never appeared during training.

Evaluation. We apply linear probes (trained on true embeddings) to the predicted embeddings and measure concept/object recognition accuracy. This tests whether predicted embeddings preserve functional binding properties, not just reconstruction fidelity. For text: we use synthetic captions ([sec:app:text95encoder95decomposition]); for images: CLEVR, CLEVR-2D, PUG:SPARE. We report the best accuracy across all MLP architectures and learning rates for each configuration.

4pt

4pt

In 4.2, we test how concept and object components each contribute to concept and object recognition in scene embeddings.

Setup. We estimate object and concept embeddings as described in Tab. 6. We use the same retrieval and probing metrics defined in [sec:app:clip95geometry95experiments]. For interventions, the target of an edited embedding is the intended counterfactual scene (or, when multiple scenes share the same object tuple due to positional variation, any scene with that object tuple is treated as correct).

Permuted-ID controls. To ensure that effects are not explained by subtracting an arbitrary but similarly-sized signal, we also run permuted-ID controls for both concept and object ablations. Concretely, let \(N_{\mathrm{c}}:=\sum_{i=1}^C |\mathcal{C}_i|\) be the total number of concept values (often \(N_{\mathrm{c}}=C\cdot V\)), and let \(N_{\mathrm{o}}:=|\mathcal{C}|=\prod_{i=1}^C |\mathcal{C}_i|\) be the number of objects (concept tuples). We sample uniform random permutations \(\pi_{\mathrm{c}}:[N_{\mathrm{c}}]\to[N_{\mathrm{c}}]\) and \(\pi_{\mathrm{o}}:[N_{\mathrm{o}}]\to[N_{\mathrm{o}}]\). When applying \(\pi_{\mathrm{c}}\), we treat each concept value \((i,v)\) as a unique ID in \([N_{\mathrm{c}}]\); when applying \(\pi_{\mathrm{o}}\), we treat each object \(\boldsymbol{o}\in\mathcal{C}\) as a unique ID in \([N_{\mathrm{o}}]\). We then form the same residual embeddings as in the main text, but with permuted components, e.g., \[f(\boldsymbol{x}_{\boldsymbol{s}}) - \sum_i \boldsymbol{u}_{\pi_{\mathrm{c}}(c_{1,i})} - \sum_i \boldsymbol{u}_{\pi_{\mathrm{c}}(c_{2,i})} \quad\text{or}\quad f(\boldsymbol{x}_{\boldsymbol{s}}) - \boldsymbol{u}_{\pi_{\mathrm{o}}(\boldsymbol{o}_1)} - \boldsymbol{u}_{\pi_{\mathrm{o}}(\boldsymbol{o}_2)}.\] We evaluate these permuted residuals with the same probes.

4pt

We provide additional results for the object-replacement interventions introduced in §4.1. These experiments test whether object embeddings can be reused across scenes to produce counterfactual scene embeddings via simple arithmetic.

Setup. Given a two-object scene \(\boldsymbol{s}=(\boldsymbol{o}_1,\boldsymbol{o}_2)\) with embedding \(f(x_{\boldsymbol{s}})\), we construct an edited embedding by replacing \(\boldsymbol{o}_1\) with a counterfactual object \(\boldsymbol{o}_1'\): \[\tilde{z} \;=\; f(x_{\boldsymbol{s}}) \;-\; k\,\boldsymbol{u}_{\boldsymbol{o}_1} \;+\; k\,\boldsymbol{u}_{\boldsymbol{o}_1'} ,\] where \(k\) controls the strength of the intervention. Setting \(k=0\) leaves the embedding unchanged, while larger \(k\) increases the contribution of the replacement. In practice, performance varies with \(k\), and we report full curves over \(k\) in Figure 13. Unless stated otherwise, results in 12 use \(k=1.0\).

We distinguish between two settings. In the different-concepts setting, the two objects in the scene do not share any concept values (e.g., a red cube and a blue sphere). In the shared-concepts setting, the objects share at least one concept value (e.g., a red cube and a red sphere). The shared-concepts setting is more challenging because the two objects occupy overlapping concept directions in the representation.

Results. Table 12 reports probing and retrieval performance for object replacement interventions at full strength (\(k=1.0\)) across datasets and models. In most settings, replacing an object embedding yields scene embeddings that support both object probing and retrieval consistent with the intended counterfactual scene. Object embeddings obtained via scene averaging and position-specific averaging perform similarly across datasets, indicating that positional effects are limited. When available, embeddings derived from single-object scenes also enable effective edits, though with reduced performance in some settings.

Performance drops substantially when the two objects in the scene share concepts, particularly for retrieval. This effect is consistent across datasets and encoders. A plausible explanation is feature interference: when objects share concept values, their contributions overlap in representation space, making it harder to selectively remove and insert a single object without affecting the other [10].

We further analyze the role of intervention strength in [fig:intervention_k_curves,fig:intervention_k_curves_single,fig:intervention_k_curves_text]. Varying the intervention coefficient \(k\) reveals that performance typically improves with stronger interventions before saturating. However, in shared-concept settings, optimal performance is often achieved at smaller values of \(k\), suggesting that partial edits mitigate interference effects.

To check that the additive decomposition of §4.1 is not limited to two-object scenes, we generate CLEVR scenes with three objects and matching text descriptions (Fig. 15). We evaluate \(R^2\) reconstruction, probing accuracy, and retrieval as in Tab. 2 and Tab. 3.

The additive decomposition extends to 3 objects: probing accuracy remains 0.91–0.93 on CLEVR even with occlusions, and reconstruction from objects consistently exceeds reconstruction from concepts, matching the pattern in Tab. 2. Retrieval drops under occlusion (0.76 \(\to\) 0.65), as expected, consistent with prior findings that VLM binding degrades as the number of objects grows [10].

Setup. To assess whether the additive decomposition extends beyond synthetic datasets, we generate images using Gemini Nano Banana 2 (gemini-3.1-flash-image-preview), spanning 5 object types and 5 patterns (625 samples). Unlike the synthetic datasets, these images exhibit variation in size, shape, color, and pattern realization, making the task substantially more challenging. We evaluate object decomposition and editing following the same protocol as in Tab. 2 and Tab. 3, using retrieval-based metrics: full retrieval among all 625 scenes, 4-way retrieval (correct scene, permuted concept combination, and two random distractors), and binary retrieval between the original and edited scene embeddings.

Results. Results are well above chance despite the increased difficulty, suggesting that the additive structure extends, to some degree, to more naturalistic scenes. We note that with a larger sample size, object embedding estimates would likely improve further.

To build intuition about the embedding geometry, we visualize MDS projections for a small set of concepts [42]. We consider two shapes (C, S) and two colors (R, B). This yields objects such as RC (‘red cube’) and BS (‘blue sphere’), and two-object scenes such as RCBS (‘red cube and blue sphere’). Object and concept embeddings are estimated as averages of embeddings of scenes that contain the corresponding component. Purple points represent sums of object embeddings (e.g. RC+BS). We connect the embeddings of the corresponding two-object scenes (e.g. RCBS) to these summed vectors.

Results. Across datasets and models, the projections exhibit a partial grid-like structure reflecting color and shape composition. This structure is most pronounced for text embeddings, while for CLIP image embeddings, it is more distorted. In DINOv2, objects with different shapes are far apart in embedding space, whereas objects with similar colors remain close.

The binding-complexity analyses in §5.1 and §5.3 uses a 1-hidden-layer MLP as the approximator. Our choice is motivated by the view that MLPs trained with SGD tend to favor simple, compressible solutions [43], [44], and that compositional mappings are precisely the kind of solutions this bias should favor [50]. If a simple compositional binding function exists, this approximator family should find it.

To verify that our findings are not specific to the MLP family, we additionally fit XGBoost and Random Forest approximators with the same setup as §5.1 (predict CLIP/DINOv2 scene embeddings from discrete concept indices, evaluate on held-out objects). For XGBoost we sweep n_estimators\(\in\{100, 500, 1000\}\), max_depth\(\in\{6, 8, 16, 32\}\), lr\(\in\{0.1, 0.01\}\). For Random Forest we sweep n_estimators\(\in\{100, 500, 1000\}\), max_depth\(\in\{\text{None}, 8, 16, 32\}\). We report the best accuracy across configurations.

In all three cases the approximator succeeds at concept-level prediction but fails at object-level binding. MLPs perform slightly better than XGBoost and Random Forest, though this is not an exhaustive search. The qualitative conclusion of §5.1 is unchanged: CLIP’s binding function does not generalize to unseen objects across this set of approximator families.

We complement §5.4 by fitting the same multiplicative Global product probe to CLIP and DINOv2 embeddings (Fig. 18). Concept recognition recovers with more training data, but object recognition stays near zero across encoders. Unlike the from-scratch models (§5.4, [sec:app:from95scratch95vision]), neither CLIP nor DINOv2 admits a multiplicative binding function, consistent with their known cross-modal binding failures.

The controlled experiments in §5.2–§5.4 use multi-hot token sequences as input to the scene encoder. To verify that the conclusions are not an artifact of the discrete input format, we replicate the entire pipeline (§5.2–§5.4) with pixel inputs and a from-scratch vision encoder.

Setup. We replace the multi-hot embedding layer of the scene encoder with a convolutional front-end and increase the number of transformer layers from 6 to 8 (we found this configuration to generalize better). The training pipeline, evaluation protocol, and binding analyses are otherwise identical to the main paper. Each object is defined by two concepts (square color and border color) with \(V=50\) values, giving up to \(6.5\!\times\!10^{6}\) object combinations. To stress the visual difficulty, we consider three levels of pixel-space complexity (Fig. 19): (i) noise-free, non-overlapping objects; (ii) speckled noise with non-overlapping objects; and (iii) noisy and overlapping objects.

Results. The two main findings from §5.3–§5.4 replicate on pixel inputs (Fig. 20). The binding maps of generalizing from-scratch vision models are well approximated by shallow MLPs (top row), and the multiplicative Global product probe recovers object recognition on held-out objects (bottom row), even under speckled noise and overlapping objects. Contrast with Fig. 18, where the same probe fails on the pretrained CLIP and DINOv2 encoders.

References↩︎