1 Introduction↩︎

Object-centric learning promises better data efficiency than image-level learning. By aligning representations at the level of objects rather than whole scenes, a model can exploit the compositional structure that scenes share, learning to assign similar representations to an object across the many scenes it appears in. Image-level self-supervised methods align augmented views of the same image. But under random cropping, two views often capture different regions of a scene, potentially featuring different objects, and a good encoder should arguably assign these regions different representations. Image-level alignment works against this: forcing the two views to agree, while avoiding collapse, leaves the encoder no choice but to rely on global, high-level semantics. Because semantics are context-dependent, each patch feature focuses more on encoding information on the surrounding scene rather than the object at its location. Object-level alignment removes this tension. Rather than matching scenes to scenes, it matches objects to objects: only representations of the same object need to agree across views, while distinct objects remain free to differ. This makes the representation modular, allowing the encoder to separate object identity from scene context. Such a representation should transfer across the many scenes an object appears in. It is our hypothesis that object-centric learning is more data-efficient than image-level learning, since the model can connect instances of the same object across scenes instead of relearning it in each new context. Beyond data efficiency through compositional generalization, object-centric encoders could advance relational reasoning, interpretability and temporal dynamic modelling [2]–[5].

Alignment between object representations is more challenging than alignment between whole images. While the latter has clearly defined positive samples (augmented views of the same image), the former requires object masks to define such samples. Previous approaches to object-centric learning tried to jointly solve partitioning and representation learning in an end-to-end manner [6], [7], however, this strategy could only work on natural images if it relied on frozen large-scale pre-trained encoders to extract patch features [8]. Building on patch features from frozen pre-trained encoders with slot attention lets both the partitioning and the object representations be learned jointly, but at the price of a restriction on each: both are a function of features from the frozen encoder. In contrast, our method accepts a fixed partitioning during training in exchange for far fewer restrictions on the representation space.

Different downstream tasks have different demands from object-centric representations. Classification calls for a largely semantic space, in which the representations of two instances with the same semantics lie close together, e.g., two horses in the same image should have similar representations. Tracking, in contrast, calls for a more instance-specific space: watching two nearly identical horses race, we still need to tell them apart. These demands are in tension, since semantic similarity pulls instances of the same category together while instance-specificity pushes them apart. In encoders trained with image-level losses, empirical evidence suggests that final-layer representations are largely semantic, and instance-specific object binding, if it occurs at all, emerges only in earlier layers [9]. Our method instead encodes both semantic and instance-specific information in the final-layer representations.

As illustrated in Figure 1, we capture both semantic and instance-specific information in each patch feature through training with two heads. The semantic head, \(f\), uses masks to aggregate patch features into a single object representation. These object representations feed into an object-centric LeJEPA loss that aligns different augmented views of the same object while preventing collapse, yielding semantic coherence. The instance head, \(\phi\), is applied to each patch feature individually and can be understood as predicting the instance representation of the object the patch belongs to. Here the aim is to align the instance predictions of patches within the same mask while keeping them separable from those of patches outside it. We achieve this with a contrastive loss [10]–[12].

2 Related Works↩︎

Our work sits at the intersection of three lines of research: general self-supervised representation learning (SSL), self-supervised pretraining at the level of image regions rather than whole scenes, and object-centric learning that jointly discovers and represents objects.

2.1 Representation Learning↩︎

Purely reconstruction-based representation learning can lead to representations that are not aligned with perception-based downstream tasks [13]. This is why contrastive and self-supervised learning paradigms have begun to dominate the field of representation learning [14]–[17]. Many of them rely on heuristics to avoid representation collapse, such as teacher–student architectures, stop-gradients, or hyperparameter schedules. The LeJEPA framework [18] replaces these heuristics with a principled anti-collapse objective that regularizes the representations toward an isotropic Gaussian through a statistical test for normality. In the image domain, LeJEPA has only been applied to embeddings for images as a whole, we adopt the LeJEPA loss to work on an object level.

2.2 Region- and Object-level SSL↩︎

A growing body of work moves self-supervised pretraining from whole images to local regions, aligning the features of corresponding regions across views instead of pooling the entire scene. VICRegL [19] and DenseCL [20] match dense or local features, whereas DetCon [21] and ODIN [22] pool features over heuristic or self-generated mask proposals and contrast the resulting region representations. Another approach incorporates object proposals into the augmentation pipeline, cropping views around proposed objects rather than at random [23]. Unlike approaches that reduce each image to a single representation and apply alignment and repulsion on an image level [23], our method extracts a separate representation for every object via its mask and applies a loss among these within-image representations, explicitly separating distinct instances. Closest to us, SlotMIM [24] couples masked image modeling with slot-style grouping to learn object-level representations, and serves as our primary object-centric baseline. Our method shares the region-pooling idea with the previous works but differs in two respects: the anti-collapse signal comes from a distributional normality test ported more naturally to variable-sized sets of objects rather than from a contrastive or teacher–student mechanism, and we add an explicit instance-separating objective so that co-occurring objects of the same category remain distinguishable.

2.3 Object-centric Learning↩︎

Ideally, an object-centric model decomposes a scene and finds representations for each object in the scene. This challenge was tackled in an end-to-end manner by combining sequential or parallelized grouping mechanisms with representation learning [6], [7]. While initial methods solely relied on reconstruction-based objectives, more recent works have explored adding contrastive objectives for object-centric representation learning [25]. Scaling these models to real-world scenes generally requires incorporating image foundation models [8] due to the cyclic dependency mentioned before. However, building on top of frozen features imposes a ceiling on what the model can express. In slot attention, each slot is a projected linear combination of patch features, so a slot can only ever express what is already present in the patch feature space. The same frozen features also dictate how the scene is partitioned, placing a strong prior on the partitioning. Training the encoder from scratch removes this ceiling, because the patch features can adapt to become more powerful in representing objects than generic features.

Training end-to-end object-centric encoders without reconstruction-based losses is difficult, since joint-embedding and contrastive objectives rely on meaningful positive (and negative) object pairs, which would not be available at the beginning of training due to unstable scene decomposition. As a remedy, some works propose image decomposition based on dataset-wide prototypes [24], [26]–[28]. These methods again rely on slot attention to match patches to the prototypes, where the assignment is soft. Even when different regions of the image favor different prototypes, no patch is assigned to a single prototype exclusively. Object information therefore bleeds across the resulting partitions, undermining the clean separation that compositional generalization requires. Hard masks avoid this leakage by enforcing binary assignments. Motivated by this, [29] obtain binary masks from external segmenters, which are now cheap to acquire at scale from promptable models [30], [31]. Their approach inherits the clean separation of hard masks but, unlike ours, still depends on a pre-trained encoder: object representations are extracted by passing mask-cropped crops through a frozen foundation model, leaving the encoder unable to adapt its features to objects. Relying on an external segmenter does mean accepting stale, fixed partitions that do not adapt during training. In return, the partitioning is clean and the encoder is no longer restricted to forming slots within a frozen feature space, so the representation can be trained end-to-end. We trade adaptive partitioning for clean, hard masks and an unconstrained, fully trainable encoder.

3 Background↩︎

The guiding principle behind self-supervised image encoders is to map similar images to similar representations. In the absence of labels indicating similarity, similar pairs are synthesized by randomly augmenting an image. Enforcing alignment across augmentations - typically photometric transforms such as color jitter together with random cropping - push the encoder towards capturing semantics.

This alignment objective alone has a trivial solution, known as collapse: mapping every image to the same representation satisfies alignment perfectly. LeJEPA [18] therefore pairs the alignment term with a regularizer, SIGReg, that wards off collapse by encouraging the representations of a batch to resemble an isotropic Gaussian distribution through checking the Epps-Pulley statistic.

LeJEPA applies both terms at the image level: it aligns whole-image representations across augmentations and regularizes them with SIGReg. Our method instead applies them per object. Rather than aligning and regularizing the representation of an entire scene, which may consist of multiple objects, we do so for each object representation, which promises greater data efficiency.

4 Method↩︎

We learn object representations in two complementary spaces. The first is a semantic space, shaped by an object-centric LeJEPA loss \(\mathcal{L}_{\mathrm{ObjectLeJEPA}}\), in which two horses are assigned similar representations. The second is an instance space, shaped by a contrastive loss \(\mathcal{L}_{\mathrm{instance}}\), in which those same two horses remain separable within a scene. Both losses build on object representations extracted from masks supplied by an external model (SAM 2 [31]). As illustrated in Figure 1, we extract one semantic representation per mask and feed it to \(\mathcal{L}_{\mathrm{ObjectLeJEPA}}\), while every patch predicts an instance-level representation of its object, which \(\mathcal{L}_{\mathrm{instance}}\) processes. At inference time no masks are needed, as the backbone yields informative patch features directly. Section 4.1 details the extraction before the two sections after it present the losses.

4.1 Object Representation Extraction↩︎

The extraction of an object’s representation relies on knowledge about its spatial location and extent. Obtaining these in an unsupervised manner in turn relies on meaningful image representations. As of now, training both in an end-to-end manner is unstable for natural images without “extra knowledge”, for instance using pre-trained encoders. Here, we focus on the representation learning part and take the object masks as given. These masks are precomputed with SAM 2 before the training to avoid repeated computations over the epochs. Concretely, we run the SAM 2 automatic mask generator on each original image, prompting it with a regular \(16\times16\) grid of point cues. More details on mask generation are provided in the appendix. Our goal is to train an image encoder \(g\) that for each patch of the image \(\mathbf{x}\) encodes both instance-level and semantic information about objects that it may belong to.

4.1.0.1 Semantic Object Representations

Following the principle that a whole is defined by its parts, we compute a semantic object representation \(\mathbf{z}\) as a function \(f\) of the patch features that fall within its mask.

As illustrated in Figure 2, we first calculate a weighted mean of independently projected patch features based on the patch-wise average-pooled object mask. The following cross attention masks out keys and values coming from outside the mask. Finally, we obtain the object representation by adding a residual connection from the weighted mean before passing it through a residual MLP.

LeJEPA [18] relies on augmentations for its alignment term and we closely follow the same augmentation pipeline. Let us denote the semantic object representation from the \(n\)-th training image, \(v\)-th augmented view, and \(k\)-th mask as \(\mathbf{z}_{n,v,k}\). Because the random augmentations involve cropping, some objects will not be visible in all views. We define an object to be present in a view, if its view-projected mask covers an area of at least \(16\times 16\) pixels, corresponding to the patch size of the ViTs we are using. We define the set of objects present in the \(v\)-th view of the \(n\)-th image as \(\mathcal{K}_{n,v} \subseteq \mathcal{K}_n\). Dually, we define the set of views where an object is present as \(\mathcal{V}_{n,k}\). A common procedure in self-supervised learning is to apply alignment on an MLP-projection of the representations, which we will refer to as \(\tilde{\mathbf{z}}_{n,v,k}\). We set the projection dimension to \(d=64\).

4.1.0.2 Instance-level Object Representations

Here we rely on the principle that a part should know which whole it belongs to. Hence, the extraction of the instance-level object predictions of \(i\)-th patch boils down to an MLP \(\phi\) applied to the \(i\)-th patch feature before applying \(\ell_2\) normalization, \[\begin{align} \mathbf{y}_{n,v,i} = \frac{\phi(g(\mathbf{x}_{n,v})_i)}{\|\phi(g(\mathbf{x}_{n,v})_i)\|_2}, \end{align}\] where \(\mathbf{x}_{n,v}\) is the \(v\)-th augmented view of the \(n\)-th image and \(g(\mathbf{x}_{n,v})_i\) is the patch feature of the \(i\)-th patch.

4.2 Object-centric LeJEPA↩︎

The object-centric LeJEPA loss combines an alignment term and a regularization term. The alignment term \(\mathcal{L}_{\mathrm{pred}}\) encourages the representations of the same object across different views to be similar. We can only align when an object appears in at least two views. Hence, we define the effective set \(\mathcal{K}_{n}^\mathrm{eff} = \{k \in \mathcal{K}_n \;| \;|\mathcal{V}_{n,k}| \geq 2\}\). For each extracted object representation \(\mathbf{z}_{n,v,k}\) that is present in at least one other view, we compute the alignment as such \[\begin{align} \ell^{\mathrm{pred}}_{n,v,k} = \frac{1}{d} \| \boldsymbol{\mu}_{n,k} - \mathbf{\tilde{z}}_{n,v,k}\|_2^2, \;k \in \mathcal{K}_{n}^\mathrm{eff}, \;v \in \mathcal{V}_{n,k} \end{align}\] where \(\boldsymbol{\mu}_{n,k} = \frac{1}{|\mathcal{V}_{n,k}|}\sum_{v \in \mathcal{V}_{n,k}} \mathbf{\tilde{z}}_{n,v,k}\) is the average of the \(k\)-th object representation across all views where it is present. The final prediction loss \(\mathcal{L}_{\mathrm{pred}}\) is an average over all object-level alignment terms \(\ell^{\mathrm{pred}}_{n,v,k}\).

The regularization term \(\mathcal{L}_{\mathrm{SIGReg}}\) [18] is applied on the projected object representations \(\mathbf{\tilde{z}}_{n,v,k}\) to prevent collapse. Instead of calculating the empirical characteristic function with respect to representations of images in a batch of size \(B\), we compute it over all object representations extracted from all \(B\) images. In the original LeJEPA paper, SIGReg uses the Epps–Pulley test statistic, which is scaled by the sample size. In our case this sample size varies significantly across batches, since the number of objects in a view is far from constant. Hence, we regularize the Epps–Pulley test statistic with a constant scale factor set to \(B\) to avoid large fluctuations in gradient magnitude and unstable training.

The two terms are combined into the object-centric LeJEPA loss \[\begin{align} \mathcal{L}_{\mathrm{ObjectLeJEPA}} = \mathcal{L}_{\mathrm{pred}} + \lambda_{\mathrm{LeJEPA}}\, \mathcal{L}_{\mathrm{SIGReg}}, \end{align}\] where we set \(\lambda_{\mathrm{LeJEPA}} = 0.05\) as recommended by [18].

4.3 Instance-level Loss↩︎

Our second object-centric objective targets instance-specific structure so that co-occurring objects can be told apart. We want all patches of the same object to agree on a common instance representation while remaining separable from those of other objects. Unlike the LeJEPA loss, this term is computed independently within each view, since separating co-occurring objects is an intra-image problem while cross-view consistency is already handled by \(\mathcal{L}_{\mathrm{pred}}\). Following the supervised contrastive formulation [12], for every patch \(i\), let \(\mathcal{A}(i)\) be all other patches in the same view and \(\mathcal{P}(i) \subseteq \mathcal{A}(i)\) those sharing its dominant mask. With temperature \(\tau = 0.1\), the per-anchor loss is defined for all patches \(i\) that share its mask with at least one other patch, \[\begin{align} \ell^{\mathrm{instance}}_{n,v,i} = -\frac{1}{|\mathcal{P}(i)|} \sum_{p \in \mathcal{P}(i)} \log \frac{\exp\!\left(\mathbf{y}_{n,v,i}^\top \mathbf{y}_{n,v,p} / \tau\right)}{\sum_{a \in \mathcal{A}(i)} \exp\!\left(\mathbf{y}_{n,v,i}^\top \mathbf{y}_{n,v,a} / \tau\right)}. \end{align}\] We average all valid granular loss terms \(\ell^{\mathrm{instance}}_{n,v,i}\) first over all patches within a view, and then jointly over all views \(v\) and images \(n\). Here we assign patches to a mask if it covers at least \(50\)% of the mask. Background patches serve as negatives but are not used as anchors.

The total training objective is the sum of the two object-centric losses, \[\begin{align} \mathcal{L} = \mathcal{L}_{\mathrm{ObjectLeJEPA}} + \mathcal{L}_{\mathrm{instance}}. \end{align}\]

5 Results↩︎

We compared three methods trained on COCO [32]: image-level LeJEPA [18], SlotMIM [24], and our own method, Object-LeJEPA, which uses mask proposals from SAM 2 [31]. As an upper bound, we also report results for DINOv3 [17], using the official checkpoint (ViT-B) trained at scale. We froze all encoders and evaluated them on a range of downstream tasks: linear probing for image classification on ImageNet [33], various dense tasks on ADE20k [34], tracking via nearest neighbours on DAVIS [35], and object re-identification from region-pooled patch features on NAVI [36]. Our downstream benchmarks span object types, scene layouts, and image distributions that differ from COCO, so they probe how well the frozen features generalize beyond the pretraining domain.

5.1 Training Details↩︎

For the main experiments we trained on COCO for 150 epochs, following [18] in setting \(\lambda_{\mathrm{LeJEPA}}=0.05\), a weight decay of \(0.05\), and a LeJEPA projection dimension of \(64\). We optimized with AdamW [37] at a batch size of 256 and a learning rate of \(5\cdot10^{-4}\), using a cosine schedule preceded by one linear warm-up epoch. For each image, we sampled \(2\) global views of size \(256\times256\) and \(8\) local views of size \(128\times128\), together with the standard LeJEPA augmentations. We used these same hyperparameters for training both image-level LeJEPA and our method. SlotMIM (800) was trained for 800 epochs on COCO as recommended in the training script provided by its authors. For comparability, we also include a SlotMIM model that was trained for 150 epochs. Unless reported otherwise, all models used the ViT-Base architecture [38] with a patch size of \(16\times16\). Further implementation details can be found in the appendix. The code will be made publicly available upon publication.

5.2 Downstream Tasks↩︎

In the following paragraphs, we introduce downstream tasks and probes on frozen patch features and discuss the results. Further details can be found in the appendix.

5.2.0.1 Instance-Awareness Probing

To test whether encoders capture instance-level information, we first assigned each patch to the mask that occupies the largest fraction of the patch. We discarded patches assigned to the background. Then we applied two postprocessing pipelines to the frozen foreground patch features on ADE20k. First, we K-Means clustered the L2-normalized features, setting the number of clusters to the number of ground-truth instance masks, and measured agreement with those masks via foreground adjusted rand index (FG-ARI) and mean IoU. Second, following Li  [9], we trained a quadratic probe on the ADE20k train set to predict whether a pair of foreground patches belongs to the same instance, and reported accuracy and AUC on the validation set. Features encoding instance-level information should both cluster into object instances and support accurate same-instance prediction.

When clustering patch features into object instances (Table 1), Object LeJEPA not only lead the COCO-trained models but overtook DINOv3, raising the FG-ARI from \(0.357\) to \(0.431\) and the mIoU from \(0.300\) to \(0.355\). The quadratic probe (Table 2) tells the same story: Object LeJEPA reached \(0.915\) accuracy and \(0.954\) AUC, surpassing both COCO baselines and effectively matching DINOv3 (\(0.914\) / \(0.957\)). Learning representations guided by explicit object information shapes a patch feature space whose geometry encodes object membership about as well as a model trained at far greater scale. Prior work [9] shows that the ability to tell whether two patches share an object emerges in pretrained ViTs such as DINO, but that this signal may reside more strongly in intermediate layers than in the final one. Our instance-level loss instead pulls this object-membership information directly into the last-layer representation that downstream tasks consume, which avoids the need to find the right layer for the right task.

5.2.0.2 Dense Semantic Tasks

To test whether the encoders capture similarities across objects in different images, we evaluated the frozen patch features on two dense prediction tasks. First, we trained a linear probe mapping frozen patch features to semantic segmentation labels on the ADE20k training set and evaluated it on the validation set, reporting mean IoU and pixel accuracy. Second, following the setup of [14], we ran a simple tracking pipeline on DAVIS in which the mask of the initial frame is propagated to subsequent frames by nearest-neighbour matching of the L2-normalized frozen patch features, reporting contour accuracy, region similarity, and their mean. Although this task also benefits from instance-level information, it requires matching objects across frames under appearance and pose changes.

On linear-probe semantic segmentation (Table 3), Object LeJEPA attained the best mIoU among the COCO-trained models (\(0.418\)), edging out SlotMIM (\(0.409\)) and clearly improving over image-level LeJEPA (\(0.339\)), while DINOv3 remained ahead at \(0.500\). On DAVIS label propagation (Table 4), Object LeJEPA improved the \(\mathcal{J}\&\mathcal{F}\) mean to \(0.682\), well above image-level LeJEPA (\(0.613\)) and SlotMIM (\(0.629\)), and narrowed the gap to DINOv3 (\(0.715\)). Our patch representations were therefore both temporally stable and object-discriminative. Notably, our instance-level loss only contrasts objects within a single view, yet the discriminability it induces together with the semantic loss seems to have transferred to separating objects across frames and scenes.

5.2.0.3 Object-level Tasks

Because our model was trained with mask-guided object-level alignment, we evaluated whether the resulting object representations transfer to object-level downstream tasks. We considered two tasks. On ADE20k, we extracted a representation for each object using its ground-truth mask. We averaged patch features within the mask and additionally computed the semantic object representations (\(\mathbf{z}\)) for our method. We trained a linear probe to predict the object class and computed top-1 and top-5 balanced accuracy on the validation set. The second task was object re-identification on NAVI, which contains multiple images per object across varying backgrounds, poses, and lighting. Here we also extracted object representations using ground-truth masks and built a memory bank with k representations per object and classified each remaining image by its nearest neighbor in the bank, using \(\ell_2\)-normalized object representations. We varied k from 1 to 10 and, for each k, repeated the memory-bank sampling 10 times and averaged.

For linear classification of individual objects (Table 5), Object LeJEPA again outperformed both baselines using averaged patch features (\(0.212\) top-1), and its native slot representation lifted this further to \(0.250\), though all COCO-trained models trail DINOv3 (\(0.367\)) on this semantically demanding task.

On NAVI instance re-identification (Figure 3), a few points stand out. First, DINOv3 was exceptionally strong, reaching \(92.2\%\) balanced accuracy from a single shot while the COCO-trained models trailed far behind. While the NAVI objects can be considered out-of-distribution for COCO, we do not know whether the same holds for DINOv3, that was trained on a much larger corpus. Second, among the COCO-trained models SlotMIM was the weakest, even below image-level LeJEPA. We suspect its patch feature representations blend global and local information to a higher extent due to the soft object assignments introduced by their slot attention module. Third, Object LeJEPA was the strongest, and its averaged patch-object features clearly beat its own object embeddings. These are trained to align objects across augmented views and, although region-specific, they also absorb surrounding context. That context is useful for category-level tasks such as object classification, where objects and backgrounds statistically co-occur, and indeed the slots won there (Table 5). In the features themselves, however, local information dominates, so for re-identification the average-pooled patch-object vectors transferred better.

5.2.0.4 Image-level Task

We also evaluated image-level classification on ImageNet-1k. We trained a linear probe to predict the class label from the image representation and reported top-1 and top-5 balanced accuracy on the validation set. Since the object-centric models did not learn a [CLS] token, we represented each image by globally average pooling its patch features.

Despite being trained to represent objects rather than whole images, our averaged patch features improved top-1 balanced accuracy over LeJEPA from 47.3% to 53.9% and trailed SlotMIM trained for 800 epochs (55.2%) by only a small margin. We attribute this gap to SlotMIM’s soft masks, which likely make its latent space more compatible with global average pooling. The appendix contains further visual results on samples from ImageNet, visually assessing the feature quality by self-similarity maps on images containing object categories that were not seen during training.

5.3 Dataset Size Ablation↩︎

Figure 4 ablates the effect of the pretraining set size and the backbone capacity across four downstream tasks, described in Section 5.2. Our central finding is one of data efficiency: trained on only \(10\%\) of COCO, Object LeJEPA with a ViT-B architecture already matched image-level LeJEPA trained on the full COCO dataset on every task. With a tenth of the data it reached \(0.648\) \(\mathcal{J}\&\mathcal{F}\) on DAVIS tracking, \(0.472\) top-1 balanced accuracy on ImageNet-1k, \(0.380\) mIoU on ADE20k segmentation, and \(0.343\) 1-shot balanced accuracy on NAVI re-identification, in each case meeting or slightly exceeding image-level LeJEPA’s full-data scores (\(0.613\), \(0.463\), \(0.339\), and \(0.337\), respectively). Object-level alignment therefore recovered, from ten times less data, what image-level alignment attained only at full scale. The remaining trends were as expected. Scaling Object LeJEPA along either axis - from \(10\%\) to \(100\%\) of COCO, or from a ViT-Small to a ViT-Base backbone - improved performance on all four tasks.

5.4 Loss & Mask Ablation↩︎

4pt

Table 7 disentangles the contribution of each loss. Object alignment \(\mathcal{L}_{\mathrm{ObjectLeJEPA}}\) on its own already outperformed image-level LeJEPA on every task. The instance separation loss \(\mathcal{L}_{\mathrm{instance}}\) added dense, instance-level supervision, boosting the instance-discrimination tasks of tracking and re-identification, but it struggled on the more semantic classification and segmentation tasks. Combining the two losses improved every task over either loss in isolation, with the sole exception of image classification, where the alignment-only setting remained marginally ahead (\(54.5\)% vs.\(53.9\)%). The final row replaces the unsupervised SAM masks with COCO ground-truth instance masks during training, isolating the effect of mask quality while keeping the objective fixed. The two were strikingly close, and neither dominated: ground-truth masks were marginally ahead on tracking and image classification, whereas SAM masks were better on segmentation and substantially better on NAVI re-identification. Crucially, Object LeJEPA does not rely on costly human annotation, and unsupervised SAM masks are sufficient for learning strong representations.

6 Conclusion↩︎

Our method extends LeJEPA by moving the alignment and regularization term from the image to the object level. This is enabled through cheap mask proposals during training and results in a much more data-efficient training. At inference time we do not require masks and, in our experiments, achieved the same downstream task performance as the image-level LeJEPA with 10% of the training data.

7 Visual Results↩︎

In this section we present visual results for models that are trained on the COCO dataset. Only DINOv3 was trained on a larger dataset. Since COCO does not include images of fish, it is interesting to investigate the encoders’ instance-separating and cross-image matching capabilities on out-of-distribution objects. We performed two different experiments. For the first experiment we applied K-Means clustering on the \(\ell_2\)-normalized patch features with varying number of clusters \(k\). In the second experiment, we selected a patch feature belonging to a fish in one image and visualized the feature similarity (cosine-similarity) to other patches from the same image and other images.

Among the COCO-trained models, our encoder, Object LeJEPA, is the only method that actually separates the two instances into different clusters (Figure 5). The other models tend to introduce clusters that contain patches from both fish. This supports the thesis that our encoder actually generalizes the instance-detection capabilities to objects not seen during training. The instance-separating capabilities of our model are also demonstrated by much sharper object boundaries in the intra-image similarity map (Figure 6). The patch features belonging to the other fish are noticeably darker. We can see that the COCO-trained models have never seen a fish during training because they do not seem to be able to distinguish them from flowers like DINOv3 does (image 5).

8 Implementation Details↩︎

8.1 Training Details↩︎

We list here the full set of hyperparameters used to train our main model. The encoder is a ViT-Base with a patch size of \(16\times16\), trained from scratch with a stochastic-depth (drop-path) rate of \(0.1\). The LeJEPA projection head is a three-layer MLP (\(768\rightarrow2048\rightarrow2048\rightarrow64\)) with synchronized batch normalization, projecting into a \(64\)-dimensional space, and the masked cross-attention pooling uses \(8\) heads.

We train on COCO for \(150\) epochs with AdamW (\(\beta\) defaults, \(\epsilon=10^{-4}\)), a learning rate of \(5\cdot10^{-4}\), and a weight decay of \(0.05\). The effective batch size is \(256\) (\(64\) images per GPU across \(4\) NVIDIA RTX A6000 GPUs). The learning rate follows a cosine schedule decaying to \(5\%\) of its peak, preceded by a single linear warm-up epoch. We use mixed-precision (AMP) training and clip gradients to a global norm of \(1.0\). Per image we sample \(2\) global views at \(256\times256\) and \(8\) local views at \(128\times128\), together with the standard LeJEPA photometric augmentations (color jitter, grayscale, Gaussian blur, Gaussian noise, and solarization).

The training objective combines three terms: the LeJEPA SIGReg regularizer on the slot projections (weight \(\lambda_{\mathrm{LeJEPA}}=0.05\)), the object-alignment loss (weight \(1.0\)), and the within-view instance-separation InfoNCE loss (weight \(1.0\), temperature \(0.1\)). SIGReg uses \(1024\) random projections and \(17\) quadrature knots.

The object masks are mask proposals precomputed once with the SAM 2.1 Hiera-Large automatic mask generator [31], run with \(16\) points per side, a predicted-IoU threshold of \(0.8\) (how confident SAM is in a mask), and a stability-score threshold of \(0.92\) (how stable the binary mask is with respect to cutoff threshold variations). The other values are left to the default configuration. We cap the number of partitions (masks) per image at \(64\), which bounds the per-batch slot count and the cross-attention memory footprint.

8.2 Downstream Task Details↩︎

All encoders are frozen and used purely as feature extractors. For each dataset we run a single forward pass per image and cache its outputs, so every probe on a given dataset reads the same features. We work with three kinds of representations. Patch features are the last-layer feature maps. At a patch size of \(16\times16\) a \(512\times512\) image yields a \(32\times32\) grid. The image-level representation is the mean of all patch features over the spatial grid. For the image-level models, Image LeJEPA and DINOv3, we additionally extract the [CLS] token. The object-level representation of a mask is the weighted mean of the patch features over the patches. The weights for this mean come from the patch-wise average-pooled mask. For Object LeJEPA we additionally evaluate the semantic slot representations \(\mathbf{z}_{n,v,k}\). Each dataset is processed at a fixed input resolution, always a multiple of the \(16\)-pixel patch size: ADE20k, COCO, and NAVI at \(512\times512\) and ImageNet-1k at \(224\times224\). DAVIS instead keeps its native \(480\)p aspect ratio, resizing each frame so that its shorter side is \(512\) pixels (rounded to a multiple of \(16\)).

8.2.0.1 Instance Clustering (ADE20k)

For each validation image we \(\ell_2\)-normalize the foreground patch features and run K-Means with \(K\) set to the number of ground-truth foreground instances in that image. We use \(5\) random initializations per image. Clusters are matched to ground-truth instances by Hungarian assignment on cluster-vs-instance IoU, and we report the mean IoU over the \(K\) matched pairs. The adjusted rand index (FG-ARI) is computed directly over all foreground patches and needs no matching, as it is permutation-invariant. Metrics are averaged over all validation images containing at least two instances. We use the 2021 ADE20k release, which provides true per-instance masks.

8.2.0.2 Same-Instance Quadratic Probe (ADE20k)

Following Li  [9], we learn a low-rank symmetric bilinear form \[\begin{align} \mathrm{IsSameObject}(x,y)=\sigma\!\left((Px)^{\top}\mathrm{diag}(g)(Py)+b\right) \end{align}\] that predicts, for a pair of raw (unnormalized) patch embeddings, whether they belong to the same object instance. Here \(P\in\mathbb{R}^{k\times d}\) projects each patch into a \(k=32\) dimensional binding subspace and \(g\in\mathbb{R}^{k}\) is a learned signed signature, giving a symmetric, rank-\(\leq k\) form with \(O(kd)\) parameters that is not constrained to be positive semi-definite. The probe is trained on the ADE20k training split with binary cross-entropy over all within-image off-diagonal patch pairs, sampling at most \(64\) foreground patches per image. We optimize with Adam (learning rate \(10^{-3}\), weight decay \(10^{-4}\), cosine schedule) for \(5\) epochs at \(32\) images per batch. At evaluation we score every foreground patch pair (up to \(256\) patches per validation image) and report, per image, the pair-classification accuracy at the decision boundary (logit \(>0\)) and the threshold-free ROC-AUC, averaged over validation images.

8.2.0.3 Semantic Segmentation (ADE20k)

We train a single \(1\times1\) convolutional head on the frozen patch features to predict the \(150\) ADE20k semantic classes, with cross-entropy and the background class ignored. Training is done at the feature-grid resolution (the label mask is nearest-neighbour downsampled), and at evaluation the logits are bilinearly upsampled back to the cached mask resolution before the argmax, so the metric follows the full-resolution protocol. We optimize with Adam (learning rate \(10^{-2}\), cosine schedule) for \(5\) epochs at a batch size of \(128\). We report mean IoU (over the classes present in each image) and pixel accuracy, averaged over the validation images.

8.2.0.4 Label Propagation (DAVIS)

We follow the label-propagation tracker of [14]. The ground-truth mask of the first frame is propagated to subsequent frames by affinity-weighted voting over \(\ell_2\)-normalized patch features: affinities are softmax-weighted with temperature \(0.1\), restricted to a spatial neighborhood of radius \(12\) patches around each query, and sparsified to the top \(5\) source patches per query. The context set for each frame is the first frame plus the \(7\) most recently predicted frames. Frames keep their aspect ratio (short-side resize) and are processed one at a time. We report region similarity \(\mathcal{J}_m\) (mask IoU), contour accuracy \(\mathcal{F}_m\) (boundary F-measure with a tolerance band of \(0.8\%\) of the image diagonal), and their mean \(\mathcal{J}\&\mathcal{F}\), first averaged over objects and frames within a video, then over videos.

8.2.0.5 Object Classification (ADE20k)

Each object is represented by its object-level descriptor, extracted with its ground-truth mask, and for Object LeJEPA additionally by its semantic object representation. We train a linear head on the training-split objects with cross-entropy, using Adam (learning rate \(10^{-2}\), cosine schedule) for \(50\) epochs at a batch size of \(4096\). The classes present in the training split are densely re-indexed, and validation objects of classes unseen in training are excluded from scoring. We report top-1 and top-5 balanced accuracy over the validation objects.

8.2.0.6 Object Re-Identification (NAVI)

On the NAVI wild set, each object instance appears in multiple images under varying background, pose, and lighting. We represent each object by its mask-pooled patch descriptor (and, for Object LeJEPA, its slot), \(\ell_2\)-normalized. For each number of shots \(k\in\{1,2,3,5,10\}\) we build a memory bank by sampling \(k\) representations per identity and classify every remaining image by its cosine-nearest exemplar across all identities; identities with \(\leq k\) images are skipped. For each \(k\) we repeat the sampling \(10\) times and report the mean balanced accuracy (with standard deviation).

8.2.0.7 Image Classification (ImageNet-1k)

The object-centric models (SlotMIM and Object LeJEPA) do not learn a usable [CLS] token, so we represent each of their images by the image-level descriptor (the global average of its patch features). For the image-level models, Image LeJEPA and DINOv3, we additionally report results using their [CLS] token. On the \(\ell_2\)-normalized, standardized representations we fit a multinomial logistic-regression probe (inverse regularization \(C=1\)) on the full training split and report top-1 and top-5 balanced accuracy on the validation split.

References↩︎