1 Introduction↩︎

Visual intelligence is not only about recognizing what is in an image, but also about recovering the dense spatial structure that makes the physical world measurable, navigable, and actionable. Across segmentation, depth, motion, and scene layout, this structure is repeatedly revealed by shapes and boundaries: object masks delineate entities, depth discontinuities expose geometry, motion silhouettes separate dynamic regions, and occlusion contours organize visible surfaces. Yet many modern visual foundation models are driven by objectives that favor semantic invariance, cross-modal alignment, or appearance reconstruction, and remain comparatively weak at the fine-grained spatial understanding that downstream physical intelligence depends on.

A central reason is how shapes and boundaries are treated. They are usually regarded as outputs of perception, recovered by task-specific heads under dedicated supervision. Such an output-centric view ties them to annotations that are expensive, ambiguous, and often unavailable, making them difficult to exploit in large-scale pretraining. As a result, foundation models rarely use shapes and boundaries as native learning signals, relying instead on view-invariant self-distillation [1]–[3], cross-modal alignment [4], [5], or masked image reconstruction [6]. We argue that this is a missed opportunity: boundaries and shape discontinuities are not merely outputs to be predicted, but fundamental organizing signals for learning dense representations.

In this work, we therefore study how randomly initialized Vision Transformers can learn shape- and boundary-aware representations from raw images alone, without human annotations, external edge detectors, or pretrained backbones. Our study builds on dense, over-parameterized field representations of boundary geometry [7], [8], which encode vectorized boundaries through pixel-wise, distance-transform-like fields. Such representations have been extensively used in supervised line-segment and wireframe detection, where they turn sparse geometric structures into dense learning signals. While they have rarely been considered for self-supervised representation learning, we find that their dense and over-parameterized nature makes them uniquely well suited to our goal: boundary geometry can be bootstrapped from raw images during pretraining, with usable boundary fields emerging even from an untrained model.

We adopt a self-distillation formulation, but extend its teacher–student mechanism from semantic token learning to boundary-aware dense token learning. Starting from a randomly initialized network, the teacher generates online targets for both global visual semantics and sub-token boundary fields, while the student learns to recover them from masked views; through EMA updates, the teacher’s boundary perception co-evolves with the student. This creates a fundamental tension: semantic self-distillation may suppress fine-grained boundary signals, while boundary modeling may in turn disrupt the invariance needed for strong semantic representations. The core of our method is a design that turns this tension into cooperation.

1.0.0.1 Boundary-Forcing Masked Modeling.

Our key idea is to let the teacher’s own boundary predictions decide what the student must reconstruct. From the teacher’s online boundary fields we identify the boundary-bearing tokens and force them into the student’s masked set, so the student must recover boundary geometry purely from surrounding context. This turns boundaries, the most information-dense and least redundant regions of an image, into the hardest and most informative prediction targets. Crucially, the masked tokens are then routed by geometry: boundary tokens are supervised by the discovered boundary fields, while the remaining masked tokens follow the standard semantic self-distillation objective. Because a semantic target is inherently ambiguous exactly where two regions meet, this routing supplies a well-posed objective precisely where conventional masked modeling is weakest. It is what allows boundary-aware and semantic representations to co-emerge rather than compete, resolving the tension above.

1.0.0.2 Categorical Reparameterization of Boundary Fields.

A second ingredient makes dense boundary self-distillation stable. Directly regressing continuous boundary fields in a teacher–student loop is prone to collapse. We instead reparameterize each boundary field into a categorical distribution over discretized bins, recasting boundary prediction as a per-pixel classification problem. This brings two benefits. First, the boundary objective becomes compatible with the same centering and sharpening mechanisms that stabilize semantic self-distillation, so boundary learning inherits the scalability of modern SSL [2]. Second, it connects to classical a-contrario detection theory: under the no-structure null hypothesis, boundary orientations are uniformly distributed, so deviation from uniformity provides a principled, parameter-free measure of how strongly a token supports a real boundary, a validation signal obtained for free.

1.0.0.3 Scaling and Results.

We instantiate masked boundary modeling at scale as LingBot-Vision, a one-billion-parameter Vision Transformer trained purely with self-supervision. Across dense spatial tasks, LingBot-Vision matches or surpasses visual foundation models up to seven times larger: it attains the best NYU-Depth v2 accuracy of all compared models, including the 7B DINOv3, stays on par with distilled dense specialists on semantic and video object segmentation, and yields boundary tokens stable enough to be tracked through video by plain cosine similarity of the frozen features. These advantages survive distillation: the ViT-L, ViT-B and ViT-S students lead dense prediction in their size classes, and the 0.3B student matches the NYU-Depth v2 accuracy of the 7B DINOv3 with roughly \(23\times\) fewer parameters. Finally, we advance our depth completion system from LingBot-Depth to LingBot-Depth 2.0 by changing nothing in its recipe but the encoder initialization and the scale of the training data, obtaining leading performance on 14 depth completion benchmarks; notably, the advantage of the LingBot-Vision encoder widens as the downstream training data grows, so better pretraining compounds instead of washing out. Together, these results show that boundary modeling, long confined to standalone contour or line-segment detection, can serve as a general and scalable pretraining principle for learning spatially structured visual representations.

1.0.0.4 Contributions.

• We advocate a boundary-centric view of self-supervised pretraining, in which boundaries are native learning signals rather than downstream outputs, and show they can be bootstrapped from raw images without human labels, external edge detectors, or pretrained backbones.

• We propose masked boundary modeling, a self-distillation framework that forces teacher-discovered boundary tokens into the student’s masked set and routes masked tokens by geometry, turning the conflict between semantic abstraction and geometric sensitivity into cooperation.

• We introduce a categorical reparameterization of boundary fields that stabilizes dense self-distillation and yields parameter-free boundary validation at no extra cost.

• We scale the framework to LingBot-Vision, a 1B Vision Transformer that rivals or surpasses foundation models up to \(7\times\) larger on dense spatial perception, distill it into a strong ViT-L/B/S family, and deliver LingBot-Depth 2.0 with leading results on 14 depth completion benchmarks.

2 Related Work↩︎

In this section, we briefly revisit the development of representation learning for vision data with a focus on self-supervised pretraining, and then discuss boundary modeling in the literature and its connection to self-supervised vision pretraining.

2.1 Self-supervised Visual Representation Learning↩︎

Learning meaningful and versatile representations from visual data has long been a central topic in computer vision and machine learning. Self-supervised learning (SSL), one of the most promising directions, aims to learn representations from the data itself, without labels, and the last decade has witnessed its great success through contrastive learning [9], [10], masked image modeling [6], [11], [12], self-distillation [1]–[3], [13]–[15], and most recently the joint-embedding predictive architectures (JEPA) [16]–[18]. Over the years, the advances in visual SSL have been shifting from global to dense representations, moving toward larger-scale training, and seeking more principled ways to avoid collapse at scale.

2.1.0.1 Pretraining using Self-Distillation.

The self-distillation lineage currently dominates. DINO [1] trains a student to match the categorical token distributions of an EMA teacher and, notably, was among the first to show object layouts and boundaries emerging in attention maps without any supervision. iBOT [11] moved the same agreement objective down to masked tokens, DINOv2 [2] scaled it on curated data into a distilled model family, Web-SSL [13] showed that language-free pretraining keeps scaling on billions of web images, and Franca [14] reworked the clustering objective itself. The weakness of the lineage is equally consistent: because the objective rewards agreement rather than spatial structure, dense feature quality is fragile at scale, and DINOv3 [3] had to introduce Gram anchoring precisely to stop dense feature maps from degrading over long training schedules. Boundaries and layout emerge in these models as a byproduct, and keeping them intact requires ever more corrective machinery.

2.1.0.2 Masked Image Modeling.

The two other families occupy opposite ends of a spatial-semantic trade-off. Masked image modeling [6], [12], [19] reconstructs missing content against pixel-level targets, which makes it spatially faithful by construction but comparatively weak as a frozen semantic representation. Language supervision, from CLIP [4] to SigLIP 2 [20], is the mirror image: text describes whole images, so the learned features excel at recognition while underperforming on dense prediction, and even finer patch-text alignment [15] recovers only part of the gap. Neither extreme delivers representations that are simultaneously semantic and spatially precise.

2.1.0.3 Predictive Architectures and World Models.

JEPA methods [16], [17], [21] predict in latent space to avoid modeling pixel-level nuisance, and LeJEPA [22] recently grounded the family in a theoretically justified objective. Their ambition raises the stakes for spatial quality: frozen encoders are becoming the perception substrate of world models and robot policies [23], [24], where an error in geometry is no longer a lost benchmark point but a wrong action.

Across all of these families the same pattern recurs: dense spatial quality is obtained indirectly and then protected by retrofits, whether post-hoc feature upsamplers [25], corrective regularization such as Gram anchoring, or agglomerative distillation from several teachers [26], [27]. LingBot-Vision takes the trend of the field to its conclusion and makes dense spatial structure the objective of pretraining itself.

2.2 Boundary Structure Modeling of Images↩︎

Boundary structures, edges and contours are among the oldest formalizations of image content, capturing the spatial intensity changes of images, and have been studied and applied since the beginning of computer vision, well known through the operators of Sobel [28] and Canny [29] and their statistical and structured successors [30], [31]. The modern practice of boundary modeling rests on learning with deep neural networks, ranging from supervised to unsupervised and self-supervised learning [7], [8], [32]–[37]. Within this body of work, our pretraining inherits two ideas directly. The first is statistical: the a-contrario theory [38] declares a structure meaningful only if it would be unlikely under a structureless null model, which gave LSD [30] its parameter-free control of false detections. The second is representational: instead of classifying pixels as boundary or not [32], the attraction-field line [7], [8], [33] encodes each line segment redundantly into a dense field in which every nearby pixel stores enough geometry to reconstruct the entire segment, turning sparse detection into dense prediction [34], [35]; subsequent work carried this representation to self-supervised training at scale [36], [37]. Throughout this literature, the boundary is the product and the field is the means. Our work inverts the relationship: the attraction field, made categorical, becomes the medium of self-supervised pretraining; its uniform distribution doubles as the a-contrario null hypothesis; and detection-quality boundaries fall out as a byproduct of representation learning rather than being its goal.

3 Boundary-centric Masked Representation Modeling↩︎

Masked modeling has been a central pretraining paradigm in both natural language processing and computer vision over the past decade. In language, text is a sequence of discrete tokens, so masking a token and predicting it from context is a well-posed problem over a finite vocabulary, the formulation behind masked language models such as BERT (and, in autoregressive form, modern LLMs). Images admit no such natural tokenization: they are continuous, highly redundant arrays of pixels with no discrete units comparable to words. Masked modeling in vision therefore first imposes a tokenization, partitioning an \(H\times W\) image into a set of non-overlapping patch tokens via a patch embedding, and predicts masked tokens from the rest. Recent masked-modeling frameworks, including MAE [6], I-JEPA and V-JEPA [16], [17], and the iBOT objective within DINOv2 [2] and DINOv3 [3], all confront the same question of which tokens to mask and predict, and converge on the same answer: random masking.

Random masking, however, is content-agnostic: it hides tokens regardless of what they depict, even though tokens differ greatly in how much structure they carry. In this section we present boundary-centric masked modeling, which lets image structure, specifically its boundaries, decide which tokens to mask and what to reconstruct there. We build on the self-distillation paradigm of DINO [1] (recapped in Sec. 3.1) and augment it with a second, geometric channel of supervision that the teacher generates for itself. Figure 2 illustrates the masking and routing at its core; the subsections that follow develop each component.

3.1 Self-Distillation Vision Pretraining↩︎

Modern self-supervised pretraining of Vision Transformers (ViTs) follows a teacher–student self-distillation paradigm [1]–[3]. Let \(f_\theta\) be the student network and \(f_{\bar\theta}\) a teacher of identical architecture whose weights are an exponential moving average (EMA) of the student, \[\bar\theta \leftarrow \lambda\,\bar\theta + (1-\lambda)\,\theta,\] with momentum \(\lambda\) annealed toward \(1\) during training. Given an image \(\mathbf{x}\), a set of augmented views is sampled, typically two global crops and several lower-resolution local crops [1]. A ViT encodes each view into a class token \(\mathbf{z}^{\texttt{cls}}\) and a sequence of patch tokens \(\{\mathbf{z}^{\texttt{p}}_i\}_{i=1}^{N}\).

3.1.0.1 Image-level distillation (DINO).

A projection head \(h\) maps the class token to a distribution over \(C\) prototypes. The teacher distribution is centered and sharpened, and the student is trained to match it across views: \[\mathbf{p}^{\texttt{t}} = \mathrm{softmax}\!\left(\frac{h_{\bar\theta}(\mathbf{z}^{\texttt{cls}}) - \mathbf{c}}{\tau_t}\right), \qquad \mathbf{p}^{\texttt{s}} = \mathrm{softmax}\!\left(\frac{h_{\theta}(\mathbf{z}^{\texttt{cls}})}{\tau_s}\right),\] \[\mathcal{L}_{\texttt{DINO}} = -\,\mathbf{p}^{\texttt{t}}\!\big(\mathbf{x}^{(1)}\big)^{\!\top} \log \mathbf{p}^{\texttt{s}}\!\big(\mathbf{x}^{(2)}\big),\] summed over mismatched view pairs, with a stop-gradient on the teacher. The temperatures satisfy \(\tau_t < \tau_s\) (teacher sharpening), and the center \(\mathbf{c}\) (an EMA of teacher outputs, or equivalently a Sinkhorn–Knopp normalization [2]) prevents the trivial collapse to a constant distribution.

3.1.0.2 Patch-level distillation (iBOT).

To learn dense representations, a subset \(\mathcal{M}\subset\{1,\dots,N\}\) of the student’s patch tokens is masked and replaced by a shared mask token [11]. With a patch projection head \(g\), the student predicts a distribution at each masked location, distilled from the teacher’s distribution for the corresponding unmasked token: \[\mathcal{L}_{\texttt{iBOT}} = -\frac{1}{|\mathcal{M}|}\sum_{i\in\mathcal{M}} \mathbf{q}^{\texttt{t}}_i{}^{\!\top}\log \mathbf{q}^{\texttt{s}}_i, \qquad \mathbf{q}_i = \mathrm{softmax}\!\big(\frac{g(\mathbf{z}^{\texttt{p}}_i)-\mathbf{c}_{\rm iBOT}}{\tau}\big).\]

3.1.0.3 The mask is chosen at random.

A defining, and rarely questioned, property of this paradigm is that the masked set \(\mathcal{M}\) is sampled at random, independently of image content. Concretely, \(\mathcal{M}\) is drawn from a block-wise random masking distribution with a target ratio \(r\), \(\mathcal{M}\sim p_{\mathrm{mask}}(\cdot\,;r)\), so that every patch is equally likely to be hidden regardless of what it depicts. This content-agnostic choice treats all tokens as interchangeable reconstruction targets. Yet the difficulty and informativeness of recovering a token vary enormously across an image: a token in a flat, homogeneous region is largely predictable from its neighbors, whereas a token straddling a boundary carries irreducible structural information.

3.2 Boundary-Forcing Masked Modeling↩︎

Sec. 3.1 leaves one degree of freedom in masked self-distillation unused: which tokens to hide. Random masking treats all tokens alike, yet boundaries carry structural information that interiors do not. We close this gap by letting the self-distillation discover where boundaries lie and masking accordingly. Classical edge and line-segment detectors read boundaries out at the pixel level, typically with convolutional decoders. Because a Vision Transformer operates at the token level, we instead attach a boundary attribute to each token: a token is a boundary token if a line or curve predicted by the teacher passes through its \(P\times P\) patch (here \(P{=}16\)). How those lines are predicted and validated is the subject of Sec. 3.4.

3.2.0.1 Boundary-forcing mask and geometry routing.

Concretely, the teacher’s predicted boundaries, produced online without labels (Sec. 3.4), are rasterized to a pixel map. A token is a boundary token when a boundary falls within its patch, giving the set \[\mathcal{B} = \big\{\, i \in \{1,\dots,N\} : \text{a predicted boundary intersects } \mathrm{patch}(i) \,\big\},\] which we obtain by max-pooling the boundary map onto the token grid. We then force \(\mathcal{B}\) into the student’s masked set, on top of the random mask \(\mathcal{M}\) of Sec. 3.1: \[\mathcal{M}^{+} = \mathcal{M} \cup \mathcal{B}.\] Every masked token is supervised by the semantic iBOT objective \(\mathcal{L}_{\texttt{iBOT}}\) as in standard self-distillation; boundary tokens additionally receive the categorical boundary objective \(\mathcal{L}_{\texttt{bnd}}\) (Sec. 3.3), and the class token follows \(\mathcal{L}_{\texttt{DINO}}\) (Fig. 2). Boundary tokens thus carry a dual target: the usual semantic reconstruction, plus an explicit geometric one. Boundaries are the least redundant regions of an image: unlike an interior token, which its neighbors largely determine, a boundary token cannot be recovered by copying context. Forcing boundary tokens into the mask therefore hides exactly the information the student cannot infer for free, and compels it to reconstruct boundary geometry from the surrounding context.

3.2.0.2 Why the geometric target carries the gain.

The geometric objective complements the semantic one rather than replacing it. A semantic codeword encodes no explicit geometry and is weakest where two or more regions meet, so standard reconstruction under-supervises exactly the boundary tokens we force into the mask; the categorical geometric target supplies the structure it misses. Empirically the two are complementary, not competing: retaining the semantic objective on boundary tokens alongside the geometric one improves both classification and dense geometry over the geometric target alone (Sec. 3.6). The geometric target is nonetheless the active ingredient. Our early-stage experiments, which sourced boundaries from an off-the-shelf detector and reconstructed the forced tokens with the semantic objective alone, only matched the random-masking baseline (Sec. 3.6); and such a setup must rely on an external detector, because without learning the boundary geometry the teacher has no boundary signal of its own. Learning that geometry is thus what makes boundaries useful to the student: the mask decides where the model confronts structure and the categorical objective decides what it recovers there, two halves of one mechanism that let boundary-aware and semantic representations co-emerge.

3.2.0.3 A bootstrapping problem.

Boundary forcing presupposes that we already know where the boundaries are, yet a network trained from scratch does not: the very tokens we most want to mask are the ones it cannot yet locate. Resolving this chicken-and-egg problem requires clearing two obstacles. First, boundaries must be available from the very first step, before any boundary geometry has been learned. A lightweight corner-point detector makes this possible, thanks to a property of the dense boundary-field representation we build on (Sec. 3.3):

Corner points thus anchor usable boundaries from the outset (Fig. 3), and the fields sharpen as training proceeds (Sec. 3.4); we detail the sampling and decoding procedure in Appendix 8. Second, the boundary target must be learnable without collapsing, which motivates the categorical reparameterization of Sec. 3.3.

3.3 Reparameterization of Boundary/Line Fields↩︎

Finding 1 shows that the holistically-attracted field proposed in [8], hereafter simply the boundary field, turns even unlearned, largely random field values into line segments once corner points are given. Thus, if we let the neural network predict the field values rather than sample them at random, the decoded segments become image-grounded boundary predictions that improve as the field improves. This section makes the field concrete, explains why the naive way of learning it fails, and derives a categorical reparameterization that makes it learnable.

3.3.0.1 The boundary field in a nutshell.

We model image boundaries as a set of straight line segments \(L = \{\ell_j\}_{j=1}^{M}\), each given by its two endpoints \(\ell_j = (x_1, y_1, x_2, y_2)\); curved boundaries are represented as chains of short segments. The boundary field lifts the sparse set \(L\) into a dense map (Fig. 4): every pixel \(p\) near a segment stores a small attribute vector \[\mathbf{a}(p) = \big(d_p,\; \theta_p,\; \phi^{1}_p,\; \phi^{2}_p\big),\] recording its distance \(d_p\) to the nearest segment, the direction \(\theta_p\) toward it, and two angles \(\phi^{1}_p, \phi^{2}_p\) that locate the segment’s endpoints as seen from \(p\). The parameterization is deliberately redundant: any single pixel in a segment’s support region \(S_\ell = \{p : d_p \le \tau_d\}\) carries enough information to reconstruct the entire segment, and conversely \(L\) is decoded from the field by letting all support pixels vote and aggregating the votes. This many-pixels-one-segment redundancy is what made the decoding of unlearned fields in Finding 1 possible, and what later keeps decoding robust to noisy predictions (Sec. 3.4).

3.3.0.2 Why line segments rather than edge maps?

An edge map, which marks photometric discontinuity pixel by pixel, may seem the more general choice. The trouble is that edge pixels can hardly be validated: each is an isolated filter response, accepted or rejected by an arbitrary threshold, with no statistic to test, and texture fires such responses as readily as true structure. A line segment, by contrast, is a single hypothesis supported by its many support pixels, so it can be tested against the no-structure null hypothesis in which orientations are uniformly distributed, and accepted only when chance cannot explain its support (Appendix 9). This difference is decisive in self-distillation, where the teacher’s own predictions become the student’s targets: unvalidated edges would feed hallucinated structure back into training, whereas validated segments keep the bootstrapped targets clean (Sec. 3.4). Segments moreover carry endpoints and orientation, which is precisely what lets corner points anchor them (Finding 1).

3.3.0.3 Why categorical reparameterization?

The field is a continuous, vector-valued map, and the obvious way to distill it (regressing \(\mathbf{a}(p)\) under an \(\ell_1\) or \(\ell_2\) loss) fails. In a teacher–student loop with an EMA teacher, continuous regression targets drift and collapse; and, unlike the semantic branch, they cannot exploit the centering and sharpening that stabilize self-distillation (Sec. 3.1). We remove this asymmetry by turning boundary prediction into classification.

3.3.0.4 From regression to per-pixel classification.

We discretize the range of each field channel into \(K\) bins and represent a scalar value as a distribution over them. For a boundary position \(p\) and channel \(c\in\{d,\theta,\phi^{1},\phi^{2}\}\), the (validated) teacher value \(a^{c}(p)\) is encoded as a soft categorical label \[\bar{y}^{c}_{k}(p) \;\propto\; \exp\!\Big(\!-\,\delta^{c}\big(k,\,a^{c}(p)\big)^{2} \big/ \tau_\ell\Big), \qquad k = 1,\dots,K,\] where \(\delta^{c}\) is the distance from the center of bin \(k\) to the value \(a^{c}(p)\) and \(\tau_\ell\) a label temperature. For the orientation channel \(\theta\), whose range wraps at \(2\pi\), the bins are circular and \(\delta^{\theta}\) is an arc distance. The label is deliberately narrow (a few bins wide): over-smoothing pushes every target toward the uniform distribution and erases the signal. The student’s boundary head predicts a distribution \(\hat{y}^{c}(p)\) per channel, and the boundary objective is a cross-entropy over the boundary positions \(\mathcal{B}\) selected by the forcing mask (Sec. 3.2), \[\mathcal{L}_{\texttt{bnd}} = -\frac{1}{|\mathcal{B}|}\sum_{p\in\mathcal{B}}\;\sum_{c}\; \bar{y}^{c}(p)^{\!\top} \log \hat{y}^{c}(p).\]

3.3.0.5 Inheriting the stability of self-distillation.

Because boundaries are now categorical, the machinery that keeps semantic self-distillation from collapsing applies unchanged: the teacher distributions are centered (or Sinkhorn–Knopp normalized) and sharpened before they supervise the student, exactly as in Sec. 3.1. This is precisely what the continuous formulation could not do, and it is why boundary learning inherits the stability and scalability of modern SSL instead of requiring bespoke regularizers.

3.3.0.6 A native null hypothesis, and free validation.

The categorical form also connects the learning target to classical a-contrario detection theory [30], [38], which validates a candidate segment by asking how unlikely its support would be under a null model containing no structure, one in which boundary orientations are uniformly distributed. That null is now literally the uniform distribution over bins. “No boundary” is thus a specific, representable prediction, and any departure from uniform is exactly the evidence that the Number-of-False-Alarms (NFA) test measures (Appendix 9). Two consequences follow at no cost: non-boundary regions need no special background class, since their target is simply uniform; and the same distributions that supervise the student can be scored for statistical significance, so boundary validation is native to the representation rather than a separate post-hoc step.

3.4 Online Generation of Boundary Targets↩︎

As the ViT is initialized from scratch, it carries no information to produce valid boundary targets, yet self-distillation needs a target from the very first step, and we have no boundary annotations to fall back on. Our targets are instead produced by the teacher itself, online, and refined as training proceeds. What makes this possible is the redundancy of the boundary field (Sec. 3.3): because each segment is over-determined by the many pixels in its support region, coherent segments can be read out of the field even when the field is far from accurate.

3.4.0.1 Boundaries emerge before they are learned.

Finding 1 already exhibited the extreme case: with corner points fixed, even field values carrying no learned information decode into plausible segments (Fig. 5). The reason is the voting redundancy above: every support pixel casts a vote for the same underlying segment, and aggregating many weak votes is robust to noise, so the decoded geometry is meaningful long before the field is pixel-accurate. This is the seed that lets boundary perception bootstrap from raw images: the teacher need not be a good boundary predictor to provide a usable signal; it only needs to be better than the student, which the EMA update guarantees.

3.4.0.2 A token-level head for a dense field.

The boundary field lives at sub-token resolution, whereas the backbone emits one feature per \(P\times P\) token; a lightweight boundary head bridges this gap without any convolution. Each patch token is processed independently by a small MLP, and its output is expanded and rearranged into an \(r\times r\) tile of positions at stride \(s = P/r\) (we use \(s{=}2\)), so a token literally unfolds into the dense positions it covers. Each position’s feature is then \(\ell_2\)-normalized onto the unit sphere and scored, per field channel, against \(K\) learned bin prototypes: the bin classifiers are bias-free linear layers with unit-normalized weights, so every logit is a cosine similarity and is bounded by construction. This prototype design mirrors the DINO projection head and complements the centering of Sec. 3.3 in preventing collapse. The teacher and the student share this head design, the teacher’s weights being the EMA copy. Finally, continuous field values are read out of the categorical predictions by taking the expectation over bin centers per channel (the circular mean for the orientation channel), so the discrete bins still yield sub-bin.

3.4.0.3 From a view to a validated target.

At each iteration, the teacher \(f_{\bar\theta}\) turns a global view into a dense target field in four steps (Fig. 5): (i) it predicts a dense boundary field over the view; (ii) it pairs the field with a sparse set of corner points localized by a frozen, single-block Vision Transformer; (iii) combining corner points with the field, it decodes a set of candidate segments \(L\) by vote aggregation (Sec. 3.3); and (iv) it discards segments that fail the a-contrario test (Appendix 9) and re-renders the survivors into a clean, validated target field \(\mathbf{a}(\cdot)\). Only this re-rendered field, not the raw prediction, supervises the student, so unsupported structure never becomes a teaching signal. Because the field is regenerated every step from the EMA teacher, the targets co-evolve with the model: as the student sharpens, so does the teacher, and the boundary targets it emits. Fig. 11 in Appendix 9 ablates the two safeguards of this pipeline on a trained model: reliable corner anchors keep the decoding clean even without validation, and the a-contrario test recovers nearly identical line segments even from weak, field-derived anchors; the decoding degenerates only when both are absent.

3.4.0.4 No boundary annotations in the loop.

Throughout, the representation backbone is trained from random initialization, with no human boundary annotation, edge detector, depth sensor, or image-classification pretraining in the loop; the boundary fields, the learned target of our method, emerge entirely from raw images. The only fixed component is the single-block Vision Transformer that seeds the corner points in step (ii): it localizes only a sparse set of them, is orders of magnitude smaller than the backbone, and never observes the boundary fields it helps decode. We therefore reserve all representational capacity for learning boundary structure from scratch, rather than importing it from a pretrained model.

3.5 Training Objective↩︎

The full objective assembles the pieces of Secs. 3.1–3.3. For each image we sample global and local views; the student processes the global views under the boundary-forced mask \(\mathcal{M}^{+}\) (Sec. 3.2) and is trained to minimize \[\mathcal{L} \;=\; \mathcal{L}_{\texttt{DINO}} \;+\; \lambda_{\texttt{i}}\,\mathcal{L}_{\texttt{iBOT}} \;+\; \lambda_{\texttt{b}}\,\mathcal{L}_{\texttt{bnd}} \;+\; \lambda_{\texttt{k}}\,\mathcal{L}_{\texttt{KoLeo}}, \label{eq:total}\tag{1}\] where \(\mathcal{L}_{\texttt{DINO}}\) distills the class token across view pairs, \(\mathcal{L}_{\texttt{iBOT}}\) is computed over all masked tokens in \(\mathcal{M}^{+}\), and \(\mathcal{L}_{\texttt{bnd}}\) is computed over the boundary positions of \(\mathcal{B}\subset\mathcal{M}^{+}\), so boundary tokens receive both losses, realizing the dual supervision of Sec. 3.2. The last term is the KoLeo regularizer [2], which spreads class-token features within a batch. Loss weights and schedules are deferred to Sec. 4.3.

All teacher quantities, the semantic distributions and the validated boundary targets alike, are produced under stop-gradient, so gradients flow only through the student; after each optimizer step the teacher is updated as the EMA of the student with momentum annealed toward one (Sec. 3.1), and the targets of every branch co-evolve smoothly with the model. Boundary targets are generated for the global views, whose resolution suffices for reliable decoding and validation; local views are supervised by the image-level distillation only. Together with the centering and sharpening applied to both the semantic and the boundary distributions, this is all that is needed to keep training stable: no bespoke regularizer is attached to the boundary branch.

3.6 Proof of Concept on ImageNet-1K↩︎

Before scaling up (Sec. 4), we validate each design choice at small scale: ViT-L/16 pretrained on ImageNet-1K [39], compared against a DINO+iBOT baseline trained under an identical configuration. We probe the two facets the method is supposed to balance: \(k\)-NN classification on ImageNet-1K for global semantics, and linear-probe depth estimation on NYU-Depth v2 [40] for dense geometry (Table 1).

3.6.0.1 Main comparison.

The full recipe improves both facets at once: over the matched DINO+iBOT baseline, ImageNet-1K \(k\)-NN top-1 rises from 81.6% to 82.4%, while NYUv2 \(\delta_1\) improves from 81.4% to 84.9% and RMSE drops from 0.474 to 0.440. Boundary supervision therefore does not trade global semantics for dense geometry: the two co-emerge, as promised in Sec. 3.2, and the semantic representation itself benefits from confronting structure.

3.6.0.2 Design ablations.

The ladder of Table 1 attributes the gains. The categorical boundary target is the active ingredient: adding it alone accounts for essentially the entire dense improvement (+3.0 points of \(\delta_1\), RMSE from 0.474 to 0.446) at no cost to classification. Dual supervision then adds +0.2 points of \(k\)-NN and +0.3 points of \(\delta_1\) (RMSE 0.446 to 0.443), confirming that the semantic and geometric objectives are complementary rather than competing (Sec. 3.2). The RoPE backbone contributes a further +0.4 on \(k\)-NN and +0.2 on \(\delta_1\); this is a backbone modernization orthogonal to our method, which we nonetheless carry into the scaled recipe (Sec. 4). Conversely, forcing boundary tokens into the mask while reconstructing them with the semantic target only performs on par with, indeed slightly below, the baseline (\(\delta_1\) 81.2%, RMSE 0.481), reproducing our early-stage observation under the final pipeline: the mask decides where, but the gain comes from what is reconstructed there.

3.6.0.3 What transfers to scale.

This study freezes the method configuration that Sec. 4 scales unchanged: boundary-forced masking with dual supervision, the categorical boundary target, and the RoPE backbone. Everything that changes from here on, namely the corpus, the model capacity, and the training systems, is external to the method.

4 LingBot-Vision: Pushing the Boundaries in Vision Pretraining↩︎

Based on the promising results in Table 1, we push boundary-centric masked representation learning toward scalable training for large vision foundation models. The method itself is kept exactly as in Sec. 3; what scaling changes is everything around it. Following the data curation pipelines of DINOv2 and DINOv3 [2], [3], we assemble a corpus of roughly 161M images, selected from 2B raw ones, that combines curated public datasets with retrieval-curated web pools (Sec. 4.1). On this corpus we train LingBot-Vision, a ViT-g/16 backbone of roughly 1.1 billion parameters (Sec. [sec:scaling:model]), supported by the distributed systems that keep the boundary branch a small fraction of the step time (Sec. 4.2) and an optimization recipe that keeps training stable at scale (Sec. 4.3). We furthermore find LingBot-Vision to be training-efficient in terms of both data and optimization budget: it consumes a corpus an order of magnitude smaller than DINOv3’s and less than one third of its training samples (Secs. 4.1 and 4.3).

4.1 Training Data Curation↩︎

Our data curation pipeline follows DINOv2 [2], implemented on our own data sources. The uncurated side is an in-house pool of roughly 2 billion web images; the curated side uses open-source datasets as seeds for query-based retrieval from that pool, with image embeddings computed by a pretrained DINOv2 ViT-B encoder [2]. The seeds include ImageNet-1k and ImageNet-21k [39], [41], Google Landmarks v2 (GLDv2) [42], and Mapillary-SLS [43], together with twenty further datasets spanning fine-grained recognition, instance retrieval, and dense scene understanding (e.g., ADE20K [44], NYU-Depth v2 [40], SUN-RGBD [45], Cityscapes [46], VOC [47]), chosen to cover a wide range of downstream tasks.

4.1.0.1 From 2B raw images to a 161M corpus.

Each seed contributes in one of two ways, mirroring DINOv2: it is either included as is, or it queries the uncurated pool for visually similar images (\(k\)-NN retrieval for large seeds, cluster-based retrieval for small ones). After retrieval and deduplication, the retrieved portion is materialized as three pools totaling roughly 143M images; the as-is portion comprises ImageNet-21k (13.15M), ImageNet-1k (1.28M), the clean split of GLDv2 (1.58M), and Mapillary-SLS (1.46M). Combined, the corpus holds 160.75M images, of which about five sixths are retrieval-curated. During training, the whole curated corpus is sampled with a single global uniform shuffle, so each image is drawn with equal probability.

4.1.0.2 Comparison with LVD-142M and LVD-1689M.

Our corpus is comparable in scale to DINOv2’s LVD-142M (142M images curated from a 1.2B pool) [2] and an order of magnitude smaller than DINOv3’s LVD-1689M (1,689M images curated from a pool of roughly 17B) [3]. While we tried our best to curate data, our pipeline is also deliberately simpler than theirs: a single retrieval encoder over a 2B pool, without the concept-rebalancing clustering of LVD-142M or the hierarchical curation of LVD-1689M. That LingBot-Vision nonetheless matches or surpasses models trained on these corpora on dense tasks (Sec. 5) suggests that the gains come from the pretraining objective rather than from a data advantage, and substantiates the training-efficiency claim above.

4.2 Distributed Training and Efficiency↩︎

On the systems side we follow standard practice, sharding the model with FSDP under bf16 mixed precision with memory-efficient attention, and we store the corpus as uncompressed Arrow shards that are memory-mapped for zero-copy random access during loading; none of this is specific to our method. What is specific is the boundary branch, which adds a dense \(K\)-bin prediction and an online target-generation pipeline on top of standard self-distillation. We describe how both are kept cheap.

4.2.0.1 Sparse and fused boundary computation.

The boundary head never runs densely. Only the boundary tokens \(\mathcal{B}\), a small fraction of all tokens per batch, are unfolded to sub-token resolution and scored against the bin prototypes, so the head’s cost scales with the number of boundary tokens rather than with the image size. Naively materializing the soft labels and the cross-entropy over four channels and \(K\) bins per position would dominate memory traffic; we instead fuse label construction (including the circular arc-distance channel) and the categorical cross-entropy into single custom kernels, and compile the remaining head operations, which removes the multi-gigabyte intermediate tensors entirely.

4.2.0.2 Online target generation as batched GPU work.

The teacher-side pipeline of Sec. 3.4, namely field decoding, corner-line pairing, the a-contrario test, and the re-rendering of validated segments, is implemented end-to-end as batched CUDA kernels, with no per-image loops and no host synchronization. Two stages required particular care. First, corner-line pairing computes the distance between every corner point and the two endpoints of every segment proposed by the boundary field; materializing this pairwise distance tensor on the GPU is prohibitively memory-intensive, so we fuse the distance computation with the argmin reduction, keeping memory linear in the number of proposed segments instead of quadratic. Second, the a-contrario test of LSD [30] is a purely sequential CPU design; we reformulate it as a per-candidate reduction over support pixels, so that all candidate segments of a batch are validated in parallel.

An earlier implementation performed segment de-duplication on the CPU and stalled every training step; eliminating all host round-trips was essential for the target generation to overlap cleanly with the rest of the step. In the final recipe, generating validated boundary targets adds only a small fraction of the total step time, and end-to-end training throughput stays close to that of the semantic-only baseline under an identical configuration.

4.3 Implementation Details↩︎

Training proceeds in three stages, following the staging of DINOv3 [3]: a long self-distillation pretraining, a Gram-anchoring phase that restores dense-feature quality late in training, and a short high-resolution adaptation. Our budget is deliberately modest. Where DINOv3 spends 1M iterations on low-resolution pretraining, 100k on Gram anchoring, and 30k on multi-resolution adaptation at a global batch size of 4,096, LingBot-Vision spends 300k, 100k, and 100k iterations on the three stages at a global batch size of 3,072; in total, less than one third of the samples DINOv3 processes. To further save compute, our high-resolution adaptation uses a single resolution of \(512\) px rather than a multi-resolution schedule.

4.3.0.1 Backbone and heads.

LingBot-Vision uses a ViT-g/16 backbone of roughly 1.1B parameters with SwiGLU feed-forward layers, rotary position embeddings computed in fp32, and four register tokens; the proof-of-concept study (Sec. 3.6) uses ViT-L/16. The boundary head follows Sec. 3.4: a three-layer per-token MLP, a head dimension of 512 at output stride \(s{=}2\), a learnable tile positional embedding, and \(K{=}32\) bins per field channel. The DINO and iBOT projection heads follow DINOv2 [2], with a separate iBOT head. The frozen corner-point detector is a single-block ViT, orders of magnitude smaller than the backbone.

4.3.0.2 Pretraining.

We pretrain for 300k iterations with AdamW at a global batch size of 3072, scaling the base learning rate with the square-root rule \(\sqrt{\mathrm{bs}/1024}\) and decaying it with a cosine schedule after a linear warmup. Weight decay ramps from 0.04 to 0.2 with a cosine schedule, the teacher temperature ramps from 0.04 to 0.07 in the first 30k iterations, and the EMA teacher momentum is annealed from 0.994 to 1.0 over the course of training. The objectives of Eq. 1 are weighted equally (\(\lambda_{\texttt{i}} = \lambda_{\texttt{b}} = 1\)), with the KoLeo weight \(\lambda_{\texttt{k}} = 0.1\), and the boundary branch uses the narrow soft-label temperature of Sec. 3.3. Two stability practices complete the branch: teacher targets at non-boundary positions are filled with uniform random labels rather than a constant background value, which removes the trivial all-background solution, and teacher-side centering is kept on throughout. Compared with the proof-of-concept configuration, scaling required only mild retuning: a higher initial EMA momentum, a lower weight-decay ceiling, fewer bins (\(K\) from 128 to 32, verified to be quality-neutral), and, following DINOv3, plain rather than weight-normalized prototype layers in all heads. During pretraining, global and local crops are sized \(256\) and \(112\) px, respectively.

4.3.0.3 Gram anchoring and high-resolution adaptation.

Over long schedules, patch-level feature quality is known to degrade even as global metrics keep improving; DINOv3 counteracts this by anchoring the Gram matrix of the student’s patch features to that of an earlier teacher snapshot [3]. We adopt this Gram-anchoring phase for LingBot-Vision, running it for 100k iterations after pretraining. Training then concludes with a 100k-iteration adaptation phase at a single resolution of \(512\) px, with a lowered learning rate and a higher EMA momentum, which adapts the backbone (and its rotary embeddings) to high-resolution inputs while the boundary branch remains active.

4.3.0.4 Model distillation for efficiency.

After the flagship ViT-g is trained, we distill it into smaller backbones, obtaining a 300M ViT-L, an 86M ViT-B and a 21M ViT-S. Following the distillation practice of DINOv2 and DINOv3 [2], [3], the frozen ViT-g replaces the EMA teacher in the same training pipeline, so the smaller students inherit both the semantic and the boundary-aware representations of the flagship. Distillation uses the same iteration budget as the pretraining stage, namely 300k iterations per student; we skip the Gram-anchoring phase, which counteracts a drift that does not arise with a frozen teacher, and apply the high-resolution adaptation at \(512\) px for an additional 100k iterations, all at a global batch size of 3,072. The resulting schedule is much shorter than DINOv3’s distillation schedule of 1M+250k iterations at a global batch size of 4,096.

5 Results↩︎

In this section, we evaluate our pretrained vision foundation model, LingBot-Vision, on various downstream tasks, following the standard protocol of probing frozen features: linear probing on dense prediction and classification tasks, and non-parametric probing (\(k\)-NN retrieval and label propagation) on global and video tasks. The promising performance on public benchmarks against the leading vision foundation models (e.g., DINOv2 [2], DINOv3 [3], SigLIP 2 [20] and V-JEPA 2.1 [18]) lays a solid foundation for spatial understanding, and culminates in LingBot-Depth 2.0 (Sec. 6).

5.1 Robust Dense Representation of Images↩︎

5.1.1 PCA Components of Patch Features↩︎

Before any probing, the structure of the frozen features is already visible to the naked eye. Fig. 6 maps the top three PCA components of the patch features to RGB for LingBot-Vision and the leading vision foundation models, computed per image. Across scenes, LingBot-Vision resolves objects as coherent, sharply bounded regions: individual cars and the lane layout of the highway, hen silhouettes cleanly separated from the wire fence, the winding contour of the snake, and fine structures such as flower stalks around the marmot. The comparison also exposes characteristic failure modes of the baselines: DINOv2 features carry per-token speckle, SigLIP 2 collapses into blocky noise away from salient objects, and V-JEPA 2.1 lets background texture bleed into foreground regions. These qualitative differences anticipate the quantitative gaps below: boundary-faithful features are precisely what pixel-dense readouts consume.

5.1.2 Linear Probing on Depth Estimation↩︎

5.1.2.1 Evaluation protocol.

We follow the standard linear probing protocol: the ViT backbone is frozen, and only the decoder is trained on the training split of each target dataset. Instead of practices that add decoding capacity (e.g., a DPT head [49] or multi-layer feature aggregation), we use a single linear layer on the frozen patch tokens as the decoder for all pretrained vision transformers, so the comparison attributes performance to the features rather than to the readout. NYUv2 [40] and KITTI [50] are evaluated at their default resolutions and reported in RMSE.

5.1.2.2 Results.

Table 2 groups the competitors by parameter count. With a 1B-parameter ViT-g/16 backbone, LingBot-Vision attains the best NYUv2 RMSE of the entire table (0.296), ahead of the 7B-parameter DINOv3 (0.309) and the 2B-parameter V-JEPA 2.1 (0.307), with 7\(\times\) and 2\(\times\) fewer parameters respectively. The margin over models at comparable scale is substantial: 13% lower RMSE than the best same-scale competitor AM-RADIOv2.5 (0.340), 16% lower than DINOv3 ViT-H+ (0.352), which is distilled from the 7B teacher, and 20% lower than DINOv2 (0.372). The strongest video-pretrained baseline at equal size, V-JEPA 2.1 ViT-g (0.350), remains 15% behind. On KITTI, LingBot-Vision obtains the best RMSE among models below 2B parameters (2.552), surpassed only by the 7B DINOv3 (2.346) and the 2B V-JEPA 2.1 (2.461). Notably, LingBot-Vision achieves these results with patch size 16, a coarser token grid than the patch-14 competitors at equal input resolution.

5.1.2.3 Analysis.

A linear decoder cannot compensate for deficiencies of the representation; it can only read out what the frozen features already encode. Accurate depth under this protocol therefore requires features that are smooth within a surface and change sharply across occlusion boundaries, exactly where depth discontinuities concentrate. This is the regime that the boundary-oriented pretraining of LingBot-Vision targets, and it is visible in Fig. 6: object interiors map to coherent regions while transitions stay crisp. Depth is consequently the task where LingBot-Vision holds the largest margin, and the margin is wider on NYUv2 than on KITTI, where the spread across all backbones compresses: outdoor driving scenes are dominated by road and sky regions with few nearby occlusion boundaries, so boundary fidelity contributes less of the error budget.

5.1.3 Linear Probing on Semantic Segmentation↩︎

5.1.3.1 Evaluation protocol.

The same single-linear-layer decoder is trained per dataset on the frozen patch tokens. Following the DINOv3 protocol, ADE20K [44] and VOC12 [47] are evaluated at \(448\times448\) for models with patch size 14 and \(512\times512\) for patch size 16, so that all models produce the same patch-token resolution; Cityscapes [46] uses its default evaluation resolution. We report the mean Intersection-over-Union (mIoU) against the ground-truth annotations.

5.1.3.2 Results.

On ADE20K, Cityscapes and VOC12 (Table 2), LingBot-Vision is on par with the distilled DINOv3 ViT-H+: it trails by 1.3 mIoU on ADE20K (53.5 vs. 54.8) while matching it on Cityscapes and leading on VOC12. Against the standard self-supervised reference at equal size, DINOv2, the improvement is a consistent 4 mIoU or more on all three benchmarks (e.g., 53.5 vs. 49.5 on ADE20K), and the equal-size video-pretrained V-JEPA 2.1 ViT-g trails by 5.7 mIoU on ADE20K. LingBot-Vision also surpasses AM-RADIOv2.5 on all three benchmarks, although the latter agglomerates several teachers including segmentation-specialized ones, and it exceeds the spatially post-trained PEspatial by 4.2 mIoU on ADE20K. The remaining gap is only to the DINOv3 family (2.4 mIoU on ADE20K to the 7B model), whose members benefit from distillation and dedicated dense-feature objectives. We emphasize that LingBot-Vision is pretrained from scratch with a single self-supervised objective, whereas the strongest dense baselines rely on distillation from a 7B teacher (DINOv3 ViT-H+) or task-oriented post-training (PEspatial).

5.1.3.3 Analysis.

Segmentation rewards two properties jointly: patches of the same semantic region must map to nearby features, and the feature transition must land exactly on the class boundary, since mIoU errors concentrate along contours and thin structures. The PCA maps of Fig. 6 show that LingBot-Vision provides both, with coherent object interiors and crisp transitions. The comparison also separates objective families: language-aligned pretraining (SigLIP 2, 42.7 mIoU on ADE20K) optimizes image-level alignment at the expense of spatial detail, and video-predictive pretraining recovers dense structure only after its refinements (V-JEPA 2 to V-JEPA 2.1, 24.4 to 47.8 mIoU). Boundary-oriented image pretraining reaches distillation-level segmentation quality without a larger teacher.

5.2 Video Understanding↩︎

We assess the temporal consistency of the frozen features in two ways: quantitatively, with training-free video object segmentation on standard benchmarks, and qualitatively, by tracking individual boundary tokens through self-captured videos and visualizing the similarity responses of the queried tokens over time.

5.2.1 Video Object Segmentation↩︎

5.2.1.1 Evaluation protocol.

We evaluate training-free video object segmentation on DAVIS-2017 [51] and YouTube-VOS [52]: the ground-truth mask of the first frame is propagated to subsequent frames by top-\(k\) attention over the frozen patch features, without any fine-tuning. Videos are resized so that the shorter side is 480 px for models with patch size 16 and 420 px for those with patch size 14, and we report the \(\mathcal{J}\&\mathcal{F}\)-Mean.

5.2.1.2 Results.

As reported in Table 3, LingBot-Vision reaches 70.0 and 73.5 \(\mathcal{J}\&\mathcal{F}\) on DAVIS and YouTube-VOS respectively, on par with DINOv3 ViT-H+ (71.1 and 74.0) and the 7B DINOv3 (71.1 and 74.1), and the best among all remaining models at any scale. At equal size, the margin over DINOv2 is 6.1 and 7.9 points, and the video-pretrained V-JEPA 2.1 ViT-g trails by 1.9 and 1.2 points despite being trained on video.

5.2.2 Boundary Token Tracking↩︎

Beyond aggregate benchmarks, we probe temporal consistency at the level of individual tokens. Fig. 7 tracks three boundary-token queries through three videos of different nature: a manipulation episode seen from a robot’s head camera, a self-captured cat video, and a camera pan through a home. At every frame, each query is matched by cosine similarity of the frozen patch features over the boundary tokens of that frame, within a local window around its previous position, and the query feature is updated from its best match while staying anchored to its first-frame template, so the tracker follows appearance changes without drifting. The queries stay locked in all three regimes: on the robot, the gripper fingers and the manipulated pitcher are held through the whole episode; on the cat, the head and flank queries follow a full turn of the body while the query on the static window frame stays put; and under the camera pan, the queries survive a large viewpoint change and re-acquire the plant when it re-enters the view. Throughout, the similarity responses remain selective rather than diffusing over the scene. Boundary tokens thus behave as stable, trackable entities in the feature space of LingBot-Vision, obtained without any temporal supervision.

5.3 Strong Distilled Models on both Global and Dense Tasks↩︎

We complete the picture with image-level recognition, evaluating both the giant teacher and the distilled LingBot-Vision family, since classification is also the axis along which we distill.

5.3.1 Linear and \(k\)-NN Probing on ImageNet↩︎

5.3.1.1 Evaluation protocol.

We evaluate ImageNet-1K [39] classification on frozen features with two readouts: a linear classifier trained on the training split, and a training-free \(k\)-NN classifier. Evaluation is token-aligned: input resolutions are set per patch size so that all models produce the same number of patch tokens, consistent with the dense protocols above. The linear probe follows the standard grid-search practice: SGD with a momentum of 0.9, trained for 10 epochs with a batch size of 1024, with the learning rate swept over \(\{1, 2, 5\} \times 10^{k}\) for \(k \in \{-4, \dots, 0\}\) and the weight decay over \(\{0, 10^{-5}\}\), selecting the best combination on the ImageNet-1K validation set; training uses Inception-style random resized crops, and evaluation uses \(224\times224\) images.

5.3.1.2 Results.

At giant scale (Table 4), LingBot-Vision obtains 86.32 (linear) and 83.39 (\(k\)-NN) top-1 accuracy, on par with DINOv2’s flagship model (87.00 and 83.68) and behind DINOv3-7B (87.87 and 85.68) and SigLIP 2 (87.33 and 84.75). The remaining gap to DINOv3-7B thus concentrates on image-level recognition, while the dense comparisons above favor LingBot-Vision: a trade-off consistent with our boundary-oriented pretraining, which invests model capacity in localized structure rather than image-level invariances.

5.3.2 Distilled Models Across Scales↩︎

5.3.2.1 Setup.

To serve different deployment budgets, we distill the giant LingBot-Vision teacher into ViT-L, ViT-B and ViT-S students, and probe them with the same global and dense protocols (Table 5).

5.3.2.2 Results.

The distilled family preserves the profile of the teacher at every size. At ViT-L, LingBot-Vision matches DINOv2 on global accuracy (86.38 vs. 86.43 linear) while reducing the NYUv2 RMSE by 25% (0.310 vs. 0.411); against DINOv3-L it concedes 0.9 points of linear accuracy but leads depth by 12% on NYUv2 and stays within 0.5 to 2.3 mIoU across the segmentation benchmarks. Notably, the 0.3B student attains a NYUv2 RMSE on par with the 7B DINOv3 of Table 2 (0.310 vs. 0.309) under the same protocol, with roughly 23\(\times\) fewer parameters. At ViT-B, LingBot-Vision obtains the best linear accuracy in its size class (85.05) together with the best depth on both datasets (0.339 NYUv2, 2.793 KITTI), trailing DINOv3-B by at most 0.3 mIoU on ADE20K and Cityscapes. At ViT-S, it again leads linear probing (82.22), NYUv2 (0.383) and ADE20K (47.01); the smallest student stays within 0.4 points on \(k\)-NN and VOC but falls behind on KITTI and Cityscapes. Across sizes, the video-pretrained V-JEPA2.1 trails on every dense benchmark, SigLIP 2 remains close on classification but far behind on dense tasks, and Franca trails on both axes.

5.3.2.3 Analysis.

Two observations stand out. First, the dense advantage transfers through distillation essentially intact: the students inherit the boundary-faithful patch features that make the teacher strong, while the residual gaps to DINOv3 concentrate on image-level accuracy, mirroring the giant-scale comparison above. Second, the family forms a practical accuracy-compute frontier: at every deployment budget, the LingBot-Vision student offers the best available depth accuracy in its class with classification within a point of the best competitor, so downstream users do not trade dense quality for model size.

The frozen-feature results above, together with the encoder-initialization study of Table 6, establish LingBot-Vision as a strong spatial backbone. Sec. 6 builds on it to develop LingBot-Depth 2.0 for metric depth completion.

6 Meet LingBot-Depth 2.0↩︎

Based on the success of LingBot-Vision on general vision pretraining, it is time to upgrade our previous LingBot-Depth [53] for depth completion, a robotics-oriented perception task, with LingBot-Vision as its spatial-perception-native backbone. To be self-contained, we first briefly recap how LingBot-Depth was trained in its previous version, and then describe our tweaks toward stronger performance. At a glance, LingBot-Depth 2.0 comes in two sizes, built on the ViT-L/16 and ViT-g/16 backbones from our LingBot-Vision pretraining, and we explore data scaling by growing the training set from the publicly released 3M samples to a newly curated 150M collection, setting the leading performance on 14 benchmarks spanning different depth patterns and camera types.

6.1 A Recap of LingBot-Depth 1.0↩︎

6.1.0.1 Masked depth modeling.

LingBot-Depth formulates depth completion as masked depth modeling (MDM), an RGB-D variant of masked autoencoding [6]. The key observation is that the missing measurements of commodity depth cameras are not nuisance noise but a natural masking signal: they concentrate exactly where imaging is hard, on specular, transparent and textureless surfaces, and thus mark the regions where geometry must be inferred from visual context. Accordingly, the RGB frame and the raw depth map are patchified by two separate embedding layers onto a shared token grid, and each token receives a shared spatial positional embedding plus a modality embedding that distinguishes the two sources. Depth tokens are masked according to sensor validity: fully invalid patches are always masked, partially valid patches are masked with high probability, and fully valid tokens are randomly added until the target masking ratio of 60% to 90% is reached. A ViT-L encoder, initialized from DINOv2 [2], jointly embeds the full set of RGB tokens and the unmasked depth tokens; the decoder discards the latent depth tokens and reconstructs the full-resolution depth map from the contextual tokens alone with a hierarchical ConvStack head adapted from MoGe [58], supervised by an L1 loss on valid ground-truth pixels.

6.1.0.2 Data curation.

MDM requires RGB-D data whose missing patterns are realistic, whereas existing datasets either avoid hard imaging conditions or provide rendered, artifact-free depth. LingBot-Depth therefore curates its own data through two parallel pipelines. The synthetic stream renders RGB, perfect depth and speckle-projected stereo pairs in Blender from self-hosted indoor scenes, then runs semi-global matching on the stereo pair so that the resulting sensor-like depth carries the artifacts of real active cameras (1M samples from 442 scenes). The real-world stream builds on a custom multi-camera capture rig spanning diverse indoor, commercial and outdoor scenes, with pseudo ground truth distilled from the stereo infrared pairs by FoundationStereo [59] under left-right consistency checks (2M captures). Combined with open-source RGB-D datasets, this yields roughly 10M training samples, of which the curated 3M were publicly released. Trained on this corpus, LingBot-Depth surpasses top-tier RGB-D cameras in both depth precision and pixel coverage.

6.2 Two Important Tweaks for LingBot-Depth↩︎

LingBot-Depth 2.0 keeps the masked depth modeling recipe of Sec. 6.1 unchanged and improves its two external ingredients: the encoder initialization and the scale of the curated training data.

6.2.0.1 Spatial-perception-native encoder.

The single external dependency of the recipe is the encoder initialization, and that is precisely where LingBot-Depth 2.0 upgrades: the DINOv2 ViT-L/14 encoder is replaced by our LingBot-Vision pretraining, at both ViT-L/16 and ViT-g/16. Table 6 isolates this choice by training the same MDM pipeline from different initializations. At ViT-L, the LingBot-Vision encoder improves over both the DINOv2 and DINOv3 initializations on nearly every benchmark, with the largest gains on the hardest block-mask patterns (e.g., 0.094 vs. 0.152 RMSE on DIODE-Indoor against DINOv2), and the advantage carries to giant scale (0.083 vs. 0.118 on DIODE-Indoor). This is the fit one would expect: depth completion concentrates its uncertainty around object boundaries and material transitions, exactly the structure that the boundary-anchored features of LingBot-Vision encode.

6.2.0.2 Scaling the curated data.

The second tweak scales the data: using the capture and pseudo-labeling machinery recapped in Sec. 6.1, the curated corpus grows from the publicly released 3M samples to 150M RGB-D samples. Fig. 8 tracks depth completion quality as the training set grows: accuracy improves monotonically with data for every encoder initialization compared, and the LingBot-Vision initialization retains its lead at every data scale. The two tweaks therefore compound: the better starting point does not get washed out by more data, it keeps paying at 150M.

6.3 Benchmarking LingBot-Depth 2.0↩︎

6.3.0.1 Datasets.

Our evaluation data falls into three categories, matching the structure of Tables 7 and 8. (1) Block-mask benchmarks are public RGB-D datasets with high-quality ground truth, from which we remove large contiguous regions of the input depth: DIODE-Indoor and DIODE-Outdoor [60], precise laser scans covering both indoor and outdoor scenes; iBims-1 [61], a laser-scanned indoor benchmark with carefully verified ground truth, particularly around depth boundaries; and NYU [40], the standard Kinect-captured indoor test split. (2) Sparse benchmarks reduce the input depth to scattered measurements, sampled following MarigoldDC [62] on VOID [63], iBims-1 and NYU, and taken from sparse SfM depth on ETH3D [64]. (3) Real-sensor benchmarks evaluate on raw, incomplete captures of commodity depth cameras: HAMMER [65], which pairs RealSense D435, L515 and time-of-flight captures of the same scenes with high-precision ground truth; ClearGrasp [66], real D415 and D435 captures of transparent objects, a worst case for active depth sensing; and our self-captured LingBot evaluation set with D415, D435 and D455 cameras. Our LingBot evaluation set comprises \(1{,}751\) frames captured across \(35\) distinct indoor scenes. The scenes span a broad range of everyday environments: offices, meeting rooms and reception areas; schools, libraries and laboratories; hospitals and clinics; hotels, guestrooms and homes; restaurants, showrooms and retail spaces; and transit areas such as stations, parking lots, garages and elevators. Many of these contain the reflective, transparent and textureless surfaces that are a worst case for active depth sensing. To obtain reliable ground truth, we run a strong stereo-matching model (FoundationStereo [59]) on the left–right image pairs of each camera; a left–right consistency check discards unreliable estimates, and a vision–language model (Qwen3.5 [67]) is used to detect and remove frames with large motion blur. Pairing this ground truth with the raw, incomplete depth of each RealSense camera lets us evaluate depth completion under realistic sensing conditions.

6.3.0.2 Evaluation protocol.

We evaluate depth completion in two input regimes on the public benchmarks above, plus the raw captures of the real-sensor suite. In the block-mask regime, large contiguous regions of the input depth are missing; in the sparse regime, only scattered depth points are given. We report RMSE and the threshold accuracy \(D_{105}\), and compare against recent depth completion systems (CDMs [54], OMNI-DC [55], Any2Full [56], PriorDA [57]) and LingBot-Depth 1.0.

6.3.0.3 Quantitative results.

On the masking patterns (Table 8), LingBot-Depth 2.0 with the ViT-L backbone attains the best RMSE on seven of the eight benchmarks: all four block-mask settings and three of the four sparse settings, with ETH3D remaining with LingBot-Depth 1.0. The step over the previous version is large where completion is hardest: on block-masked DIODE-Indoor the RMSE is halved (0.132 to 0.062), and DIODE-Outdoor improves from 3.404 to 2.440. The external baselines are strong only within their preferred regime: CDMs collapses under sparse inputs and Any2Full degrades on outdoor block masking, whereas LingBot-Depth 2.0 is consistent across both. On real sensors (Table 7), 2.0 is the best on six of the eight camera configurations, matches CDMs on Hammer D435, and is particularly strong on the transparent-object ClearGrasp captures (0.010 and 0.012 RMSE), the classic failure case of active depth sensing. Scaling the backbone to ViT-g pushes further on most pattern benchmarks (0.060 on DIODE-Indoor, 0.114 and 0.113 on NYU in the two regimes) and on the LingBot sensor suite (0.228, 0.345 and 0.375 across the three cameras), with ClearGrasp remaining in favor of the ViT-L variant.

6.3.0.4 Qualitative results.

Fig. 9 probes the two classic failure modes of active depth sensing, mirrors and glass, on captured video sequences. The raw sensor depth is missing exactly where the scenes are hardest: window panes, a glass balustrade and reflective floors return no measurements at all. LingBot-Depth 2.0 completes these regions with geometry that is consistent as a surface, not merely plausible per pixel: in the refined point clouds, the window and the balustrade appear as flat, contiguous planes in both the front and top views, and the completion stays stable across consecutive frames of each sequence. Fig. 10 compares against the baselines on transparent tabletop objects, outdoor signage, dark walls with thin structures, and a bright window scene. The differences are most visible in the point-cloud rows: the baselines either leave the missing regions open, bend large planar surfaces, or scatter floating points around depth discontinuities, and LingBot-Depth 1.0 still exhibits residual outliers near windows and object rims. LingBot-Depth 2.0 keeps walls straight, planes intact and object boundaries sharp, with occlusion edges in the completed depth that align with the object contours of the RGB frame, the signature of its boundary-aware encoder.

7 Conclusion↩︎

This report presented LingBot-Vision, a vision foundation model pretrained to be spatial-perception native. The central idea is to make boundary structure a first-class citizen of self-supervised learning: image boundaries are represented as dense categorical boundary fields, boundary-bearing tokens are forced into the masking pattern, and the training targets are bootstrapped online from the teacher’s own predictions, validated by a parameter-free a-contrario test whose null hypothesis coincides with the uniform categorical distribution. Trained at ViT-g scale on curated data, LingBot-Vision delivers the strongest frozen-feature dense prediction in its class, including the best NYUv2 linear-probing RMSE of all compared models, ahead of the 7B DINOv3, video object segmentation on par with the strongest distilled models, competitive image-level recognition, and boundary tokens stable enough to be tracked through video. Distillation carries these properties to ViT-L, ViT-B and ViT-S students, so the same recipe serves every deployment budget.

The value of a vision foundation model is ultimately measured downstream. Upgrading LingBot-Depth to LingBot-Depth 2.0 changed nothing in the masked depth modeling recipe except the encoder initialization and the scale of the curated data, and delivered the leading depth completion performance across masking patterns and real depth cameras, with the advantage of the LingBot-Vision initialization widening as the training data grows. We take this as evidence that boundary-oriented pretraining is a practical route toward robotic spatial perception, and we release the pretrained models of LingBot-Vision to the community.

Acknowledgments↩︎

We gratefully acknowledge the optical testing team at Orbbec Inc. for their professional support. We also thank Yubin Hu, Changkun Liu, Xingyi He, Mengfei Xia, Jianyuan Wang, Shangzhan Zhang, Yuanbo Yang for helpful discussion. The large-scale curated RGB-D data for LingBot-Depth 2.0 cannot be successful without the support of Yongtao Huang, Yibo Lu, Chongjun Zhong, Han Zhang and Xiage Qin.

Appendix↩︎

8 Sampling Boundary Fields↩︎

This appendix details the demonstration behind Finding 1 and Fig. 3: decoding line segments from boundary fields that carry no learned information.

8.0.0.1 Setup.

Images are resized to \(512\times512\) and the boundary field is defined at output stride 2, with every channel normalized to \([0,1]\): the distance channel by the support threshold \(\tau_d\) (5 px at the field resolution) and the three angular channels by their angular ranges (Sec. 3.3). Corner points are produced by the frozen corner-point detector used throughout pretraining (Sec. 3.4) and are kept fixed across all draws.

8.0.0.2 Sampling.

For each draw, the field channels are sampled i.i.d.per position from \(\mathcal{U}(0,1)\), with the sub-pixel refinement offsets set to zero. In the bottom rows of Fig. 3, all channels are random, including the direction channel \(\theta\) that orients each position’s chord proposal; randomly oriented proposals rarely agree, so the decoded segments stay short, yet they remain anchored on the corner points. In the top rows, only \(\theta\) is replaced by a parameter-free guidance that carries no learned information: the image’s level-line orientation, the direction orthogonal to the local intensity gradient that underlies classical a-contrario line detection [30], [38]. Collinear level-lines are grouped into line-support regions [30], the resulting segments are rendered into a boundary field (the lifting of Sec. 3.3), and the \(\theta\) channel of this rendering replaces the random draw; the distance and endpoint channels stay random.

8.0.0.3 Decoding.

Decoding follows Sec. 3.4 with the a-contrario validation disabled: every position proposes a chord from its attributes \((d, \theta, \phi^{1}, \phi^{2})\), the proposal endpoints are attached to their nearest corner points, votes are accumulated over every corner pair, and the pairs with sufficient votes become the decoded segments. The five columns of Fig. 3 are independent draws, rendered while sweeping the minimum vote count from 2 to 10; the decoded line segments are essentially unchanged across both the draws and the sweep. The randomness in \(d\), \(\phi^{1}\) and \(\phi^{2}\) is thus fully absorbed by corner anchoring and vote aggregation, while the direction channel controls the completeness of the recovery: the fully random field still anchors on the corners but yields only short fragments (bottom rows), and the parameter-free level-line guidance recovers the complete line segments (top rows).

9 A-contrario Validation↩︎

The a-contrario framework [30], [38] formalizes the Helmholtz principle: a geometric structure is meaningful only if it would be highly unlikely under a null model that contains no structure. For a candidate segment with \(n\) support pixels, of which \(k\) have orientations aligned with the segment within a tolerance, the Number of False Alarms (NFA) is \[\mathrm{NFA}(n,k) = N_{\mathrm{t}} \cdot \sum_{i=k}^{n}\binom{n}{i}\, p^{\,i}\,(1-p)^{\,n-i},\] where \(p\) is the probability that a pixel aligns with the segment by chance and \(N_{\mathrm{t}}\) is the number of tested candidates. A segment is accepted iff \(\mathrm{NFA} \le \varepsilon\); the canonical choice \(\varepsilon = 1\) (at most one false alarm per image on average) makes the test essentially parameter-free. The null hypothesis, uniformly distributed orientations, is exactly the uniform distribution over bins of our categorical boundary field (Sec. 3.3).

9.0.0.1 Why the classical test does not parallelize.

In LSD [30], candidate formation and validation are entangled. A candidate rectangle only exists after a greedy region-growing pass: starting from a high-gradient seed pixel, the region absorbs neighbors whose level-line orientation agrees within tolerance and marks them as consumed, so that later seeds cannot reuse them; a rectangle is then fitted to the grown region and submitted to the NFA test. Regions are grown one at a time, the outcome depends on the seed order, and the control flow branches on every pixel, which is why a-contrario detectors have remained sequential CPU designs.

9.0.0.2 Corner anchoring decouples candidates from validation.

Our pipeline obtains the candidate set without any grouping. Because the boundary field is deliberately redundant, a complete chord proposal, with both endpoints, can be read out densely at every position (Sec. 3.3). Corner points then anchor these proposals: a batched line-segment matcher snaps the two endpoints of every proposal to their nearest corner points, with the pairwise distance computation fused into the argmin reduction so that memory stays linear in the number of proposals, and each corner pair accumulates the votes of the proposals assigned to it. The candidate set is simply the resulting corner-pair graph. Validation thereby becomes a pure per-candidate reduction with no shared state: one GPU thread per candidate counts the support and aligned pixels in a fixed-width rectangle around its segment and evaluates the binomial tail above, so all candidate segments of a training batch are validated in parallel (Sec. 4.2). This decoupling, dense endpoint readout plus corner anchoring in place of region growing, is what makes a-contrario validation cheap enough to run inside the training loop at every iteration.

9.0.0.3 Implementation constants.

The orientation evidence is the image’s level-line field, computed with LSD’s \(2\times2\) finite-difference gradient on the grayscale view, batched over the global crops; pixels with negligible gradient magnitude are marked as undefined and never count as aligned. Each candidate uses a rectangle of fixed width \(3\) px, and orientations are compared modulo \(\pi\) with an alignment tolerance of \(\pi/16\), i.e., \(p = 1/16\), stricter than the \(\pi/8\) of LSD. We keep the classical number of tests \(N_{\mathrm{t}} = (HW)^{5/2}\), an upper bound over all possible rectangles; this is conservative for our candidate set, which contains only the corner pairs, and we drop the extra factor LSD adds for its width refinements since our width is fixed. The acceptance threshold is the canonical \(\varepsilon = 1\), and candidates whose aligned-pixel density falls below \(0.5\) are rejected before the tail is evaluated. During pretraining the test runs at every iteration on every decoded segment, and images retaining fewer than \(10\) validated line segments are excluded from the boundary loss for that iteration. The same reduction applies unchanged when the orientation evidence is taken from the model’s own boundary field instead of the image gradient, which is how the boundary tokens of Fig. 1 are validated.

References↩︎