1 Introduction↩︎

Encoders are critical for a variety of modern applications, ranging from large-scale data curation and retrieval-augmented generation to recommendation systems. However, encoder architectures have remained largely unchallenged since BERT, most improvements stem from scaling training datasets. The research community has focused on elaborate post-training pipelines utilizing pre-trained models merely as initialization [1]. This heavy reliance on downstream finetuning does not show the shortcomings of the pre-trained backbone, hindering its development. [2] recently demonstrated the cost of neglecting pre-training: frontier pre-trained backbones like ModernBERT [3] and NeoBERT [4] are vastly overtrained relative to their size.

Concurrently, the vision encoder community has continued to improve pre-trained backbones via scaling strategies [5]–[8] and novel training recipes [9]–[16]. Since these encoders are often utilized frozen, i.e. without modifications, evaluation standards have naturally prioritized frozen benchmarks. Therefore encouraging pre-training innovations. Effective representation learning is defined by the interpretability and versatility of frozen embeddings.

Motivated by recent architectural insights in the vision community, we revisit the design and evaluation of text encoders. Specifically, we conduct a comprehensive scaling analysis, evaluating the representations of frozen pretrained backbones alongside standard finetuning protocols. This shift in perspective reveals a counterintuitive phenomenon: as modern BERT models scale, their features become increasingly unexploitable by frozen probes. To resolve this bottleneck, we introduce CrossBERT. Inspired by [9], [16], CrossBERT decouples representation learning from token reconstruction by appending a lightweight cross-attention predictor to the final backbone layers. This architectural shift ensures that the backbone focuses entirely on feature extraction, while token reconstruction is isolated within the predictor. We demonstrate that this novel design preserves representation quality as the model scales. Furthermore, even when evaluated in a data-constrained contrastive setup using only MS-MARCO, CrossBERT mitigates saturation and exhibits a significantly superior scaling trends compared to standard baselines.

Throughout the paper, our baseline BERT is a modern implementation (see 5.1). BERT and CrossBERT differ only on the architecture structure: flat vs bipartite. In 2 we explain our conjecture for the failure mode of the BERT architecture and how CrossBERT solves it. In 3 we detail the architecture of CrossBERT and present its advantages compared to the usual BERT models. In Section 4 we summarize the new frozen evaluation methods that we use to measure solely the pre-training performance. Finally we present our experimental results and discussion in Section 5 and conclude with future work in Section 8.

Contributions

1. The CrossBERT Architecture. We introduce a bipartite encoder that ensures consistent performance scaling on frozen evaluations. To the best of our knowledge, this is the first Masked Autoencoder for text.

2. High-Efficiency Training via High Masking Ratio. We demonstrate CrossBERT’s robustness at masking ratios \(>50\%\), accelerating training throughput by \(\approx 1.5\text{--}2\times\). Furthermore, this tolerance enables a Complementary Masking Strategy (CMS), which processes the inverse mask in parallel. Effectively doubling the sample efficiency by collecting gradients from every token in the sequence.

3. Scaling Laws under Frozen Evaluation. We conduct a scaling analysis (\(2 \times 10^{18}\) to \(1 \times 10^{21}\) FLOPs) to quantify intrinsic representation quality. This exposes a performance gap in standard BERTs and validates CrossBERT’s superior extractability, an advantage that we show extends to MTEB tasks.

2 CrossBERT: Intuition↩︎

2.0.0.1 Problem.

BERT representations worsen as we scale compute as can be seen in Fig.  1.

2.0.0.2 Explanation.

We conjecture that the reason behind such degradation stems from the Masked Language Modeling (MLM) objective [17] applied on the flat design of the BERT architecture.

MLM trains an encoder to reconstruct a corrupted input sequence. Given a sequence of tokens \(X = \{x_1, \dots, x_N\}\), a subset of indices \(\mathcal{M}\) is selected for masking. The tokens at these positions are replaced by a special token \(\texttt{<MASK>}\), yielding a corrupted sequence \(\tilde{X}\). The model processes \(\tilde{X}\) to generate contextualized representations, and the objective is to minimize the negative log-likelihood of the original tokens \(x_m\) at the masked positions:

\[\mathcal{L}_{\text{MLM}} = - \sum_{m \in \mathcal{M}} \log P(x_m \mid \tilde{X})\]

where \(P(x_m \mid \tilde{X})\) denotes the probability assigned to the true token \(x_m\) by the prediction head. Hence the objective only measures the token reconstruction ability of the model but not the actual quality of its representations.

Since the “flat" design of the BERT architecture does not explicitly separate representation creation from token reconstruction, the representations remain overly”grounded" in the local signal required to predict missing tokens instead of being versatile high-level abstractions. Some evidence of this phenomenon is displayed in 1, where the BERT encoder is probed at different depths. Notably, retrieving representations from earlier layers, rather than the final output, yields slightly improved performance, showcasing the over-specialization of these last layers.

2.0.0.3 Solution.

Inspired by Masked Auto-Encoders (MAE) approaches in vision [9], [16], we propose CrossBERT: a bipartite architecture that separates the heavy lifting of representation creation (Encoder) from the specific task of token reconstruction (Predictor), as illustrated in Fig. 2.

3 CrossBERT: Architecture & Advantages↩︎

3.1 Architecture↩︎

A sketch of the architecture is shown in the right panel of 2. The input sequence is partitioned into a visible set (processed by the encoder) and a masked set (processed by the predictor). To preserve sequence order, we encode the position of each token through RoPE, ensuring the encoder and predictor are aware of which positions are missing. The predictor is a few transformer blocks that only cross-attend to the encoder representations. By removing self-attention between masked tokens, we force the predictor to act strictly as a “readout" interface that must satisfy its objective solely by querying the encoder’s embeddings. Additionaly we implement modern architectural optimizations [3], [4] such as RMSNorm.

To set the size of the encoder and predictor we align with MAE [16]. The predictor shares the encoder’s hidden dimension, but is significantly shallower (approximately one-forth of the encoder’s depth). Ablation on the predictor’s shape and size can be found in 9. We gain two insights from this ablation. First, the specific aspect ratio (width vs. depth) of the predictor is not significant. Second, while increasing the predictor’s capacity can yield some performance gains (with diminishing returns), it comes at the cost of slower training. Since our objective is to obtain rich frozen features, we decide to allocate most of the compute to the encoder.

3.2 Emerging Advantages↩︎

This design choice leads to several advantages: predictor’s transfer learning; robustness to higher masking ratios; the ability to use the Complementary Masking Strategy (CMS); better data efficiency and lower training costs.

3.2.0.1 Transferability.

The predictor learns to extract information only from the encoder representation during pre-training. Hence, the predictor functions as a learned pooling mechanism. It can be effectively re-used as a warm-started module for downstream finetuning, serving as an efficient bridge between the frozen encoder features and the target task.

3.2.0.2 Robustness to higher masking ratios.

BERT architectures typically suffer performance degradation when masking ratios exceed \(20\text{--}40\%\) [18] (see 11). We challenge this limitation by profiling the frozen representation quality of CrossBERT across multiple masking ratios. As shown in 2, CrossBERT exhibits remarkable stability: increasing masking from \(20\%\) to \(50\%\) incurs a negligible drop in GLUE performance (\(-0.7\%\)). This resilience confirms that our bipartite architecture successfully insulates representation learning from the difficulty of the reconstruction task. Increasing to higher masking ratios directly translates into reduced training costs, which opens the door to a Complementary Masking Strategy (CMS).

3.2.0.3 CMS.

Complementary Masking Strategy is depicted in the left panel of 2. Every batch is augmented with its inverse mask, i.e. creating two complementary views of the original token sequence. This allows the model to predict and learn from every token in the sequence. The model ensures that information from the original sequence does not “leak" into the complementary view within the same batch by applying a unique attention mask on each view. Hence, both views can be processed simultaneously. For BERT, this approach does not mix well with the 20% – 40 % masking ratio requirement as the inverse view would land on the 60% – 80% ratio. Because CrossBERT remains effective beyond \(50\%\) masking, it can learn from both the original and the complementary sequence efficiently.

3.2.0.4 Sample efficiency.

MLM is known for being sample-inefficient; the model learns only from a fraction of tokens per pass since gradients are only computed on masked tokens. However by utilizing CMS, CrossBERT processes the visible tokens and its inverse, effectively seeing all the tokens in the sequence and predicting all the tokens from the predictor in one go. Empirically, our results (10) confirm that CMS does not damage the performance of the baseline compared to standard masking. Instead, CMS effectively doubles sample efficiency, requiring half the training data to reach the same performances.

3.2.0.5 Computational efficiency.

Even if BERT were to employ CMS, the computational cost would be prohibitively expensive: two forward-backward passes on the full sequence. On the other hand, the encoder of CrossBERT drops masked tokens. Hence the cost of training with CMS is just one forward-backward pass of the encoder and predictor on the full sequence.

4 Method↩︎

In this section we aim to describe how we prove that CrossBERT works better than the current BERT recipe.

Historically, encoder evaluations have relied heavily on full finetuning for downstream tasks ranging from classification to information retrieval. Doing so, makes it hard to judge whether the final downstream performance is a result of finetuning or original pretrained representation quality. Therefore, we evaluate performance by freezing the encoder, which allows us to directly measure the pretrained representation quality.

We focus on two evaluation benchmarks. While GLUE [19] served as the gold standard for early pre-training, the field, especially within the contrastive finetuning landscape, has since adopted the MTEB benchmark [20] to better assess embedding quality. We now describe how to adapt these established benchmarks to a frozen evaluation protocol.

4.1 GLUE frozen evaluation↩︎

Given the historical importance of classification in encoder evaluation, we use classification tasks to probe feature quality throughout the training process. Specifically, the output of the last layer is averaged across all tokens to generate a single representation of the sequence. A classifier is then optimized on these frozen features. To ensure efficiency, these probes are fitted in parallel, necessitating only a single forward pass over the training set. Two distinct types of classifiers are used:

4.1.0.1 Linear probing.

The linear probes are optimized via Ridge Regression, which minimizes the standard least-squares error augmented by a \(L_2\) penalty term. The loss function is defined as: \[\mathcal{L}_{\text{Ridge}}(\mathbf{W}) = \| \mathbf{Y} - \mathbf{X}\mathbf{W} \|_F^2 + \lambda \| \mathbf{W} \|_F^2\] where \(\mathbf{X}\) represents the frozen encoder representations, \(\mathbf{Y}\) the target labels, and \(\lambda\) the regularization coefficient controlling the penalty strength. The optimal weights \(\mathbf{W}^*\) are computed directly via the closed-form analytical solution: \[\mathbf{W}^* = (\mathbf{X}^\top \mathbf{X} + \lambda \mathbf{I})^{-1} \mathbf{X}^\top \mathbf{Y}\] This approach allows for rapid, deterministic fitting across multiple regularization strengths without the need for iterative optimization. Moreover, this procedure is done on GPU leveraging torch.linalg.solve making it even faster. In practice, we sweep multiple logspaced \(\lambda\) as we notice that some dataset, and especially small ones, are very sensitive to this hyperparameter.

4.1.0.2 kNN probing.

As a non-parametric complement to linear evaluation, k-Nearest Neighbors (kNN) assess the intrinsic geometry of the representation space. For a given query \(z\), the prediction is determined by a majority vote among the set of its \(k\) closest neighbors \(\mathcal{N}_k(z)\), identified by minimizing the distance metric \(d\) (either \(L_2\) or Cosine): \[\begin{align} \mathcal{N}_k(z) &= \underset{\mathcal{S} \subset \mathcal{D}, |\mathcal{S}|=k}{\operatorname{arg\,min}} \sum_{x_j \in \mathcal{S}} d(z, x_j) \\ \hat{y} &= \operatorname*{arg\,max}_{c \in \mathcal{C}} \sum_{x_i \in \mathcal{N}_k(z)} \mathbb{1}(y_i = c) \end{align}\] where \(\mathcal{D}\) is the set of all evaluation points and \(\mathcal{C}\) the set of all classes. The implementation runs on GPU and is heavily based on the released code of CAPI [12].

In practice, these evaluations are computationally negligible. For a 250M parameter model, the entire linear and kNN probing process on GLUE takes less than 5 minutes on a single H100. The cost is dominated by the forward pass over the dataset.

4.2 MTEB frozen evaluation↩︎

We evaluate the richness and adaptability of our frozen representation using the Massive Text Embedding Benchmark [20], which spans seven distinct task downstream categories: Classification, Clustering, Pair Classification, Semantic Textual Similarity (STS), Reranking, Retrieval, and Summarization.

4.2.0.1 Contrastive probing.

The above downstream tasks require sentence/document representations and a coherent, well-structured representation space, which is not tackled by the pre-training objective. Contrastive finetuning aims to address these gaps. While existing literature typically performs this via contrastive finetuning on an unfrozen backbone, we explore two configurations: a standard unfrozen backbone and a frozen-backbone approach where only an attention based pooler is optimized on top of fixed features.

The token-level encoded representations are pooled into a single sentence/document representation via a learnable lightweight adapter, consisting of a few Transformer blocks with cross-attention. Let the pooled representations of a passage (resp. query) be \(p\) (resp. \(q\)). Given a query \(q\), the contrastive loss, aims to pull closer a related passage \(p\) and the query \(q\) in representation space while pushing away semantically similar unrelated passages \(p^-\) (hard-negatives). The contrastive loss, as defined in [10], is \[\mathcal{L}_{c} = -\frac{1}{N} \sum_{i=1}^{N} \log \frac{e^{ \phi(q_i, p_i) }}{ e^{ \phi(q_i, p_i) } + \sum_{n \in \mathcal{S}_i} e^{ \phi(q_i, p_{in}^-) } } \label{eq:contrastive95loss}\tag{1}\] where \(\phi(q_i, p_i)\) is the cosine similarity between a query \(q_i\) and a passage \(p_i\), and \(p_{in}^-\) denotes the hard negatives for each query.

Since CrossBERT already features an adaptable component (the predictor), we re-purpose it to warm-start the adapter for the contrastive learning phase, while keeping the encoder frozen. As detailed in our ablation study (12), this strategy accelerates convergence and improves MTEB scores by \(\approx 2\)%.

4.3 Evaluations at different scales↩︎

Model performance and optimal hyperparameters follow predictable power-law trends relative to the total compute budget \(C\) [21], [22]. We define \(C\) as: \[C = F_N D\] where \(D\) is the number of pre-training tokens and \(F_N\) are the FLOPs per token for a forward-backward pass. For a standard BERT transformer with sequence length \(S\), layers \(L\), and non-embedding parameters \(N\) [2] we have: \[F_N = 6 N + 12 d L S\] However, CrossBERT modifies \(F_N\) based on the masking ratio \(r\), linearly interpolating between the encoder (\(F^{\text{enc}}_N\)) and the cross-attention predictor (\(F^{\text{pred}}_N\)): \[F_N = F^{\text{enc}}_N (1-r) + F^{\text{pred}}_N r\] This demonstrates that as the masking ratio increases, the total computational cost linearly interpolates between the encoder and the predictor. To scale up we need to increase model sizes \(F_N\) and dataset sizes \(D\) in tandem.

5 Experiments↩︎

5.1 Setup↩︎

5.1.0.1 Models.

We train three models that share the same corpus, tokenizer, and encoder backbone: BERT, Electra, and CrossBERT. BERT and Electra are flat, whereas CrossBERT is bipartite (3). BERT and CrossBERT use the MLM objective of 2, while Electra replaces it with replaced-token detection (RTD) [23]: a small auxiliary generator substitutes a fraction of the input tokens with plausible alternatives, and the encoder predicts at each position whether the token is original or replaced. For context, we also report the published results of ModernBERT [3], NeoBERT [4], DeBERTaV3 [24], and OptiBERT [2], all of which are trained with substantially more data and compute.

5.1.0.2 Data.

We choose a subset of DCLM [25] as our training corpus (\(\approx 4\text{T}\) tokens). Across all experiments, the masking ratio is set to \(20\%\) for BERT and \(50\%\) for CrossBERT. Additionally, all CrossBERT models leverage CMS (see 3.2). All models rely on the RoBERTa tokenizer [26].

5.1.0.3 Codebase.

Our codebase is based on PyTorch [27] and Lingua [28], opting for Fully Sharded Data Parallelism (FSDP) and torch.compile for maximum throughput and reduced memory footprint. Handling Dynamic Shapes: Compilation requires static graphs, which conflicts with random MLM masking. We resolve this by enforcing a fixed count of masked tokens per GPU (Per GPU Batch size \(\times\) Mask Ratio) and permuting each mask in the sequence via random permutations (torch.randperm). This ensures static tensor shapes without compromising masking randomness.

5.1.0.4 Hyperparameters.

Following [22], we fit power laws for both Batch Size (BSZ) in total tokens and Learning Rate (LR) sweeping model sizes ranging from (50M to 700M) with a LR (resp. BSZ) logspaced from \(10^{-4}\) to \(10^{-2}\) (resp. \(10^4\) to \(5\times10^6\)). After keeping only the top 3 performing models for the fitting, we obtain the following results for CrossBERT: \[\begin{align} \text{BSZ}_{\text{CrossBERT{}}}(C) &= 10^4 \times C^{0.092} \\ \text{LR}_{\text{CrossBERT{}}}(C) &= 564.6 \times C^{-0.279} \end{align}\] For the BERT setup, we simply reuse the hyperparameters found by [2] \[\begin{align} \text{BSZ}_{\text{BERT}}(C) &= 17.38 \times C^{0.24} \\ \text{LR}_{\text{BERT}}(C) &= 69.18 \times C^{-0.24} \end{align}\] 16 shows the hyperparameter heatmap for the CrossBERT sweeping.

5.1.0.5 Eval on GLUE.

For Linear Probing, we fit linear heads on frozen features along logarithmic data regimes (\(\{1, \dots, 10^4\}\) samples) and L2-regularization strengths (\(\lambda \in [1, 10^4]\)). For each task we report the best score across all \(\lambda\) for the largest available sample size. For kNN Probing, non-parametric evaluation with kNN are conducted using diverse configurations by sweeping across distance metrics (\(L_2\), Cosine) and neighborhood sizes (\(k \in \{1, 3, 10, 30\}\)). We report the maximum score across all \(k\) and distances.

5.1.0.6 Eval on MTEB.

We use the MS-MARCO training set (500k queries) with hard negatives. No instruction templates are applied. We finetune contrastively for one epoch, with a batch size of 512 and a learning rate of \(5 \times 10^{-5}\) (cosine decay to \(5 \times 10^{-7}\)). To be comparable with current published models, we use both an unfrozen and frozen backbone.

5.1.0.7 Single-scale runs.

All three models share the same encoder backbone (28 layers, hidden dimension 768); CrossBERT additionally couples this encoder to a 6-layer cross-attention predictor. The models are trained on 50B tokens with a data-to-model ratio of \(35:1\) that deliberately exceeds the compute-optimal \(\approx 15:1\) to avoid undertraining. The learning rate is fixed at \(6 \times 10^{-4}\) and the global batch size at \(\approx 393\text{k}\).

5.1.0.8 Scaling sweep.

[2] showed that the data-to-model ratio \(F_N / D\) heavily impacts performance and should be kept around 10:1 to 100:1. In all our scaling experiments we fix the ratio at \(20:1\) and generate a suite of models spanning a total compute range from \(2\times10^{18}\) to \(1\times10^{21}\) FLOPs. To optimize performance at every scale, we determine batch size and learning rate by sweeping values across smaller models (\(50\text{M}\) to \(700\text{M}\)) and extrapolating the optimal settings via a power law fitting). In total we train 8 BERT and 9 CrossBERT models, see 18.

5.2 Results: Single-scale Runs↩︎

3 reports the single-scale runs of our three models alongside the current literature.

5.2.0.1 Training efficiency and throughput.

CrossBERT demonstrates superior computational efficiency. In terms of training throughput, it achieves 207k tokens/sec compared to the baseline’s 123k tokens/sec, representing a \(1.68\times\) speedup in wall-clock time on H100 GPUs. Furthermore, when compared to the existing literature, CrossBERT remains highly competitive despite utilizing approximately \(100\times\) less total compute than models like ModernBERT and NeoBERT. Against OptiBERT, a model of similar scale, CrossBERT requires \(\approx 40\%\) fewer FLOPs while delivering higher performance on both GLUE and MTEB(eng, v1).

5.2.0.2 Full-finetuning performances.

Despite this massive reduction in compute, CrossBERT does not compromise on quality. On the GLUE benchmark, it matches our robust baseline (\(86.6\) vs. \(86.4\)). More notably, on the MTEB(eng, v1) benchmark (Full Contrastive Finetuning), CrossBERT achieves the highest average score of 54.5, outperforming both NeoBERT (\(51.3\)) and ModernBERT (\(46.9\)) which used much more finetuning data.

5.2.0.3 Robustness of frozen representations.

The most significant advantage of CrossBERT lies in the versatility of its frozen features. We observe three critical behaviors:

• Minimal degradation: When switching from full finetuning to frozen adaptation, the standard BERT baseline suffers a substantial drop of over 10 points on MTEB(eng, v1) (\(53.9 \to 42.1\)). In contrast, CrossBERT retains 99% of its performance, scoring 53.6 in the frozen setting. This indicates that the pretrained features are easily adaptable.

• Retrieval capability: This robustness is most visible in the Retrieval task, where CrossBERT more than doubles the score of the baseline (\(40.7\) vs. \(19.2\)), proving that the model captures semantic similarity without needing deep task-specific adaptation.

• Linear separability: On the GLUE Linear Probe, CrossBERT surpasses the BERT baseline by a large margin (\(+6.4\) points). This confirms that our approach prevents the over-specialization often seen in standard MLMs, producing high-level abstractions that are linearly separable and directly usable for downstream tasks. While a performance gap of \(\approx 13\) points remains between the linear probe and full finetuning, this is partly attributable to the simplicity of averaging encoder outputs; utilizing more sophisticated probing mechanisms, such as attention-based pooling, would likely reduce this gap further. Electra scores even higher than CrossBERT on this probe, but this advantage does not carry over to the sentence-embedding tasks examined next.

5.2.0.4 MLM versus RTD on a flat backbone.

On token-level probes, Electra outperforms BERT and CrossBERT, reaching \(87.8\) on GLUE full finetuning and \(76.9\) on the GLUE linear probe. However, on sentence-level embedding tasks it collapses. The unfrozen backbone benchmarks at \(49.0\) on MTEB(eng,v1) whereas the frozen one drops to \(22.4\) with a Retrieval score of \(0.0\). This collapse mirrors prior findings that RTD distorts sentence-embedding geometry [3], [29]. Two conclusions follow. First, neither swapping the objective on a flat backbone (Electra) nor keeping MLM on a flat backbone (BERT) yields versatile frozen representations; the bipartite separation of representation from reconstruction is what produces them. Second, GLUE-style classification probes alone cannot diagnose representation quality, since MTEB exposes failures that GLUE hides.

5.3 Results: Scaling Sweep↩︎

1 aggregates the GLUE and MTEB(eng,v2) scores of BERT and CrossBERT, while [fig:scaling_trends_glue,fig:scaling_trends_mteb] break them down per task. Our sweep reveals four insights into the scaling behavior of masked language models.

5.3.0.1 Degradation of frozen BERT representations.

The most striking trend is the sharp divergence in scaling laws between the two models. While standard BERT benefits from scaling when it is unfrozen and fully finetuned, its frozen performance suffers a brutal drop as model size increases. Specifically, for the two largest configurations (\(>500\)M parameters), BERT’s frozen performance drops below that of the smallest model in the sweep (\(10\times\) smaller). Since the data-to-model ratio (20:1) is held above the compute-optimal ratio (15:1), undertraining cannot explain the drop. This confirms a severe misalignment between the standard MLM objective and semantic embedding tasks: as the model scales, it becomes increasingly specialized for token reconstruction at the expense of versatile, high-level abstractions.

5.3.0.2 Task-specific sensitivity on GLUE.

This degradation manifests non-uniformly across tasks (4). On the GLUE benchmark, the sudden decline of performance does not necessarily appear at the same point. For example MNLI accuracy drops only for the last model while performances for the rest seem to drop earlier.

5.3.0.3 MTEB(eng, v2) superior scaling and saturation profiles.

In contrast, CrossBERT exhibits robust, monotonic scaling across all metrics. Most notably, CrossBERT fundamentally alters the relationship between pre-training and adaptation:

• The adaptation gap & equalization on MTEB(eng, v2): Standard BERT requires full finetuning to bridge a massive performance deficit (\(\approx15\) points) between its frozen and unfrozen states. While this heavy downstream adaptation can eventually equalize performance, i.e. masking pre-training deficiencies by bringing BERT closer to CrossBERT, it comes at a significant compute cost. In contrast, CrossBERT’s frozen representations are naturally aligned, sitting at worse around one point below the fully fine-tuned optimum.

• Structural retrieval capability: We observe a fundamental distinction in retrieval tasks. Standard MLM effectively flatlines near zero at all scales, indicating a structural inability to learn dense retrieval without supervision. In contrast, CrossBERT builds these capabilities naturally, scaling linearly with compute.

• Frozen outperforming unfrozen finetuning: At larger scales, CrossBERT with frozen adaptation begins to outperform even the unfrozen baselines. This indicates that CrossBERT scales more effectively than standard BERT, avoiding early saturation. It produces representations that are naturally richer, rendering the heavy process of unfrozen finetuning unnecessary, and eventually inferior, to a lightweight adaptation of the frozen features.

5.3.0.4 Training stability.

Self-supervised pre-training can suffer from unstable optimization, so we monitor training across the full scaling range. Two observations indicate that CrossBERT trains stably. First, the pre-training validation loss decreases smoothly and tracks the expected scaling law at every compute budget (14). Second, downstream probing performance rises steadily over the course of training rather than oscillating or collapsing (6). We observe no divergence across scales. In contrast higher scale BERT trainings suffer from significant instability and loss spikes (see 17).

6 Related Works↩︎

6.0.0.1 Standard Encoders.

Since BERT [17] and RoBERTa [26], encoders have relied on flat architectures where representation and reconstruction are mixed. Recent updates like ModernBERT [3] and NeoBERT [4] scale this design but inherit its structural inefficiency. As a result, brute-force scaling of this approach yields diminishing returns for representation quality [2].

6.0.0.2 Contrastive Learning.

To compensate for this misalignment, the field relies on heavy post-training [1], [30], [31] or token-level objectives like MEXMA [32]. These methods effectively treat the symptoms, but we posit that a better aligned pre-training can reduce the amount of adaptation needed to obtain the final model.

6.0.0.3 Architecture Design.

In computer vision, Masked Autoencoders [16] established the efficacy of asymmetric designs, where a lightweight decoder reconstructs pixels from highly masked inputs. [9] further demonstrated that reducing interaction between representation and reconstruction, using cross attention only, enhances produced features. In NLP, while T5 [33] also utilizes a bipartite structure, it relies on a heavy decoder optimized for autoregressive text generation employing both self and cross attention. In contrast, CrossBERT adopts asymmetry strictly for representation learning: rather than generating text, we employ a lightweight, cross attention only, predictor solely to offload the reconstruction burden, ensuring the encoder optimizes for semantic abstraction rather than token prediction.

7 Limitations↩︎

Our analysis is empirical. We identify the failure mode of flat MLM through controlled experiments (1 and 1), but we do not give a theoretical account of why coupling representation and reconstruction degrades frozen features as models grow. Our conclusions also rest on models no larger than a few billion parameters; every trend we observe is monotonic across this range, yet we cannot formally exclude qualitatively different behavior at the much larger scales typical of decoders. Finally, we find that RTD (Electra) yields strong token-level probes but poor sentence embeddings, and a mechanistic explanation of this collapse lies outside the scope of this work.

8 Conclusion & Future Work↩︎

We revisited the design of text encoders by shifting the evaluation focus from full finetuning to frozen representation quality. This change in perspective revealed that flat BERT-like architectures trained with MLM suffer from a fundamental misalignment: as compute scales, representations become increasingly unexploitable for downstream tasks, overspecializing on reconstruction at the expense of versatility. CrossBERT resolves this by insulating representation learning from the token prediction task. Beyond improving training efficiency, our results show an interesting trend: at large scales, CrossBERT produces frozen features that outperform fully finetuned baselines. This result challenges the expensive downstream contrastive adaptation of standard encoders, demonstrating that the right pre-training incentives can produce significantly richer representations needing less heavy adaptation to be effective.

Several directions follow naturally from these findings. The bipartite design is agnostic to the reconstruction objective, so MLM could be swapped for alternative pretext tasks better suited to representation learning. A systematic comparison with T5-style pre-training, where the predictor is replaced by an auto-regressive decoder would be an interesting avenue to explore.

Acknowledgements↩︎

We thank Badr Youbi Idrissi and João Maria Janeiro for the helpful discussions and feedback that shaped this work.

Impact Statement↩︎

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

9 Predictor shape design ablation↩︎

4 analyzes the trade-off between predictor size and downstream performance. At the lower bound, the 8M parameter predictor acts as a representational bottleneck, noticeably impairing model quality. However, increasing capacity to just 25M yields a substantial boost. Beyond this point, we observe diminishing returns; scaling further to 50M or 85M incurs higher computational costs for only marginal performance gains. While we ultimately selected the 42M configuration (one-fourth of the encoder depth, grey row) to maintain dimensional alignment with the backbone, the data suggests that the 25M variant remains a highly competitive alternative, promising faster pretraining speed.

10 Complementary Masking ablation↩︎

11 BERT Masking ablation↩︎

12 Predictor MTEBv2 contrastive finetuning ablation↩︎

13 MTEB(eng,v2) Results of the Single-scale runs.↩︎

14 Learning curve of scaling laws↩︎

15 Downstream performance evolution during training↩︎

16 Learning rate and batch size sweeps↩︎

17 Large scale training stability↩︎

Because scaling BERT resulted in a severe degradation of downstream performance, we further investigated this collapse across various training setups. To determine if this instability is specific to BERT, we additionally scaled Electra to 1B parameters. As shown in 9, both BERT and Electra exhibit a similar performance collapse at the 1B scale. For BERT, altering standard hyperparameters—including data and model seeds, learning rate, and batch size—failed to prevent the degradation. While specific hyperparameter tuning allowed Electra to avoid collapsing, this extreme sensitivity highlights the inherent brittleness of training standard ‘flat’ architectures at scale.

Conversely, CrossBERT is highly robust, achieving strong out-of-the-box performance without requiring such exhaustive hyperparameter sweeps. When scaling further to 2B parameters, standard configurations initially failed to converge. However, we found that simply reducing the initialization standard deviation from the conventional 0.02 (standard across ViT and BERT implementations) to 0.015 successfully stabilized the training. Furthermore, while Electra historically demonstrates superior performance at smaller scales, CrossBERT 1B significantly outperforms Electra 1B on GLUE, confirming the superior scaling trajectory of our proposed architecture.

18 List of Models with hyperparameters↩︎

BERT Model Configurations 4pt

CrossBERT Model Configurations

References↩︎