1 Introduction↩︎

Brazilian Portuguese is spoken natively by over 200 million people, yet a practitioner who needs to deploy a sentence-embedding model for it, whether for semantic search, classification, or retrieval-augmented generation, has no comprehensive native benchmark to guide that choice. The dominant evaluation frameworks, MTEB [1] and its multilingual successor MMTEB [2], do include Portuguese tasks, but they are a small fraction of MMTEB’s 500-task suite, and the most heavily reported Portuguese retrieval task is mMARCO-PT [3], a machine translation of English MS MARCO. Translation introduces systematic artifacts (translation noise, domain drift, idiomatic flattening) that can mask real differences between models on native text [4].

Language-specific MTEB extensions have closed this gap for Chinese [5], Scandinavian [6], French [7], Polish [8], Russian [9], Persian [10], Dutch [11], German [12], Vietnamese [4], and Japanese [13], but no comprehensive native one existed for Portuguese. We close that gap, and add a layer of statistical rigor that complements the established MTEB methodology. We make four contributions:

1. A native, non-translated task suite. We curate 22 tasks across seven MTEB categories from existing Brazilian-Portuguese resources spanning legal, medical, tax, scientific, encyclopedic, and social-media domains, excluding machine-translated benchmarks (notably mMARCO-PT and mkqa-PT) by construction.

2. A large and diverse model panel. We evaluate 93 models: 73 open-weight (23M–27B parameters) and 20 closed commercial APIs, covering the proprietary embedding services alongside the open ecosystem.

3. A statistical-rigor layer. For every headline comparison we report per-task bootstrap confidence intervals, paired-bootstrap \(p\)-values, Item Response Theory discrimination at the task and instance levels, and a Borda-count robustness check. This layer shows the benchmark cleanly orders \(78.7\%\) of all model pairs into about a dozen distinguishable tiers and places the leader above 87 of 92 models, while the top six converge into one unresolved frontier tier (§6).

4. A cross-leaderboard agreement analysis. We correlate MTEB-PT against the live HuggingFace MTEB multilingual leaderboard (\(\rho = 0.75\), over the \(n = 55\) models common to both) and find it only a moderate predictor of Portuguese rank: one model ranks 3rd of those 55 on that board but 49th of them here (§7).

All code (Apache-2.0, archived at https://doi.org/10.5281/zenodo.21087217), the full results dataset (CC-BY 4.0, archived at https://doi.org/10.57967/hf/9377), and an interactive leaderboard¹ are released openly.

2 Related Work↩︎

Embedding evaluation has converged on a lineage of benchmarks of growing scope: SentEval [14] for STS and classification probes, BEIR [15] for heterogeneous retrieval, MTEB [1] for eight task categories over 58 datasets, and MMTEB [2] for over 500 tasks across 250+ languages. MMTEB introduced the Borda aggregation and the inter-task-correlation task-selection device we build on, though its per-language coverage for lower-resource languages remains limited, Portuguese among them.

These language-specific extensions vary in scope and rigor: as originally published, most reported on the order of 40–50 models, and few included more than one or two closed commercial APIs. Closer to the retrieval-only tradition, MIRACL [16] offers native multilingual retrieval over 18 languages (Portuguese not among them), and BEIR-PL [17] exemplifies the translate-and-evaluate workflow we avoid. Portuguese also has prior benchmark collections, but they evaluate task-fine-tuned language models rather than embeddings: Napolab [18] gathers native NLU datasets (entailment, similarity, hate speech), and PORTULAN ExtraGLUE [19] machine-translates the GLUE suite into Portuguese. Neither measures the frozen sentence embeddings MTEB-PT compares.

These benchmarks have prioritized scope and coverage, with statistical reporting a secondary concern. Several have begun to add it: the Scandinavian Embedding Benchmark (SEB) [6] reports per-model rank confidence intervals, MTEB-French reports Critical Difference diagrams, and MMTEB introduced inter-task correlation as a task-selection device. We build on this direction for Brazilian Portuguese by reporting four complementary statistics together: per-task confidence intervals, paired-bootstrap significance across model pairs, IRT per-task discrimination, and a quantitative cross-leaderboard correlation.

3 Benchmark Design↩︎

The benchmark’s single inclusion criterion is native source. Every task draws on a corpus produced for, or naturally occurring in, Brazilian Portuguese, whether human-created, naturally found, or LM-generated and verified.² Machine-translated corpora are excluded by construction: translation noise, domain drift, and idiomatic flattening introduce artifacts that can mask real differences between models. Within that constraint we select for domain diversity, covering settings where an embedding model is realistically deployed: social-media classification, legal and tax retrieval, medical retrieval and clustering, open-domain retrieval, and semantic similarity and inference. The filter admits one LM-generated task, ToxSynPT, synthesized and human-verified directly in Brazilian Portuguese; synthetic generation may nonetheless introduce artifacts of its own.

The 22 headline tasks span seven MTEB categories: 4 classification, 1 multilabel classification, 2 pair classification, 2 STS, 5 clustering, 6 retrieval, and 2 reranking; these 22 formulations draw on 17 distinct corpora, as four sources (ASSIN, Quati, JurisTCU, MedPT) recur under multiple protocols. The retrieval weighting (6 of 22, against 1 of 8 in the original MTEB) reflects both the practical centrality of search and retrieval-augmented generation and the discrimination analysis of §8, where retrieval is the category that most sharply separates models; it is carried by four retrieval corpora beyond the reused Quati and JurisTCU sources (MedPTRetrieval, FaqBacen, FaQuAD-IR, BR-TaxQA-R). Table 1 gives each task’s source dataset, license, evaluation-set size, corpus or label structure, and citation.

3.0.0.1 Tasks.

Classification covers content moderation, fact-checking, and legal text: HateBR [20] (Instagram hate speech), ToxSynPT [22] (synthetic toxicity), FactckBr [21] (fact-check classification), and PortuLexRRIP [35] (8-way legal rhetorical-role identification); BrighterEmotion [36] adds multilabel emotion classification. Pair classification and STS draw on the ASSIN families: AssinRTE [23] and InferBR [24] (entailment), and AssinSTS [23] and Assin2STS [25] (similarity). Clustering spans medical (MedPTClustering [26]), encyclopedic (the Wikipedia-category task introduced here), legal (JurisTCUClustering [28]), scientific (SciELOClustering [29]), and technical (StackoverflowClustering [30]). Retrieval spans open-domain (Quati [32]), legal (JurisTCU [28]), tax (BRTaxQA [34]), banking FAQ (FaqBacen [33]), factual QA (FaQuADIR [31]), and medical (MedPTRetrieval [26]); reranking reformulates the Quati and JurisTCU corpora as top-\(k\) re-ranking. We report each model’s score as the unweighted mean of its primary metric across the 22 headline tasks, hereafter the 22-task mean (written Mean\(_{22}\) in tables), following the averaging convention of MTEB.

3.0.0.2 Exclusions.

Several candidate datasets are kept out of the headline suite. mMARCO-PT and mkqa-PT are machine translations of English benchmarks and fail the native-source filter. A second, empirical hazard reinforces this. Because MS MARCO is ubiquitous in retrieval-model training and mMARCO is its translation, scores on translated mMARCO-PT partly reflect prior exposure to that corpus rather than embedding quality, rising monotonically with a model’s MS MARCO training (Appendix 18). The native retrieval tasks it displaces give a less train-on-test measurement of the same capability. We further exclude a clustering set that is degenerate (every model scores 1.000, as each cluster is a distinct news event with non-overlapping vocabulary, leaving no ranking signal), a news-clustering set that mirrors copyrighted content under no permissive license, and a classification variant with unclear provenance and an undeclared license. Beyond these named cases, the headline suite is the survivor of a wider screen. We tested many more Brazilian-Portuguese candidate tasks than we keep, dropping them by construction for degeneracy, low discrimination, unclear quality, or licensing. One filter was the per-task discrimination analysis of §8, which removes tasks at or below the random-baseline floor of §6. The discriminations reported in §8 are therefore those of the survivors, not of the candidate pool, and are biased upward; we report them to document the screen, not to claim that every retained task discriminates well. The final discrimination cut leaves the ranking essentially unchanged (Spearman \(0.99\) between the 22-task means before and after it).

3.0.0.3 Diversity diagnostic.

To rule out accidental near-duplication, we follow BEIR [15] (also used by MTEB-NL [11] and FaMTEB [10]): we embed 100 documents per task and correlate the per-task mean embeddings. So that the result reflects corpus content rather than a single encoder’s idiosyncrasy, we average over a panel of three architecturally diverse models: a multilingual encoder (multilingual-E5-base), a decoder-LLM embedder (Qwen3-Embedding-0.6B), and a Portuguese-specific encoder (Serafim-100m); the per-model similarity matrices agree closely (mean pairwise Spearman \(\rho = 0.82\)). Raw cosine is inflated because embedding vectors tend to share a common direction (anisotropy), which compresses every pair into a narrow high-similarity band, so we mean-center the task embeddings before correlating. After centering, tasks are on average nearly unrelated (mean pairwise similarity \(-0.04\)), and tasks in the same category are more alike than tasks across categories (\(+0.04\) versus \(-0.06\)); the single \(1.00\), AssinRTE–AssinSTS, which share the ASSIN corpus and differ only in label, stands out as the lone genuine near-duplicate (Figure 1). The inter-task ranking-agreement analysis of §8 reaches the same conclusion from model rankings rather than corpus content.

4 Model Coverage↩︎

We evaluate 93 models spanning three orders of magnitude in size, from 23M (all-MiniLM-L6-v2) to 27B (harrier-oss-v1-27b) parameters, grouped into five families; full per-model attributes (parameters, embedding dimension, context length) are given in Table 4 and released with the results.

Decoder-LLM embedders (32) adapt autoregressive language models as embedding backbones through contrastive fine-tuning: the Qwen3-Embedding [37], Octen [38], F2LLM-v2 [39], harrier-oss [40], SFR-Embedding [41], and gte-Qwen2 [42] series, Linq-Embed-Mistral [43], KaLM-Embedding-Gemma3-12B [44], Llama-Embed-Nemotron-8B [45], BidirLM [46], the BOOM checkpoint, and the Brazilian-legal-tuned jua models [47], among others. Multilingual encoders (12) are from-scratch encoder-only models: the multilingual-E5 [48], paraphrase-multilingual [49], granite-embedding [50], and Jina-v5 [51] families, plus BGE-M3 [52] and Snowflake-arctic-embed-l-v2.0 [53]. Proprietary APIs (20): Google Gemini-Embedding (001 and -2) and text-embedding-005, OpenAI text-embedding-3 (small and large), the Voyage 3, 3.5, and 4 generations (with their context, finance, and law variants), Mistral-embed and Codestral-embed, and Amazon Titan-Embed-Text-v2, each via its official endpoint at list price (§9). We evaluate every commercial embedding endpoint we could reach through a public API during the evaluation window, including Voyage’s domain-specialized and context variants; coverage is consequently broader for vendors that expose more endpoints. Portuguese-specific encoders (16): the PORTULAN Serafim [54] and Albertina [55] families, BERTimbau [56], the Legal-BERTimbau and stjiris-legal-BERT variants [57], and the MedLink biomedical bi-encoder. Sentence-Transformer baselines (13) anchor the low end: LaBSE [58], all-mpnet-base-v2 and all-MiniLM [49], embeddinggemma-300m [59], GTE-small [42], PIXIE-Rune-v1.0 [60], and PwC-Embedding [61], alongside the GIST, BGE-small, E5-small, and mxbai-embed-large public checkpoints; every model’s identifier and attributes appear in Table 4.

5 Evaluation Protocol↩︎

We adopt the per-category metrics of MTEB [1], inherited unchanged by every language-specific extension, so MTEB-PT scores remain comparable to global MTEB up to the language-domain shift: classification is accuracy from a logistic-regression probe over 10 train/test resamples; multilabel classification is subset accuracy (the fraction of instances with every label predicted correctly); pair classification is held-out accuracy; STS is the Spearman rank correlation (1 means identical order) between cosine similarity and human judgment; clustering is V-measure (a clustering-quality score from 0 to 1) over 10 \(k\)-means runs; retrieval is nDCG@10 (a ranking score from 0 to 1 that rewards placing relevant documents in the top 10); and reranking is mean average precision (MAP, precision averaged along the ranking). For every (task, model) pair we publish the headline score, the per-instance score distribution where it is well-defined (per-query nDCG@10, per-experiment V-measure, per-resample accuracy), and the dataset revision SHA, all as Parquet in the results repository.

5.0.0.1 The statistical layer.

For every headline comparison we report a set of complementary quantities, each answering a plain question about the ranking. Bootstrap confidence intervals ask how much a score would move if the benchmark were re-run on a resampled task suite: a 95% percentile-bootstrap CI from 10,000 resamples of the underlying score distribution. (The two pair-classification and two STS tasks lack per-instance distributions in mteb 2.12, so we bracket them with standard analytic confidence intervals (Fisher-\(z\) for the STS correlations, Wilson for the pair accuracies), tight for the leading models at \(\pm 0.013\)–\(0.016\).) Paired significance asks whether one model reliably beats another on the same tasks, by resampling the 22 tasks with replacement (10,000 resamples) and recomputing the paired difference. From the same resamples we read two plain summaries used in §6: the probability that a model is best (the fraction of resamples in which it attains the top mean) and pairwise resolution (the fraction in which one model outranks another); models that no pair separates at \(95\%\) form a tier. Item Response Theory treats the score matrix as a test: a task’s discrimination \(a_t\) measures how sharply it separates models of similar ability, and a model’s ability \(\theta\) is a skill estimate that weights each task by that discrimination (§8); retrieval tasks score \(a_t \in [1.49, 2.05]\) and clustering tasks \(a_t \in [0.67, 1.22]\). Borda count is a preference-vote ranking, each task casting a vote, robust to the odd outlier task [62] and agreeing with the 22-task-mean ranking at Kendall \(\tau = 0.91\) (a rank-agreement measure where 1 is identical order).

6 Headline Results↩︎

Table 2 ranks the top 15 of 93 models by the 22-task mean, with 95% bootstrap confidence intervals. Across the full panel, the 22-task mean spans \(0.248\) (paraphrase-multilingual-MiniLM-L12-v2) to \(0.682\) (gemini-embedding-001), against a random-embedding baseline whose 22-task mean is \(0.18\), with per-task chance levels that vary sharply by category (near zero for retrieval, higher for classification and clustering; Appendix 13); the complete 93-model ranking is on the interactive leaderboard. The leaders mix closed commercial APIs and large open-weight decoder-LLM embedders.

6.0.0.1 Resolution across the field.

A useful benchmark should order the models it is given, and MTEB-PT does so across most of the field. Of all 4,278 model pairs, \(78.7\%\) are cleanly separated (one model’s 22-task mean beats the other’s in over \(95\%\) of task-bootstrap resamples); adjacent-rank pairs are the hard case, and only about \(3\%\) of them resolve, so what the benchmark orders cleanly is distant pairs, not near-neighbours. Grouping models it cannot separate yields roughly a dozen tiers (8 to 15 as the resolution threshold moves from \(0.99\) to \(0.90\)), and the leader, gemini-embedding-001, stands above 87 of the other 92. The 22-task mean spans 43 points end to end, with tight rank intervals through the mid-field (Table 2).

Only at the very top do the models stop separating. A typical top-six pair differs by about \(0.005\) in mean while individual tasks vary by roughly \(0.05\), so separating adjacent frontier ranks would take hundreds of tasks. That is an upper bound rather than a goal, since a gap so small may itself be noise; finer resolution at the top is better sought through harder or more discriminating tasks (§8) than through more tasks of the present difficulty.

6.0.0.2 The converged frontier tier.

Within that cluster the leaderboard converges. The six leaders’ means fall within \(0.020\) of one another, and the paired bootstrap does not resolve their order. gemini is nonetheless the leader: it leads the unweighted mean and every robust aggregation we tried, a per-task z-score mean, a discrimination-weighted mean, and Borda count, and across task-resampled suites it holds first place in \(66\%\) of draws against \(13\%\) for the next. The one summary that ranks it second is a least-squares Item Response Theory ability fit (§8), and only by under \(0.001\) standard deviations. This convergence is a property of these frontier models, not of the benchmark, which separates the cluster decisively from the rest of the field; within it the choice turns on cost (§9), license, latency, and context length rather than score.

6.0.0.3 Robust to aggregation and within-task noise.

The frontier convergence is not an artefact of how scores are combined. First, the paired bootstrap resamples the 22 tasks, treating each task’s score as fixed. Its interval therefore measures how much the ranking depends on which tasks are in the suite, not sampling error within a task. That within-task noise is small: even the smallest task (Quati, 50 queries) has a per-query standard error of \(0.035\), which contributes only \(0.0016\) to the 22-task mean, well below the \(0.039\) spread the resample already captures across tasks. A paired per-query two-level bootstrap confirms this: resampling within tasks barely moves the top-tier interval. Second, the ranking is robust to the aggregation weight: a category-balanced mean (equal weight to each of the seven categories rather than to each task) reorders the panel at Kendall \(\tau = 0.91\), eight of the top ten unchanged, so the unweighted-mean convention (inherited from MTEB) is not driving the result despite the \(100\times\) spread in evaluation-set sizes.

Qwen3-Embedding-8B sits inside this cluster at rank two, alongside KaLM-Embedding-Gemma3-12B at three. It is open-weight (Apache-2.0) and self-hostable [37], yet the median of the other 92 models beats it on only one of the 22 tasks, and the closed APIs that lead retrieval do so by margins inside the bootstrap band. An open-weight model thus sits among the closed leaders on the cost–quality frontier. At the other end of the scale axis, the 308M embeddinggemma-300m ranks 13th of 93, within \(0.033\) of the leader: near-frontier quality at edge-deployable size.

6.0.0.4 Retrieval as the dominant separator.

Per-category means show retrieval spreading models most widely: it carries the highest per-task discrimination and the largest cross-model score variance of any category (§8, Table 3). No single model leads all of retrieval, classification, and clustering, and the clustering tasks separate models weakly (the lowest-discriminating tasks are split between clustering and two classification probes, §8). We take up the discrimination-theoretic basis for this in §8.

6.0.0.5 Portuguese-specific encoders and retrieval.

Brazilian-Portuguese-specific encoders trail general multilingual encoders on the retrieval and reranking tasks, but the gap reflects training objective and domain, not language specialization. The 16 Portuguese-specific encoders average \(0.331\) across these eight tasks against \(0.517\) for the 12 general multilingual encoders, a 0.186-point gap (Welch’s \(t\)-test \(t = -3.45\), \(p = 0.003\); a rank-based Mann–Whitney test agrees, \(p = 0.001\)). That comparison is confounded by training objective: the Portuguese-specific group includes four masked-language-model encoders never fine-tuned for retrieval (BERTimbau and Albertina, mean \(0.258\)) alongside legal-domain encoders specialized away from open-domain search. The general-purpose, retrieval-tuned Serafim encoders, by contrast, show no detectable gap from the multilingual group (\(0.516\) versus \(0.517\), \(\Delta \approx 0\), \(p = 0.99\)); with only three IR-tuned Serafim variants the sample is too small to prove they are equal, so we read this as finding no gap rather than as demonstrated equivalence, but it still places the difference in training objective and domain rather than language. Scale and breadth of contrastive training, not language-specific pretraining, plausibly govern retrieval quality here [48], [52].

6.0.0.6 The classification mirror image.

The same masked-language-model encoders that trail on retrieval rank among the leading models on classification, where MTEB fits a logistic-regression probe on frozen embeddings and so rewards linearly separable features rather than metric geometry. Averaged over the four classification tasks, the 335M BERTimbau-large [56] ranks 3rd of 93 (second among open-weight models, behind only the 27B harrier-oss) and the 110M BERTimbau-base 9th, both ahead of every open decoder-LLM embedder except harrier and of most closed APIs, though neither was fine-tuned for embeddings. An encoder can thus carry probeable class signal without the metric geometry open-domain retrieval demands: the same training-objective account read from the other side.

7 Cross-Leaderboard Agreement↩︎

Does a single-language benchmark add information beyond the existing multilingual leaderboards? If MTEB-PT rankings tracked the global HuggingFace MTEB multilingual leaderboard closely, a practitioner could treat the latter as a proxy and this benchmark would be largely presentational. We test the proxy hypothesis directly. Matching the 55 headline-panel models that appear on both leaderboards (HuggingFace snapshot of 2026-06-29; models matched by normalized name), the Spearman correlation between the HuggingFace MTEB multilingual rank and the MTEB-PT 22-task-mean rank is \(\rho = 0.754\) (\(n = 55\)), with a model-resampling 95% bootstrap interval of \([0.54, 0.89]\); the Pearson correlation is \(0.766\). The multilingual ranking carries real information about Portuguese performance, but even the upper end of that interval falls short of a usable proxy and roughly 40% of the variance is unexplained (\(R^2 \approx 0.59\)); the live board is a drifting, differently-composed reference, so the correlation should be read as moderate, not exact.

The agreement is uneven across categories: STS transfers most cleanly (\(\rho = 0.80\)), followed by reranking (\(0.75\)) and retrieval (\(0.74\)), with pair classification (\(0.73\)) and classification (\(0.68\)) in the middle and clustering least (\(0.62\)). Even STS, the highest-agreement category, stays near \(\rho = 0.80\), so the multilingual ranking leaves substantial residual variance in every category, and language-specific extensions are not redundant restatements of it.

The largest shift is Llama-Embed-Nemotron-8B [45] (Figure 2), a multilingual model ranking 3rd of these 55 on the multilingual leaderboard yet 49th of them on MTEB-PT, the largest such divergence in the panel. The divergence concentrates in retrieval: on the six MTEB-PT retrieval tasks it falls far below the panel median, while on classification and clustering the gap is smaller but in the same direction. One plausible cause is the composition of the multilingual leaderboard, which is dominated by English-mined data and mMARCO with little native low-resource retrieval; a model strong there need not transfer to native Brazilian-Portuguese retrieval. Part of this divergence is moreover a designed consequence: because MTEB-PT excludes translated retrieval such as mMARCO by construction, a model strong on that style of data necessarily diverges here, so the low correlation partly reflects our task selection rather than a deficiency in the multilingual board.

A second high-multilingual model diverges the same way for a different reason: e5-mistral-7b-instruct diverges less sharply, mid-pack on the multilingual board (26th of these 55) but 48th on MTEB-PT, its English-only Mistral-7B base carrying little native Portuguese. Nor is the divergence specific to this one live snapshot. The published MMTEB headline [2], built from a different task pool, model set, and aggregation, ranks multilingual-E5-large-instruct first among its open models and reports that broad multilingual pretraining compensates for model size. Yet that model places 19th of all 93 on MTEB-PT, behind the decoder-LLM embedders that lead native Brazilian-Portuguese; a second, independently-constructed multilingual reference reorders the same way.

The multilingual leaderboard remains a useful first filter (a model outside the global top tier rarely leads here), but it cannot substitute for native evaluation. We do not claim the MTEB-PT ranking is closer to real deployment quality: proving that would need a separate downstream task, which is outside this study.

8 Task Discrimination via Item Response Theory↩︎

A benchmark of twenty-two tasks raises a measurement question: which tasks actually separate models, and which contribute redundant or saturated signal? Item Response Theory (IRT), developed in psychometrics to estimate how sharply a test item distinguishes examinees of differing ability [63], answers this directly. We treat each task as an item and each model as an examinee and fit a two-parameter logistic (2-PL) response function, \[\hat{s}_{mt} = \frac{1}{1 + \exp\!\big(-a_t\,(\theta_m - b_t)\big)},\] where \(\hat{s}_{mt} \in (0,1)\) is the predicted normalized score of model \(m\) on task \(t\), recovering a discrimination \(a_t\) (the slope of the task’s response curve) and a difficulty \(b_t\) for every task, and a latent ability \(\theta_m\) for every model. IRT has been applied to NLP evaluation before (to scale item difficulty [64], to select informative examples for cheaper evaluation [65], and to re-weight leaderboard examples [66]), but always at the instance level, where items are individual test examples and the response is dichotomous. We apply it at two granularities: here at the task level, as a benchmark-design diagnostic over the suite’s twenty-two tasks, and at the canonical instance level in §8.1. To our knowledge no benchmark in the MTEB family reports either.

Because benchmark scores are continuous in \([0,1]\) rather than dichotomous, we fit the 2-PL by least squares (gradient descent, 5,000 iterations, \(\theta\) mean-centered for identifiability) on the 93-model panel, rather than by the standard dichotomous-response maximum likelihood. We report the recovered discriminations \(a_t\) and difficulties \(b_t\) in Table 3; the model abilities \(\theta_m\) reproduce the 22-task-mean ranking (Spearman \(\rho = 0.99\)).

Retrieval tasks discriminate models sharply; clustering and the weakest classification tasks do not. The six retrieval tasks occupy the top six positions in Table 3, led by MedPTRetrieval (\(a_t = 2.05\)) and FaqBacen (\(a_t = 1.88\)). The two lowest of all are classification tasks: fact-check verification (FactckBr, \(0.47\)) and legal rhetorical-role identification (PortuLexRRIP, \(0.51\)), followed by three of the five clustering tasks (\(a_t\) between \(0.67\) and \(0.81\)). Averaged by category, classification is the lowest at \(0.83\) (\(47\%\) of retrieval’s \(1.75\)), but that average is pulled down by those two probes while HateBR (\(1.13\)) and ToxSynPT (\(1.21\)) discriminate well; the more uniform clustering category averages \(0.89\), and multilabel classification, STS, pair classification, and reranking fall in between (all near or above \(1.0\); Table 3). This has a practical reading: a fixed budget for separating similar-ability models is better spent on retrieval coverage than on clustering or the low-signal classification probes, which carry little of the ability-ranking signal at the headline level.

Bootstrap CIs on \(a_t\) (resampling the 93-model panel 200 times, so they gauge stability under panel composition rather than classical estimation error; Table 3) make the category-level separation robust (Quati at \(1.78\) \([1.62, 1.95]\) does not overlap MedPTClustering at \(0.71\) \([0.60, 0.84]\)), while adjacent within-category ranks overlap, so the ordering is reliable by category, not as an exact per-task rank. Read the \(a_t\) as a descriptive, panel-relative diagnostic, not a psychometric estimate. It is a least-squares 2-PL fit on continuous scores with a single latent ability, so its values are meaningful only in their ordering and ratios. As expected for a single-factor fit, they track the raw per-task score variance closely (\(\rho = 0.88\), Table 3). The task-level \(a_t\) thus compactly restate which tasks spread models widely. The generative uses of IRT (efficient item selection, misfit as a data-quality signal) we reserve for the instance level (§8.1), where the response is dichotomous.

8.0.0.1 Model ability.

Fitting the same 2-PL as an ability model, a latent \(\theta_m\) per model estimated jointly with the task parameters, recovers the 22-task-mean ranking closely over the full panel (Kendall \(\tau = 0.91\)). At the very top the two disagree: this least-squares ability fit, which we read as descriptive rather than psychometric, places voyage-context-4 marginally above gemini, by under \(0.001\) standard deviations, where the mean and the other aggregations place gemini first. The disagreement falls inside the converged frontier tier (§6) and does not settle its order.

A second view of redundancy, complementary to the corpus-content diagnostic of §3, is direct task-to-task ranking agreement: it ignores corpus text and reads only the score matrix. Figure 5 (Appendix 19) reports the Spearman rank correlation between every pair of tasks over the 93-model panel. The mean off-diagonal correlation is \(\bar{\rho} = 0.53\), ranging from \(-0.16\) (MedPTClustering vs.PortuLexRRIP, unrelated categories) to \(0.97\) (Quati vs.QuatiReranking, the same corpus under two protocols). Within-category agreement (\(\bar{\rho} = 0.67\)) exceeds across-category agreement (\(0.51\)); the clustering tasks are the exception, agreeing among themselves at only \(\rho = 0.47\), the lowest of any category and the same conclusion the IRT fit reaches by a different route. IRT asks which tasks separate models of similar ability; Spearman asks which tasks agree on the ordering. The two answer adjacent questions; since both are computed from the same score matrix, their agreement is expected rather than independent corroboration.

Finally, the headline ranking does not depend on the choice of aggregation. Across the 93 models, the Borda count (each task casting a preference vote with a tournament tie-break [62]) agrees with the simple per-task mean at Kendall \(\tau = 0.91\); the two orderings differ only for models with high cross-task variance, which Borda penalizes and the mean does not, and never by more than a few positions.

Nor does it depend on the suite’s shared-corpus task pairs. The ASSIN sentences seed both AssinRTE and AssinSTS, and the JurisTCU and Quati corpora each recur under multiple protocols (retrieval, reranking, and, for JurisTCU, clustering), so the 22-task mean treats some correlated signal as independent. Dropping the redundant reformulations (the two corpus-derived reranking tasks, the JurisTCU clustering task, and one of the two ASSIN tasks) leaves a ranking that agrees with the full 22-task mean at Kendall \(\tau = 0.97\) (Spearman \(0.997\), with \(7\) of the top ten unchanged), and leaves both the top-cluster spread and the cross-leaderboard correlation (\(\rho \approx 0.75\)) intact. The suite carries redundant pairs, but the headline conclusions do not rest on counting them as independent tasks.

8.1 Instance-level discrimination↩︎

The task-level fit asks which task types separate models. The same model applied within a task (items are individual test instances, the response is whether a model answers each instance correctly) is the canonical dichotomous 2-PL of the IRT-for-NLP literature [64], [65] and answers a sharper question: which instances carry the discriminative signal. We fit it by penalized marginal maximum likelihood (EM with Gauss–Hermite quadrature; a weak \(\log a \sim \mathcal{N}(0,1)\) prior regularizes low-variance items) on the four dichotomous tasks whose per-instance predictions are released (HateBR and ToxSynPT for classification, AssinRTE and InferBR for pair classification), over the 35–37 models with per-sample dumps. Recovered abilities reproduce per-task accuracy (Kendall \(\tau = 0.74\)–\(0.96\)). Two benchmark-relevant uses follow.

8.1.0.1 Efficient evaluation.

Ranking instances by Fisher information (a measure of how much a single test example reveals about a model’s ability) and scoring models on only the most informative ones recovers the full-task ranking with far fewer instances than random sampling. Across the four tasks the \(K = 50\) highest-information instances recover the full-task ranking at Kendall \(\tau = 0.66\)–\(0.90\), versus \(0.42\)–\(0.65\) for random selection of the same size; the margin is largest on the synthetic ToxSynPT (\(0.90\) vs \(0.50\), of \(5{,}208\) instances), whose LM-generated items may be more template-separable, and smaller on the human-annotated tasks (HateBR \(0.70\) vs \(0.52\)), so the latter is the conservative estimate. The advantage holds for \(K = 25\)–\(200\) and survives cross-model validation: selecting items on one half of the models and ranking the disjoint other half still beats random (\(\tau = 0.82\) vs \(0.55\) on held-out models at \(K = 50\), ToxSynPT). As a sanity check, selecting the least-informative instances instead drives the recovered ranking negative (\(\tau = -0.76\) at \(K = 25\)): informative items help and the least-informative ones actively hurt, which confirms the fit is finding real signal rather than noise [65].

8.1.0.2 A label-quality audit.

Instances whose correctness correlates negatively with model ability (strong models systematically wrong, weak models right) are item-misfit flags, \(11\%\) (ToxSynPT) to \(30\%\) (AssinRTE) of active items. Inspecting the worst HateBR cases is suggestive rather than a re-annotation (confirming them would require a held-out human audit), but they consistently surface annotation-subjective boundary instances rather than random noise or memorized duplicates, yielding a ranked, auditable list of the most fit-anomalous instances per task; on near-ceiling tasks some of these reflect model misfit rather than mislabeling, so the list is a label-review screen, not a verdict.

This is a case study, not a benchmark-wide claim. Per-instance predictions are released for seven of the twenty-two tasks; we fit the four with binary instance-level correctness (classification and pair classification), leaving aside the two continuous STS tasks and one further classification task. The fit spans 35–37 of the 93 models, the efficiency result depends on a correctly regularized fit (an unregularized joint-ML fit is degenerate), and pair-task correctness is threshold-dependent. Full per-task results are in Table 7.

9 Resource-Aware Model Selection↩︎

For deployment, the headline finding is that frontier quality does not require paying a per-token price: a free, self-hostable open-weight model, Qwen3-Embedding-8B (Apache-2.0, \(0.670\) on the 22-task mean), reaches the converged frontier tier (§6) alongside the top commercial APIs, at no per-token cost. The two classes are priced differently: a closed API charges per token, an open-weight model in the compute and memory its parameters demand. Both proxy the cost of producing an embedding, so we place each class on its own resource axis against the shared quality axis (Figure 3): the 20 closed endpoints by list price (USD per 1M input tokens), the open-weight models by parameter count.

Among the 20 priced endpoints (Figure 3a), four are Pareto-optimal: voyage-3.5-lite ($0.02/1M, \(0.620\)), voyage-4 ($0.06, \(0.643\)), voyage-context-4 ($0.12, \(0.668\)), and gemini-embedding-001 ($0.15, \(0.682\)); the other sixteen are dominated, and three of the four frontier points belong to Voyage. The frontier is shallow. A \(7.5\times\) increase from the cheapest to the top-scoring frontier endpoint ($0.02 to $0.15) buys only \(0.062\) in the 22-task mean, about three times the converged-frontier spread of §6, and pricing rather than scale shapes it: text-embedding-3-large ($0.13, \(0.645\)) is dominated outright by voyage-context-4, which is both cheaper and better. These positions hold under the 512-token evaluation cap (§5), which leaves voyage-context-4’s defining long-context capability inactive, so its frontier rank reflects short-input quality rather than the contextualized long-document embedding it is priced for. List prices are vendor-controlled and change without notice; the finding here is the structure of the frontier, its shallowness and the identity of the dominated endpoints, not the absolute dollar values, which we report as a dated snapshot.

The open-weight panel (Figure 3b) shows the effect of model size. Quality rises steeply with scale at the small end and then flattens: the frontier climbs from sub-100M encoders to a knee at embeddinggemma-300m (\(0.649\)), beyond which two further orders of magnitude in parameters add little, and the largest open models (harrier-27b, F2LLM-14B, llama-embed-nemotron-8b) fall below the frontier rather than extending it. Scale buys retrieval quality on average (§10) but is neither necessary nor sufficient at the top: a 0.3B model reaches near-frontier quality, and the frontier plateaus from there through the multi-billion range that Qwen3-Embedding-8B tops.

Read together, the two panels locate the models worth comparing and reduce the choice to operational axes. Within each frontier the scores fall inside the converged band of §6, so the frontier orders the resource axes, not the models. The benchmark does not resolve the models’ pairwise order there, and the decision then turns on latency, context length, region, and license. License in particular constrains the open option: several otherwise-competitive open models (Linq-Embed-Mistral, SFR-Embedding, Jina-v5) ship under non-commercial terms that bar production use, whereas Qwen3-Embedding is Apache-2.0. Where self-hosting is acceptable, an Apache-2.0 model at the frontier removes per-token cost entirely; where it is not, the priced frontier is shallow enough that the cheapest Pareto endpoint sits close to the best.

10 Discussion, Limitations, and Ethics↩︎

10.0.0.1 Discussion.

MTEB-PT discriminates the model field sharply. It resolves about a dozen distinguishable tiers and reports each with calibrated uncertainty rather than a bare point estimate. It is precise enough to establish that the six frontier models are statistically indistinguishable: a point-estimate ranking would invent an order among models that the benchmark places in a single tier. For the practitioner this makes the decision more precise, not less. Across the field the benchmark settles the ranking; within the frontier tier it identifies the models worth comparing on operational axes it does not measure, such as cost and license. And it answers that comparison favorably for open models, since a free, self-hostable open-weight model reaches the frontier tier at no per-token cost.

A transferable component of this work is the statistical layer. It lets the benchmark report a genuine tie as a tie and a real gap as a gap, and as embedding models cluster near the top in every language, uncertainty-aware evaluation becomes necessary rather than optional. We offer the layer, bootstrap intervals, paired significance, tier and probability-of-best estimates, IRT discrimination, and a Borda check, as a template for any language-specific MTEB extension, and we will contribute it upstream. Its discrimination estimates also point toward cheaper evaluation: the instance-level analysis demonstrates it directly, recovering the ranking from the most informative instances within a task, and the task-level discriminations suggest the same economy in choosing tasks, though we screen for discrimination rather than separately demonstrate task-subset recovery.

Finally, the results chart a path to better Portuguese embeddings. The apparent deficit of Portuguese-specific encoders on retrieval is a training-objective effect, not an effect of language: retrieval-tuned encoders on the same language match the multilingual models, while masked-language-model and legal-domain encoders trail. The productive route to stronger native retrieval is to fine-tune capable multilingual or decoder-LLM backbones on native Portuguese retrieval data; the mirror-image strength of those same encoders on classification shows that model choice should follow the task.

A further result concerns specialization. The apparent retrieval deficit of Portuguese-specific encoders (§6, \(\Delta = -0.186\), \(p = 0.003\)) is confounded by training objective: once the comparison is restricted to the general-purpose, retrieval-tuned Serafim encoders the gap is no longer detectable (\(\Delta \approx 0\), \(p = 0.99\)). With only three IR-tuned Serafim variants this is an underpowered null, not a demonstration of equivalence; moreover the group \(t\)-test treats models as independent, but the panel is a convenience sample with several correlated families (multiple Serafim and Legal-BERTimbau variants share checkpoints and training data), so its effective sample size is smaller than the model count and the reported \(p\)-values are optimistic. We therefore read this as relocating the apparent language gap onto training objective and domain, not as proving that language-specific pretraining is irrelevant. In our panel, parameter count correlates with retrieval quality (Spearman \(\rho = +0.63\) over the open models), consistent with the scale and breadth of contrastive training, rather than the language label, governing retrieval here; scale is nonetheless confounded with architecture, since the larger models are decoder-LLM embedders, and is neither necessary nor sufficient, with a 0.3B model outranking several encoders above 5B parameters.

10.0.0.2 Limitations.

• Translation exclusion: our native-source filter drops mMARCO-PT and mkqa-PT, the two largest machine-translated Portuguese retrieval corpora, which preserves signal but narrows retrieval-corpus diversity; we argue the trade-off is worth it (§3) while acknowledging it is contestable.

• Document-length bias: median document lengths are short (186 words for Quati, 116 for FaQuAD-IR, 43 for both JurisTCU and MedPTRetrieval), and only BR-TaxQA-R, at a 752-word median, exercises long-document retrieval; because we cap every input at 512 tokens (Appendix 14), even its documents are truncated, so models tuned for long contexts cannot show that advantage and are not stress-tested. A controlled check bounds the cost: on BR-TaxQA-R, lifting one model’s (bge-m3) cap from 512 to 2048 tokens raised its nDCG@10 by nearly a quarter, so the uniform cap has a measurable long-document penalty that we accept in exchange for cross-model comparability. Since retrieval is the category that most separates models (§8) and a frontier endpoint, voyage-context-4, is a long-context model, the cap conditions both the retrieval ranking and the cost-Pareto, not only an unmeasured ceiling.

• Small retrieval corpora: some retrieval tasks index small candidate pools (FaQuAD-IR over 244 paragraphs, MedPTRetrieval over 500), so their nDCG@10 measures ranking within a small set rather than open-domain search and is easier than large-corpus retrieval; that said, the IRT discrimination of §8 places both among the more separating tasks, so the small corpus does not flatten them.

• Register skew: the suite leans toward institutional and legal Portuguese. The JurisTCU corpus recurs three times (retrieval, reranking, clustering), and together with BR-TaxQA-R, PortuLexRRIP, and FaqBacen the legal and regulatory register accounts for six of the twenty-two formulations. Conversational, dialogue, and product-review text are not represented, so a score here speaks less directly to those deployment settings.

• Closed-API drift: closed APIs evolve silently; our scores are pinned to vendor identifiers and an evaluation window (closed-API queries and list prices are from June–July 2026), and should be re-checked at deployment.

• Pretraining contamination: several corpora are public web text (the Wikipedia-category, SciELO, and Stack Overflow clustering sets) and are likely present in the pretraining data of many evaluated models, so scores there may partly reflect memorized content rather than genuine embedding quality; the instance-level misfit audit (§8) targets mislabeling, not memorization, so we cannot rule this out for the web-sourced tasks.

• Bootstrap resolution: the clustering tasks have only 10 \(k\)-means experiments per model, limiting per-task bootstrap power, though the headline findings are robust to it.

• Small evaluation sets: the retrieval and reranking tasks built on Quati rest on 50 queries, and several classification probes on a few hundred test instances; the per-instance bootstrap CIs reflect this, and the smallest tasks carry correspondingly wide intervals.

• Descriptive group tests: the model panel is a convenience sample of available encoders rather than a random draw, so group-level comparisons such as Portuguese-specific versus multilingual (§6) describe this panel rather than a population of models.

10.0.0.3 Ethics.

The content-moderation tasks (HateBR, ToxSynPT) contain offensive text; we redistribute it as the dataset authors do, without further filtering, and recommend human-in-the-loop screening before any downstream deployment. The social-media source (HateBR, drawn from Instagram) may carry personal identifiers present in the original posts; we redistribute it only under the source dataset’s license and terms, add no new personal data, and recommend the same care before deployment. No native-speaker labor was uniquely employed beyond the original dataset authors’, credited to the original sources in §3 and Table 1. All code (Apache-2.0) and results (CC-BY 4.0) are released openly.

11 Conclusion and Future Work↩︎

We have introduced MTEB-PT, a native Brazilian-Portuguese text-embedding benchmark of 22 tasks across seven MTEB categories and six domains, curated from existing Portuguese resources and excluding machine-translated corpora by construction. We evaluate 93 models on it, 73 open-weight (from 23M to 27B parameters) and 20 closed commercial APIs. Its distinguishing element is a statistical layer that reports the leaderboard as a measurement with explicit uncertainty rather than a bare ranking: per-task bootstrap confidence intervals, paired-bootstrap significance for every headline comparison, Item Response Theory discrimination at the task and instance levels, and a Borda-count robustness check.

Several findings bear directly on the model-selection decision. The 22-task mean spans 43 points, from \(0.25\) to \(0.68\), and the benchmark cleanly orders \(78.7\%\) of model pairs into about a dozen tiers, placing the leader above 87 of 92 models. The six leaders fall within \(0.020\) and converge into one frontier tier that gemini leads but the benchmark does not resolve, so within that tier the choice turns on cost and license rather than score. A self-hostable open-weight model sits inside that leading cluster at no per-token cost, and a 308M model ranks 13th of 93, within \(0.033\) of the leader, placing near-frontier quality at edge-deployable size. The apparent deficit of Portuguese-specific encoders on retrieval tracks training objective and domain rather than language. And a model’s rank on the global multilingual leaderboard predicts its Portuguese rank only moderately (\(\rho = 0.75\) over 55 shared models, one of them 3rd there but 49th here), so a native benchmark measures something the multilingual boards do not.

These results come with limits: the panel is a convenience sample evaluated at a single point in time, the benchmark covers Brazilian Portuguese rather than the wider lusophone world, and it establishes that native and multilingual boards measure partly different constructs without adjudicating which is more deployment-relevant. The Item Response Theory layer also points beyond this benchmark, supporting adaptive evaluation that ranks models from fewer, higher-discrimination tasks and an overfitting diagnostic that tracks discrimination drift on near-duplicate data, both applicable to MTEB at large; natural next steps include a European-Portuguese dialect axis and contributing the statistical layer upstream to the mteb library as a reusable component.

For Brazilian Portuguese, a language of over 200 million speakers that until now had no consolidated native embedding benchmark, MTEB-PT lets practitioners choose models on native evidence rather than on translated or English-centric proxies, and gives researchers an extensible task suite and a reusable statistical template. All code (Apache-2.0), the full results dataset (CC-BY 4.0), and an interactive leaderboard are released openly under permanent DOIs, and we intend to keep the leaderboard current as new models appear.

Data and Code Availability↩︎

The evaluation code is released under Apache-2.0 and archived at https://doi.org/10.5281/zenodo.21087217. The full results dataset, comprising per-model per-task scores and per-query metrics, is released under CC-BY 4.0 and archived at https://doi.org/10.57967/hf/9377. An interactive leaderboard is available at https://huggingface.co/spaces/mteb-pt/leaderboard, and the task suite is distributed as an installable extension of the mteb library.

Acknowledgment↩︎

We gratefully acknowledge Verda for the GPU compute credits that supported this work.

12 Model Catalog↩︎

Table 4 lists every evaluated model with its parameter count, embedding dimension, maximum context length, open/closed status, and (for closed APIs) list price. Sizes span \(23\) M (all-MiniLM-L6-v2) to \(27\) B (harrier-oss-v1-27b).

13 Full Score Matrix↩︎

Table 5 gives every model’s score on each of the 22 headline tasks (the per-task primary metric), ranked by the 22-task mean. The same matrix, together with the per-instance and per-query parquet artifacts behind the confidence intervals, is released openly in the results repository.

1.2pt

14 Reproducibility Details↩︎

All evaluations are inference-only. Models are encoded on a single NVIDIA A100-80GB (Verda); the 20 closed APIs are called over HTTP and use no local GPU. We fix max_seq_length to 512 tokens across all models and tasks, encode in the model’s native precision, and apply each model’s documented pooling and padding configuration (mean, CLS, or last-token pooling, and left or right padding, per the model card; we verified the canonical configuration for every decoder-LLM embedder, where the wrong padding side silently degrades scores). Instruction-tuned models receive their default task-type prompt as shipped in the pinned mteb version; we add no custom instructions, so the prompt configuration is fixed by the model card and the mteb release rather than chosen by us. Every dataset is pinned to a HuggingFace revision SHA in its task wrapper. Classification uses a logistic-regression probe over an 80/20 stratified split with 10 resamples (the mteb default); clustering uses mini-batch \(k\)-means with n_init \(=10\) over 10 random seeds (random_state \(=0\ldots9\)). Per-instance bootstrap confidence intervals use \(10{,}000\) resamples; paired-bootstrap \(p\)-values use the per-task or per-instance difference vectors. We evaluated all models on mteb 2.12 (pair-classification accuracy computed per our released code); a three-seed check on HateBR (two models) returned a score standard deviation of \(0.0000\), confirming the classification evaluator is deterministic at fixed batch size on GPU. Closed-API scores were collected against the vendor identifiers in Appendix 12 during the evaluation window and should be re-checked at deployment, since endpoints evolve silently.

15 Construction of the Tasks Introduced Here↩︎

Several headline tasks are constructed or reformulated by this work from raw Brazilian-Portuguese sources; we document their construction below.

15.0.0.1 WikiCatClusP2P

is constructed by us from the Brazilian-Portuguese Wikipedia (wikimedia/wikipedia, 20231101.pt snapshot) via the MediaWiki categorymembers and extracts APIs. We traverse 15 broad subject categories (História, Geografia, Política, Esporte, Música, Cinema, Literatura, Religião, Ciência, Tecnologia, Animais, Plantas, Medicina, Filosofia, Astronomia) to depth 2, and take the first paragraph (between 80 and 500 characters) of each discovered article, labelled by the root category through which it was first reached. The released dataset (mteb-pt/wikipedia-categories, CC-BY-SA-3.0 inherited from Wikipedia) has 2,873 article paragraphs across the 15 clusters. Label noise can arise when an article legitimately belongs to several categories; we keep the first-discovery label and report this as a known limitation.

15.0.0.2 MedPTRetrieval and MedPTClustering

reformulate the AKCIT/MedPT medical question–answer corpus (CC-BY-4.0) into MTEB task formats. For retrieval we draw a stratified sample of 500 question–answer pairs (random_state \(=42\), balanced across the 7 question_type categories), using the 500 questions as queries and their 500 paired answers as the corpus with a \(1{:}1\) gold mapping. For clustering we sample \(\approx\!600\) questions across the 12 broad medical specialties (about 50 per specialty) and cluster by specialty.

15.0.0.3 SciELOClusteringP2P, StackoverflowPtClustering, and JurisTCUClusteringP2P

cluster native Brazilian-Portuguese texts by a coarse topic label drawn from each source’s own taxonomy. SciELO clusters scientific abstracts from the SciELO Brazil open-access library into 8 broad research areas; StackoverflowPt clusters technical question titles from the Portuguese Stack Overflow (pt.stackoverflow.com) into 10 technology tags; JurisTCU clusters Federal Court of Accounts (TCU) jurisprudence excerpts into 10 legal areas. Each released dataset (mteb-pt/scielo-clustering, mteb-pt/stackoverflow-clustering, mteb-pt/juristcu-clustering) carries the source label as the gold cluster.

15.0.0.4 FaqBacenRetrieval and FaQuADIR

are native retrieval tasks. FaqBacen pairs 373 citizen questions about Brazilian financial regulation with a pool of 1,673 unique answers from the Banco Central do Brasil public FAQ (mteb-pt/faq-bacen); FaQuADIR reformulates FaQuAD as academic retrieval, mapping 900 questions about Brazilian higher education to 244 source paragraphs drawn from 18 official documents (mteb-pt/faquad-ir). The two corpus-derived reranking tasks (mteb-pt/quati-reranking, mteb-pt/juristcu-reranking) reuse the Quati and JurisTCU corpora, restricting each query to its pool of human-judged candidate passages.

16 Statistical Significance at the Top↩︎

Table 6 reports the paired-bootstrap two-sided \(p\)-values (\(10{,}000\) resamples of the 22 headline tasks) for every pair among the ten leading models, the converged-tier detail behind the field-level resolution of §6. Of the 45 comparisons among the top ten, 39 fail to reject an equal mean at \(\alpha = 0.05\); of the six that do, all involve a model ranked seventh or lower, so no pair within the top six resolves. One survives a Holm–Bonferroni correction for the 45 tests: voyage-context-4 over voyage-3-large (rank four versus eight, \(p \approx 0.001\), below the corrected threshold of \(0.0011\)); the next-strongest, gemini over SFR-Embedding-Mistral (rank one versus ten, \(p \approx 0.002\)), sits just above it. The remaining four are marginal (\(p \in [0.02, 0.05]\)) and consistent with the \(\approx 2\) false positives expected at \(\alpha = 0.05\), so those should not be over-read. The pairwise-resolution and tier statistics of §6 derive from these same resamples: \(P(A{>}B)\) is the fraction in which one model’s 22-task mean exceeds another’s, a pair is resolved at \(P>0.95\), and tiers are formed greedily, a new tier beginning at the first model the current tier-leader beats at that threshold. Of the \(4{,}278\) pairs, \(78.7\%\) resolve; the top six do not, which is why they read as one converged tier.

2.6pt

17 Instance-Level IRT: Per-Task Results↩︎

Table 7 reports the per-task numbers behind the instance-level analysis of §8.1.

18 Translated Retrieval and Train–Test Overlap↩︎

We evaluate translated mMARCO-PT (unicamp-dl/mmarco, Google translation [3]) as a dense-retrieval task under the same nDCG@10 metric and encoding pipeline as the native retrieval tasks: 150 dev-small queries against a 30k-passage corpus of the relevant passages plus per-query BM25 top-200 hard negatives, over 73 open models spanning the ability range; the rank comparison below is over the 71 of these that also have native-retrieval scores. Because mMARCO-PT and the native tasks are encoded identically, each model’s mMARCO-versus-native difference is a within-model comparison.

On this panel mMARCO-PT separates models, and its ranking correlates with native retrieval (Spearman \(\rho = 0.84\)). Within that agreement, scores are systematically higher for models trained on MS MARCO. Sorting models into three cohorts by declared training data yields a monotonic rank-shift (Figure 4): models trained on the translated mMARCO, which includes the Portuguese split (F2LLM [39], and the Serafim-IR encoders, fine-tuned on the \(\sim\)​40M-triple Brazilian-Portuguese portion [54]), gain a mean of \(+6.2\) ranks on mMARCO-PT relative to native retrieval; models trained on English MS MARCO gain \(+1.2\); models trained on neither lose \(-3.2\). F2LLM tops mMARCO-PT at every size while ranking mid-pack natively, and Serafim-IR over-performs while its non-IR sibling does not. A model whose training explicitly excludes MS MARCO [50] ranks lower on mMARCO-PT than natively (Table 8), the no-exposure control.

Cohorts are assigned from self-declared training data, so any undisclosed MS MARCO training falls into the “neither” group and narrows the observed gap; the effect is a lower bound. The web-sourced clustering tasks (§10) carry a weaker, corpus-level exposure risk, since their text may appear in pretraining; but no model trains on their task signal, and native Portuguese is a small fraction of these models’ predominantly English pretraining, so the translated-retrieval overlap is the more direct one.

4pt

19 Inter-Task Ranking Agreement↩︎

Figure 5 gives the full Spearman rank-correlation matrix between the 22 headline tasks over the 93-model panel, the model-ranking-agreement diagnostic of §8. Retrieval and reranking tasks agree most (the \(0.97\) Quati–QuatiReranking pair is the highest), while the clustering tasks are the least correlated with the rest of the suite.

References↩︎