Graded Entity-Familiarity Readouts in Language Models:
Polish Adaptation, Cross-Language Robustness, and Refusal Steering1
July 15, 2026
Abstract
Can a language model estimate its familiarity with an entity before generating an answer? We study activations at the final prompt token in twelve instruction-tuned models from the Bielik, PLLuM, Gemma-4, and Qwen3 families, using a new dataset of 1,440 Polish entities spanning four domains and ten Wikipedia-pageview deciles, plus fabricated controls. Familiarity-probe scores separate real from fabricated entities in every family; in the Polish-adapted Bielik and PLLuM families they additionally track entity popularity (model-mean Spearman \(\rho\) 0.28–0.57, versus at most 0.11 in Gemma-4 and Qwen3), a pattern more strongly associated with Polish adaptation than with parameter count in this model sample. In a paired experiment on two families, probes retain 96–101% of within-language AUROC when the Polish question stem is replaced with an English one around unchanged entity names, showing robustness to prompt language in this setting. In Gemma-4-12B, the only model that natively refuses, adding a one-dimensional familiarity direction at a single layer moves refusal rates monotonically in both directions (0.24 to 1.00 on well-known entities; 0.73 to 0.00 on unknown ones). Finally, a calibrated familiarity probe is competitive among pre-generation abstention gates, although post-generation detectors better predict behavioral error on average. These results support a graded pre-generation entity-familiarity readout, and a separation between representational familiarity and the policy that converts it into abstention.
1 Introduction↩︎
When a language model is asked about an entity it has never encountered, it can either abstain or confabulate. Prior work shows that the model’s internal states contain information relevant to this choice [1]–[3], and [4] localized an entity-recognition mechanism in sparse-autoencoder space. Closest to our own work, [5] showed that supervised probes over an entity’s pre-generation hidden states predict graded QA accuracy. Most detection methods, however, score generated answers [6]–[9]. A recent study of the Polish Bielik family [10] instead measured the prompt point (the last token of the question, before any answer token exists) and found that unsupervised activation-dispersion metrics separate well-known from fabricated Polish entities at AUROC 0.95–1.00 across scales from 1.5B to 11B parameters, while the same signal says much less about whether the answer will be factually correct.
That result was binary (known vs.fabricated), single-family, single-language, and purely correlational. It therefore left open the questions that determine what the signal is:
RQ1 (language). Is the signal about the entity or the question’s surface form? If a probe trained on Polish questions transfers to English questions about the same entities, the readout is at least robust to the question-stem language.
RQ2 (gradation). Is familiarity a bit or a quantity? If the score tracks a continuous popularity axis, it behaves as a graded quantity, and the predecessor’s ceiling is a property of an easy contrast, not of the signal.
RQ3 (causality). Does the model itself read this signal, or is the familiarity–refusal link merely correlational?
RQ4 (application). Can the signal gate generation (abstain or route to retrieval before producing an answer) competitively with post-generation detectors, at a fraction of their cost?
We answer all four with one fixed one-forward-pass measurement protocol (§3), applied to a new popularity-graded dataset of Polish entities (§4) and twelve instruction-tuned models from four families spanning a Polish-pretraining axis, from Polish-native Bielik through PLLuM (Polish continual pretraining on Llama-3.1 and Mistral-NeMo bases) to the multilingual, non-Polish-centric Gemma-4 and Qwen3. Coverage varies by question: gradation (RQ2) uses all twelve models on the new dataset (v2) and gating (RQ4) the largest model per family on v2, while language transfer (RQ1; six models) and steering (RQ3; Gemma-4-12B) predate v2 and use the predecessor’s three-condition dataset (v1; §4).
Our contributions, in narrative order: (1) a QID-resolved, decile-stratified Polish entity dataset with fabricated anchors, token-length matched to the real distribution (§4); (2) evidence that the familiarity score is graded in the Polish-adapted families and that its gradation is more strongly associated with Polish adaptation than with scale in this model sample, with a behavioral mirror in answer correctness (§5); (3) zero-shot cross-language probe transfer at 96–101% of within-language performance in the two families tested, arguing against a large question-stem-language effect in this paired setting (§6); and (4) an intervention-plus-application study: a single rank-one familiarity direction steers Gemma-4-12B refusal monotonically in both directions (0.24\({\to}\)1.00 / 0.73\({\to}\)0.00), extending [4] with a dose–response characterization (§7), and a risk–coverage comparison against adapted post-generation baselines that positions the calibrated probe within the pre-generation gate class and quantifies the dissociation between representational familiarity and behavioral abstention policy (§8, §9).
2 Related Work↩︎
2.0.0.1 Internal-state hallucination detection.
Supervised probes on hidden states detect falsehoods and hallucinations [2], [3], [11]–[15]; semantic entropy measures answer instability across samples [9], [16], [17]; SelfCheckGPT is sampling-based and black-box [18]; unsupervised linear truth directions [19], [20] and verbalized confidence [21], [22] offer complementary readouts. Closest to our setting are question-only approaches: [1] elicit self-knowledge behaviorally, [23] train question-side probes, [24] find answerability encoded before generation, [25] detect hallucinations from pre-generation artifacts, and [4] identify entity-recognition features that causally modulate refusal in Gemma 2. [5] estimate entity knowledge from pre-generation hidden states with supervised probes that predict graded QA accuracy; this is the nearest precedent to our readout. We differ in the popularity-decile design with a fabricated anchor and in adding cross-family, cross-language, and interventional axes; of our two readouts, only dispersion is unsupervised and closed-form, while our best-performing probe is, like theirs, a supervised classifier. [26] carry entity-recognition probes into real-time detection. Linear representations are known to track pretraining term frequency [27], [28], and internal states reflect knowledge recall rather than truthfulness [29], converging with our familiarity-not-truth framing. Recent single-generation detectors such as D2HScore [6], EigenTrack [7], and MIND [8] score the answer’s hidden states; we compare against documented adaptations of all three (§8).
2.0.0.2 Popularity and the long tail.
Factual accuracy tracks entity popularity [30] and pretraining co-occurrence [31]. We turn this external axis into a probe of the internal signal: if familiarity is a readout of exposure, its score should be monotone in log-popularity.
2.0.0.3 Activation steering.
Adding difference-of-means directions to the residual stream steers behavior [32], [33]; inference-time intervention steers truth-related directions [34]; refusal in particular is mediated by a single direction [35], and conditional or surgical variants refine where and when to steer [36], [37]. We apply this methodology not to jailbreaking but to the epistemic axis: does moving along the familiarity direction move the model’s own abstention decision?
2.0.0.4 Polish evaluation.
Mu-SHROOM [38] and MultiHaluDet [39] cover Polish hallucination annotation; Bielik [40], [41] and PLLuM [42] are the Polish-adapted model families we study, alongside Gemma-4 [43] and Qwen3 [44]; PLLuM’s 8B and 12B variants build on Llama-3.1 [45] and Mistral-NeMo [46].
3 Familiarity Readouts↩︎
All measurements are taken at the prompt point: the final token of a one-sentence Polish question (Kim jest {entity}? Odpowiedz jednym zdaniem. for people; Czym jest … for places), in one forward pass, before generation. Following [10] we use two signal classes plus a baseline: dispersion (unsupervised: per-layer inverse participation ratio and Shannon entropy over the post-SwiGLU MLP activation vector; best (metric, layer) cell selected on an anchor contrast, here top-decile-vs-fabricated, oriented so higher = more familiar), a probe (logistic regression on the residual-stream hidden state, layer selected by 5-fold CV, training-contrast scores always out-of-fold), and first-token entropy. Conventions: separability \(=\max(\text{AUROC}, 1-\text{AUROC})\); bootstrap 95% CIs; fixed seeds. Functionally analogous readouts appear in every family (on Gemma-4-12B the best layers sit in the same mid-depth relative band as Bielik, 0.64–0.71 on athletes, and the per-head picture is equally diffuse: rank consistency 0.57 vs., 2 of 768 heads in all four domain top-20 sets), so one measurement recipe applies to every family without retuning. Because each readout is trained or selected per model, similar performance and layer placement do not establish that the families share an identical representation; cross-model transfer or representational-similarity tests would be needed for that stronger claim.
4 A Popularity-Graded Dataset↩︎
4.0.0.1 Why graded.
The predecessor’s known-vs-fabricated contrast is at ceiling already at 1.5B, capping what scale comparisons can reveal. A pilot on its 336 real entities found the familiarity score is not a bit but a graded popularity readout: Spearman correlation with log 12-month pl.wiki pageviews averaged \(\rho=0.75\) across 6 models \(\times\) 4 domains, surviving within the well-known tier alone; pageviews and Wikidata sitelinks were interchangeable. This motivated sampling the popularity axis continuously.
4.0.0.2 Design.
For each of four domains (athletes, cities, writers, musicians) we harvest a full candidate pool from Wikidata SPARQL (occupation- or class-filtered, Polish citizenship or location, Polish-Wikipedia article required), resolve every entity by QID
(excluding disambiguation pages by construction) and stratify 300 real entities into ten deciles of log 12-month pl.wiki pageviews (30 per decile), plus 60 fabricated anchor entities per domain: morphologically valid invented Polish names, screened for
non-existence by exact page lookup and full-text search, and selected so that real-vs-fabricated token-count AUROC stays near chance under both the smallest and largest Bielik tokenizers (achieved 0.50–0.57, target \(<0.65\)); a post-hoc check confirms the screen generalizes to all four family tokenizers (pooled token-count AUROC 0.49–0.52, worst single domain cell 0.56). All sampling is deterministic (random.Random(0)); the
totals are 1,200 real + 240 fabricated = 1,440 entities. The steering and language-transfer experiments (§6, §7) predate v2 and use the predecessor’s three-condition dataset (42
well-known / 42 obscure-real / 42 fabricated entities per domain), which we call v1.
4.0.0.3 Models.
Twelve instruction-tuned checkpoints: Bielik v3.0 (1.5B, 4.5B, Minitron-7B [47], 11B), Gemma-4 (E4B, 12B), PLLuM (4B; Llama-PLLuM-8B; 12B, Mistral-NeMo-based; all instruct-2512), and Qwen3 (1.7B, 4B, 14B, with thinking disabled). The families span the axis we care about: Polish-native pretraining (Bielik), Polish continual pretraining over non-Polish bases (PLLuM), and multilingual non-Polish-centric training (Gemma-4, Qwen3).
5 RQ2: Familiarity Is Graded and Aligned with Polish Adaptation↩︎
| Family | Model | B | disp. | probe |
|---|---|---|---|---|
| 1.5B | 1.5 | 0.367 | 0.351 | |
| 4.5B | 4.5 | 0.434 | 0.431 | |
| Minitron-7B | 7.0 | 0.405 | 0.441 | |
| 11B | 11.0 | 0.381 | 0.567 | |
| E4B | 4.0 | 0.111 | 0.093 | |
| 12B | 12.0 | 0.189 | 0.092 | |
| 4B | 4.3 | \(-\)0.002 | 0.294 | |
| Llama-8B | 8.0 | 0.280 | 0.359 | |
| 12B | 12.25 | 0.273 | 0.277 | |
| 1.7B | 1.7 | 0.039 | 0.076 | |
| 4B | 4.0 | 0.164 | \(-\)0.005 | |
| 14B | 14.0 | 0.130 | 0.111 |
4pt
5.0.0.1 The signal is graded.
Figure 1 shows per-decile familiarity curves; Table 1 and Figure 2 summarize gradation strength. Classifying each of the 48 model\(\times\)domain decile-vs-fabricated AUROC curves by their trend (Spearman \(\rho\) of the ten AUROCs vs.decile index: rising \(\rho\ge0.5\), falling \(\rho\le-0.5\), else flat), 27 of 48 are rising and none crosses the falling threshold. This trichotomy is coarse (ten deciles, an arbitrary \(\rho\) cutoff), so we read it as a descriptive summary rather than primary evidence; the per-cell correlations with uncertainty are in the released artifacts.
5.0.0.2 Gradation aligns with Polish adaptation, not scale, in this sample.
Within Bielik, the probe’s mean gradation grows from \(\rho=0.351\) (1.5B) to \(0.567\) (11B). But scale alone does not produce gradation: Qwen3 moves from \(0.076\) to \(0.111\) between 1.7B and 14B, and Gemma-4-12B sits at \(0.092\). The most informative observational comparison is PLLuM: after continual pretraining on Polish text, the Llama-3.1-based 8B reaches \(\rho=0.359\) and the Mistral-NeMo-based 12B \(0.277\), exceeding the matched-size non-Polish models by 0.17–0.27 absolute (Gemma-4-12B \(0.092\), Qwen3-14B \(0.111\)). We now have controlled before/after evidence for both PLLuM architectures: each base checkpoint grades well below its Polish-adapted descendant at matched architecture, tokenizer, and scale. Mistral-NeMo-12B (base of PLLuM-12B) scores mean prompt-point \(\rho=0.102\) against PLLuM-12B’s 0.277, and Llama-3.1-8B (base of PLLuM-8B) scores 0.193 against PLLuM-8B’s 0.359 – probe lifts of 0.175 and 0.166 from Polish continual pretraining alone (dispersion lifts 0.094 and 0.223). The Mistral-NeMo base sits at the non-Polish level, so the adaptation, not the architecture, produces the gradation. An exact permutation test over the twelve model-level mean \(\rho\) values (\(\binom{12}{7}=792\) family-label assignments) puts the Polish-vs-non-Polish gap at 0.315 for the probe (Cohen’s \(d\approx4.1\); \(p=0.0013\), the exact null’s resolution floor) and 0.179 for dispersion (\(p=0.019\)); the partial correlation with Polish adaptation controlling for log-size is 0.905. Because checkpoints within a family share corpora, tokenizers, and training pipelines, they are not independent replications: the effective sample is closer to four families than twelve models, so we emphasize the effect sizes over the checkpoint-level \(p\)-values.
5.0.0.3 Behavioral mirror.
We next test whether the internal gradient corresponds to anything behavioral. On the 320-entity evaluation subset per model, we sampled five answers per real entity and judged all of them with a strict factual-correctness rubric (claude-opus-4-8; 4,800 calls). Figure 3: Bielik-11B’s per-decile correctness rises monotonically from 0.06 (decile 0) to 0.68 (decile 9), mirroring the internal score’s gradient. Both correctness and the internal score share popularity as a plausible common cause, so we read the mirror as consistency between the two gradients; whether familiarity predicts correctness beyond pageviews and domain is an incremental-validity question we leave open (§10). Qwen3-14B is near floor everywhere (0.16 even at decile 9): it answers every Polish long-tail question with confident confabulation. PLLuM-12B is weak except at the very head (0.43 at decile 9). Gemma-4-12B is flat, but confounded: it is the only model that refuses, and the strict judge marks explicit refusals as factually correct \({\sim}88\%\) of the time. Conditioning on answering, Gemma-4’s correct-of-non-refused rises from 0.00 (decile 0) to 0.25 (decile 9), noisy but directionally consistent. A second judge outside the Claude family (gpt-5.2, byte-identical rubric) re-judged a stratified 20% of these verdicts (960 answers): pooled agreement is 89.8% at the answer level (\(\kappa=0.646\)) and 91.7% at the entity level (\(\kappa=0.693\)), and Bielik-11B’s decile-correctness monotone survives relabeling (Spearman 0.914 vs. on the identical subset). The one low-agreement cell (Gemma-4, \(\kappa=0.469\)) is precisely the refusal-scoring divergence above: the second judge scores refusals as correct far less often (positive rate 0.204 vs.Opus’s 0.371), corroborating the confound diagnosis. The mirror is not merely a popularity artifact: familiarity retains predictive value for behavioral error after controlling for log-pageviews and domain (likelihood-ratio \(p=0.013\) for Bielik-11B, \(p<0.001\) for Qwen3-14B), and a calibrated probe gate outperforms a popularity-only gate on Bielik (AURC 0.662 vs.).
6 RQ1: The Signal Is Robust to a Polish/English Stem Substitution↩︎
If the familiarity signal were a property of the question’s surface form (Polish morphology, tokenization quirks, template wording), it should degrade when the question language changes. We kept every entity string unchanged and switched only the question stem (Kim jest …? \(\to\) Who is …?), then trained the probe in one language at its CV-best layer and scored the other language zero-shot at the same layer, on identical entities (six models: four Bielik scales, two Gemma-4 scales; four domains; v1 conditions).
Figure 4: pooled over domains, cross-language AUROC retains 96–101% of within-language AUROC on known-vs-fabricated for every model (Bielik 0.961–1.005; Gemma-4 0.983–0.985), and 92–100% on the fabricated-string-free known-vs-obscure contrast. Every model is robust to the stem substitution in this paired setting, consistent with reports of language-agnostic latent processing in multilingual transformers [48]. Two design limits bound what this shows: the entity name, the largest semantic part of the prompt, is unchanged across conditions, and the cross-language evaluation reuses the entities the probe was trained on in the other language. To test transfer to unseen entities we re-ran the probe with entity-disjoint 5-fold splits (train one language on one entity set, test the other language on held-out entities, both directions). Transfer retains 98–100% of within-language AUROC on the same held-out split for the four Bielik scales, but only 74–93% for the two Gemma-4 scales: robustness to the stem swap generalizes to unseen entities in the Polish-adapted family and degrades for the multilingual model. A single template per language remains a limit (§10).
The design also argues against a specific alternative: if Gemma-4’s weaker Polish signal reflected an English-centric model struggling with Polish questions, English stems should strengthen its within-language signal. They do not (mean probe delta EN\(-\)PL: \(-0.034\) to \(-0.001\) across contrasts and scales). What limits Gemma-4 here is its familiarity with the entities, not the question language. One caveat is inherent: the language switch also changes the template. A within-Polish neutral-template control in the predecessor study showed the signal is robust to a template swap at these scales, which bounds, but does not eliminate, this confound (§10).
7 RQ3: The Familiarity Direction Is Causally Potent↩︎
| Refusal rate [Wilson 95% CI] | |||
|---|---|---|---|
| 2-4 Condition | \(\alpha{=}0\) | fam.\(\alpha{=}{+}8\) | fam.\(\alpha{=}{-}24\) |
| Known | 0.24 | 0.00 | 1.00 |
| [.14,.39] | [.00,.08] | [.92,1.0] | |
| Unknown-real | 0.81 | 0.00 | — |
| [.67,.90] | [.00,.08] | ||
| Fabricated | 0.64 | 0.00 | — |
| [.49,.77] | [.00,.08] | ||
| Refusal direction, L44\(\alpha{=}0\) \(\alpha{=}{-}12\) | |||
| Fabricated | 0.64 [.49,.77] | 0.31 [.19,.46] | |
| Unknown-real | 0.81 [.67,.90] | 0.40 [.27,.56] | |
3pt
7.0.0.1 Correlational prelude.
On Gemma-4-12B, the only model with a refusal behavior to predict, we first asked whether refusal is readable from the familiarity signal (v1 athletes, 42 entities per condition). Within the well-known condition it is: dispersion at the independently chosen known-vs-fabricated layer ranks refusals at AUROC 0.955 [0.878, 1.000] (zero-leakage probe 0.910; graded Spearman \(\rho\approx-0.55\)): the seven “known” entities Gemma-4 refuses are concentrated among those its activations assign lower familiarity. Within the unfamiliar conditions the mid-band score does not rank refusals (AUROC \({\sim}0.6\), CIs spanning 0.5), yet refusal remains strongly decodable from late prompt-point layers (out-of-fold 0.96–0.98; raw dispersion 0.89–0.94 at L42–47; Figure 5), suggesting a mid-depth familiarity representation and a late abstention readout downstream of it.
7.0.0.2 Intervention.
Correlation alone cannot settle RQ3, so we intervene. We extracted two rank-one directions from prompt-point activations (difference of condition means, unit norm): a familiarity direction at layer 30 (\(\text{mean(known)} - \text{mean(unknown-real} \cup \text{fabricated)}\)) and a refusal direction at layer 44 (\(\text{mean(refused)} - \text{mean(non-refused)}\) within the unfamiliar conditions). The two directions are geometrically distinct: cosine \(-0.14\), against an SD of \(0.016\) for random unit pairs in this residual space, a modest negative alignment rather than a shared axis, so the familiarity direction is not a re-labeled refusal direction. During generation we add \(\alpha \cdot \mathrm{rms}(h) \cdot d\) to the residual stream at every position of the single target layer, where \(\mathrm{rms}(h)=\lVert h\rVert_2/\sqrt{3840}\) keeps doses comparable across layers [32], [33], [35]. We ran 5,040 greedy generations over a 13-point dose grid, with three equal-norm random directions per layer as controls and a four-way degeneration guard (empty, repetitive, non-Polish, token-salad outputs).
7.0.0.3 Results (Figure 6, Table 2).
Pushing well-known entities toward “unfamiliar” (\(\alpha<0\) along familiarity) raises refusal monotonically from 0.238 to 1.000 at \(\alpha=-24\) (dose-curve Spearman \(\rho=-0.99\)); pushing unknown and fabricated entities toward “familiar” suppresses refusal from 0.726 to 0.000 at \(\alpha\ge+8\) (\(\rho\le-0.97\)). The model then fluently invents biographies: a confabulation proxy (coherent non-refusal answers about unknown entities) rises from 0.27 to 1.00. No steered output triggers any of the four degeneration heuristics; the heuristics screen for gross corruption (empty, repetitive, non-Polish, token-salad outputs) and do not certify preserved semantics or factuality beyond that. Random equal-norm directions never produce a single Polish refusal at any amplitude and instead destroy the output (degeneration 0.39–1.00; at the amplitudes where familiarity steering saturates refusal, random controls are 100% degenerate), so the effect is not a generic norm artifact. Random directions are a coarse specificity control, however: they show that arbitrary directions are destructive at these amplitudes, not that the familiarity direction is the unique semantic direction with this effect. Five-sample confirmations at temperature 0.7 reproduce the endpoints (known at \(\alpha=-24\): mean refusal 0.976, all 42 entities refuse at least once; unfamiliar at \(\alpha=+8\): mean refusal 0.019–0.024), and a 60-generation external audit of the refusal marker (claude-haiku-4-5) agrees with the marker metric at 0.95. These results converge with [4], who causally modulated refusal via entity-recognition SAE directions in a Gemma 2 chat model; we add the dose–response characterization, equal-norm random controls with degeneration accounting, sampled confirmations, and the late-layer refusal direction below.
7.0.0.4 The late refusal direction, and a reframing.
The L44 refusal direction is causally potent but asymmetric: pushing away from refusal works (0.726\({\,\to\,}\)0.357 at \(\alpha=-12\); confirmed at five samples), pushing toward refusal adds only +0.05 to +0.07 (baseline 0.73–0.81 is near ceiling), and on well-known entities it does nothing. The strict hypothesis motivating it, that the mid-band familiarity direction is causally inert where it fails to rank refusals, is falsified: familiarity steering saturates refusal in both directions within the unfamiliar conditions too. Correlational dissociation does not imply causal inertness. The familiarity direction is causally potent for refusal in all three conditions; the late refusal representation adds an independently steerable, mostly suppressive handle. We note the scope of this claim: the intervention is applied at every generation position, not only at the prompt point, and both directions are estimated on the same 42 entities per condition they are evaluated on. A held-out-entity check (correlational, saved activations only) shows the direction is not an artifact of the specific entities: built on a training subset, it separates known from unfamiliar held-out entities at AUROC 0.800 versus 0.802 in-sample, and train-fold directions have cosine 0.999 with the full-set direction. What is demonstrated is that a one-dimensional direction derived from familiarity contrasts strongly steers Gemma-4-12B’s refusal behavior; showing that the model’s natural prompt-time familiarity computation feeds the refusal decision would additionally require prompt-point-only interventions on held-out entities, or mediation-style evidence (§10).
8 RQ4: Gating Generation with a One-Pass Signal↩︎
| Method (cost) | -11B | -12B | -14B | -12B |
|---|---|---|---|---|
| (a) Real-vs-fabricated entity discrimination | ||||
| Our probe (1 pass) | 0.934 [.91,.96] | 0.871 [.83,.91] | 0.859 [.81,.91] | 0.903 [.87,.94] |
| Our dispersion (1 pass) | 0.746 [.69,.80] | 0.657 [.59,.73] | 0.666 [.60,.73] | 0.707 [.64,.77] |
| First-token entropy (1 pass) | 0.652 [.59,.72] | 0.560 [.49,.62] | 0.407 [.34,.48] | 0.627 [.56,.70] |
| Semantic entropy (5 gen + API) | 0.750 [.69,.80] | 0.477 [.40,.54] | 0.480 [.41,.55] | 0.585 [.52,.65] |
| D2HScore (1 gen)\(a\) | 0.629 [.55,.69] | 0.438 [.36,.51] | 0.506 [.44,.58] | 0.489 [.42,.56] |
| EigenTrack (1 gen, sup.)\(b\) | 0.621 [.55,.69] | 0.648 [.58,.71] | 0.535 [.46,.61] | 0.443 [.36,.52] |
| MIND (1 gen, sup.)\(c\) | 0.653 [.58,.72] | 0.677 [.61,.74] | 0.717 [.65,.78] | 0.601 [.53,.67] |
| (b) Behavioral: majority-incorrect vs.correct (strict judge, real entities only) | ||||
| Our probe (1 pass) | 0.764 [.70,.83] | 0.409 [.34,.48] | 0.914 [.80,.99] | 0.722 [.62,.81] |
| Our dispersion (1 pass) | 0.675 [.58,.77] | 0.262 [.20,.34] | 0.866 [.64,1.0] | 0.656 [.52,.78] |
| First-token entropy (1 pass) | 0.661 [.57,.75] | 0.355 [.29,.43] | 0.441 [.24,.67] | 0.629 [.51,.74] |
| Semantic entropy (5 gen + API) | 0.805 [.74,.86] | 0.761 [.71,.81] | 0.806 [.64,.95] | 0.782 [.67,.89] |
| D2HScore (1 gen)\(a\) | 0.696 [.62,.77] | 0.768 [.70,.83] | 0.409 [.21,.61] | 0.229 [.13,.33] |
| EigenTrack (1 gen, sup.)\(b\) | 0.684 [.60,.77] | 0.748 [.68,.81] | 0.503 [.26,.73] | 0.490 [.36,.63] |
| MIND (1 gen, sup.)\(c\) | 0.528 [.44,.62] | 0.778 [.72,.84] | 0.782 [.52,.98] | 0.858 [.79,.92] |
| Gate (cost) | B-11B | G-12B | Q-14B | P-12B | macro |
|---|---|---|---|---|---|
| SE (5g+API) | 0.697 | 0.381 | 0.944 | 0.811 | 0.708 |
| MIND (1g, sup.) | 0.785 | 0.353 | 0.967 | 0.841 | 0.737 |
| EigenTrack (1g, sup.) | 0.712 | 0.383 | 0.964 | 0.909 | 0.742 |
| D2HScore (1g) | 0.728 | 0.405 | 0.986 | 0.972 | 0.773 |
| Probe (1p, pre-gen) | 0.701 | 0.712 | 0.923 | 0.852 | 0.797 |
| Dispersion (1p) | 0.719 | 0.771 | 0.916 | 0.852 | 0.815 |
| FT entropy (1p) | 0.722 | 0.745 | 0.985 | 0.886 | 0.834 |
| Base error (no gate) | 0.841 | 0.600 | 0.978 | 0.922 | — |
1.7pt
| Answers refused | Per-answer refusal | |||
|---|---|---|---|---|
| 2-3(lr)4-5 Model | real | fabr. | people | cities |
| -11B | 0/1200 | 0/400 | 0.00 | 0.00 |
| -12B | 533/1200 | 246/400 | 0.52–0.64 | 0.03 |
| -14B | 0/1200 | 0/400 | 0.00 | 0.00 |
| -12B | 0/1200 | 0/400 | 0.00 | 0.00 |
3pt
8.0.0.1 Setup.
The applied question is whether the one-pass signal can gate: decide, before generating, whether to answer from memory, retrieve, or abstain. On the 320-entity evaluation subset per model (six real entities per decile plus 20 fabricated, per domain; stratified, fixed seed) we compare our probe (Platt-calibrated logistic probe on prompt-point residuals, trained only on non-eval v2 rows; it never sees an evaluated entity) and dispersion against first-token entropy, discrete semantic entropy over five samples [9], [16], and three post-generation internal-state detectors: D2HScore [6], EigenTrack [7], and MIND [8], reimplemented from the papers with all adaptations documented in the caption of Table 3 and the released code.
8.0.0.2 Real-vs-fabricated discrimination.
On fabricated-vs-real (Table 3a) the calibrated probe outperforms our adapted implementations of every baseline on every model (0.859–0.934), beating the best generation-time baseline by 0.14–0.30 AUROC per model (paired bootstrap over entities: every per-model margin’s 95% CI excludes zero), despite the supervised baselines training on eval-pool entities while the probe never does. On this target, the prompt point is easier to read than the generated answer’s internal states. Purely lexical classifiers on the entity string top out at AUROC 0.786 (character n-grams; single length and token-count features stay \(\le\)0.61 under all four family tokenizers), so the probe’s 0.859–0.934 sits 0.07–0.15 above the lexical-naturalness ceiling and is not explained by surface name form. We stress that the target is operational: it certifies discrimination of our real/fabricated construction, not possession of answer-relevant knowledge; a model can register that an entity exists while knowing no usable facts about it (§10).
8.0.0.3 Behavioral prediction and the Gemma-4 inversion.
On predicting strict-judge incorrectness (Table 3b) semantic entropy is the most consistent (0.76–0.81 on all four models): it directly measures answer instability. The probe transfers well where behavior follows knowledge (Qwen3 0.914, Bielik 0.764), though no per-cell best-vs-runner-up difference in this contrast is individually significant under a paired bootstrap (e.g., Qwen3 probe-vs-MIND CI \([-0.12, +0.39]\)), so we read block (b) as pattern-level rather than cell-level evidence. On Gemma-4 it appears to invert (0.409), but this is a label-semantics artifact, not a property of the signal: the strict judge scores explicit refusals as correct, and Gemma-4’s refusals fall mostly on unfamiliar entities. Separating refusal from correctness with the published marker list (which reconciles to Table 5’s 533/1200) shows the inversion is a labeling artifact, not a signal property: among answered items Gemma-4’s base error is 0.90, and only 13 of 149 answered entities are majority-correct, too few to estimate a behavioral AUROC in either direction. The dissociation that matters here is therefore carried by the refusal behavior itself (Table 5), not by an inverted AUROC: the familiarity signal tracks the entity’s status under our operational measure; behavioral error is additionally shaped by the model’s policy. More generally, folding refusals into “correct” conflates abstention with factual accuracy, and every block (b) number, and the risk–coverage analysis below, inherits this target definition for the one abstaining model; a clean analysis would separate answered-vs-abstained from correctness-given-answering, which we flag wherever Gemma-4 is affected and leave as the principal evaluation revision.
8.0.0.4 Selective answering.
Framed as risk–coverage [49], [50] (Figure 7, Table 4), base error rates on this long-tail benchmark are severe: 0.84 (Bielik-11B), 0.60 (Gemma-4), 0.98 (Qwen3), 0.92 (PLLuM); Gemma-4’s 0.60 counts its refusals as correct, and among answered items its base error is 0.90 (§10). The gates also differ in inference category (Figure 8; we report pass and generation counts rather than measured latency or FLOPs): semantic entropy achieves the best macro AURC (0.708) but requires five sampled generations plus an external clustering call and can only act after answering. Every post-generation detector beats the probe’s macro AURC (0.708–0.773 vs.). The probe’s claim is a within-class one: it is the best or effectively tied gate in the pre-generation class on three of four models (the PLLuM margin over dispersion sits at the fifth decimal; on Qwen3, dispersion is better by 0.007), and pre-generation is the only class that can route before generating. The post-generation detectors beat the probe only on Gemma-4, but this advantage is largely a labeling artifact: once refusals are treated as abstentions and risk is measured among answered items, no gate separates on Gemma-4 (AURC 0.83–0.93, against a 0.90 answered base error). Calibration: against the behavioral target the probe’s raw ECE is 0.41–0.64, a target mismatch (it is calibrated to the real-vs-fabricated target, not answer incorrectness); a one-dimensional out-of-fold Platt recalibration [51], [52] onto the behavioral target yields ECE 0.039 / 0.063 / 0.000 / 0.049 on Bielik/Gemma-4/Qwen3/PLLuM. At these base rates a constant predictor attains ECE 0.000 by construction, so low recalibrated ECE certifies that calibration is achievable, not that the score discriminates; ranking quality is carried by the AURC analysis above.
8.0.0.5 Who acts on the signal?
Table 5: only Gemma-4 refuses at all, and selectively by domain type: 52–64% per-answer refusal on real people vs.% on cities, a biography-caution policy layered on top of the knowledge signal. Bielik, Qwen3, and PLLuM produce zero refusals in 1,600 answers each: whatever familiarity information their activations carry, their generation policies never convert it into abstention.
9 Discussion↩︎
The four results compose into one claim: prompt-point activations carry a graded entity-familiarity readout that is robust to a Polish/English stem substitution, is more strongly associated with Polish adaptation than with scale in this sample, and can strongly steer the one native abstention policy we observe. Gradation (§5) shows the quantity co-varies with Polish adaptation rather than parameter count, observationally. Language transfer (§6) argues against a large question-stem-language effect in the paired setting, consistent with entity-level recognition features [4]; entity-disjoint transfer remains to be tested. Steering (§7) shows the familiarity direction is causally potent for refusal, though establishing that the model’s natural prompt-time computation flows through it requires mediation-style evidence. The gating experiments (§8) show the value and the boundary: the signal separates real from fabricated entities better than our adapted post-generation baselines and can act before a single token is generated, but predicting behavioral error additionally requires the model’s policy, which varies from Gemma-4’s domain-selective caution to the absence of refusal behavior in the other three families. What distinguishes a family that consumes the signal from one that leaves it inert, for instance whether the familiarity direction is coupled into the model’s output pathway, is an open mechanistic question we are pursuing in ongoing work.
The dissociation has a deployment corollary for Polish and other medium-resourced languages. Polish-adapted models carry more graded familiarity information but never abstain; Gemma-4 abstains but its readout is less graded. A one-forward-pass calibrated probe supplies the missing piece externally: a familiarity-aware gate that routes long-tail queries to retrieval [53] before a single token is generated. The representational prerequisite for calibrated abstention is thus present in every family we measured; what is missing in three of them is a policy that consumes it. Whether such a policy can be installed in a never-refusing model, by activation-level intervention or by the refusal-aware tuning studied for English models [54], [55], is the natural next question; we leave a systematic comparison to future work.
10 Limitations↩︎
Familiarity is an operational construct. Our primary discrimination target is real-vs-fabricated status, not validated knowledge: a model can register that an entity exists while holding no answer-relevant facts, and the probe’s positive training class (top popularity deciles) couples its score to the popularity axis by construction. The fabricated names are token-length matched, but we did not control character n-grams, name-component frequency, or morphological likelihood, so a lexical-naturalness shortcut cannot be fully excluded; non-neural lexical baselines and real-entity-only controls (e.g., post-cutoff entities) are the appropriate follow-up. Lexical baselines bound the shortcut concern: character-n-gram models reach AUROC 0.786, so the probe’s 0.859–0.934 is above but not fully independent of surface form. Familiarity also retains predictive value for behavioral error after controlling for log-pageviews and domain (likelihood-ratio \(p=0.013\)/\(p<0.001\) for Bielik-11B/Qwen3-14B), so the behavioral mirror is not purely a popularity common-cause, though popularity remains a strong baseline.
Single steering model and domain. The steering result is established on one model (Gemma-4-12B, the only one with a refusal behavior to steer) and one domain (athletes, \(n{=}42\) per condition; Wilson CIs are wide); whether familiarity steering can install abstention in the never-refusing families is untested. Both directions are estimated from the entities they are tested on, so effect sizes may be optimistic; the intervention acts at every generation position, so it demonstrates the direction’s potency rather than isolating the natural prompt-point pathway; and the familiarity direction contrasts known entities with obscure-real and fabricated ones pooled, so it may encode correlated features (popularity, real/fabricated status, lexical form) alongside familiarity. High-dose specificity rests on the degeneration contrast plus low-dose non-degenerate controls, and the degeneration heuristics do not certify preserved semantics.
Judge circularity and the refusal label. Correctness labels come from a single model family (claude-opus-4-8). A second judge (gpt-5.2, identical rubric) covers a stratified 20% of the v2 verdicts at \(\kappa=0.65\)–\(0.69\) (§5), substantial but not perfect; behavioral numbers inherit the residual uncertainty, and no human audit was performed. The strict judge scores explicit refusals as correct \({\sim}88\%\) of the time, conflating abstention with factual accuracy for Gemma-4; this affects the block (b) AUROCs, base error rates, risk–coverage/AURC, and behavioral calibration wherever Gemma-4 appears. We report a marker-based separation of answered-vs-abstained from correctness-given-answering (§8): it reconciles to Table 5 and shows Gemma-4’s answered base error is 0.90, not the refusal-inflated 0.60, and that the post-generation gates’ Gemma-4 advantage is largely a labeling artifact. An LLM three-way re-judge (refuse/correct/incorrect; claude-opus-4-8; all 4,800 real-entity answers) confirms this: it agrees with the marker list at 0.99 per answer (Gemma-4 542 refusals vs.the marker’s 533) and reproduces the answered-only picture (Gemma-4 base error 0.88, 14 of 131 answered entities majority-correct).2 A human audit remains future work.
Baseline fidelity. Our MIND variant omits its unsupervised data pipeline and is trained supervised with CV (likely flattering it); EigenTrack’s unspecified feature set is instantiated as 16 features with a shortened window; D2HScore runs as specified but its drift term is anti-predictive for one-sentence answers. All adaptations are documented and the supervision asymmetry favors the baselines.
Subsample sizes. Gating and behavioral results use 320 entities per model; per-decile correctness estimates average 24 entities; the behavioral contrast is heavily imbalanced for Qwen3 (7 majority-correct of 240) and PLLuM (25/240).
Popularity proxy. Pageviews proxy pretraining exposure imperfectly: the 12-month window is contemporary while the models’ training cutoffs differ and predate it, and pageviews correlate with article length, link density, and recency. The bottom decile contains zero-pageview stubs with no internal gradient; sitelink counts give materially identical results.
Family-level inference. Only four model families are compared, and checkpoints within a family are correlated replications, so the exposure-vs-scale conclusion is sample-limited (§5). We add controlled before/after evidence for both PLLuM architectures: the Mistral-NeMo-12B and Llama-3.1-8B bases grade at \(\rho=0.102\) and 0.193 versus their descendants’ 0.277 and 0.359, isolating Polish continual pretraining from architecture and scale.
Refusal metric. Refusal is detected by a Polish marker list validated by two independent audits (agreement 0.95–1.00), but steering-induced phrasing outside the list would be missed; the metric is untested on other abstention styles.
Language-transfer scope. The EN/PL switch changes both language and template, and only one template per language is tested; a within-Polish neutral-template control bounds the template effect but was not repeated for every v2 model. Entity strings remain Polish in both conditions by design, so EN prompts are code-mixed rather than fully English. We do test entity-disjoint transfer (§6); it holds for Bielik but weakens for Gemma-4. Multiple paraphrases, translated entity forms where aliases exist, and additional languages are still needed before any language-independence claim.
Ethics and Availability Statement↩︎
We use only public information about public entities, plus invented names screened against real-person collisions. Steering shows abstention can be suppressed as easily as induced, so deployed abstention policies are not robust to activation-level manipulation; familiarity gating abstains more on long-tail entities, a coverage-disparity risk. Code, the CSV dataset, and provenance artifacts, including pinned model checkpoints and judge identifiers, are available online.3 The experiments were orchestrated with AI coding assistance. All results were produced and verified by the author.
References↩︎
google/gemma-4-E4B-it and google/gemma-4-12b-it; no technical report was available at the time of writing“Gemma 4
model card.” https://ai.google.dev/gemma/docs/core/model_card_4, 2026.Preprint. Under review.↩︎
Binary and three-way judges agree on 86% of the 4,800 answers (cf.the 89.8% cross-family agreement, §5). Disagreements are of two kinds. The binary judge splits the 543 refusals inconsistently (327 scored correct, 216 incorrect), where the three-way judge labels all
refuse; this is the source of the correction. The other 142 are borderline factual answers on long-tail entities (133 flip incorrect\({\to}\)correct, 9 the reverse), a small non-directional shift: the same entity often draws opposite flips on different samples, so per-entity majority labels and Bielik’s decile monotone are unchanged. The residual correct/incorrect variation is thus judge noise of the same order as the cross-family disagreement, not a systematic bias.↩︎