1 Introduction↩︎

Despite progress in computer vision, visual recognition models often fail to maintain performance under environmental changes at deployment, when test samples deviate from training distribution [1], [2]. Unlike domain adaptation [1], [3], which has access to target distribution during training, domain generalization (DG) [4] addresses the stricter scenario of generalizing to unseen environments without any prior exposure. Under this constraint, DG aims to learn domain-invariant representations that preserve only essential semantics of the input.

To pursue such invariance, standard DG frameworks train across multiple source domains, under the assumption that reducing distributional discrepancies among diverse sources will expose an underlying invariant subspace. For instance, aligning a simple sketch with a real photograph of an apple is expected to reveal the shared semantic “apple-ness” beneath domain-specific variations. Accordingly, traditional DG methods primarily align feature distributions across domains [4]–[12] or employ robust training strategies to mitigate domain-specific correlations [13]–[15].

Recently, the field has turned to large-scale vision–language models (VLMs) such as CLIP [16], motivated by their strong zero-shot generalization ability. Because standard fine-tuning can compromise this robustness under distribution shift [17], current strategies aim to preserve the original zero-shot representations while adapting them to the DG task [18]—often freezing the text encoder while adapting the visual branch—via knowledge distillation [19]–[21], prompt tuning [21]–[28], or weight ensembling [18], [29], [30].

These approaches share a common premise that preserving the pre-trained visual knowledge of VLMs (e.g., CLIP) benefits generalization through zero-shot robustness or semantic expressiveness. But does preserving the visual features truly benefit domain generalization? Consider the ‘apple’ examples in 1 (a). Highly expressive visual encoders entangle the core semantics (\(\bigstar\)) with visual styles, such as sketch strokes, photographic textures, or artistic abstractions. Without an explicit criterion to distinguish true semantics from these variations, models preserving such visual representations tend to retain any feature that is predictive of labels in the training domains. Consequently, domain-specific cues are absorbed alongside true semantics, expanding the range of a class beyond its core and blurring its boundary under domain shifts. In contrast, the ideal representation retains only core semantics by dropping domain spurious cues.

To avoid inheriting domain entanglement from visual guidance, we propose to discard it altogether. Instead, we elevate the text space to serve as the primary source of domain invariance, drawing on both the stability of its anchors and their semantic structure (3). Concretely, as shown in 1 (b), features from diverse domains are constrained to align with text-defined anchors through a class-conditional Information Bottleneck (IB). These anchors remove domain-induced styles not shared across modalities while preserving information shared between input images and text.

Extensive experiments demonstrate that our approach consistently achieves the state-of-the-art performance across representative DG benchmarks with clearer separation in the learned embedding space, as illustrated in 1 (c). Importantly, our evaluation spans diverse backbone architectures, further demonstrating the generality of the proposed framework.

Our contributions are summarized as follows:

1. Revisiting the visual guidance in DG, we reveal that highly expressive visual encoders can propagate domain-specific cues and hinder domain invariance under distribution shift.

2. We propose a purely text-guided approach based on IB theory, suppressing domain-specific variations while preserving shared semantics.

3. Through extensive experiments across diverse DG benchmarks and backbone architectures, we demonstrate consistent state-of-the-art performance and improved reliability under domain shift.

4. We further analyze guidance signals in DG and highlight supervision design as a key factor for learning invariant representations.

2 Related Work↩︎

Domain Generalization (DG). Visual recognition models often suffer from real-world distribution shift. To evaluate these failures, established benchmarks span diverse shifts: style variation (e.g., sketch vs. photo) [31]–[33], differences in dataset acquisition process [34] or camera location [35], and background shifts [36]. Together, these benchmarks expose the inherent brittleness of models under unseen environments.

DG seeks representations that remain invariant across such domain discrepancies without access to target data. Classical approaches primarily extract invariance by aligning source-domain feature distributions via statistical matching [5], [37], adversarial learning [6], [37], [38], or feature disentanglement [4], [11], [12], [39]–[43]. Others instead regularize optimization dynamics [7], [8], gradient constraints [14], [29], ensembling [15], [44]–[46], and consistency guidance [13]. These approaches define invariance based on patterns shared across the training domains, yet such shared structure often mix true semantics with spurious correlations, limiting generalization beyond source-like distributions.

Recent approaches increasingly build upon CLIP [16], adapting its visual encoder while attempting to preserve zero-shot robustness. Representative strategies include prompt optimization [21]–[28], robust fine-tuning and weight ensembling [18], [29], [30], [47], and knowledge distillation [19]–[21]. Across these methods, the visual encoder is typically the component being updated during adaptation. Meanwhile, its pretrained knowledge is preserved and reused through mechanisms such as regularization, ensembling, or distillation.

In contrast, the text encoder is usually kept frozen and serves as a relatively stable semantic reference. Depending on the method, text embeddings are used (i) as auxiliary alignment targets alongside visual supervision [19]–[21], [30], (ii) as regularizers to constrain visual drift from the original zero-shot space [47], or (iii) as diversified prompts to steer visual representations via multimodal alignment [26]–[28]. We instead revisit the functional roles of CLIP’s visual and text encoders in DG, minimizing reliance on visual guidance and treating textual semantics as the primary basis for defining invariance.

Information Bottleneck in DG. Empirical Risk Minimization (ERM) [48] often struggles under distribution shift due to spurious correlations. While Invariant Risk Minimization [7] enforces a shared classifier across environments, it does not fully eliminate such biases. To further constrain representations, several classical DG methods adopt the IB principle. As the IB objective is intractable, these approaches rely on variational formulations, introducing KL-based regularization toward simple, typically non-semantic priors [8], [10], with meta-learning [9], or coupling it with feature disentanglement objectives [11]. In contrast, we anchor the bottleneck to fixed text embeddings as an explicit semantic prior. Existing IB priors are either uninformative (e.g., \(\mathcal{N}(0, \mathbf{I})\)), giving no signal to separate semantic from spurious factors, or learned from source images, inheriting domain bias. In contrast, our approach is class-conditional yet label-derived, signaling what to preserve without injecting domain bias.

3 Motivation↩︎

To assess whether visual and textual modalities provide structurally reliable basis for domain-invariant guidance, we investigate the following questions: 1) Do their embedding spaces remain stable across domains? 2) How much domain-invariant and domain-specific information do the modality-specific encoders encode? 3) How do visual and textual guidance signals affect learning dynamics?

3.1 Empirical Examination↩︎

Cross-domain Reliability of Embedding Space. We examine the geometry of multimodal embedding spaces across domains by extracting visual and textual CLIP features from image–caption pairs in the four PACS domains [31] (art painting, photo, cartoon, sketch). To obtain textual embeddings as rich as their visual counterparts, we generate captions with semantic and stylistic details (App. 7), instead of using predefined prompts like "a [domain] of [class]", and feed them into a frozen CLIP text encoder. This provides rich instance-level descriptions and enables a fair comparison between modalities.

As shown in 2 (a), the visual space (top) exhibits pronounced domain-dependent dispersion. Even for the same class (e.g., person, highlighted with black boxes), the corresponding clusters significantly shift across domains. In contrast, text embeddings (bottom) remain relatively stable, largely insensitive to stylistic variations present in the captions. This contrast reveals a notable gap: while visual encoders capture fine-grained details including domain-specific cues, text embeddings remain semantically structured.

Encoded Information in Embedding Spaces. The previous experiment implies that the textual embedding space likely encode more domain-invariant information compared to the visual counterpart. Also, one might wonder whether this behavior is specific to CLIP. To answer these questions, we report in 2 (b) the normalized mutual information between the learned embeddings and both class labels (\(x\)-axis) and domain labels (\(y\)-axis) across multiple encoders.

Across visual backbones (ViT [49], CLIP-Image [16], and DINOv2 [50]), their embeddings consistently encode substantial domain information \(I(Z;\text{domain})\) alongside class information \(I(Z;\text{class})\). In contrast, language models (MPNet [51], MiniLM [52], and CLIP-Text [16]) exhibit near-zero domain information, while primarily encoding class semantics. This result strongly implies that domain entanglement is indeed inherent to expressive visual representations. Preserving fine-grained variations, visual encoders inadvertently keep some domain-specific factors as well. In contrast, textual representations exhibit much less stylistic variation, leading them to align more consistently with semantic structure. Considering that domain generalization strictly requires isolating domain-invariant semantics from such variations, a natural hypothesis from this experiment is that textual space would be more appropriate to guide DG models.

Learning Dynamics under Domain-Dependent Guidance. A natural next question is if the guidance signals contain domain-dependent variations, how does this actually affect learning dynamics? To investigate this, we estimate the local Lipschitz constant, approximated by the norm of the gradient of learned features with respect to unseen target inputs [53]. This metric measures how sensitively the representation reacts to small perturbations under domain shift.

As shown in 2 (c), student models (ViT [49] and ResNet-50 [54]) distilled with visual signals exhibit consistently higher Lipschitz values than those trained with textual guidance, indicating more input-sensitive mappings. This result verifies our earlier hypothesis that visual guidance would expose the student to cross-domain conflicting cues (2 (a)). Fitting to such inconsistent signals makes the representation more sensitive to small input changes and compromises stability. In contrast, textual guidance mitigates cross-domain conflict, yielding smoother and more stable representations under domain shift.

3.2 Information-theoretic Perspective↩︎

Beyond our empirical diagnostics, we further provide an information-theoretic perspective on our two design choices: removing domain-contaminated visual guidance and introducing an explicit information bottleneck. We interpret this through the lens of finite model capacity (formal assumptions and derivations are provided in App. 8).

An input image \(X\) can be decomposed into domain-invariant and domain-specific (spurious) components: \(X = (X_{\text{inv}}, X_{\text{sp}})\). Denoting the output of the student model \(\phi\) as \(S = \phi(X)\) with finite capacity \(C\), we have \(I(S;X_\text{inv}) + I(S;X_\text{sp}) \leq C\). This constraint implies a trade-off between invariant semantics and spurious domain styles. Because visual guidance depends on the input image, it carries domain-specific variation \(X_\text{sp}\) into the supervision signal. Learning from such guidance increases \(I(S;X_\text{sp})\), thereby reducing the capacity available for \(X_\text{inv}\). Since labels depend only on \(X_\text{inv}\), this limits the attainable predictive information and degrades generalization to unseen domains. This provides additional theoretical support for removing visual guidance.

However, the removal alone does not fully resolve the capacity allocation problem. Although domain signals \(X_\text{sp}\) are no longer injected through supervision, they are still present in the input and may be encoded by the model during training. In practice, a high-capacity model can learn separate domain-specific feature pathways that reach the same semantic target, leaving the representation entangled with \(X_\text{sp}\). To explicitly restrict this spurious capacity allocation, we introduce a Text-Anchored Information Bottleneck in the next section.

4 Method↩︎

Our previous diagnostics verify that domain invariance is largely determined by the structure of supervision. Instead of relying on visually entangled guidance, which inevitably propagates spurious domain variations, we propose to anchor the training entirely to fixed, domain-invariant text embeddings, namely, Text-Anchored Information Bottleneck, illustrated in 3. We first formalize the domain generalization problem and introduce the Information Bottleneck formulations (4.1), then present our text-anchored framework with two mechanisms: preserving semantic structure in the text space and suppressing spurious visual cues (4.2).

4.1 Preliminary↩︎

We first formulate the DG problem and introduce the Information Bottleneck and its conditional variant, which governs our semantic purification strategy.

Problem Formulation. In domain generalization, training data from \(K\) source domains \(\mathcal{D}_S = \{\mathcal{D}_1, \mathcal{D}_2, \dots, \mathcal{D}_K\}\) are given, where \(\mathcal{D}_k = \{(x_i^{(k)}, y_i^{(k)})\}_{i=1}^{N_k}\) from each domain \(k\) consists of \(N_k\) samples and follows a probability distribution \(P_k(X, Y)\). The core challenge is distribution shift, where \(P(Y|X)\) remains invariant while the marginals differ across domains (\(P_i(X)\neq P_j(X)\)). The task aims to learn a feature encoder \(Z=f_\theta(X)\) that generalizes to an unseen target domain \(\mathcal{D}_T\), where \(\mathcal{D}_T\cap\mathcal{D}_S=\emptyset\).

Information Bottleneck. To learn robust representations \(Z\), we adopt the Information Bottleneck (IB) [55], which seeks features that are maximally predictive of labels \(Y\) while compressing the input \(X\): \[\mathcal{L}_\text{IB} = -I(Z;Y) + \beta I(Z;X), \label{eq:ib}\tag{1}\] where \(I(\cdot;\cdot)\) denotes mutual information and \(\beta > 0\) controls the trade-off between prediction and compression. Assuming the Markov chain \(Y\leftrightarrow X\leftrightarrow Z\), IB encourages \(Z\) to retain label-relevant information while discarding irrelevant variations in \(X\).

Minimizing \(I(Z;X)\) alone, however, does not explicitly distinguish label-relevant semantics from domain-specific factors, since domain cues may still be predictive of \(Y\) within the training domains. Therefore, we minimize the Conditional Information Bottleneck (CEB) [56]: \[\mathcal{L}_\text{CEB} = -{I(Z;Y)+\beta I(Z;X|Y)}, \label{eq:ceb}\tag{2}\] which replaces \(I(Z;X)\) with \(I(Z;X|Y)\). By conditioning on the label \(Y\), CEB removes input information that is unnecessary given \(Y\). This provides a direct mechanism for suppressing spurious domain-specific variations, as labels serve as semantic anchors that isolate task-relevant information.

4.2 Text-Anchored Information Bottleneck↩︎

3 presents our architecture-agnostic training framework for optimizing the CEB under fixed textual semantics. With \(K\) classes, we obtain semantic anchors \(T = [\mathbf{t}_1, …, \mathbf{t}_K ]^\top \in \mathbb{R}^{K \times d}\), where each \(\mathbf{t}_k \in \mathbb{R}^d\) is the frozen CLIP text embedding of the prompt ‘a photo of a [class]’. Although we have used rich captions in 3 to fairly match image richness, here we adopt the simplest prompt, which strips instance-level noise while keeping class identity. A trainable visual encoder \(f\) maps an input image \(\boldsymbol{x}\) to \(\boldsymbol{z}=f_\theta(\boldsymbol{x}) \in \mathbb{R}^d\). Motivated by 3, we treat these fixed text embeddings as the primary source of domain invariance, explicitly anchoring the representation space to domain-stable semantics rather than relying on invariance to emerge implicitly or using text as auxiliary guidance. We derive concrete forms of the two complementary objectives, maximizing predictive sufficiency \(I(Z;Y)\) and minimizing anchor-inconsistent variation \(I(Z;X|Y)\), comprising our CEB formulation in 2 . This yields three terms: a semantic distillation loss \(\mathcal{L}_\text{sem}\) that maximizes \(I(Z;Y)\), and compression and alignment losses \(\mathcal{L}_\text{comp}\), \(\mathcal{L}_\text{align}\) that minimize \(I(Z;X|Y\)), together forming our final objective (7 ). We further detail each below.

Maximizing \(I(Z;Y)\). The mutual information term \(I(Z; Y)\) in 2 quantifies the dependency between the representation \(Z\) and the target \(Y\), defined as \[I(Z; Y) = \mathbb{E}_{Z, Y} \left[ \log \frac{p(Z,Y)}{p(Z)p(Y)} \right] = \mathbb{E}_{Z, Y} \Big[\log p(Y|Z)\Big] + H(Y),\] where \(H(Y)\) is constant. Thus, maximizing \(I(Z; Y)\) is equivalent to maximizing \(\mathbb{E}_{Z, Y} [\log p(Y|Z)]\), i.e., minimizing the cross-entropy loss [57]. We parameterize \(p(Y|Z)\) using fixed text embeddings as class prototypes. For a sample with label \(k\), we define: \[\mathcal{L}_\text{sem} = -\log \frac{\exp(\cos(\boldsymbol{z}, \mathbf{t}_k)/\tau)}{\sum_{k' \in K} \exp(\cos(\boldsymbol{z}, \mathbf{t}_{k'})/\tau)},\] where \(\mathbf{t}_k\) is the text embedding of class \(k\) and \(\tau\) is a temperature. This objective pulls representations toward their class anchors and pushes them away from others, thereby distilling the semantic structure of the textual space into the learned representation (3, upper right).

Minimizing \(I(Z;X|Y)\). We minimize the second term \(I(Z;X|Y)\) in 2 to suppress spurious domain variations. We first derive its variational upper bound, and then convert it into a tractable training objective.

Starting from the definition of conditional mutual information, \[I(Z;X|Y) \mathrel{\vcenter{:}}= \mathbb{E}\left[\log \frac{p(X,Z| Y)}{p(X| Y)p(Z| Y)}\right] = \mathbb{E}\left[\log \frac{p(Z| X,Y)}{p(Z| Y)}\right] = \mathbb{E}\left[\log \frac{p(Z| X)}{p(Z| Y)}\right],\] where the last equality follows from the Markov chain \(Y\leftrightarrow X \leftrightarrow Z\) assumption, implying \(p(Z|X, Y) = p(Z|X)\). Since the true marginal \(p(Z|Y)\) is intractable, we introduce a variational approximation \(q(Z|Y)\). By non-negativity of KL divergence, \[I(Z;X|Y) \le \mathbb{E}[\log p(Z|X)]-\mathbb{E}[\log q(Z|Y)] = \mathbb{E}\big [\mathrm{KL} (p(Z|X)\|q(Z|Y)) \big]. \label{eq:ub}\tag{3}\]

We now derive a stable training objective from this bound. For samples belonging to class \(k\), 3 becomes \[\mathbb{E}_{X|y=k}\!\big[\mathrm{KL} (p(Z|X)\|q(Z|y=k))\big]= \mathbb{E}_{X|y=k}\big[\mathbb{E}_{Z}\big[\log p(Z|X)-\log q(Z|y=k)\big]\big], \label{eq:kl952ndterm}\tag{4}\]

Since CLIP embeddings are \(l_2\)-normalized and lie on the unit hypersphere, we model both \(p\) and \(q\) as von Mises–Fisher (vMF) distributions [58], parameterized by a mean direction \(\boldsymbol{\mu}\) and concentration \(\kappa\). A vMF can be viewed as the spherical analogue of a Gaussian, concentrating probability mass around a direction on the unit sphere. Specifically, \(q(Z|y=k)\) is defined as a vMF centered at the fixed text embedding \(\mathbf{t}_k\) obtained from the frozen text encoder, with a fixed concentration \(\kappa_q\), while \(p(Z|x)\) is centered at the image feature \(\boldsymbol{z}=f_\theta(x)\), with concentration \(\kappa_p\). With \(\kappa_p\) fixed, the first term in 4 , \(\mathbb{E}_Z[\log p(Z|X)]\), becomes constant with respect to \(\theta\). Thus, the optimization reduces to minimizing the second term, \(-\mathbb{E}_{X|y=k}\mathbb{E}_Z[\log q(Z|y=k)]\).

Approximating \(Z\sim p(Z|x)\) by its mean feature \(\boldsymbol{z}=f_\theta(x)\) (with a large \(\kappa_p\)), and using the vMF density \(q(\boldsymbol{z}|y=k)=C_d(\kappa_q)\exp{(\kappa_q\boldsymbol{t}_k^\top\boldsymbol{z})}\), the log-density can be substituted into 4 . Applying the empirical expectation over a minibatch \(\mathcal{B}_k\) yields \[-\frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}[\log C_d(\kappa_q)+ \kappa_q\mathbf{t}_k^\top \boldsymbol{z}_i] = \text{const}- \kappa_q\, \mathbf{t}_k^\top\left(\frac{1}{|\mathcal{B}_k|}\sum_{i\in \mathcal{B}_k}\boldsymbol{z}_i\right),\] where both \(\kappa_q\) and \(C_d(\kappa_q)\) are constants. Let \(\bar{\boldsymbol{z}}_k =\sum_{i\in \mathcal{B}_k}\boldsymbol{z}_i/|\mathcal{B}_k|\) denote the mean feature of the class \(k\), the objective reduces to \[-\boldsymbol{t}_k^\top \bar{\boldsymbol{z}}_k =-\,\|\boldsymbol{t}_k\|\|\bar{\boldsymbol{z}}_k\|\cos(\boldsymbol{t}_k,\bar{\boldsymbol{z}}_k)= -\,\|\bar{\boldsymbol{z}}_k\|\cos(\boldsymbol{t}_k,\bar{\boldsymbol{z}}_k),\] since \(\|\boldsymbol{t}_k\|=1\). Empirically, however, directly optimizing this multiplicative form couples the magnitude and the cosine terms, which can lead to poorly conditioned gradients during training (e.g., the cosine term may quickly saturate, weakening gradients acting on the \(\|\bar{\boldsymbol{z}}\|\) term). We therefore adopt a separable surrogate objective that independently encourages (i) a large intra-class resultant length \(\|\bar{\boldsymbol{z}}_k\|\), and (ii) strong directional alignment \(\cos(\mathbf{t}_k,\bar{\boldsymbol{z}}_k)\) with the text anchor \(\mathbf{t}_k\), to stabilize optimization. From the magnitude term, we define: \[\mathcal{L}_{\mathrm{comp}} = -\frac{1}{|K_{\mathcal{B}}|}\sum_{k \in K_{\mathcal{B}}}\|\bar{\boldsymbol{z}}_k\|, \label{eq:loss95comp}\tag{5}\] where \(K_\mathcal{B}\) denotes the set of classes present in a mini-batch \(\mathcal{B}\). This encourages features of the same class to concentrate along a coherent direction (3, lower right). From the cosine term, we define \[\mathcal{L}_{\mathrm{align}} = -\frac{1}{|K_{\mathcal{B}}|}\sum_{k \in K_{\mathcal{B}}}\cos\! \left(\mathbf{t}_k, \frac{\bar{\boldsymbol{z}}_k}{\|\bar{\boldsymbol{z}}_k\|} \right) \label{eq:loss95align}\tag{6}\] which aligns each class mean with its text anchor. The final objective is \[\mathcal{L}=\mathcal{L}_{\mathrm{sem}}+\beta_1 \mathcal{L}_{\mathrm{align}}+\beta_2 \mathcal{L}_{\mathrm{comp}}, \label{eq:loss95full}\tag{7}\] where \(\beta_1\) and \(\beta_2\) balance the two effects.

Unlike prior IB-based DG methods [9]–[11] that learn explicit Gaussian parameters (e.g., \(\boldsymbol{\mu}\), \(\boldsymbol{\sigma}\)) from each image to match a simple unconditional prior (e.g., \(\mathcal{N}(0, \mathbf{I})\)), irrespective of class semantics, we impose a class-conditional prior anchored by fixed text embeddings, enabling more reliable compression of the representations.

5 Experiments↩︎

5.1 Experimental Setting↩︎

Datasets. We evaluate on six standard DG benchmarks: TerraIncognita [35], OfficeHome [32], VLCS [34], PACS [31], DomainNet [33], and NICO⁺⁺ [36]. The first four contain four domains each with 10, 65, 5, and 7 classes, respectively. For large-scale evaluation, we use the latter two, DomainNet (6 domains, 345 classes) and NICO⁺⁺ (6 domains, 60 classes).

Baselines. Across diverse backbones, representative DG methods vary by architecture. For consistent comparison, we focus on a set of key baselines. Linear probing (LP) evaluates frozen features, while MIRO [13] represents classical DG without external guidance. RISE [19] and VL2V [20] perform cross-modal distillation, explicitly using a CLIP image encoder as the primary teacher with simple alignment objectives for text supervision. In contrast, CLIP-dedicated methods such as CLIPood [30] do not employ an external teacher but implicitly retain original CLIP visual features through zero-shot preservation. When applied to non-CLIP backbones, however, this preservation operates on backbone-specific weights, so CLIP visual features are no longer involved in adaptation and guidance is provided only by CLIP text embeddings.

Implementation Details. We follow the DomainBed [59] protocol. All datasets use the standard leave-one-domain-out setting, where one domain is held out for testing while training on the remaining domains. For NICO⁺⁺, we adopt a leave-one-group-out protocol, where multiple domains are grouped and one entire group is reserved for evaluation. Model selection is based on validation accuracy using a 20% split of training data. We use \(\beta_1\) = 0.1 and \(\beta_2\) = 1.0 based on ResNet-50 on PACS, and apply them across all datasets and backbones (sensitivity in App. 9). Results are averaged over three runs, with standard deviations reported in App. 10. See App. 11 for more details.

5.2 Comparison with Baselines↩︎

6pt

3pt

Main Results. ¿tbl:tab:main? shows that our method achieves the state-of-the-art performance across various standard backbones. Existing approaches exhibit architectural bias: KD methods (e.g., RISE, VL2V) favor conventional encoders, while CLIP-specialized methods favor CLIP backbones, and both degrade when evaluated beyond their original setup. In contrast, ours consistently improves the performance without backbone-specific design, demonstrating its universal robustness.

Robustness to Background Shift. We further compare the performance of competing DG models in 1 on NICO⁺⁺, which focuses on background-driven shift. Notably, the margin becomes more pronounced; ResNet-50 reaches 95.7% accuracy, nearly closing the gap to CLIP (97.5%). This suggests that this setting is particularly better-aligned with our text-anchored bottleneck, since background variations are more readily separable from foreground semantics than the style shifts in ¿tbl:tab:main?, where object appearance itself can vary. In such cases, varying background is more likely to be filtered as spurious style. We further discuss in App. 13 that this purification improves overall robustness, though it may fail in rare cases where contextual cues are genuinely required for identification.

Generalization across Backbones. While ¿tbl:tab:main? evaluates standard architectures, 4 extends the comparison to a broader range of backbones, including lightweight models, CNNs, and diverse transformers. Across all backbones, ours is the only method that consistently maintains a clear margin over LP.

Interestingly, when the pretrained backbone is already strong (i.e., high LP performance), other state-of-the-art methods often shrink to near-zero or even negative margins (e.g., MIRO on SwinT and RISE on DINOv2). This implies that KD approaches (RISE, VL2V) risk distorting well-structured semantic spaces, while regularization-based methods (MIRO, CLIPood) largely preserve them without actively removing domain-specific factors, yielding only marginal gains. In contrast, our text-anchored compression explicitly filters spurious variations, sustaining positive improvements regardless of backbone strength.

Recent work [62] notes that some DG benchmarks may be affected by data leakage, as supervised backbones could have encountered similar domains during large-scale pretraining. To partially control for this effect, we additionally evaluate on DINOv2, a self-supervised backbone less likely to exhibit such leakage. While most methods remain close to the LP baseline in this setting, our method maintains a clear margin, suggesting that the gains are not solely attributable to inherited exposure, but instead stem from the intended domain-invariance mechanism.

5.3 Further Analysis↩︎

Ablation on Loss Components. 2 ablates the three terms across three backbones on OfficeHome, PACS, and DomainNet. Adding \(\mathcal{L}_\text{comp}\) to \(\mathcal{L}_\text{sem}\) drives the largest gains, while \(\mathcal{L}_\text{align}\) provides a consistent further improvement; their combination performs the best across all backbones and datasets, confirming that compression and alignment are complementary. App. 14 extends this ablation to other backbones.

3.5pt

Does the bottleneck indeed suppress domain-specific information? To verify this, we visualize the information flow for a specific class (house) using t-SNE in 5. Across the cartoon and sketch domains in PACS, the inputs (top) are clearly domain-separated. Our features (blue in the middle, \(Z\)) concentrate into a unified cluster successfully, discarding domain information. On the other hand, the baseline (RISE) features (orange in the middle, \(Z\)) largely retain the domain-dependent separation. Within each single domain, our embedding space (right) forms more compact class clusters, indicating that the text-anchored bottleneck removes cross-domain variation, enhancing class separability with larger margins.

To examine the underlying mechanism, we track the CEB term \(I(Z;X|Y)\) during training. 6 (a) estimates this quantity using MINE [63]. In practice, \(I(Z;X|Y)\) is approximated by the alignment and compression terms (\(\mathcal{L}_{\text{align}}\) and \(\mathcal{L}_{\text{comp}}\) in 7 ), which decrease together during training, indicating that the objective suppresses dependence on input-specific variations. This mechanism also facilitates semantic learning. 6 (b) compares the full model with a variant trained only with \(\mathcal{L}_{\text{sem}}\). By filtering domain-specific variations, the model allows \(\mathcal{L}_{\text{sem}}\) to converge to a lower value, resulting in higher accuracy.

Does visual guidance reintroduce domain entanglement? In 6 (c), we test this by injecting CLIP image supervision into our objective in 7 with varied guidance ratios. As the ratio increases, the accuracy consistently drops, with CNNs (RegNet, RN50) degrading more than Transformers (DeiT, SwinT), likely because such guidance propagates domain-specific statistics that CNNs are more sensitive to. This indicates that aligning with expressive visual features disrupts the text-driven abstraction. Interestingly, this trend contrasts with prior KD approaches [19], [20], where a small amount of textual guidance often improves performance, suggesting different roles of text under the two settings.

Why do text embeddings serve as effective anchors? Since our bottleneck relies on textual anchors, we examine whether it depends on specific CLIP properties or reflects a more general characteristic of language spaces. ¿tbl:tab:proto? (a) compares anchors from different language encoders (MiniLM [52], MPNet [51]) and prompt templates on OfficeHome. All variants perform similarly, indicating that the framework is largely agnostic to particular language model and not tied to CLIP-specific representations (see App. 15 for richer prompt variants explored; App. 16 for class-similarity analysis).

Even random anchors achieve competitive performance, surpassing the previous state-of-the-art (76.6 vs.), suggesting that stable external anchors are the key to the framework’s effectiveness, while pretrained text embeddings further benefit from their learned semantic structure.

Crucially, this semantic advantage grows as class discrimination becomes harder; on DomainNet (345 classes), the gap between CLIP and random anchors widens sharply (58.3 vs. 36.4). It is most evident in open-set generalization, where unseen test classes make random anchors inapplicable, yet text anchors still generalize well (78.8 vs. CLIPood 78.2; 3).

How should anchors be integrated into training? ¿tbl:tab:proto? (b) varies three factors on OfficeHome and PACS: the anchor source (random vs. CLIP), whether anchors are kept fixed (), and the alignment objective. Freezing inconsistently helps for contrastive and L2 while consistently under ours, as learnable anchors otherwise absorb domain-specific statistics and drift from invariant references. Likewise, our CEB-based objective is the only one that turns semantic CLIP anchors into reliable gains, via compression and directional constraints. Overall, domain invariance emerges only when stable external anchors are paired with an objective that explicitly enforces such invariance.

6 Conclusion↩︎

Expressive visual features can propagate spurious cues, challenging the common belief that greater capacity ensures robustness. We introduce a Text-Anchored Information Bottleneck that explicitly grounds supervision in external anchors to directly address the source of domain bias. Rather than distilling invariance from domain-shifting visual data, we utilize stable external signals to prove that the structural source of supervision is more critical than model capacity. While fixed anchors lack contextual cues and benchmark data leakage [62] currently obscures true generalization gains, it will be an interesting future direction to explore adaptive anchors and alternative signals to resolve these limitations and better isolate core semantics.

Acknowledgments↩︎

This work was supported by Samsung Electronics, Youlchon Foundation, National Research Foundation of Korea (NRF) grants (RS-2021-NR05515, RS-2024-00336576, RS-2023-0022663), and the Institute for Information & Communication Technology Planning & Evaluation (IITP) grants (RS-2022-II220264, RS-2024-00353131) funded by the Korean government.

Appendix↩︎

7 Details on Caption Generation↩︎

To better understand the structure of CLIP text embeddings, we visualize image-conditioned captions generated using LLaVA-v1.5-7B² in 7.

Each image from the PACS dataset is fed to the captioner with the following prompt:

“USER: <image> Generate a short and clear caption for the image. ASSISTANT:”.

The generated captions capture a mix of high-level semantic concepts (e.g., house, home) and domain-specific visual styles (e.g., a painting of, a cartoon of), along with peripheral details present in the images. As shown in 7, for example, the surrounding descriptions somewhat vary in style and content, but the core semantics associated with the class label (e.g., house) is consistently preserved in the textual embeddings across all domains.

8 Information-Theoretic Analysis↩︎

We formalize the intuition in Sec. 3 that image-dependent guidance can increase reliance on spurious domain information, thereby reducing the predictive information available for unseen-domain generalization.

Notation. An input image \(X\) can be decomposed into domain-invariant and domain-specific components: \(X = (X_{\text{inv}}, X_{\text{sp}})\). We denote the class label by \(Y\) and the student representation by \(S = \phi(X)\). The text teacher provides guidance \(G_{\text{t}}=g_{\text{t}}(Y)\), while the image teacher provides guidance \(G_{\text{i}} = g_{\text{i}}(X)\). Their mixture is denoted by \(G_{\alpha} = \alpha G_{\text{i}}+(1-\alpha)G_{\text{t}}\), with a weight \(\alpha \in [0,1]\).

Assumptions. We assume the following:

• (A1) \(X\) is generated from \(X_\text{inv}\) and \(X_\text{sp}\), the label \(Y\) is determined by \(X_\text{inv}\), and \(S\) is a function of \(X\). Consequently, \(Y \perp S \mid X_{\text{inv}}\) and \(X_{\text{inv}} \perp X_{\text{sp}}\) (see 8).

• (A2) The text guidance is independent of the spurious component, whereas image guidance is not, i.e., \(I(G_{\text{t}};X_{\text{sp}})=0\) and \(I(G_{\text{i}};X_{\text{sp}})>0\). This reflects our setting, where \(G_{\text{t}}\) is conditioned on \(Y\), whereas \(G_{\text{i}}\) depends on \(X\).

• (A3) As \(\alpha\) increases, spurious information in the mixed guidance is non-decreasing and is partially transferred to the learned representation. Formally, we assume that \(I(G_\alpha;X_\text{sp})\) is non-decreasing in \(\alpha\), and that there exists \(c\in(0,1]\) such that \(I(S;X_\text{sp}) \approx c\, I(G_\alpha;X_\text{sp})\).

Proposition 1. Under (A2)–(A3), \(I(S; X_{\mathrm{sp}})\) is non-decreasing in \(\alpha\).

Proposition 2. Let \(C\) denote the model capacity of \(\phi\). Then the information encoded in the representation is bounded as \(I(S;X) =(S;X_{\mathrm{inv}})+I(S;X_{\mathrm{sp}}\mid X_{\mathrm{inv}})\le C\).

Proposition 3. Under \(X_{\mathrm{inv}} \perp X_{\mathrm{sp}}\) (A1), \(I(S;X_{\mathrm{sp}} \mid X_{\mathrm{inv}})\ge I(S;X_{\mathrm{sp}})\).

Proposition 4. \(I(S;X_\text{inv})\ge I(S;Y)\).

Proposition 5. A decrease in \(I(S;Y)\) increases the lower bound on the test error \(\mathcal{E}_\text{test}\).

Interpretation. Propositions 1–5 collectively delineate a trade-off mechanism between image guidance and model generalization. Specifically, P1 demonstrates that increasing the weight on image guidance causes the representation \(S\) to absorb more spurious information \(X_\text{sp}\). Under finite capacity, this reduces the information budget available for \(I(S;X_\text{inv})\) (P2–3). Consequently, the label-predictive information in the representation is diminished (P4), which, via Fano’s inequality, leads to a higher lower bound on the test error (P5). In summary, \[\alpha \uparrow \;\Rightarrow\; I(S; X_\text{sp}) \uparrow \;\Rightarrow\; I(S; X_\text{inv}) \downarrow \;\Rightarrow\; I(S; Y) \downarrow \;\Rightarrow\; \mathrm{LB}(\mathcal{E}_\text{test}) \uparrow.\]

9 Hyperparameter Sensitivity↩︎

In 9, we evaluate the sensitivity of our framework to the hyperparameters \(\beta_1\) and \(\beta_2\), searched over {0.0, 0.1, 0.5, 1.0}. Based on the best ResNet-50 validation accuracy on PACS, we select a fixed configuration of \(\beta_1=0.1\) and \(\beta_2=1.0\).

Using this single configuration (denoted by the triangle in 9), our method remains competitive to, and often surpasses, the strongest SOTA baselines (dashed line). Although dataset-specific tuning could yield marginal gains on certain benchmarks such as VLCS or TerraIncognita, we intentionally keep these hyperparameters fixed across all datasets and architectures.

This choice is motivated by recent observations that extensive hyperparameter tuning may introduce evaluation biases in domain generalization benchmarks [62]. While we do not modify the standard evaluation protocol, the results suggest strong performance under a shared hyperparameter configuration.

10 Standard Deviation across Seeds↩︎

All results are averaged over three seeds; 4 reports per-method standard deviations on the CLIP-ViT-B/16 backbone.

11 More Experimental Details↩︎

Following common practice, we train for 5K iterations with batch size 32 (64 for OfficeHome) for TerraIncognita, OfficeHome, PACS, and VLCS, and for 15K for DomainNet and NICO⁺⁺ with a batch size of 64. We use cosine annealing without warmup. Learning rates are set to \(10^{-4}\) for ResNet/EfficientNet, \(10^{-6}\) for CLIP backbones, and \(10^{-5}\) otherwise.

Our mapper is implemented as a lightweight Transformer encoder that projects image features into the text embedding space, and its learning rate is fixed to \(10^{-4}\) in all experiments. To stabilize training on OfficeHome, we freeze the pretrained image backbone for the first 1K iterations and optimize only the mapper before full end-to-end training. Our implementation is based on PyTorch 1.13 with CUDA 11.8, and all experiments are conducted on a single NVIDIA RTX A6000 GPU.

We report MIRO, CLIPood, RISE, and VL2V results using their official codebases, and extend them to previously unreported backbones. For these unreported backbone settings, we tune only the learning rate around each method’s default values. Specifically, we search {\(5 \cdot 10^{-6}\), \(10^{-5}\), \(3 \cdot 10^{-5}\), \(5 \cdot 10^{-5}\), \(10^{-4}\)} for MIRO (default: {\(10^{-5}\), \(3 \cdot 10^{-5}\)}), {\(10^{-5}\), \(5 \cdot 10^{-5}\), \(10^{-4}\), \(5 \cdot 10^{-4}\), \(10^{-3}\)} for RISE (default: {\(10^{-3}\)}), and {\(5 \cdot 10^{-6}\), \(10^{-5}\), \(5 \cdot 10^{-5}\), \(10^{-4}\)} for both CLIPood (default: {\(5 \cdot 10^{-6}\), \(10^{-5}\)}) and VL2V (default: {\(5 \cdot 10^{-5}\)}). Model selection for these baselines follows the standard DomainBed protocol used in prior work, where hyperparameters are selected separately for each target-domain split. In contrast, our method uses a single universal hyperparameter setting across domains, datasets and architectures, as described in 9.

12 Performance Across Diverse Backbones↩︎

3pt

5 provides the detailed results corresponding to 4 in the main text, covering diverse CNN and Transformer backbones with different pretraining sources.

13 Failure Analysis: Trade-off with Visual Context↩︎

We compare the predictions of VL2V and of our method on OfficeHome in 10. Our method fails on 7.1% of samples where VL2V is correct, while it succeeds on 13.4% of samples where VL2V fails, suggesting that image guidance is overall more harmful than beneficial under domain shift.

We further inspect the 7.1% failure subset in 10. These cases often involve visually ambiguous objects whose local appearance alone is not sufficiently distinctive. For example, our method predicts printer for an oven, spoon for toys, and bucket for trash can. Broader image cues can then help disambiguate the correct class. These examples highlight a trade-off of our approach: reducing non-essential visual variation improves overall robustness, but can hurt recognition when additional visual cues are needed for disambiguation.

14 Loss Ablation Across Backbones↩︎

To further validate the effectiveness of our loss design across backbones, 6 extends the ablation study to diverse architectures. Starting from \(\mathcal{L}_{\text{sem}}\), while adding \(\mathcal{L}_{\text{comp}}\) generally is more impactful than adding \(\mathcal{L}_{\text{align}}\) alone, their combination enforces directional consistency and reduces intra-class variance, facilitating convergence to higher accuracy across all backbones.

15 Exploration on Prompt Types↩︎

We evaluate richer, AI-generated prompts on four benchmarks. We consider two richer description styles, one based on lexical and hierarchical semantics and another on class-specific shape and function, against the simple class template. As shown in Tab. 7, increasing prompt richness yields no consistent gain over the simple template, indicating that class-level prompts already suffice for separability.

16 Class Similarity Comparisons Across Text Encoders↩︎

In 8, we report the top-5 nearest classes for several OfficeHome query labels using CLIP and MiniLM text embeddings. The two encoders capture similarity in different ways. CLIP often favors visual or shape-based resemblance; for example, for helmet, it retrieves kettle and bucket, and for fork, it retrieves pen and knives. In contrast, MiniLM tends to emphasize semantic or functional relatedness: for clipboards, it retrieves paper clip and post-it notes, and for fork, it retrieves spoon and knives. This comparison highlights that different text encoders induce different neighborhood structures, yet both provide semantically organized anchor spaces.

References↩︎

---

1. Corresponding author↩︎

2. H. Liu et al., Improved Baselines with Visual Instruction Tuning, CVPR 2024.↩︎