1 Introduction↩︎

Deep neural networks have achieved enormous success in a wide range of computer vision tasks, usually assuming that the training and test samples are drawn from the same data distribution and label space. However, real-world application scenarios often introduce unpredictability, potentially placing the model at risk of performance degradation when the above constraints are not satisfied [1]. Domain generalization (DG) [2] is then motivated as a more realistic setting to deal with data distribution shift, which refers to using multiple source domains to obtain a model with the generalization ability that can be directly applied to arbitrary unseen target domains.

Most current domain generalization researches [3]–[6] are based on the assumption of close set recognition, i.e., the classes of the source domains are consistent with those of the target domains. However, in practice, the deployed model is often exposed to some new classes that have never been encountered during the training phase [7]. For example, in medical imaging, some diseases are so rare [8] that obtaining their training samples is unrealistic. In close set classification, objects are forced to be assigned into a known class, which introduces potential risks to the model’s robustness and security. To mitigate this issue, it is essential to explore a more practical setting called open set domain generalization (OSDG), which aims to recognize unknown classes while maintaining original classification accuracy of known classes.

In open set domain generalization [9], [10], the key is to address both domain shift and category shift simultaneously. However, traditional open set recognition models are not easily applicable for domain generalization tasks because they tend to generate biased decision boundaries. i.e., only modeling the training data while neglecting the out-of-distribution samples [11], [12]. For example, the multi-binary classifier [13], [14] consists of multiple one-vs-all binary classifiers to define a decision boundary for each known class. If a given sample is classified as negative by all sub-classifiers, it is considered to have a high probability of belonging to unknown classes. As shown in 1, the limited data distribution of the positive samples (i.e., from only one corresponding class) and the more diverse distribution of the negative samples (i.e., from all other classes) can increase the risk of predicting inputs as positive rather than negative. This causes the decision boundary asymmetrically biased to the positive samples, potentially rejecting all out-of-distribution samples as unknown and misclassifying known classes in the unseen target domain.

To establish a balanced decision boundary across domains and classes, our attention turns to meta-learning [15], a simple yet effective approach for handling domain shift. Prior work on meta-learning-based domain generalization [3], [16] seeks an optimal balance among domains by matching gradients across tasks sampled from them. This domain-wise meta-learning can mitigate the risk of exhibiting excessive bias towards particular domains. As shown in 2, the rationale is that if the angles between the gradients are small, which implies that optimizing one task does not interfere with other tasks, then it is possible to achieve a win-win outcome by optimizing their combined gradient. In contrast, large angles between gradients indicate conflicting objectives, where updating one task can adversely impact the optimization procedure of others.

We propose learning positive and negative samples in such a balanced way, to place the decision boundary at the middle zone of them, attaining a more rational separation between known and unknown classes in the target domain. Concretely, we introduce a novel meta-learning strategy called dualistic MEta-learning with joint DomaIn-Class matching (MEDIC). Instead of simply adding extra iterations for inter-domain or inter-class meta-learning, we take a step further to achieve gradient matching between domains and classes simultaneously. For tasks selected from different domains, we additionally split and recombine them at the category level to construct inter-class pairs. By matching the gradients of these recombined tasks, we expect the model to not only generalize well across domains, but also to grasp a more precise understanding of class-wise relationships, which is beneficial for both close set generalization and open set recognition. This article extends our original work [17] from an initial insight to a generalized framework with accompanying theory and experiments.

• We investigate inter-class gradient matching for open set domain generalization. The method is introduced from a special case (i.e., two steps per inner loop) to a general one (i.e., multiple steps per inner loop) with an integrated task scheduling strategy for hard class pairs.

• We provides a more precise theoretical analysis of step-wise gradient matching compared to original proofs [18] [16], as eliminating reliance on mathematical expectation. Our strategy could achieve task-wise gradient matching close to its maximum value with fewer steps.

• Extensive experiments show that our method not only outperforms several state-of-the-art methods in the open set scenario, but also maintains remarkable accuracy in the conventional domain generalization setting.

2 Related Work↩︎

2.1 Domain Generalization↩︎

Domain Generalization (DG) is intended to train a model on multiple source domains that can generalize to unseen target domains without extra retraining process. Existing methods mainly focus on three directions: (i) Feature Representation, learning domain-invariant features through techniques such as domain adversarial learning [19]–[22], invariant risk minimization [23], [24], or causality-based feature disentangling [25], [26]. (ii) Data Augmentation, enhancing training diversity via domain transfer, mixing or Fourier transforms [5], [27]–[29], adversarial generation [30], [31], or stochastic noise injection [32], [33]. (iii) Learning Strategy, applying meta-learning [34]–[36], ensemble learning [37]–[40], or regularization [16], [41]–[43] where some can be efficiently achieved via meta-learning.

2.2 Open Set Recognition↩︎

Open Set Recognition (OSR) focuses on detecting novel classes not included in the training set. Based on the use of additional data, existing methods can be classified into two categories. (i) Artificial Classes. Some methods [46], [47] augment the training data with auxiliary classes to improve distinction among known classes, but their effectiveness depends heavily on the quality of these samples. Others [48], [49] propose to use generative models to guess unknown class samples, yet the resulting images are often low in quality and far from realistic, making them less effective on complex datasets [50]. (ii) Discriminative Models. OpenMax [51] replaces the softmax layer and estimates unknown probabilities using EVT [52]. Self-supervised methods [53]–[55] leverage reconstruction errors, as they believe that known-class samples are usually reconstructed more accurately than unknown ones. Metric learning [11], [56], [57] is also widely used to enhance feature discrimination. However, these approaches often misclassify all out-of-distribution samples as unknown, limiting their direct application to domain generalization.

2.3 Meta-Learning↩︎

Meta-learning, also known as learning to learn [58], [59], aims to equip models with the ability to generalize across tasks by finding an initialization that can be quickly adapted with minimal updates. The model-agnostic meta-learning (MAML) [60] and first-order meta-learning (Reptile) [18] divide the model learning process into inner and outer loops. The inner loop is for task-specific adaptation, while the outer loop seeks a globally optimal initialization for the task in the inner loop. In domain generalization, meta-learning has been applied to balance optimization across diverse domains [3], [16], [36]. MLDG [3] simulats domain shifts via meta-train and meta-test splits. Fish [16] introducs a first-order strategy to reduce computational cost. Unlike these domain-level strategies, our approach further samples tasks at the class level to prevent biased decision boundaries and better distinguish known from unknown classes in the target domain.

2.4 Open Set Domain Generalization↩︎

Open Set Domain Generalization (OSDG), summarized in ¿tbl:tab:32pbs-cmp?, aims to address both domain and class shifts. Previous studies mainly focus on training highly discriminative models [9], [10], [61] or rejecting unknown classes at test time [62]. A key limitation of prior methods is their separate treatment of the two shifts. For example, DAML [9], based on domain augmentation and meta-learning, primarily targets the data shift between source domains. CrossMatch [63] adopts consistency regularization between the close set classifier and multi-binary classifier without considering the domain shift. And our object is to tackle both shifts within a unified framework.

More recent methods [64]–[66] use stronger models to improve open set performance. Some employ stable diffusion [67] to synthesize unknown classes, while some generate class descriptions using rules or GPT-4o [68] and integrate them into CLIP [69]. Concurrently, several meta-learning methods have been developed based on MEDIC [17]. L2OT [70] adds a regularization term to penalize similar distributions between different classes. EBiL-HaDS [71] incorporates noisy samples into training and proposes a scheduler to select challenging domain–class pairs to separate tasks. HyProMeta [72] detects noisy samples using class prototypes and split meta-train and test sets according to clean and noisy samples. These studies show the adaptability of MEDIC, and the MEDIC++ proposed in this paper is a more generalized baseline, of which MEDIC can be regarded as a special case.

3 Method↩︎

3.1 discuss some definitions in open set domain generalization and the training process of meta-learning. 3.2 and 3.3 present our MEDIC++ framework. 3.4 establishes the adaptive task sampling strategy. Finally, 3.5 and 3.6 introduce the multi-binary classifier and its inference methodology.

3.1 Preliminary↩︎

Problem definition. For open set domain generalization, we are provided with \(S\) source domains \(\mathcal{S} = \lbrace \mathcal{D}_1,\mathcal{D}_2,...,\mathcal{D}_S\rbrace\) with a label space \(\mathcal{C}\) and \(T\) unseen target domains \(\mathcal{T} = \lbrace \mathcal{D}_{S+1},\mathcal{D}_{S+2},...,\mathcal{D}_{S+T}\rbrace\) with an extended label space \(\mathcal{C}\cup\mathcal{U}\) which satisfies \(\mathcal{C}\cap \mathcal{U}=\varnothing\). The \(s\)-th domain consisting of \(N_s\) samples is represented as \(\mathcal{D}_s=\lbrace(x^s_i,y^s_i)\rbrace_{i=1}^{N_s}\), where \(x^s_i\) denotes the \(i\)-th sample and \(y^s_i\) can take values from \(\mathcal{C}\) for \(\mathcal{S}\) and \(\mathcal{C}\cup\mathcal{U}\) for \(\mathcal{T}\), which refers to the corresponding label in the source or target domains. Our objective is to utilize these source domains \(\mathcal{S}\) to develop a model that can generalize to any unseen domain \(\mathcal{T}\). Below are several key terms.

Task splits, also termed task partition of a dataset \(\mathcal{S}\) is defined as a collection of non-empty subsets \(\{\mathcal{S}_1,\mathcal{S}_2,\dots,\mathcal{S}_t\}\) such that \(\mathcal{S}_i \cap \mathcal{S}_j = \varnothing \;(i \neq j)\), and \(\bigcup_{i=1}^{t} \mathcal{S}_i = \mathcal{S}\). That is, the subsets are mutually disjoint and their union constitutes the entire dataset \(\mathcal{S}\). For example, consider a dataset with domain labels \(\{a,b\}\) and class labels \(\{1,2\}\): \(\mathcal{S} = \{(a,1), (a,2), (b,1), (b,2)\}.\) Two valid partitions can be constructed in different ways. Partitioning by domain yields \(\big\{\{(a,1),(a,2)\},\{(b,1),(b,2)\}\big\},\) while partitioning by class yields \(\big\{\{(a,1),(b,1)\},\{(a,2),(b,2)\}\big\}.\) These represent two different criteria for dividing the same dataset. In this paper, a task refers to a mini-batch of samples drawn from a specific subset of such a partition.

MLDG-like meta-learning [3] requires to split the source domains \(\mathcal{S}\) into the meta-train set \(\mathcal{S}_{\mathcal{F}}\) and meta-test set \(\mathcal{S}_{\mathcal{G}}\), ensuring that \(\mathcal{S}_{\mathcal{F}} \cup \mathcal{S}_{\mathcal{G}}=\mathcal{S}\) and \(\mathcal{S}_{\mathcal{F}} \cap \mathcal{S}_{\mathcal{G}}=\varnothing\). The losses on these two sub-datasets \(\mathcal{S}_{\mathcal{F}}\) and \(\mathcal{S}_{\mathcal{G}}\) are defined as \(\mathcal{F}(\Theta)\) and \(\mathcal{G}(\Theta)\) respectively, with \(\Theta\) representing the parameters of the training model. The optimization is conducted in a bi-level manner. First, the parameters \(\Theta\) are updated to \(\Theta'\) with the loss of \(\mathcal{F}(\Theta)\). Then the loss of \(\mathcal{G}(\Theta')\) is combined with \(\mathcal{F}(\Theta)\) to update the model’s original parameters \(\Theta\).

Reptile-like meta-learning [18] segments the optimization process into an inner loop and an outer loop. Each iteration can be summarized as follows: A task consists of a batch of data sampled from a particular data distribution, and a step aggregates several tasks to create a larger batch. During the inner loop, the model is sequentially updated with steps to reach parameters \(\hat{\Theta}\). Then in the outer loop, the original parameters \(\Theta\) are updated in the direction of \(\hat{\Theta}-\Theta\). The optimization of MLDG-like meta-learning could also be approximated using the two-step reptile scheme, with one step for meta-train and the other for meta-test. In this case, \(\Theta-\hat{\Theta} = \alpha\mathcal{F}(\Theta)+\beta\mathcal{G}(\Theta')\) where \(\alpha\) and \(\beta\) are inner loop learning rates.

Gradient matching refers to searching parameters where the gradient directions induced by different tasks are well aligned. The goal is to ensure that optimizing one task does not cause significant interference with others, thereby promoting consistent improvement. Directly enforcing gradient matching as an explicit regularization term would require computing second-order derivatives, which introduces substantial computational overhead. In practice, meta-learning is commonly adopted to achieve implicit gradient matching. A detailed discussion will be provided in the following sections.

3.2 MEDIC as a Special Case↩︎

To clarify the mechanism of gradient matching across multiple attributes (i.e., domain and class), we begin by introducing a special case called MEDIC, where both of the domains and classes are separated into two parts, yielding four tasks in total, and each step selects two of them.

Given two sub-datasets \(\mathcal{S}_{\mathcal{F}}\) and \(\mathcal{S}_{\mathcal{G}}\), along with their corresponding loss functions \(\mathcal{F}(\Theta)\) and \(\mathcal{G}(\Theta)\), our objective is to reach a consensus on their gradients \({\mathcal{F}'(\Theta)}\) and \({\mathcal{G}'(\Theta)}\) to ensure an unbiased optimization direction for both of them. The underlying principle is that if the angle between the directions of \({\mathcal{F}'(\Theta)}\) and \({\mathcal{G}'(\Theta)}\) is small which means optimizing one task does not adversely affect the other, then updating with their combined gradient (i.e., the sum of gradients in practice) can yield enhanced performance. Conversely, if the angle between their gradients is large which indicates a conflict between these two tasks, then updating one task would lead to an inferior optimization process for the other. The core idea of gradient matching is to find an area in the parameter space where the angle between the gradients of \(\mathcal{S}_{\mathcal{F}}\) and \(\mathcal{S}_{\mathcal{G}}\) is minimized, which can be accomplished by maximizing the dot product of \({\mathcal{F}'(\Theta)}\) and \({\mathcal{G}'(\Theta)}\). Moving the model towards this region, \(\mathcal{S}_{\mathcal{F}}\) and \(\mathcal{S}_{\mathcal{G}}\) will converge on gradient direction, where both tasks benefit from shared updates rather than competing each other.

Current gradient-based domain generalization methods typically treat \(\mathcal{S}_{\mathcal{F}}\) and \(\mathcal{S}_{\mathcal{G}}\) as separate domains in order to find an optimization direction only among domains [3], [16]. However, these methods often overlook the inter-class relationships, which are important in open set scenarios. Instead of simply adding extra iterations to mitigate biased prediction between classes, we propose a novel meta-learning strategy that performs gradient matching across both inter-domain and inter-class splits simultaneously, which aims to learn generalizable decision boundaries that maintain balance across all tasks.

As illustrated in 3, for tasks \(\mathcal{S}_{\mathcal{F}}\) and \(\mathcal{S}_{\mathcal{G}}\) sampled from different domains, we further divide them into \(\mathcal{S}_{\mathcal{F}_1}, \mathcal{S}_{\mathcal{F}_2}\) and \(\mathcal{S}_{\mathcal{G}_1}, \mathcal{S}_{\mathcal{G}_2}\) by class and define their loss functions as \(\mathcal{F}_1, \mathcal{F}_2\) and \(\mathcal{G}_1, \mathcal{G}_2\). The label spaces of \(\mathcal{S}_{\mathcal{F}_1}\) and \(\mathcal{S}_{\mathcal{F}_2}\), as well as \(\mathcal{S}_{\mathcal{G}_1}\) and \(\mathcal{S}_{\mathcal{G}_2}\) are both disjoint. Besides, we require \(\mathcal{S}_{\mathcal{F}_1}\) and \(\mathcal{S}_{\mathcal{G}_1}\) to share the same label space, and likewise for \(\mathcal{S}_{\mathcal{F}_2}\) and \(\mathcal{S}_{\mathcal{G}_2}\). To simultaneously apply gradient matching between domains and classes, we utilize \((\mathcal{S}_{\mathcal{F}_1}, \mathcal{S}_{\mathcal{G}_2})\) as meta-train set and \((\mathcal{S}_{\mathcal{F}_2}, \mathcal{S}_{\mathcal{G}_1})\) as meta-test set. The final meta-objective function of MEDIC is defined as follows: \[\label{eq:medic-objective} \mathop{\rm argmin}_{\Theta}\, [{\mathcal{F}_1(\Theta)} + {\mathcal{G}_2(\Theta)}] + \beta [{\mathcal{F}_2(\hat{\Theta})} + {\mathcal{G}_1(\hat{\Theta})}].\tag{1}\] This objective approximates the two-step reptile scheme, with one step for the meta-train set \((\mathcal{S}_{\mathcal{F}_1}, \mathcal{S}_{\mathcal{G}_2})\) and the other for the meta-test set \((\mathcal{S}_{\mathcal{F}_2}, \mathcal{S}_{\mathcal{G}_1})\), where \(\beta\) controls the weight between the two meta sets and \(\hat{\Theta}\) is the optimized model parameters on the meta-train set with learning rate \(\alpha\): \[{\hat{\Theta}} = \Theta - \alpha({\mathcal{F}_1'(\Theta)} + {\mathcal{G}_2'(\Theta)}).\] To validate MEDIC’s capability to perform gradient matching between domains and classes at the same time, similar to the analysis in [3], we conduct a first order Taylor expansion for the second term in 1 : \[\begin{align} {\mathcal{F}_2(\hat{\Theta})} &= {\mathcal{F}_2(\Theta)} - \alpha ( {\mathcal{F}_1'(\Theta)} + {\mathcal{G}_2'(\Theta)}) \cdot {\mathcal{F}_2'(\Theta)}, \\ {\mathcal{G}_1(\hat{\Theta})} &= {\mathcal{G}_1(\Theta)} - \alpha ( {\mathcal{F}_1'(\Theta)} + {\mathcal{G}_2'(\Theta)}) \cdot {\mathcal{G}_1'(\Theta)}, \end{align}\] and the objective function becomes: \[\label{eq:taylor-objective} \begin{align} & \mathop{\rm argmin}_{\Theta}\, [{\mathcal{F}_1(\Theta)} + {\mathcal{G}_2(\Theta)} + \beta ({\mathcal{F}_2(\Theta)} + {\mathcal{G}_1(\Theta)})] \\ &- \beta \alpha [({\mathcal{F}_1'(\Theta)} + {\mathcal{G}_2'(\Theta)})\cdot( {\mathcal{F}_2'(\Theta)} + {\mathcal{G}_1'(\Theta)})]. \end{align}\tag{2}\] The first term of 2 involves optimizing the model with the expected losses of each task, while the second term is the product of gradient sums. By expanding this part we derive the following regularization term: \[\label{eq:regularization} \mathcal{L}_{\rm reg} = -(\mathcal{F}_1' \cdot \mathcal{F}_2' + \mathcal{F}_1' \cdot \mathcal{G}_1' + \mathcal{G}_2' \cdot \mathcal{F}_2' + \mathcal{G}_2' \cdot \mathcal{G}_1').\tag{3}\] Note that we omit parameters \(\Theta\) for the sake of simplicity. As previously discussed, maximizing the dot product of gradients can regularize the optimization process to match the updating directions of different tasks. Taking task \(\mathcal{S}_{\mathcal{F}_1}\) as an example, the multiplier \(\mathcal{S}_{\mathcal{G}_1}\) contains the same classes but from different domains, whereas \(\mathcal{S}_{\mathcal{F}_2}\) includes different classes from the same domain, the two factors in any of the gradient products are either from different domains or different classes to enable domain-wise and class-wise matching simultaneously. In contrast to conventional methods primarily concerned with inter-domain relationships, the dot product between \(\mathcal{S}_{\mathcal{F}_1}\) and \(\mathcal{S}_{\mathcal{F}_2}\) bridges the gap in class-wise gradient matching inside each domain, which enables fine-grained model optimization to learn more rational decision boundaries.

3.3 The MEDIC++ Framework↩︎

MEIDC++ is a flexible meta-learning strategy that involves several tasks at each step. It can be extended into a general learning framework by creating additional tasks with finer division between domains and classes, while the number of tasks per step can also be customized as we need.

Existing meta-learning-based DG methods usually overlook the distinction between tasks and steps, with a preference for sampling only one task at each step. Fish [16] theoretically demonstrates the effectiveness of this approach in pairing tasks with one another, allowing for maximum gradient matching across all tasks. However, this strategy can become complicated when dealing with a large quantity of tasks, as the number of steps increases linearly with them. Besides, the batch normalization-based models over training only normalize the mean and variance of each single batch. As shown in 5, updating with only one task per step leads to similar statistics across classes, making their features indistinguishable and may negatively affect classification performance. Conversely, if we just take a single large step that encompasses all the tasks, the number of paired steps will be reduced to zero, leading to no gradient matching between any tasks.

We argue that taking multiple steps is essential for task-wise gradient matching, but there is no need for a large number of them. To this end, we propose our MEDIC++ as follows: During the inner loop, the dataset is partitioned into \(t\) tasks \(\lbrace \mathcal{S}_{\mathcal{H}_1}, \mathcal{S}_{\mathcal{H}_2}, ..., \mathcal{S}_{\mathcal{H}_t}\rbrace\) at both the domain and the class levels, with their loss functions represented as \(\lbrace\mathcal{H}_1, \mathcal{H}_2, ..., \mathcal{H}_t\rbrace\). We define the number of steps as \(n\) and the corresponding loss functions as \(\lbrace\mathcal{L}_1, \mathcal{L}_2, ..., \mathcal{L}_n\rbrace\). The number of tasks included in the \(i\)-th step is denoted as \(m_i\) that satisfies \(\sum_{i=1}^{n}m_i = t\). In each step, tasks are randomly sampled without repetition for gradient update, with the condition that \(n\) is less than \(t\). We assume a uniform distribution of tasks per step, i.e., tasks are distributed as \(\lfloor \frac{t}{n} \rfloor\) in some steps while \(\lceil \frac{t}{n} \rceil\) in others. The loss function of the \(i\)-th step is expressed as: \[\label{eq:step-loss} \mathcal{L}_i = \sum_{k=1}^{m_i}\mathcal{H}_{a_i^k},\tag{4}\] where \(a_i^k\) is the index of the \(k\)-th task chosen in the \(i\)-th step. Following the inner loop, the model’s parameters are updated from \(\Theta\) to \(\hat{\Theta}\), and we adopt the final parameters as \(\Theta \leftarrow \Theta + \epsilon(\hat{\Theta}-\Theta)\), where \(\epsilon\) can be considered as the negative learning rate of the outer loop. This reptile-like meta-learning [16], [18] conducts pairwise gradient matching between each step with scaling factor \(\gamma\): \[\label{eq:reptile-objective} \mathop{\rm argmin}_{\Theta}\, \sum_{i=1}^{n}\mathcal{L}_i(\Theta)-\gamma\sum_{i, j \in \mathcal{N}}^{i\neq j}\mathcal{L}_i'(\Theta) \cdot \mathcal{L}_j'(\Theta),\tag{5}\] where \(\mathcal{N}\) is the indices of \(\lbrace 1,2,...,n\rbrace\), thus the number of step-wise gradient matching can be calculated as \(\frac{n(n-1)}{2}\). We provide a more precise proof of 5 and please refer to 4.1 for more details. When substituting 4 and its derivative into 5 , we can obtain the objective: \[\label{eq:task-objective} \mathop{\rm argmin}_{\Theta}\, \sum_{i=1}^{t}\mathcal{H}_i(\Theta)-\gamma\sum_{i, j \in \mathcal{N}}^{i\neq j} \left\{ \sum_{k=1}^{m_i}\mathcal{H}_{a_i^k}'(\Theta) \cdot \sum_{k=1}^{m_j}\mathcal{H}_{a_j^k}'(\Theta) \right\},\tag{6}\] implying that matching gradients between steps is equivalent to matching gradients across all inter-step tasks.

Considering the number of task-wise gradient matching as \(t_n\), we show in 4.3 that \(t_n\) is positively correlated with the step count \(n\). By using \(\frac{t}{n}\) to approximate the number of task-wise gradient matching per step, we could derive an estimate for \(t_n\) as: \[\label{eq--} f(n) = \frac{n(n-1)}{2} \cdot (\frac{t}{n})^2 = \frac{1}{2}t^2 \cdot (1-\frac{1}{n}).\tag{7}\] Note that if \(t\) is divisible by \(n\), then \(t_n\) and \(f(n)\) coincide. The derivative of \(f(n)\) is: \[\label{eq---} f'(n) = \frac{t^2}{2n^2},\tag{8}\] which indicates that when \(n\) is small, there is a rapid increase in task-wise gradient matching. However, as \(n\) becomes larger, the growth rate converges to zero. As shown in 5 (d), if we consider a total of \(9\) tasks, \(t_n\) reaches its maximum value of \(36\) when \(n\) equals \(9\), while even with \(n\) as small as \(3\), \(t_n\) still achieves a substantial value of \(27\), revealing that the inner loop doesn’t need to be excessively long. In practice, a relatively small number of steps is sufficient to obtain gradient matching \(t_n\) close to its maximum value. Furthermore, since we perform additional task sampling at the class level, updating gradients with multiple tasks allows unified batch normalization across diverse classes, which enables the model to capture specific spatial distribution of each class in the feature space.

Remark 1. We believe that the rationality behind MEDIC++ can draw inspiration from the sampling strategies of modern gradient descent algorithms, as the sequence of steps mirrors such gradient update operation during each inner loop. Please be aware that this is merely an informal analogy and should be taken much less seriously than the previous analysis.

We begin by comparing the three task sampling strategies in 4 with gradient descent algorithms [73], assuming that the intra-task samples can be interpreted as a collective data point due to their greater similarity in gradient patterns compared to the inter-task samples. In this case, the single-step strategy is similar to batch gradient descent (BGD), which utilizes all data points for each gradient update. The single-task-per-step strategy resembles a narrower variant of stochastic gradient descent (SGD), involving a single data point per gradient step. Our MEDIC++, following a multiple-task-per-step strategy, is analogous to mini-batch gradient descent (MBGD) with each gradient computed using a small batch of randomly selected data points. Modern gradient descent algorithms often prefer MBGD, which aims to seek a balance between exploration and stability [74]–[76], so opting for an multiple-task-per-step approach seems to be a wise choice.

Remark 2. We recommend that one essential contribution of MEDIC++ is to broaden the notion of a task. In domain generalization, it is natural to treat each domain as a separate task, since source domains are predefined. However, if we instead regard the entire training set as a single global distribution, it can be partitioned into tasks along multiple dimensions. We explore class-wise partitioning as an alternative and provide evidence of its effectiveness. In this sense, task can be a design choice rather than an intrinsic property of the dataset, which needs not be limited to explicit domain or class splits. More broadly, our work suggests that any dataset capable of being decomposed into multiple sub-distributions may benefit from the learning paradigm of MEDIC++.

3.4 Adaptive Task Sampling↩︎

We introduce an adaptive task sampling strategy to enhance the frequency of gradient matching between easily confusable classes. Inspired by the method in [77], we first construct an asymmetric class transition probability matrix for each domain, where \(p_{ij}\) represents the average prediction probability of misclassifying class \(i\) as class \(j\): \[\label{eq-1} p_{ij}=\frac{p(j|c=i)}{\sum_{k\neq i}p(k|c=i)}.\tag{9}\] As shown in 7, we start by randomly selecting a class. Then, based on the transition probabilities from the selected class to others, we sample the next class without replacement. This process is repeated, with the sampled classes cyclically assigned to different tasks. The rationale behind is that a higher transition probability from class \(i\) to class \(j\) suggests that the decision boundary between these classes is more likely to be biased, thereby increasing the frequency of gradient matching between them. Since gradient matching only occurs across different tasks, where the adjacent classes are always placed, it is reasonable to use the class transition probabilities to guide the sampling of the next class.

3.5 Open Set Loss Function↩︎

For open set recognition, we adopt a multi-binary classifier [13] to serve as a supplement to the close set classifier. As illustrated in 3, the proposed classifier consists of \(|\mathcal{C}|\) one-vs-all classifiers, with each classifier trained to detect whether a given sample belongs to its corresponding class. Let \(p(\hat{y}^k|x)\) denote the output probability that an instance \(x\) is an inlier of the \(k\)-th sub-classifier. For a given sample \((x, y)\), its loss on the multi-binary classifier can be formulated as: \[\label{eq:ovaloss} \mathcal{L}_{\rm ova}(x, y) = -{\rm log}(p(\hat{y}^y|x)) - \mathop{\rm min}_{j \neq y}\,{\rm log}(1-p(\hat{y}^j|x)).\tag{10}\] The second term denotes that it updates only the most challenging binary classifier when used as a negative sample. We simply adopt this loss and train a close set classifier using cross-entropy loss denoted as \(\mathcal{L}_{\rm ce}\). Then the overall open set loss function can be defined as follows: \[\label{eq:overalloss} \mathcal{L}_{\rm all} = \mathcal{L}_{\rm ce} + \mathcal{L}_{\rm ova}.\tag{11}\] 11 is employed as the objective for each task in our meta-learning paradigm. Implementing inter-class gradient matching can stabilize the training process of both positive and negative samples, thus seeking a balance between close set generalization and open set recognition.

Remark. The close set classifier can also be considered as a general form of multi-binary classifier, where each channel serves as a single-output binary classifier. The introduction of the two-channel binary classifiers for open set classification is due to the fact that the cross entropy loss is not well-suited for single-output binary classifiers since it optimizes only the positive class but does not address the negative classes.

3.6 Inference↩︎

In the test phase, each target sample is first predicted by the close set classifier to obtain a probability distribution \(p(\hat{y}|x)\) over known classes. The model either (i) chooses the value of its maximum likelihood: \[\label{eq:confcls} {\rm conf}_{\rm cls}(x) = {\rm max}_{i=1}^{|\mathcal{C}|}(p(\hat{y}|x)_i),\tag{12}\] or (ii) then refers to the corresponding one-vs-all classifier and chooses the value on its positive output channel as the confidence score [13]: \[\label{eq:confblcs} {\rm conf}_{\rm bcls}(x) = p(\hat{y}^{{\rm argmax}_{i=1}^{|\mathcal{C}|}(p(\hat{y}|x)_i)}|x).\tag{13}\] If the score is greater than a preset threshold \(\mu\), then classify the sample into known classes, otherwise judge it as unknown. Experimental results for these two inference modes are both reported in 5.

4 Theoretical Analysis↩︎

4.1 proves that reptile-like meta-learning conducts pairwise gradient matching between each step. 4.2 explains how to scale the learning rate within the outer loop. 4.3 demonstrates that task-wise gradient matching is positively correlated with the step count.

4.1 Step-wise Gradient Matching↩︎

We prove 5 through mathematical induction. Let’s start by revisiting the definitions of the \(n\) steps inner loop, during which the model’s parameters transition from \(\Theta\) to \(\hat{\Theta}\). We represent the loss at each step as \(\lbrace \mathcal{L}_1, \mathcal{L}_2, ..., \mathcal{L}_n\rbrace\), and the parameter updating trajectory as \(\lbrace \theta_1, \theta_2, ..., \theta_{n+1}\rbrace\), with \(\theta_1\) and \(\theta_{n+1}\) corresponding to \(\Theta\) and \(\hat{\Theta}\) respectively. We use \(\mathcal{L}_i(\theta_j)\) to denote the loss of the \(i\)-th step on parameters \(\theta_j\). During the inner loop, the update process is performed with a small learning rate \(\alpha\): \[\label{eq:update-process} \begin{align} \theta_2 &= \theta_1 - \alpha\mathcal{L}_1'(\theta_1) \\ \theta_3 &= \theta_2 - \alpha\mathcal{L}_2'(\theta_2) \\ &\vdots \\ \theta_{n+1} &= \theta_n - \alpha\mathcal{L}_n'(\theta_n). \end{align}\tag{14}\] While in the outer loop, the model is updated from \(\theta_1\) to the final parameters: \[\label{eq:theta-final} \Theta \leftarrow \theta_1-\epsilon(\theta_1-\theta_{n+1}).\tag{15}\] We then sum the formulas from 14 and obtain: \[\label{eq:update-process-sum} \theta_1-\theta_{n+1} = \alpha\sum_{i=1}^n\mathcal{L}_i'(\theta_i).\tag{16}\] Plugging 16 into 15 yields the original representation for the reptile-like meta-learning objective: \[\label{eq:reptile-objective-ori} \mathop{\rm argmin}_{\theta_1}\, \alpha \sum_{i=1}^n\mathcal{L}_i(\theta_i).\tag{17}\]

Objective. To prove that the gradient of any step is matched with those of the other \(n\)-1 steps, it is adequate to demonstrate that for any positive integer \(i=k\), step \(k\) is gradient-matched with the previous \(k\)-1 steps as: \[\label{eq:lossk-thetak} \mathcal{L}_k(\theta_k) = \mathcal{L}_k(\theta_1) - \alpha\sum_{i=1}^{k-1}\mathcal{L}_i'(\theta_1) \cdot \mathcal{L}_k'(\theta_1) + \mathcal{O}(\alpha^2),\tag{18}\] where gradient matching between two steps can be expressed by their dot product at \(\theta_1\). Proving 18 only requires that the following equation holds for any loss function \(\mathcal{L}\): \[\label{eq:loss-thetak} \mathcal{L}(\theta_k) = \mathcal{L}(\theta_1) - \alpha\sum_{i=1}^{k-1}\mathcal{L}_i'(\theta_1) \cdot \mathcal{L}'(\theta_1) + \mathcal{O}(\alpha^2).\tag{19}\]

Base Case. When \(i\) equals 1, it is evident that \(\mathcal{L}(\theta_i) = \mathcal{L}(\theta_1)\), so 19 holds. When \(i\) equals 2, we can substitute 14 into \(\mathcal{L}(\theta_2)\) and conduct a first order Taylor expansion on it: \[\label{eq:loss-theta2} \mathcal{L}(\theta_2) = \mathcal{L}(\theta_1) - \alpha\mathcal{L}_1'(\theta_1) \cdot \mathcal{L}'(\theta_1) + \mathcal{O}(\alpha^2),\tag{20}\] thus 19 also holds.

Inductive Step. Assuming that 19 is true for arbitrary \(i \leq k\), we prove its validity when \(i\) equals \(k+1\). Plugging 14 and 19 into \(\mathcal{L}(\theta_{k+1})\) yields: \[\label{eq:loss-thetak431} \begin{align} \mathcal{L}(\theta_{k+1}) =\;&\mathcal{L}(\theta_k) - \alpha\mathcal{L}_k'(\theta_k) \cdot \mathcal{L}'(\theta_k) + \mathcal{O}(\alpha^2) \\ =\;&\mathcal{L}(\theta_1) - \alpha\sum_{i=1}^{k-1}\mathcal{L}_i'(\theta_1) \cdot \mathcal{L}'(\theta_1) + \mathcal{O}(\alpha^2) \\ &- \alpha(\mathcal{L}_k'(\theta_1) + \mathcal{O}(\alpha))(\mathcal{L}'(\theta_1) + \mathcal{O}(\alpha)) + \mathcal{O}(\alpha^2) \\ =\;&\mathcal{L}(\theta_1) - \alpha\sum_{i=1}^{k}\mathcal{L}_i'(\theta_1) \cdot \mathcal{L}'(\theta_1) + \mathcal{O}(\alpha^2). \end{align}\tag{21}\] Note that we substitute \(\mathcal{L}_k'(\theta_k)\) into 19 to obtain: \[\label{eq----} \mathcal{L}_k'(\theta_k) = \mathcal{L}_k'(\theta_1) - \alpha\sum_{i=1}^{k-1}\mathcal{L}_i'(\theta_1) \cdot \mathcal{L}_k''(\theta_1) + \mathcal{O}(\alpha^2),\tag{22}\] which is simplified as \(\mathcal{L}_k'(\theta_1) + \mathcal{O}(\alpha)\) within 21 . The derivation of \(\mathcal{L}'(\theta_k)\) follows the same process.

Conclusion. We prove that 19 holds for all positive integers \(i=k\) and any loss function \(\mathcal{L}\). Plugging 19 into 17 and the meta-objective is transformed to: \[\label{eq:reptile-objective-gamma} \mathop{\rm argmin}_{\theta_1}\, \sum_{i=1}^{n}\mathcal{L}_i(\theta_1)-\alpha\sum_{i, j \in \mathcal{N}}^{i\neq j}\mathcal{L}_i'(\theta_1) \cdot \mathcal{L}_j'(\theta_1).\tag{23}\] By replacing the learning rate \(\alpha\) with \(\gamma\) and initial parameters \(\theta_1\) with \(\Theta\), we ultimately obtain 5 .

Remark. Our analysis is originally motivated by MLDG [3], which adopts a two-step procedure and derives gradient matching between two domain task splits. This observation strongly inspire MEDIC [17] and lead us to investigate whether such behavior could be extended to multi-step settings. The background explains why our derivation differs substantially from previous Reptile-based analyses [16], [18], [78]. Prior works formulate the objective in terms of mathematical expectation, specifically through a quantity referred as AvgGradInner [18]. They first derive the gradient at step \(i\) as: \[\label{eq:32gradient-step-i} \mathcal{L}'_i(\theta_i) = \mathcal{L}'_i(\theta_1) - \alpha \mathcal{L}''_i(\theta_1) \sum_{j=1}^{i-1}\mathcal{L}'_j(\theta_1) + \mathcal{O}(\alpha^2),\tag{24}\] and then isolate a single term from the second component and compute its expectation: \[\label{eq:32avggradinner} \begin{align} \mathop{\rm AvgGradInner} &= \mathbb{E}_{i,j}(\mathcal{L}''_i(\theta_1) \mathcal{L}'_j(\theta_1)) \\ &= \mathbb{E}_{i,j}(\mathcal{L}''_j(\theta_1) \mathcal{L}'_i(\theta_1)) \\ &= \frac{1}{2}\mathbb{E}_{i,j}(\mathcal{L}''_i(\theta_1) \mathcal{L}'_j(\theta_1) + \mathcal{L}''_j(\theta_1) \mathcal{L}'_i(\theta_1)) \\ &= \frac{1}{2}\mathbb{E}_{i,j}((\mathcal{L}'_i(\theta_1) \mathcal{L}'_j(\theta_1))') \end{align}\tag{25}\] The result is therefore expressed in terms of an expectation \(\mathbb{E}\), which cannot be eliminated from their formulation. In contrast, we establish 18 which directly yields: \[\label{eq:32gradinner} \mathcal{L}'_i(\theta_i) = \mathcal{L}'_i(\theta_1) - \alpha\sum_{j=1}^{i-1}(\mathcal{L}_j'(\theta_1) \cdot \mathcal{L}_i'(\theta_1))' + \mathcal{O}(\alpha^2).\tag{26}\] By removing the operator \(\mathbb{E}\), we characterize the exact pairwise gradient matching among tasks, rather than its expectation. This distinction is central to our theoretical development, as it ensures that gradient matching holds on the current path instead of merely on average.

4.2 Scaling the Learning Rate↩︎

We discuss how to set learning rate \(\epsilon\) for the outer loop. Plugging 4 into 17 leads to the loss function as: \[\label{eq-----} \mathcal{L}_{\rm outer} = \alpha \sum_{i=1}^n \sum_{k=1}^{m_i}\mathcal{H}_{a_i^k}(\theta_i),\tag{27}\] while the standard loss of empirical risk minimization (ERM) [79] without meta-learning can be expressed as: \[\label{eq------} \begin{align} \mathcal{L}_{\rm erm} &= \frac{1}{t} \sum_{i=1}^t \mathcal{H}_{i}(\theta_1) \\ &= \frac{1}{t} \sum_{i=1}^n \sum_{k=1}^{m_i}\mathcal{H}_{i}(\theta_1), \end{align}\tag{28}\] which implies that the coefficient of the loss in the outer loop is \(\alpha t\) times that of ERM, thus \(\epsilon\) needs to be scaled to \(\frac{1}{\alpha t}\) of the default learning rate.

4.3 Relationship Between \(t_n\) and \(n\)↩︎

In 3.3, the estimate of the gradient matching count \(t_n\) increasing with the step count \(n\) is presented in continuous form. Here, we provide a discrete proof. As shown in 8, we construct a graph model where tasks are represented as nodes, task-wise gradient matching as edges, and steps as node partitions \(\lbrace \mathcal{P}_1, \mathcal{P}_2, ..., \mathcal{P}_n \rbrace\). The number of tasks for the \(i\)-th step can also be written as the number of nodes \(|\mathcal{P}_i|\). Assuming a uniform task count distribution per step, it holds for any \(i\) and \(j\) that: \[\label{eq:uniform-distribution} |\mathcal{P}_i| \leq |\mathcal{P}_j| + 1.\tag{29}\] Matching gradient between steps \(i\) and \(j\) can be regarded as generating a complete bipartite graph \(K(\mathcal{P}_i, \mathcal{P}_j)\) with a total of \(|\mathcal{P}_i| \cdot |\mathcal{P}_j|\) edges. When the number of steps transforms from \(n+1\) to \(n\), it is equivalent to uniformly dividing \(\mathcal{P}_{n+1}\) into \(n\) sub-partitions \(\lbrace \mathcal{P}_1', \mathcal{P}_2', ..., \mathcal{P}_n' \rbrace\), and allocating them to the remaining partitions according to their respective indices. The change in the number of edges is expressed as: \[\label{eq:delta-edge} \begin{align} \Delta_{n+1 \rightarrow n} &= \sum_{i, j \in \mathcal{N}}^{i \neq j}|\mathcal{P}_i'| \cdot |\mathcal{P}_j'| - \sum_{i=1}^n|\mathcal{P}_i'| \cdot |\mathcal{P}_i| \\ &= \frac{1}{2}\sum_{i=1}^n| \mathcal{P}_i'| \sum_{j \in \mathcal{N}}^{j \neq i}|\mathcal{P}_j'| - \sum_{i=1}^n|\mathcal{P}_i'| \cdot |\mathcal{P}_i| \\ &= \frac{1}{2} \sum_{i=1}^n|\mathcal{P}_i'| \left( \sum_{j \in \mathcal{N}}^{j \neq i}|\mathcal{P}_j'| - 2|\mathcal{P}_i|\right). \end{align}\tag{30}\] The first term is the edges unique to step \(n\), while the second term is the edges unique to step \(n+1\). It is evident that each term of the summation in 30 equals \(0\) when \(|\mathcal{P}_i'|=0\), For any \(|\mathcal{P}_i'| \geq 1\), it follows from 29 that: \[\label{eq-------} \sum_{j \in \mathcal{N}}^{j \neq i}|\mathcal{P}_j'| = |\mathcal{P}_{n+1}| - |\mathcal{P}_i'| \leq |\mathcal{P}_{n+1}| - 1 \leq |\mathcal{P}_{i}|,\tag{31}\] thus the corresponding terms are less than \(0\). Because of this, the value of 30 is negative, so the number of edges from step \(n+1\) to \(n\) decreases, which shows a positive correlation between gradient matching count \(t_n\) and step count \(n\).

5 Experiment↩︎

5.1 Datasets↩︎

We evaluate on seven standard DG datasets whose details are described as follows: (i) PACS [80] contains 4 domains (photo, art-painting, cartoon, sketch) with 7 classes and 9,991 images. (ii) Office-Home [81] comprises 4 domains (art, clipart, product, real-world) with 65 classes and 15,588 images. (iii) VLCS [82] consists of 4 domains (pascal, labelme, caltech, sun) with 5 classes and 10,729 images. (iv) TerraIncognita [83] is composed of 4 domains (location38, location43, location46, location100) with 100 classes and 24,788 images. (v) DomainNet [84] includes 6 domains (clipart, infograph, painting, quickdraw, real, sketch) with 345 classes and 586,575 images. (vi) Digits-DG [85] is an aggregation of 4 domains (mnist [86], mnist-m [87], svhn [88], syn [87]) with 10 classes and 48,000 images. (vii) CMNIST [23] consists of 3 domains, 2 classes, and 60,000 images.

5.2 Implementation Details↩︎

Basic details. Each training set is randomly segmented into 3 parts by both domain and class to obtain a total of 9 tasks. The inner loop comprises 3 steps, each of which contains 3 tasks with a fixed learning rate of \(0.01\). For datasets other than Digit-DG, we employ ResNet50 [95] and GFNet [96] pretrained on ImageNet [97] as our backbone networks. For Digits-DG, we adopt a lightweight convolutional architecture called ConvNet from [85]. The leave-one-domain-out evaluation protocol is applied to all benchmarks, i.e., picking one target domain for testing and using the rest for training and validation. We set aside 20% of the samples for validation from each source domain and choose the model that maximizes the accuracy on the overall validation set, which is same as the training-domain validation set recommended in [98].

DomainBed benchmark. We first follow the close set protocol proposed in [98], including the hyperparameter search space, model structure and Adam optimizer. For datasets like PACS which contains 3 training domains with a default batch size of \((32, 96)\), we configure our batch size to \((12, 108)\). For DomainNet which includes 5 training domains with a default batch size of \((32, 160)\), we select our batch size as \((18, 162)\). For CMNIST with a default batch size of \((64, 128)\), we set our batch size as \((32, 128)\). The first value in the parentheses is the batch size per domain or per task, while the second value is their combined sum. The default learning rates are \(5\times 10^{-5}\) for non-digits and \(1\times 10^{-3}\) for digits, with our rates set to \(6\times 10^{-4} \times 0.09\) and \(2\times 10^{-2} \times 0.04\). As used in [38], we triple the number of iterations for DomainNet from 5,000 to 15,000 since 5,000 iterations is inadequate for convergence. Notably, the multi-binary classifier is excluded in this benchmark.

Additional configurations. To further improve the model’s performance, we fine-tune hyperparameters and incorporate the multi-binary classifier into training. The number of iterations is doubled to 30,000 for DomainNet and 10,000 for TerraIncognita, while the batch size per task is uniformly set to 16 across all datasets. Using a SGD optimizer, the initial learning rates of the outer loop are configured as follows: \(0.02\) for PACS and Office-Home, \(0.01\) for VLCS, TerraIncognita and DomainNet, \(0.1\) for Digits-DG and CMNIST. Each learning rate is then reduced to 10% in the last 20% iterations. For open set experiments, the classes are organized in alphabetical order, with the former part as known classes and the latter as unknown. The split rates for known and unknown classes on each dataset are detailed in the corresponding table caption. We use close set validation accuracy for model selection.

For ablation studies, all methods use the same number of samples per iteration and scaled learning rate. Except for the specific ablation involving multi-binary classifiers, all methods are implemented with them to ensure a fair comparison.

5.3 Evaluation Metrics↩︎

We choose three evaluation metrics that take both known and unknown class accuracy into account: (i) Acc represents the typical close set accuracy. (ii) H-score [100] quantifies the harmonic mean of known class accuracy \({\rm acc}_{\rm k}\) and unknown class accuracy \({\rm acc}_{\rm u}\) as follows: \[\label{eq:h-score} {\rm H-score}=2 \cdot \frac{{\rm acc}_{\rm k}\cdot{\rm acc}_{\rm u}}{{\rm acc}_{\rm k}+{\rm acc}_{\rm u}}.\tag{32}\] When \({\rm acc}_{\rm k} + {\rm acc}_{\rm u}\) remains constant, the closer \({\rm acc}_{\rm k}\) and \({\rm acc}_{\rm u}\) are, the larger H-score will be. Compared with the weighted average, H-score puts more emphasis on the balance between close set classification and open set recognition. Nevertheless, the manually designed threshold to reject unknown classes is not applicable for a random target domain. We propose a threshold-independent metric (iii) open set classification rate (OSCR) [46] which plots the true positive rate against the false positive rate using an ever-moving threshold. Different from area under the receiver operating characteristic (AUROC) [49] that neglects known class accuracy, OSCR considers only correctly labeled samples as true positive ones.

5.4 Close Set Results↩︎

We compare our strategy with closely related meta-learning methods such as MLDG [3] and Fish [16], as well as gradient-based methods like RSC [41], AND-mask [92] and Fishr [94]. Firstly, we use the same model architecture and default hyperparameters. As ¿tbl:tab:domainbed-close? illustrates, our method achieves the highest average performance, outperforming other methods on three datasets namely PACS, OfficeHome and TerraIncognita, surpassing the second-best method by 1.1%, 0.9% and 1.5% respectively. Secondly, by involving the multi-binary classifier, the model’s average performance is further boosted by 2.0%, suggesting that the binary classifiers which separate inter-class samples also play a role in improving close set accuracy.

5.5 Open Set Results↩︎

We conduct open set experiments, and the results on PACS, Digits-DG, and DomainNet are shown in ¿tbl:tab:32pacs6-res50?, ¿tbl:tab:32digits6-conv?, ¿tbl:tab:32pacs6-gfnet?, and ¿tbl:tab:32domainnet100-res50?, respectively. Our strategy outperforms other DG and OSR methods in both close set and open set scenarios. MEDIC++ shows superior performance compared to MEDIC, achieving a improvement in close set accuracy by 1.29% in ¿tbl:tab:32pacs6-res50?. These results indicate that our method is capable of producing generalizable and discriminative representations, benefiting both DG and OSR tasks.

We also compare with several open set recognition methods such as OpenMax [51] and ARPL [56]. Note that we exclude the calculation of OSCR for OpenMax [51] and OneRing [61] due to their threshold-independent property, i.e., the classifier is configured with \(|\mathcal{C}| + 1\) output channels, one of which is dedicated to the probability of unknown classes. However, the H-score of them is still below average, further highlighting that the hard inference derived from source domains is not suitable for the unseen target domains. It can be observed that ARPL [56], which is one of the state-of-the-art approaches for open set recognition, fails to perform well compared to the standard DG methods. This may indicate that the deep learning models can exhibit a natural inclination to recognize unknown classes, so the close set classification under distribution shift remains crucial in open set domain generalization.

5.6 Ablation Study↩︎

Varying the number of steps. As shown in 1, we investigate the influence of tasks per step on the performance of model. We observe that both accuracy and OSCR initially increase, but then decline as the number of steps rises. This indicates that when the step count is relatively smaller, the notable expansion of gradient-matched tasks contributes to a rapid improvement. However, in the later stages, the differences between classes across steps lead to high similarity in the normalized features of different classes, making it challenging for the model to distinguish between them. Note that conventional meta-learning-based domain generalization methods do not experience this phenomenon because tasks from different domains are normalized separately, which actually aids in the extraction of domain-invariant features. From the second to the fourth rows are all variants of MEDIC++, which is based on the core idea that the step matters, but not too many. The similar results actually reflect the robustness of our method. In contrast, existing methods typically fall into one of two extremes: either using a single step or restricting each step to a single task. These approaches either completely lack gradient matching or are constrained by batch normalization and cost. As illustrated in the first and last rows of the table, both of these strategies significantly underperform MEDIC++.

The effect of different domain generalization paradigms. We compare our method with the baseline ERM [79], as well as meta-learning paradigms MLDG [3] and Fish [16]. Both MLDG and Fish share the same concept of simulating virtual target domains, but ignoring the relationship among classes. When using the same loss function and model, the strategy becomes the only variable between our method and others. As shown in ¿tbl:tab:32cls-cmp?, both MEDIC and MEDIC++ outperform above methods no matter which option is uniformly appointed, demonstrating the critical role played by dualistic gradient matching in open set recognition. Moreover, after transitioning from the cross-entropy loss (i.e., training with close set classifier only) to open set loss function (i.e., training with the two classifiers), our method ushers in the largest performance gain on the average of OSCR by \(2.9\%\) and \(2.8\%\) for MEDIC and MEDIC++ respectively, indicating that our strategy has better compatibility with the multi-binary classifier to learn a more generalizable boundary for each known class.

Varying the proportion of known to unknown. We conduct experiments on the Office-Home [81] dataset using the multi-binary classifier across all strategies. The OSCR results are visually presented in 9. It is evident that increasing the number of known classes introduces greater challenges to the classification task with lower accuracy. Notably, MEDIC++ achieves optimal performance across most split rates, which highlights the robustness of our method in diverse scenarios. We further construct class-wise meta-learning variants for each method. Interestingly, they consistently outperforms traditional domain-wise meta-learning strategies. In some cases, such as when the number of known classes is 30, it even surpasses MEDIC++. This thereby emphasizes the importance of finding optimal balance between classes, making it a crucial consideration in the design of meta-learning strategies.

On the effect of different meta-learning paradigms. Ablation studies are also conducted to evaluate different optimization, task partitioning, and sampling techniques as detailed in ¿tbl:tab:32strategy-cmp?. The baseline is to split source data without considering domains or classes, gradually extending to domain-wise and class-wise partitions with added sampling strategies. We find that both task partition methods benefit most optimization strategies except for MAML [60]. This may be due to its dependence on only the final gradient during the inner loop, which is more random and thus requires more careful optimization. The similarity between [77] and MEDIC++ is that both select confusable class pairs, while we further assign them to different steps rather than training together. Compared to others, our strategy (i.e., the last line), which separates easily confusable class pairs across different steps, achieves the best open set performance. This suggests that enhancing gradient matching between these pairs improves the generalizability of unbiased decision boundaries.

5.7 Analysis & Discussion↩︎

Time and Memory Costs. The primary cost across different steps is a trade-off between time and memory. Model training mainly involves four operations: forward, backward, step, and zero_grad. Most computation occurs in forward and backward, whereas step and zero_grad are lightweight. In MEDIC++, since the total data processed in the forward and backward passes is equivalent to that of ERM, the overall computational cost remains comparable. 10 illustrates how the training time and memory utilization change with the number of steps, measured over 5000 iterations on a single Nvidia RTX 2080Ti GPU with a per-task batch size of 8. In practice, training time increases as the number of steps grows, because smaller batch size can reduce GPU utilization and incur higher kernel launch overhead. However, since each batch’s computational graph is released after backward pass, peak memory usage is reduced, leading to improved memory efficiency.

Verify unbiased decision boundaries. We adopt confidence score to reflect the model’s decision tendency. As illustrated in ¿tbl:tab:conf-cmp?, conf\(_{\rm p}\) and conf\(_{\rm n}\) denote the average activation of the multi-binary classifier on the positive and negative output channels. It can be observed that ERM [79], MLDG [3], and Fish [16] exhibit reduced confidence for positive samples than for negative samples, which explains the pattern in left panel of 1. In contrast, MEDIC and MEDIC++ produce comparable activation magnitudes for both classes, corresponding to the right panel. These results suggest that inter-class gradient matching effectively promotes unbiased predictions. We also provide t-SNE feature results in 11. It can be seen that the unknown classes are generally clustered around the centralized region. For both MEDIC and MEDIC++, the overlap between known and unknown classes seems smaller, with more space allocated for potential unknown classes. Compared to MEDIC, MEDIC++ has clearer boundaries across known classes, which helps to explain its superior close set performance.

Discuss unbiased decision boundaries. We then discuss why a classifier that exhibits balanced output may improve open set behavior, as it induces maximal uncertainty in regions between known classes under mild linear assumptions. To simplify the problem, consider binary classification with two classes \(\mathcal{C}_0\) and \(\mathcal{C}_1\) and model \(f: \mathbb{R}^d \to [0,1]\), which assigns label 0 to \(\mathcal{C}_0\) and label 1 to \(\mathcal{C}_1\). Maximum predictive uncertainty occurs when \(f(x)=0.5\). We define balanced output as the sum of average predictions for both classes equals 1: \[\frac{1}{|\mathcal{C}_0|} \sum_{x^{c_0}\in \mathcal{C}_0} f(x^{c_0}) + \frac{1}{|\mathcal{C}_1|} \sum_{x^{c_1}\in \mathcal{C}_1} f(x^{c_1}) = 1. \label{eq:1}\tag{33}\] Consider two correctly classified samples \(x^{c_0}_i \in \mathcal{C}_0\) and \(x^{c_1}_j \in \mathcal{C}_1\). We examine the midpoint as: \[x_{ij} = \frac{x^{c_0}_i + x^{c_1}_j}{2}. \label{eq:2}\tag{34}\] Such points lie between known classes and are thus plausible candidates for unknown class regions. We define the average midpoint prediction as: \[\lambda(\mathcal{C}_0, \mathcal{C}_1, f) = \frac{1}{|\mathcal{C}_0||\mathcal{C}_1|} \sum_{x^{c_0} \in \mathcal{C}_0} \sum_{x^{c_1} \in \mathcal{C}_1} f\!\left( \frac{x^{c_0}+x^{c_1}}{2} \right). \label{eq:3}\tag{35}\] Smaller \(|\lambda - 0.5|\) indicates higher average uncertainty in inter-class regions. Assume that \(f\) is locally linear such that, along the segment connecting two samples, it admits that: \[f\!\left( \frac{x^{c_0}+x^{c_1}}{2} \right) \approx \frac{1}{2} \left( f(x^{c_0}) + f(x^{c_1}) \right). \label{eq:4}\tag{36}\] Under this approximation, \[\begin{align} \lambda & \approx \frac{1}{2|\mathcal{C}_0||\mathcal{C}_1|}\sum_{x^{c_0} \in \mathcal{C}_0} \sum_{x^{c_1} \in \mathcal{C}_1}\left( f(x^{c_0}) + f(x^{c_1}) \right) \\ &= \frac{1}{2} \left( \frac{1}{|\mathcal{C}_0|} \sum_{x^{c_0} \in \mathcal{C}_0} f(x^{c_0}) + \frac{1}{|\mathcal{C}_1|} \sum_{x^{c_1} \in \mathcal{C}_1} f(x^{c_1}) \right). \end{align} \label{eq:5}\tag{37}\] Therefore, \(|\lambda - 0.5|\) is minimized when 33 holds. This suggests that balanced output across known classes serves as a sufficient condition for inducing maximal uncertainty in inter-class regions under local linear assumption. We emphasize that this argument provides intuition rather than a strict guarantee, as the output of deep networks is highly likely nonlinear.

Sharing parameters between classifiers. Each known class is linked to a unique one-vs-all classifier and a single output channel in the close set classifier. The function of this channel is similar to that of the positive output channel in the corresponding binary classifier, which is activated by samples from the same class. This raises the question of whether parameter sharing between these channels is feasible. As demonstrated in ¿tbl:tab:32param-share?, sharing parameters yields performance comparable to the original architecture, while reducing the total number of output channels from \(3|\mathcal{C}|\) to \(2|\mathcal{C}|\).

Partial domain generalization. We consider the presence of domain-specific classes, where splitting domain and class over the entire set is not recommended, as this may create invalid pairs. In 6, we thus first split by domain in the third line, and then partition within each domain-specific class set in the fifth line. As illustrated in 2, each source domain consists of four known classes, with any two domains sharing two classes and differing in the other two (i.e., \(\{1,2,3,4\}\) for domain \(1\), \(\{1,2,5,6\}\) for domain \(2\), \(\{3,4,5,6\}\) for domain \(3\)). MEDIC++ continues to achieve strong performance, indicating the potential of this finer-grained task-level balance approach for other problem settings.

Single domain generalization. In this scenario, the model is trained on a single domain and evaluated on all other domains. Our method then reduces to pure class-wise gradient matching, while MLDG and Fish degenerate to standard ERM. We thus compare with CrossMatch [63], which also uses multi-binary classifiers for open set single domain generalization. As shown in 3, MEDIC++ still performs well in this setting. Since no inter-domain tasks is involved, the gain comes directly from class-wise gradient matching.

6 Conclusion↩︎

In this paper, we introduce the problem setting of open set domain generalization, which aims to tackle both challenges of domain shift and category shift in the unseen target domain. We propose a simple yet powerful meta-learning-based framework, which incorporates domain-wise and class-wise gradient matching simultaneously, accompanied by a multi-binary classifier to learn a balanced decision boundary for each known class. We conduct experiments on multiple benchmarks to demonstrate the superior performance of our approach in both close set and open set scenarios.

References↩︎