0.0.1 Introduction↩︎

This paper focuses on the unsupervised domain adaptation (UDA) task, where the learner is confronted with a source domain and a target domain and the algorithm is allowed to access to a labeled training sample from the source domain and an unlabeled training sample from the target domain. The goal is to find a predictor that performs well on the target domain.

A main obstacle in such a task is the discrepancy between the two domains. Some recent works [1]–[9] have proposed various measures to quantify such discrepancy, either for the UDA setting or for the more general domain generalization tasks, and many learning algorithms are proposed. For example, [9] uses a (reverse) KL divergence to measure the misalignment of the two domain distributions, and motivated by their generalization bound, they design an algorithm that penalizes the KL divergence between the marginal distributions of two domains in the representation space. Despite that this “KL guided domain adaptation” algorithm is demonstrated to outperform many existing marginal alignment algorithms [6], [10]–[12], it is not clear whether KL-based alignment of marginal distributions is adequate for UDA, and more fundamentally, what role the unlabelled target-domain sample should play in cross-domain generalization. Notably, most UDA algorithms are heuristically designed and intuitively justified. Moreover, most existing generalization bounds are algorithm-independent. Then there appears significant room for both deeper theoretical understanding and more principled algorithm design.

In this paper, we analyze the generalization ability of hypotheses and learning algorithms for UDA tasks using an information-theoretic framework developed in [13], [14]. The foundation of our technique is the Donsker-Varadhan representation of KL divergence (see Lemma 1). We present novel upper bounds for two notions of generalization errors. The first notion (“population-to-population (PP) generalization error”) measures the gap between the population risk in the target domain and that in the source domain for a hypothesis, and the second (“expected empirical-to-population (EP) generalization error”) measures the gap between the population risk in the target domain and the empirical risk in the source domain for a learning algorithm. We show that the PP generalization error for all hypotheses are uniformly bounded by a quantity governed by the KL divergence between the two domain distributions, which, under bounded losses, recovers the the bound in [9]. We then show that this KL term upper-bounds some other measures including Total-Variation distance [1], Wasserstein distance [6] and domain disagreement [7]. Thus, minimizing KL-divergence forces the minimization of other discrepancy measures as well. This, together with the ease of minimizing KL [9], explains the effectiveness of the KL-guided alignment approach. For expected EP generalization error, we develop several algorithm-dependent generalization bounds. These algorithm-dependent bounds further inspire the design of two new and yet simple strategies that can further boost the performance of the KL guided marginal alignment algorithms. Experiments are performed to verify the effectiveness of these strategies.

0.0.2 Related Work↩︎

0.0.2.0.0.1 Domain Adaptation

Many domain adaptation generalization bounds have been developed [1]–[3], [5]–[8], [15], and various discrepancy measures are introduced to derive these bounds including total variation [1]–[3], [15], Wasserstein distance [6], domain disagreement [7] and so on. In particular, bounds based on \({\cal H}\Delta {\cal H}\) in [2] are restricted to a binary classification setting and assume a deterministic labeling function. Furthermore, [2] also assumes the loss is the \(L_1\) distance between the predicted label and true label (which is bounded). Our bounds work for the general supervised learning problems with any labelling mechanism (e.g., stochastic labelling), and we do not require the specific choice of the loss (even unbounded). Recently, [16] proposed generalization bounds using Jensen-Shannon (JS) divergence, which bear a relation to our Corollary 2. While other algorithm-dependent bounds have been proposed for different transfer learning settings (e.g., [17]), they are not directly comparable to our own bounds. For more details about the domain adaptation theory, we refer readers to [18] for a comprehensive survey. In addition, the most common methods for domain adaptation involve aligning the marginal distributions of the representations between the source and target domains, for example, using an adversarial training mechanism [6], [8], [10] or aligning the first two moments of the representation distribution [11]. There are numerous other domain adaptation algorithms, and we refer readers to [19]–[21] for recent advances.

0.0.2.0.0.2 Information-Theoretic Generalization Bounds

Information-theoretic analysis is usually used to bound the expected generalization error of supervised learning, where the training and testing data come from the same distribution [13], [14], [22]–[26]. Exploiting the chain rule of mutual information, these bounds are successfully applied to characterize the generalization ability of stochastic gradient based optimization algorithms [24], [27]–[32]. Recently, this framework has also been used in other learning settings including meta-learning [33]–[36], semi-supervised learning [37], [38] and transfer learning [33], [39]–[42]. In particular, [39], [40] consider a different problem setup with ours. Specifically, their expected generalization error is the gap between the target population risk and a weighted empirical risk combining both the source and the target empirical risks, while our “EP” error is the gap between the target population risk and the source empirical risk. That is, we focus on the role of the unlabelled target data in cross-domain generalization when the source empirical risk is taken as a training objective, whereas their works assume the existence of labelled target data and study their role in domain adaptation.

0.0.3 Preliminary↩︎

Unless otherwise noted, a random variable will be denoted by a capitalized letter, and its realization is denoted by the corresponding lower-case letter. Consider a prediction task with instance space \(\mathcal{Z}=\mathcal{X}\times\mathcal{Y}\), where \(\mathcal{X}\) and \(\mathcal{Y}\) are the input space and the label (or output) space, respectively. Let \(\mathcal{F}\) be the hypothesis space of interest, in which each \(f\in \mathcal{F}\) is a function or predictor mapping \(\mathcal{X}\) to \({\cal Y}\). We assume that each hypothesis \(f\in \mathcal{F}\) is parameterized by some weight parameter \(w\) in some space \({\cal W}\) and may write \(f\) as \(f_w\) as needed.

Let \(\mu\) and \(\mu'\) be two distributions on \(\mathcal{Z}\), unknown to the learner, where \(\mu\) characterizes the source domain and \(\mu'\) characterizes the target domain. We may also write \(\mu\) as \(P_Z\) or \(P_{XY}\) and \(\mu'\) as \(P_{Z'}\) or \(P_{X'Y'}\), which defines random variables \(Z=(X, Y)\) and \(Z'=(X', Y')\), respectively. Let \(S=\{Z_i\}_{i=1}^n \sim \mu^{\otimes n}\) be a labeled source-domain sample and \(S'_{X'} = \{X'_j\}_{j=1}^m \sim P_{X'}^{\otimes m}\) be an unlabelled target-domain sample. The objective of UDA is to design an algorithm \(\mathcal{A}\) that takes \(S\) and \(S'_{X'}\) as the input and outputs a weight \(W\in\mathcal{W}\), giving rise to a predictor \(f_W\in \mathcal{F}\) that “works well” on the target domain. Note that the algorithm \(\mathcal{A}\) is characterized by a conditional distribution \(P_{W|S,S'_{X'}}\).

Let \(\ell:\mathcal{Y}\times \mathcal{Y} \rightarrow \mathbb{R}_0^+\) be a loss function. The population risk for each \(w\in \mathcal{W}\) in the target domain is defined as \[R_{\mu'}(w)\triangleq \mathbb{E}_{Z'}{[\ell(f_w(X'),Y')]}\] and a good UDA algorithm hopes to return a weight \(w\) that minimizes this risk. Since \(\mu'\) is unknown, one often uses recourse to the empirical risk in the source domain, defined as \[R_S(w) \triangleq {\frac{1}{n}\sum_{i=1}^n\ell(f_w(X_i),Y_i)}.\] Generalization error in this setting measures how well the hypothesis returned from the algorithm generalizes from the source-domain training sample to the target-domain unknown distribution \(\mu'\). Taking into account the stochastic nature of the algorithm \(\mathcal{A}\), a natural notion of generalization error for UDA can be defined by \[\begin{align} \label{eq:expect-error} {\mathrm{Err}}\triangleq {\ifx&W,S& \mathbb{E} \else {\mathbb{E}_{W,S}} \fi \left[R_{\mu'}(W)-R_S(W)\right]}={\ifx&W,S,S'_{X'}& \mathbb{E} \else {\mathbb{E}_{W,S,S'_{X'}}} \fi \left[R_{\mu'}(W)-R_S(W)\right]}, \end{align}\tag{1}\] where the expectation in the first expression is taken over the joint distribution of \((W,S)\sim P_{W|S}\times\mu^{\otimes n}\), and the expectation of the second expression is taken over the joint distribution of \((W,S,S'_{X'})\sim P_{W|S,S'_{X'}}\times\mu^{\otimes n}\times P_{X'}^{\otimes m}\).

There is another notion of generalization error, more traditional in the domain adaptation literature, defined as the gap between the population risk in the target domain and that in the source domain: \[\begin{align} \widetilde{\mathrm{Err}}(w)\triangleq R_{\mu'}(w)-R_{\mu}(w). \label{eq:unexpect-error} \end{align}\tag{2}\] where \(R_{\mu}(w)\triangleq \mathbb{E}_Z{[\ell(f_w(X),Y)]}\). It is apparent that \(\widetilde{\mathrm{Err}}(w)\) and \(\mathrm{Err}\) are related by the following triangle inequality: \[|R_{\mu'}(w) - R_S(w)|\le |R_{\mu'}(w) - R_{\mu}(w)| + |R_{\mu}(w)-R_{S}(w)|.\] where the second term on the right hand side is the standard generalization error in the source domain, which can be bounded by classical learning-theoretic tools, e.g., Rademacher complexity [43]. Thus, bounding \(\widetilde{\mathrm{Err}}(w)\) helps bounding \(\mathrm{Err}\).

This paper studies both notions of generalization error for UDA. Specifically, starting from Section 0.0.5, we will mainly use information-theoretic tools to bound \(\mathrm{Err}\) directly, without going through \(\widetilde{\mathrm{Err}}(w)\). For the ease of reference, we refer to \(\widetilde{\mathrm{Err}}(w)\) as the population-to-population (PP) generalization error for \(w\) and \(\mathrm{Err}\) as the expected empirical-to-population (EP) generalization error.

The following definitions are useful.

Note that the conditional mutual information \(I(X;Y|Z)=\mathbb{E}_{Z}{I^Z(X;Y)}\).

0.0.4 Upper Bounds for PP Generalization Error↩︎

We now present some upper bounds for \(\widetilde{\mathrm{Err}}(w)\). The key techniques used in developing these bounds are the information-theoretic tools in the style of Lemma 1. These bounds adopt certain KL divergence to measure the discrepancy between the source and target domains. Notably, some previously established bounds are recovered under weaker conditions. Additionally, we demonstrate that under certain conditions, the KL-based bound is an upper bound of several other discrepancy measures and hence minimizing the KL divergence forces the minimization of these other measures.

We first list some common assumptions on the loss function, which we consider in this paper.

0.0.4.1 Generalization Bounds via the Subgaussian Condition

The following generalization bound is established by combining Lemma 1 and Assumption 2, and its corresponding sample complexity bound is discussed in Appendix 1.0.2.8.

Notably this result can be turned into a generalization upper bound providing guidance to algorithm design, and at the same time it provides a lower bound of the generalization error, highlighting some fundamental difficulty of the learning task. To illustrate this, we present a corollary while noting that similar development can also be applied to other bounds presented later in this paper.

Consider that each \(f_w\) is expressed as the composition \(g\circ h\), where \(h\) is a function mapping \({\cal X}\) to a representation space \({\cal T}\) and \(g\) is a function mapping \({\cal T}\) to \({\cal Y}\). For any given \(h:{\cal X} \rightarrow {\cal T}\), denote by \(\mu_{h}\) the distribution on \({\cal T}\times {\cal Y}\) obtained by pushing forward \(\mu\) via \(h\), that is, \(\mu_{h} (t, y)= \int \delta(t-h(x)) d\mu(x, y)\), where \(\delta\) is the Dirac measure on \({\cal T}\). Similarly, let \(\mu'_{h}\) denote the distribution on \({\cal T}\times {\cal Y}\) obtained by pushing forward \(\mu'\) via \(h\).

In this result, the lower bound of \(R_{\mu'}(w)\) indicates a fundamental difficulty in UDA learning in that, using the same predictor mapping \(f_w\), there is no way for the population risk in the target domain to be lower than that of the source domain less than a constant which depends only on the domain difference. On the other hand, the upper bound suggests that it is possible to squeeze the gap between the two population risks by choosing an appropriate representation map \(h\) - evidently such a map should be attempting to align \(\mu'_h\) with \(\mu_h\) or to align their respective proxies.

It is also noteworthy that under Assumption 1 and due to Remark 1, Theorem 1 implies \[\begin{align} \label{ineq:cor-bound-subgaussian-bounded} \left|\widetilde{\mathrm{Err}}(w)\right| \leq {\frac{M}{\sqrt{2}}}\sqrt{\mathrm{D_{KL}}(P_{X'}||P_{X})+\mathrm{D_{KL}}(P_{Y'|X'}||P_{Y|X})}. \end{align}\tag{3}\] Similarly applying this result in the representation space \(\mathcal{T}\), we see that Eq. (3 ) recovers the bound in Proposition 1 of [9]. Notice that unlike [9], Theorem 1 ( or Eq. (3 )) does not require the loss to be the cross-entropy loss.

Theorem 1 and [9] both use the KL divergence from source domain to target domain, \(\mathrm{D_{KL}}(\mu'||\mu)\), and in fact, \(\left|\widetilde{\mathrm{Err}}(w)\right|\) can also be upper bounded by \(\mathrm{D_{KL}}(\mu||\mu')\). This can be done by invoking the subgaussianality of \(\ell(f_w(X'),Y')\) (rather than \(\ell(f_w(X),Y)\)); for bounded loss, the subgaussianality of \(\ell(f_w(X'),Y')\) is also satisfied. Then we obtain the following corollary.

In UDA, since \(Y'\) is completely unavailable to the algorithm \(\mathcal{A}\), it is impossible to minimize the misalignment of conditional distributions, i.e. \(\mathrm{D_{KL}}(P_{Y'|T'}||P_{Y|T})\) where \(T\) and \(T'\) are representations of source domain and target domain, respectively. A common method is to assign pseudo labels to target data based on a learned source classifier [47]. However, it may also cause additional issues [48]. For concreteness, suppose the trained model \(Q\) can well approximate the real mapping between \(X\) and \(Y\) on source domain (i.e. \(Q_{Y|T}=P_{Y|T}\)), which is usually the training objective. Let \(\hat{Y'}\) be the pseudo label of \(T'\) generated by the trained model, i.e., \(Q_{\hat{Y'}|T'}=Q_{Y|T}\). Let \(Q_{T',\hat{Y'}}=P_{T'}Q_{\hat{Y'}|T'}\), then the following holds, \[\begin{align} \mathrm{D_{KL}}(P_{T',Y'}||P_{T,Y})=\mathbb{E}_{P_{T',Y'}}\log\frac{P_{T',Y'}Q_{T',\hat{Y'}}}{Q_{T',\hat{Y'}}P_{T,Y}}=\mathrm{D_{KL}}(P_{T'}||P_{T})+\mathrm{D_{KL}}(P_{Y'|T'}||Q_{\hat{Y'}|T'}). \label{eq:pseudo-label} \end{align}\tag{4}\] For a specific \(t'\), if \(P(Y'=y'|T'=t')\neq 0\) and \(Q(\hat{Y'}=y'|T'=t')= 0\), then the second term in RHS of Eq. (4 ), \(\mathrm{D_{KL}}(P_{Y'|T'}||Q_{\hat{Y'}|T'})\rightarrow\infty\). In this case, even when the marginal distributions are perfectly aligned, the overall value of the upper bound is large. Thus, incorrect pseudo labels may even have negative impact on the target domain performance.

In fact, the misalignment of the conditional distributions appears to be the main difficulty of UDA [1], [8]. The next corollary suggests that this difficulty may be alleviated when the loss function satisfies the triangle property, namely, Assumption 4. It can be verified that this assumption is satisfied by the 0-1 loss ²; this assumption has also been considered in previous works [3], [6].

Here \(\lambda^*\) measures the possibility of whether the domain adaptation algorithm will succeed under the oracle knowledge of \(\mu\) and \(\mu'\). In particular, if the hypothesis space is large enough, the minimizer \(w^*\) for the “joint population risk” \(R_{\mu'}(w)+R_\mu(w)\) may give rise to \(R_{\mu'}(w^*)=R_\mu(w^*)=0\), then we’re likely to generalize well on the target domain. Then the KL divergence \(\mathrm{D_{KL}}(P_{X'}||P_{X})\) between the two \(\mathcal{X}\)-marginals alone bounds the PP generalization error uniformly for all \(w\in \mathcal{W}\).

This theorem motivates the strategy of penalizing \(\mathrm{D_{KL}}(P_{T'}||P_{T})\) in the representation space for UDA. The next theorem suggests that such an approach also penalizes other notions of domain discrepancy, for example, the key quantity in the PAC-Bayes type of domain adaptation generalization bounds [7], that is defined as \[\begin{align} \mathrm{dis}(P_X,P_{X'})\triangleq\left|{\ifx&W,W'& \mathbb{E} \else {\mathbb{E}_{W,W'}} \fi \left[{\ifx&X'& \mathbb{E} \else {\mathbb{E}_{X'}} \fi \left[\ell(f_{W}(X'),f_{W'}(X'))\right]}\right]}-{\ifx&W,W'& \mathbb{E} \else {\mathbb{E}_{W,W'}} \fi \left[{\ifx&X& \mathbb{E} \else {\mathbb{E}_{X}} \fi \left[\ell(f_{W}(X),f_{W'}(X))\right]}\right]}\right|. \label{eq:domain-agree} \end{align}\tag{5}\]

Note that unlike [7], here we do not require the loss function to be the 0-1 loss.

0.0.4.2 Generalization Bounds via the Lipschitz Condition

We now present such generalization bound for UDA under the Lipschitz continuity assumption of the loss function, where \({\mathbb{W}}(\cdot, \cdot)\) denotes the Wasserstein distance.

Theorem 4 can be related to the KL-based bounds in the previous section when the Wasserstein distance is defined with respect to the discrete metric \(d\). In this case and under bounded loss function, which is also Liptschitz continuous, Theorem 4 follows. On the other hand, Wasserstein distance is also equivalent to the total variation in this case, while the latter is connected to the KL divergence via Pinsker’s inequality [49] and the Bretagnolle-Huber inequality [50]. Thus, we arrive at the following result.

Note that results here are inspired by the work of [26]. Corollary 3 provides a tighter bound than the one in Eq. (3 ), as can be directly verified.

Parallel to Theorem 2, if the loss function satisfies the triangle property, we may establish the bound below, which recovers a similar result in [6] but without restricting the task to be binary classification or requiring the loss to be the \(L_1\) distance.

These results justify the strategy of minimizing domain discrepancy in the representation space. Since the KL-based bounds upper-bound those based on other measures of domain differences, penalizing the KL divergence will also penalize those other measures. This is practically advantageous since it is usually easier and more stable to minimize the KL divergence [9].

0.0.5 Upper Bounds for EP Generalization Error and Applications↩︎

There are two limitations in the bounds on the PP generalization error developed so far and in the traditional analysis of UDA. First, such bounds are independent of \(w\) and hence algorithm-independent. Second, although these bounds may inspire strategies to exploit the unlabelled target sample, e.g., aligning the source and target distributions in the representation space, they only provide very limited knowledge on the role that the unlabelled target sample plays. Inspired by the works of [24] and [51], we derive upper bounds for the EP generalization error that take better advantage of the dependence of the algorithm’s output on the unlabelled target data. Applications of these bounds in designing the learning algorithms are also presented.

0.0.5.1 EP Generalization Bounds

The theorem can be turned into a version that is more practically relevant, in which the KL term is replaced with their representation-space counter-part (following a similar argument used for deriving Corollary 1). In addition, note that although larger sample sizes allow better estimation of that KL term, utilizing pseudo-labels for estimation may have a negative impact (as discussed in Section 0.0.4), which can be amplified by the larger sample size.

This bound is tighter than the bound in Theorem 6, as can be indicated by the following corollary.

Notice that to recover Theorem 6 from Corollary 5 (under Assumption 1), we can use Jensen’s inequality to move the expectation over \(Z_i\) to inside the square root function.

0.0.5.2 Gradient Penalty as an Universal Regularizer

The algorithm-dependent bound in Theorem 6 tells us that one can reduce the EP error by limiting the disintegrated mutual information \(I^{X_j'}(W;Z_i)\). In the stochastic gradient based optimization algorithms, this term can be controlled by penalizing the gradient norm. To see this, we now consider a “noisy” iterative algorithm for updating \(W\), e.g., SGLD. At each time step \(t\), let the labelled mini-batch from the source domain be \(Z_{B_t}\), let the unlabelled mini-batch from the target domain be \(X'_{B_t}\), and let \(g(W_{t-1},Z_{B_t},X'_{B_t})\) be the gradient at time \(t\). Thus, the updating rule of \(W\) is \(W_t=W_{t-1}-\eta_t g(W_{t-1},Z_{B_t},X'_{B_t})+N_t\) where \(\eta_t\) is the learning rate and \(N_t\sim\mathcal{N}(0,\sigma^2\mathrm{I}_d)\) is an isotropic Gaussian noise. Inspired by [27], we have the following bound.

Theorem 8 hints that to reduce the generalization error, one can simply restrict the gradient norm at each step (so that \(||G_t-{\ifx&Z_{B_t}& \mathbb{E} \else {\mathbb{E}_{Z_{B_t}}} \fi \left[G_t\right]}||^2\) is reduced). This strategy will also restrict the distance between the final output \(W_T\) and the initialization \(W_0\), effectively shrinking the hypothesis space accessible by the algorithm. We also note that the importance of gradient penalty has been theoretically justified in the supervised learning setting [24], [28], [30]–[32], [51], [52].

Indeed, adding gradient penalty can be applied to any existing UDA algorithm and it is simple but effective in practice. Later on we will show that even when the algorithm \(\mathcal{A}\) does not access to any target data, in which case \(I(W;Z_i|X_j')\) reduces to \(I(W;Z_i)\) and \(g(W_{t-1},Z_{B_t},X_{B_t}')\) becomes \(g(W_{t-1},Z_{B_t})\), minimizing the empirical loss of source domain sample while penalizing gradient norm will still improve the performance. Notice that gradient penalty has been used in standard supervised learning as a regularization technique [53], [54]. It is also used in Wasserstein distance based adversarial adaptation [6], [55], and their motivation is to stabilize the training to avoid gradient vanishing problem. Here we suggest, with strong theoretical justification, that gradient penalty is a universal technique for improving the generalization performance in UDA for any gradient-based learning method.

Notably the bound in Theorem 8 only depends on the size \(n\) of labelled source sample and does not explicitly depend on \(m\), the size of unlabelled target sample. With a more careful design, if we consider the mutual information as the expected KL divergence of a posterior and a prior, based on \(I^{X_j'}(W;Z_i)\) in Theorem 6, it is possible to create a target-data-dependent prior and derive a tighter bound based on some quantity similar to "gradient incoherence" in [24].

0.0.5.3 Controlling Label Information for KL Guided Marginal Alignment

Consider instances in the representation space, \(Z=(T,Y)\) and \(Z'=(T',Y)\). Theorem 6 also encourages us to align the distributions of two domains in the representation space, as argued earlier. Then the KL guided marginal alignment algorithm proposed in [9] can be invoked here. One may notice that Theorem 6 uses \(\mathrm{D_{KL}}(\mu||\mu')\) while [9] uses \(\mathrm{D_{KL}}(\mu'||\mu)\). As already discussed in Section 0.0.4, this inconsistency can be ignored when the loss is bounded (see Corollary 4).

Most domain adaptation algorithms aim to align the marginal distributions of two domains in the representation space. However, without accessing to \(Y'\), it remains unknown if an UDA algorithm will work well since we cannot guarantee that discrepancy between conditional distribution \(P_{Y|T}\) and \(P_{Y'|T'}\) won’t become too large when we align the marginals. In [9], the authors show that \(\mathrm{D_{KL}}(P_{Y'|T'}||P_{Y|T})\) can be upper-bounded by \(\mathrm{D_{KL}}(P_{Y'|X'}||P_{Y|X})\), if \(I(X;Y)=I(T;Y)\). The authors then argue that penalizing the KL divergence of the marginals is safe.

We now argue that in practice the condition \(I(X;Y)=I(T;Y)\) can be difficult to satisfy if the cross-entropy loss is used to define the source-domain empirical risk.

By data processing inequality on \(Y-X-T\), we know that \(I(X;Y)\geq I(T;Y)=H(Y)-H(Y|T)\). Thus, to let \(I(T;Y)\) reach its maximum, one must minimize \(H(Y|T)\). On the other hand, let \(Q_{Y|T,W}\) be the predictive distribution of labels in the source domain generated by the classifier. The expected cross-entropy loss for each \(Z_i\) in the representation space is then \[{\ifx&W,Z_i& \mathbb{E} \else {\mathbb{E}_{W,Z_i}} \fi \left[\ell(f_W(T_i),Y_i)\right]} = {\ifx&Z_i& \mathbb{E} \else {\mathbb{E}_{Z_i}} \fi \left[{\ifx&W|Z_i& \mathbb{E} \else {\mathbb{E}_{W|Z_i}} \fi \left[-\log{Q_{Y_i|T_i,W}}\right]}\right]},\] which also decomposes as [56], [57] \[\begin{align} \label{eq:cross-entropy32decomposition} {\ifx&W,Z_i& \mathbb{E} \else {\mathbb{E}_{W,Z_i}} \fi \left[\ell(f_W(T_i),Y_i)\right]} = H(Y_i|T_i)+{\ifx&T_i,W& \mathbb{E} \else {\mathbb{E}_{T_i,W}} \fi \left[\mathrm{D_{KL}}(P_{Y_i|T_i,W}||Q_{Y_i|T_i,W})\right]}-I(W;Y_i|T_i). \end{align}\tag{6}\] Then minimizing the expected cross-entropy loss may not adequately reduce \(H(Y_i|T_i)\) but rather cause \(I(W;Y_i|T_i)\) to significantly increase, particularly when the model capacity is large. This may have two negative effects. First, the condition \(I(X;Y)=I(T;Y)\) is significantly violated, and \(\mathrm{D_{KL}}(P_{Y'|T'}||P_{Y|T})\) is no longer upper bounded by \(\mathrm{D_{KL}}(P_{Y'|X'}||P_{Y|X})\). Hence, aligning the two marginals alone may not be adequate. Second, large \(I(W;Y_i|T_i)\) indicates \(W\) just simply memorizes the label \(Y_i\), resulting a form of overfitting and hurting the generalization performance.

The key take-away from the above analysis is that when aligning the marginals in UDA, controlling the source label information in the weights can be important to achieve good cross-domain generalization. A similar message can also be deduced from Theorem 6, when it is viewed in the representation space and noting \(I^{T_j'}(W;Z_i)=I^{T_j'}(W;T_i)+I^{T_j'}(W;Y_i|T_i)\).

To control label information, [57] proposed an approach called LIMIT. However, this method is rather complicated and arguably hard to train in domain adaptation (see Appendix 1.0.3.8). We now derive a simple alternative strategy for this purpose.

Notice that \(I^{T_j'}(W;Y_i|T_i) \leq\inf_Q{\ifx&T_i& \mathbb{E} \else {\mathbb{E}_{T_i}} \fi \left[\mathrm{D_{KL}}(P_{W|Y_i,T_i,T_j'=t_j'}||Q_{W|T_i,T_j'=t_j'})\right]},\) which is a simple extension of variational representation of mutual information [49]. Here \(Q\) could be any distribution. By assuming \(P=\mathcal{N}(W,\sigma^2\mathrm{I}_d|Y_i,T_i,T_j'=t_j')\) and taking \(Q=\mathcal{N}(\widetilde{W},\tilde{\sigma}^2\mathrm{I}_d|T_i,T_j'=t_j')\), we have \[I^{T_j'}(W;Y_i|T_i)\leq\inf_Q{\ifx&T_i& \mathbb{E} \else {\mathbb{E}_{T_i}} \fi \left[\mathrm{D_{KL}}(P_{W|Y_i,T_i,T_j'=t_j'}||Q_{\tilde{W}|T_i,T_j'=t_j'})\right]}\propto ||W-\widetilde{W}||^2.\] Thus, we may create an auxiliary classifier \(f_{\widetilde{w}}\) that is not allowed to access to the real source label \(Y\). In each iteration, we use the pseudo labels of target data (and source data) assigned by \(f_{{w}}\) to train \(f_{\widetilde{w}}\) and adding \(||W-\widetilde{W}||^2\) as a regularizer in the training of \(W\). The algorithm is given in the Appendix. Remarkably the regularizer here resembles “Projection Norm” designed in [58] for out-of-distribution generalization.

0.0.6 Experimental Results↩︎

We perform experiments to verify the proposed techniques inspired by our theory.

0.0.6.0.0.1 Datasets

We select two popular small datasets, RotatedMNIST and Digits, to compare the different methods. RotatedMNIST is built based on the MNIST dataset [59] and consists of six domains, each containing \(11,666\) images. These six domains are rotated MNIST images with rotation angle \(0^{\circ},15^{\circ},30^{\circ},45^{\circ},60^{\circ}\) and \(75^{\circ}\), respectively. We will take the original MNIST dataset (\(0^{\circ}\)) as the source domain and take other five domains as target domains. Hence, there are five domain adaptation tasks on RotatedMNIST. Digits consists of three sub-datasets, namely MNIST, USPS [60] and SVHN [61], and the corresponding domain adaptation tasks are MNIST\(\rightarrow\)USPS (M\(\rightarrow\)U), USPS\(\rightarrow\)MNIST (U\(\rightarrow\)M), SVHN\(\rightarrow\)MNIST (S\(\rightarrow\)M).

0.0.6.0.0.2 Compared Methods

Baseline methods are some popular marginal alignment UDA methods including DANN [10], MMD [12], CORAL [11], WD [6] and KL [9]. We also choose ERM as another baseline, in which only the source-domain sample is accessible during training. To verify the strategies inspired by our theory, we first add the gradient penalty to the ERM algorithm (ERM-GP), and we then combine gradient penalty (GP) and controlling label information (CL) with the recent proposed KL guided marginal alignment method, which are denoted by KL-GP and KL-CL, respectively.

0.0.6.0.0.3 Implementation Details

Most of our implementation is based on the DomainBed suite [62]. Other settings exactly follow [9] and the results of baseline methods are taken from [9]. Specifically, each algorithm is run three times and we show the average performance with the error bar. Every dataset has a validation set, and the model selection scheme is based on the best performance achieved on the validation set of target domain during training (oracle). The hype-parameter searching process is also built upon the implementation in the DomainBed suite. Other details and additional experiments can be found in Appendix.

0.0.6.0.0.4 Results

From Table 1, we first notice that gradient penalty allows ERM to perform more comparably to other marginal alignment methods. For example, on RotatedMNIST, ERM-GP outperforms CORAL and performs nearly the same with DANN. On Digits, ERM-GP outperforms WD. When GP and CL combined with KL guided algorithm, we can see that the performance can be further boosted. This justifies the discussion in Section 0.0.5.2 and Section 0.0.5.3.

0.0.7 Conclusion↩︎

Despite that the numerous learning techniques have been developed for domain adaptation, significant room exists for more in-depth theoretical understanding and more principled design of learning algorithms. This paper presents the information-theoretic analysis for unsupervised domain adaptation, where we query two notions of the generalization errors in this context and present novel learning bounds. Some of these bounds recover the previous KL-based bounds under different conditions and confirm the insights in the learning algorithms that align the source and target distributions in the representation space. Our other bounds are algorithm-dependent, better exploiting the unlabelled target data, which have inspired novel and yet simple schemes for the design of learning algorithms. We demonstrate the effectiveness of these schemes on standard benchmark datasets.

Acknowledgments

This work is supported partly by an NSERC Discovery grant and a National Research Council of Canada (NRC) Collaborative R&D grant (AI4D-CORE-07). Ziqiao Wang is also supported in part by the NSERC CREATE program through the Interdisciplinary Math and Artificial Intelligence (INTER-MATH-AI) project. The authors would like to thank the anonymous reviewers for their careful reading and valuable suggestions.

1 Appendix↩︎

1.0.1 Some Prerequisite Definitions and Useful Lemmas↩︎

The key quantity in the most information-theoretic generalization bounds is the mutual information between algorithm’s input and output. Specifically, the core technique behind these bounds is the well-known Donsker-Varadhan representation of KL divergence [49].

Similar to [14], we need the following lemma as a main tool.

The following lemma is the Kantorovich–Rubinstein duality of Wasserstein distance [63].

To connect total variation with KL divergence , we will use Pinsker’s inequality [49] and Bretagnolle-Huber inequality [50] in this paper, for more discussion about these two inequalities, we refer readers to [67].

Below is the variational formula (or golden formula) of mutual information.

1.0.2 Omitted Proofs and Additional Results in Section 0.0.4↩︎

1.0.2.1 Proof of Theorem 1

1.0.2.2 Proof of Corollary 2

1.0.2.3 Proof of Theorem 2

1.0.2.4 Proof of Theorem 3

1.0.2.5 Proof of Theorem 4

1.0.2.6 Proof of Corollary 3

1.0.2.7 Proof of Theorem 5

1.0.2.8 Additional Results: Sample Complexity Bounds

One of the main ingredients to derive our sample complexity bound is the following lemma, where a concentration bound for a class of unbounded functions is given.

Below is another useful lemma for the bounded case, which comes from [69] with a slight modification (by invoking a different VC-dimension based generalization bound from [70]).

We are now in a position to state our sample complexity bound.

With Theorem 9 and Theorem 1, we immediately have the following corollary.

1.0.2.9 Additional Discussions on the Convergence of Empirical KL Divergence

Characterizing the convergence of the empirical KL divergence to the real KL is a challenging task that often requires several additional assumptions, as demonstrated in Theorem 9. However, it is worth noting that the convergence rate of the empirical distribution to the real distribution in the KL sense is already established in the discrete space. This fact is supported by a classic result in [71], which we state in the following theorem:

Thus, it suffices to ensure that the empirical KL converge to the real KL with the similar rate, although we do not know if there might exist a faster convergence rate.

1.0.2.10 Generalize to Approximate Triangle Inequality

In Section 0.0.4, some results require that the loss obeys the triangle inequality (i.e. Assumption 4), such as Theorem 2 and Theorem 5. While the \(0-1\) loss satisfies Assumption 4, some other loss may not. Thus, to generalize Theorem 2 and Theorem 5, we invoke an approximate triangle inequality, which is originally defined in [72].

Thus, Theorem 2 can be easily generalized below.

Theorem 5 can be generalized in the similar way. While Theorem 11 strictly speaking is not a generalization bound, as it includes \(R_\mu\) in the bound, it shares the same underlying concept as Theorem 2. Namely, to minimize the population risk in the target domain, it is essential for the source domain and target domain to be similar, and for both \(R_\mu\) and \(\lambda^*\) to be kept small.

1.0.3 Omitted Proofs and Additional Discussions in Section 0.0.5↩︎

1.0.3.1 Additional Discussion on Theorem 6

To derive the bound in Theorem 6, we need to make use of the second equality in Eq. (1 ). In fact, by the definition of \(\mathrm{Err}\) (the first equality in Eq. (1 )), the unlabelled sample \(S'_{X_j'}\) does not explicitly appear, so one can easily apply the similar information-theoretic analysis starting from the first equality in Eq. (1 ), and obtain an upper bound that consists of \(I(W;Z_i)\) and \(\mathrm{D_{KL}}(\mu||\mu')\). Precisely, the following bound holds,

The proof of Theorem 12 is nearly the same to the proof of [39] and [41] Corollary 1.

It’s important to note that although \[I(W;Z_i)\leq I(W;Z_i|X_j')={\ifx&X_j'& \mathbb{E} \else {\mathbb{E}_{X_j'}} \fi \left[I^{X_j'}(W;Z_i)\right]},\] the bound in Theorem 6 is incomparable to the bound based on \(I(W;Z_i)\). This is mainly due to the fact that we use the disintegrated version of mutual information, \(I^{X_j'}(W;Z_i)\), and the expectation over \(X_j'\) is outside of the square root, which is a convex function. Using \(I^{X_j'}(W;Z_i)\) instead of \(I(W;Z_i)\) allows us to figure out more details about the role of unlabelled target data in the algorithm. Additionally, one can also prove a bound based on \(I(W;Z_i|X_j')\) (e.g., simply applying Jensen’s inequality to Theorem 6), which is close to an individual and UDA version of [42].

Furthermore, the first term in Theorem 6 characterize the expected generalization gap on the source domain (i.e. \({\ifx&W,S& \mathbb{E} \else {\mathbb{E}_{W,S}} \fi \left[R_\mu(W)-R_S(W)\right]}\)), then the bound suggests us that it’s possible to invoke the unlabelled target data to further improve the performance on source domain, and the simplest case is the semi-supervised learning (when \(\mu=\mu'\)).

1.0.3.1.0.1 Compared with [39], [40].

Notably, bounds in [39], [40] fail to characterize the dependence between \(W\) and \(S'_{X'}\). More precisely, the algorithm-dependent term in their bounds is \(I(W;Z_i)\) or \(I(W;S)\), while our algorithm-dependent term is \(I^{X_j'}(W;Z_i)\) that directly depends on the unlabelled target data. Moreover, while the disintegrated mutual information \(I^{X_j'}(W;Z_i)\) and the unconditional mutual information \(I(W;Z_i)\) cannot be directly compared, recent work by [73] provides empirical evidence comparing similar terms in the supervised learning setting. Specifically, they demonstrate that when the empirical risk is small, such as in a realizable case, the disintegrated mutual information is smaller than the unconditional mutual information. Conversely, when the empirical risk is large, the unconditional mutual information is the smaller of the two.

1.0.3.1.0.2 More Discussion on the Vanishing of \(I(X_j';Z_i|W)\) in Remark 4.

Note that \(S\) depends on \(S_{X'}'\) given \(W\), so intuitively the dependence between each individual instance \(Z_i\) and \(X_j'\) is weaker when \(n\) and \(m\) become larger. More precisely, W.L.O.G let \(i=j=1\), and recall that \(W=\mathcal{A}(S_{X'}',S)\), when \(n,m\rightarrow \infty\), taking \(S\) and \(S_{X'}'\) as the input of the algorithm is nearly equivalent to computing \(W\) based on the source distribution \(\mu\) and the target distribution \(P_{X'}\), thus, \(W\) will only depend on the two distributions, without depending on the realizations \(Z_1\) and \(X'_1\) drawn respectively from the two distributions, that is, \(I(Z_1;X_1'|W)=I(Z_1;X_1'|\mathcal{A}(\mu,P_{X'}))=I(Z_1;X_1')=0\). In addition, one may argue that what if \(W=constant\) that does not really depend on the input data. In this case, \(I(Z_1;X_1'|W)=I(Z_1;X_1')=0\) will hold trivially. In the other extreme, if \(n=1\) and \(m=1\), then \(W=\mathcal{A}(X_1',Z_1)\), and the quantity \(I(Z_1;X_1'|\mathcal{A}(X_1',Z_1))\) should be large. When \(n\) and \(m\) increase, it becomes \(I(Z_1;X_1'|\mathcal{A}(X_{1:m}',Z_{1:n}))\). Now we want to guess \(Z_1\) from \(X_1'\), this should be easier when having the knowledge of \(\mathcal{A}(X_1';Z_1)\) compared with when having the knowledge of \(\mathcal{A}(X_{1:m}';Z_{1:n})\).

1.0.3.2 Proof of Theorem 6

1.0.3.3 Proof of Corollary 4

1.0.3.4 Proof of Theorem 7

1.0.3.5 Proof of Corollary 5

1.0.3.6 Proof of Theorem 8

1.0.3.7 Derivation of Eq. (6 )

Recall the expected cross-entropy loss, we have \[\begin{align} {\ifx&W,Z_i& \mathbb{E} \else {\mathbb{E}_{W,Z_i}} \fi \left[\ell(f_W(T_i),Y_i)\right]} &= {\ifx&Z_i,W& \mathbb{E} \else {\mathbb{E}_{Z_i,W}} \fi \left[-\log{Q_{Y_i|T_i,W}}\right]}\\ &={\ifx&Z_i,W& \mathbb{E} \else {\mathbb{E}_{Z_i,W}} \fi \left[\log{\frac{P_{Y_i|T_i,W}}{Q_{Y_i|T_i,W}P_{Y_i|T_i,W}}}\right]}\\ &=H(Y_i|T_i,W) +{\ifx&X_i,W& \mathbb{E} \else {\mathbb{E}_{X_i,W}} \fi \left[\mathrm{D_{KL}}(P_{Y_i|T_i,W}||Q_{Y_i|T_i,W})\right]}\\ &={\ifx&Z_i,W& \mathbb{E} \else {\mathbb{E}_{Z_i,W}} \fi \left[\log{\frac{P_{Y_i|T_i}P_{W|T_i}}{P_{Y_i|T_i,W}P_{Y_i|T_i}P_{W|T_i}}}\right]} +{\ifx&T_i,W& \mathbb{E} \else {\mathbb{E}_{T_i,W}} \fi \left[\mathrm{D_{KL}}(P_{Y_i|T_i,W}||Q_{Y_i|T_i,W})\right]}\\ &={\ifx&Z_i,W& \mathbb{E} \else {\mathbb{E}_{Z_i,W}} \fi \left[\log{\frac{P_{Y_i|T_i}P_{W|T_i}}{P_{Y_i,W|T_i}P_{Y_i|T_i}}}\right]} +{\ifx&T_i,W& \mathbb{E} \else {\mathbb{E}_{T_i,W}} \fi \left[\mathrm{D_{KL}}(P_{Y_i|T_i,W}||Q_{Y_i|T_i,W})\right]}\\ &=H(Y_i|T_i)-I(W;Y_i|T_i)+{\ifx&T_i,W& \mathbb{E} \else {\mathbb{E}_{T_i,W}} \fi \left[\mathrm{D_{KL}}(P_{Y_i|T_i,W}||Q_{Y_i|T_i,W})\right]} \end{align}\]

1.0.3.8 Additional Discussion on LIMIT

In Section 0.0.5, we discussed the LIMIT approach proposed by [57] as a means of controlling label information memorization during training. Roughly speaking, to update the classifier parameters, LIMIT constructs an auxiliary network that predicts gradients instead of using the true gradients, which avoids direct use of the true labels for training. To obtain accurate gradients, the auxiliary network needs to be trained using the true labels. We found that the training of LIMIT is unstable and difficult to tune the hyperparameters when used under UDA settings. Therefore, we opted to use the pseudo label strategy proposed in Section 0.0.5 instead of the pseudo gradient strategy.

1.0.4 Experiment Details↩︎

We implemented our approach using PyTorch [74] and conducted all experiments on NVIDIA Tesla V100 GPUs with 32 GB of memory. Our code builds largely on the implementation from [62]³ and [9]⁴.

1.0.4.1 Objective Functions of Gradient Penalty and Controlling Label Information

For every iteration, the objective function after adding the gradient penalty becomes \[\min_{W} \hat{L}(W,Z_{B_t},X'_{B_t})+\lambda_1||g(W,Z_{B_t},X'_{B_t})||^2,\] where \(\hat{L}(W,Z_{B_t},X'_{B_t})\) is some loss function for the source and target domain data in the current mini-batch and \(\lambda_1\) is the trade-off coefficient. For example, if we combine ERM with gradient penalty then \(\hat{L}(W,Z_{B_t},X'_{B_t})=\frac{1}{|B_t|}\sum_{k\in B_t}\ell(f_W(X_k),Y_k)\) and \(\ell\) could be the cross-entropy loss. Moreover, if we combine KL guided marginal alignment algorithm [9] with gradient penalty then the objective function is \[\min_{W,\theta} \frac{1}{|B_t|}\sum_{k\in B_t}\ell(f_W(T_k),Y_k)+\beta_1\mathrm{D_{KL}}(P_{T'}||P_T)+\beta_2\mathrm{D_{KL}}(P_T||P_{T'})+\lambda_1||g(W,Z_{B_t},X'_{B_t})||^2,\] where \(\theta\) is the parameters of the representation network and the gradient is \[g(W,Z_{B_t},X'_{B_t})= \frac{1}{|B_t|}\sum_{k\in B_t}\nabla_{W,\theta}\ell(f_W(T_k),Y_k)+\beta_1\nabla_\theta\mathrm{D_{KL}}(P_{T'}||P_T)+\beta_2\nabla_\theta\mathrm{D_{KL}}(P_T||P_{T'}).\]

In [9], the representation distribution is modelled as an Gaussian distribution, i.e., \(T\sim \mathcal{N}(\mu_\theta, \sigma^2_\theta \mathrm{I}_d|X)\) and \(T'\sim \mathcal{N}(\mu_\theta, \sigma^2_\theta \mathrm{I}_d|X')\). Additionally, let the batch size be \(b=|B_t|\), the empirical KL divergence is estimated by the mini-batch data, as given in [9], \[\begin{align} &\beta_1\mathrm{D_{KL}}(P_{T'}||P_T)+\beta_2\mathrm{D_{KL}}(P_T||P_{T'}) \\ \approx& \beta_1\frac{1}{b}\sum_{k\in B_t} [\log{P_{T_k'}}-\log{P_{T_k}}]+\beta_2\frac{1}{b}\sum_{k\in B_t} [\log{P_{T_k}}-\log{P_{T'_k}}]\\ \approx& \beta_1\frac{1}{b}\sum_{k\in B_t} \left[\log{\frac{1}{b}\sum_{k\in B_t} P_{T'_k|X'_k}}-\log{\frac{1}{b}\sum_{k\in B_t} P_{T_k|X_k}}\right]+\beta_2\frac{1}{b}\sum_{k\in B_t} \left[\log{{\frac{1}{b}\sum_{k\in B_t} P_{T_k|X_k}}}-\log{\frac{1}{b}\sum_{k\in B_t} P_{T'_k|X'_k}}\right], \end{align}\] where \(P_{T_k|X_k}=\mathcal{N}(\mu_\theta, \sigma^2_\theta \mathrm{I}_d|X_k)\) and \(P_{T'_k|X'_k}=\mathcal{N}(\mu_\theta, \sigma^2_\theta \mathrm{I}_d|X'_k)\). To be more precise, \(\mu_\theta\) and \(\sigma_\theta\) are the outputs of the representation network. Since the forward pass requires the sampling of \(T_k\) and \(T_k'\), we need to use the reparameterization trick [75] for the backward pass.

When we train the model with controlling label information, the objective function becomes \[\min_{W} \hat{L}(W,Z_{B_t},X'_{B_t})+\lambda_2||W-\widetilde{W}||^2,\] where \(\widetilde{W}\) is the auxiliary classifier and \(\lambda_2\) is the trade-off hyperparameter.

Similarly, when we combine KL guided marginal alignment algorithm with controlling label information, then the objective function in every iteration is \[\min_{W,\theta} \frac{1}{|B_t|}\sum_{k\in B_t}\ell(f_W(T_k),Y_k)+\beta_1\mathrm{D_{KL}}(P_{T'}||P_T)+\beta_2\mathrm{D_{KL}}(P_T||P_{T'})+\lambda_2||W-\widetilde{W}||^2.\] In addition, the training objective for the auxiliary classifier is \[\begin{align} \label{eq:cl-obj} \min_{\widetilde{W}} \frac{1}{|B_t|}\sum_{k\in B_t}\ell(f_{\widetilde{W}}(T'_k), f_W(T'_k))+\frac{1}{|B_t|}\sum_{k\in B_t}\ell(f_{\widetilde{W}}(T_k), f_W(T_k)). \end{align}\tag{16}\] In practice, removing the second term would not affect the performance. Note that we need to disenable the automatic differentiation of \(T\), \(T'\) and \(W\) when executing the backward pass for the auxiliary classifier. The detailed algorithm of controlling label information is given in the next section.

1.0.4.2 Algorithm of Controlling Label Information and Additional Results of ERM-CL

If we only provide the pseudo labels for the target domain data to the auxiliary classifier, i.e. removing the second term in Eq (16 ), the Algorithm 2 is the algorithm for combining any marginal alignment algorithm with controlling label information.

Even without incorporating with the marginal alignment algorithm, e.g., ERM, in which case \(L_r\) is removed, Algorithm 2 still boosts the performance in practice.

Table 2 shows that ERM-CL can overall outperform the basic ERM and is close to the performance of ERM-GP.

1.0.4.3 Architectures and Hyperparameters

The network architecture in this work is the same as in [62] and [9], where a simple CNN is used.

Other settings are also the same as [62] and [9], for example, each algorithm is trained for \(100\) epochs. To select the hyperparameters (\(\lambda_1\) and \(\lambda_2\)) for ERM-GP, ERM-KL, KL-GP and KL-CL, we perform random search. Specifically, \(\lambda_1\) is searched between \([0.1, 0.9]\) and \(\lambda_2\) is searched between \([10^{-6}, 0.8]\). Other hyperparameters searching range could be found in the source code of [9].

1.0.4.4 Additional Experimental Results

1.0.4.5 Ablation Study on the Effect of Gradient Penalty Hyperparameter

Our study includes an ablation analysis to investigate the impact of the hyperparameter \(\lambda_1\) in the context of KL-GP. Specifically, we conduct experiments on both RotatedMNIST and Digits datasets, where the source and target domains are set to 0°/60° and SVHN/MNIST, respectively. Table 3 summarizes the results. It is worth noting that setting \(\lambda_1\) to zero effectively reduces KL-GP to KL, and our results confirm the efficacy of including the gradient penalty term in KL-GP.

1.0.4.6 Visualization Results

To visualize the representations of models trained using KL, KL-GP, and KL-CL, we employ t-SNE [76]. Figure 3 displays the visualization results when SVHN is used as the source domain and MNIST as the target domain. Our findings indicate that the incorporation of additional regularizers yields a slight improvement in representation alignment. However, it is essential to note that these regularization terms are primarily designed to enhance the performance of the classifier network, rather than the representation network.

1.0.4.7 Results on VisDA17

We also conduct experiments on the VisDA17 dataset [77], which is a real-world classification task with \(280K\) images from \(12\) classes. Particularly, the source domain contains synthetic images and the target domain contains real images. Table 4 presents our experimental results. Notably, our regularization techniques, namely KL-GP and KL-CL are still capable of improving the performance of the KL guided marginal alignment algorithm to some extent.

1.0.4.8 Dynamics of Jeffrey’s divergence

The representation space version of Corollary 2 suggests that a small Jeffrey’s divergence can lead to a low testing error. Figure [fig:svhn-mnist] demonstrates that the dynamic of Jeffrey’s divergence, as computed in the representation space, can effectively characterize the evolution of the testing error throughout the training phase. Additionally, Figure [fig:sample] reveals that the number of target data used has an impact on testing performance. Specifically, when less than half of the available unlabelled target data is used, performance increases with the number of data. However, when more than half of the unlabelled target data is used, there is only marginal improvement on performance.

References↩︎