1 Introduction↩︎

Classification problems are relevant in a multitude of domains, such as computer vision, biomedical, and telecommunications engineering [1]–[3]. In general, classification refers to the estimation of a discrete vector \(\mathbf{x}\) (i.e., the class) given an observation vector \(\mathbf{y}\). In the Bayesian framework, the optimal method to solve classification problems is derived from the maximum a posteriori probability (MAP) principle [4], [5]. Classical estimation theory uses a model-based approach to derive the estimation algorithm. Then, the MAP algorithm is well-defined and applicable when the posterior density is known. If the posterior probability is unknown, the first fundamental step towards solving the MAP problem consists of learning the posterior density from the data. In this direction, deep learning (DL) approaches learn the model and solve the estimation task directly via data observation. This is achieved by leveraging artificial neural networks, whose ability to model probability density functions (pdfs) makes them particularly suited for this task, as shown in [6]–[9]. DL models require the design of two main elements: the network architecture, which defines the class of functions the network can estimate, and the objective function that is exploited during the training phase to learn the optimal parameters of the network. In reference to neural network-based classification techniques, most of the previous work focused on the conceievement of the network architecture [10]–[14]. Contrarily, a smaller part of classification literature is dedicated to the objective function design. In most cases, classification is achieved through the minimization of the cross-entropy loss function between the data empirical probability density function \(p_{data}\) and the probability density function output of the neural network \(p_{model}\) [15], [16]. The minimization of the cross-entropy corresponds to the minimization of the Kullback-Leibler (KL) divergence between the same two probability distributions. The KL divergence is defined as \[\label{eq:DKL32definition} D_{KL}(p_{data} || p_{model}) = \mathop{\mathrm{\mathbb{E}}}_{p_{data}} \left[ \log \left( \frac{p_{data}}{p_{model}} \right) \right] ,\tag{1}\] while the cross-entropy (CE) is \(\mathop{\mathrm{\mathbb{E}}}_{p_{data}} \left[- \log \left( p_{model} \right) \right]\).
Alternatively, some papers propose the usage of proper losses, which is a family of losses characterized by Bregman divergences [17]. For instance, the authors in [18] compare the square loss and the cross-entropy for supervised classification tasks, while in [19], the authors propose two novel objective functions based on logistic regression.
Another popular class of divergence functions is the \(f\)-divergence, which has been used for various classification algorithms. The authors in [20] propose a min-max game for the objective function design of deep energy-based models, where they substitute the minimization of the KL with any \(f\)-divergence. In [21], the authors propose a max-max optimization problem to tackle classification with noisy labels. They maximize the \(f\)-mutual information (a generalization of the mutual information) between the classifier’s output and the true label distribution. In [22], the \(f\)-divergence is used as a regularization term in a min-max optimization problem, for the design of fair classifiers (i.e., minimize the classifier discrepancy over sub-groups of the population).

In this paper, we propose to estimate the conditional posterior probability (needed in the MAP classifier) by expressing it as a density ratio (see 7 ) and by using a discriminative learning approach [23]. Density ratio estimation approaches have been used in a wide variety of applications [21], [24], [25]. However, unlike known classification objective functions, the main idea behind the proposed estimators is not to use a divergence minimization-based technique. Instead, to solve the classification task, we estimate the ratio between joint and marginal probability densities with a discriminator network and maximize such a ratio (corresponding to the posterior probability) with respect to the class elements. Therefore, contrarily to other \(f\)-divergence-based approaches that need a double training optimization procedure (e.g., max-max or min-max), our approach relies on a single training maximization formulation needed to learn the posterior density. Additionally, it enables the use of the \(f\)-divergence to obtain a broader set of classifiers beyond the conventional approach based on the exploitation of the cross-entropy.

In more detail, the contributions of this paper are fourfold: a) we design a class of posterior probability estimators that exploits the variational representation of the \(f\)-divergence [26]; b) we present a second class of posterior probability estimators formulated using a bottom-up approach; c) we propose a new objective function for classification tasks that corresponds to the variational representation of a novel \(f\)-divergence. The proposed divergence is analyzed theoretically and experimentally, and a comparison with the \(f\)-divergences known in the literature follows, showing that the new one achieves the best performance in almost all the considered scenarios; d) finally, we conceive two specific network architectures that are trained with the proposed objective functions.

2 MAP-based Classification Through Posterior Probability Learning↩︎

In this section, we describe the approach that we propose to tackle classification tasks and the notation used. Let \(X\) and \(Y\) be two random vectors described by the probability density functions \(p_X(\mathbf{x})\) and \(p_Y(\mathbf{y})\), respectively. Let \(X\) and \(Y\) be the input and output of a stochastic model (referred to as \(H(\cdot)\)), respectively, as shown in Fig. 1. In a classification context, \(X\) is discrete and represents the class type with alphabet \(\mathcal{A}_x\), while \(Y\) is the observation of the class elements.
The MAP estimator (classifier) is formulated as: \[\label{eq:MAP95problem} \hat{\mathbf{x}} = \mathop{\mathrm{arg\,max}}_{\mathbf{x} \in \mathcal{A}_x} p_{X|Y}(\mathbf{x}|\mathbf{y}) ,\tag{2}\] where \(p_{X|Y}(\mathbf{x}|\mathbf{y})\) is the posterior probability density. In this paper, we propose to adopt a discriminative formulation. Therefore, we express \(p_{X|Y}(\mathbf{x}|\mathbf{y})\) as the ratio between \(p_{XY}(\mathbf{x}, \mathbf{y})\) and \(p_Y(\mathbf{y})\) by using the Bayes theorem.
Since the classification task is an estimation problem, we formulate it with the MAP approach because it is the optimal approach for estimation problems in the Bayesian framework [5], [27]. A key advantage of the proposed MAP-based approach is that it does not need a double training optimization procedure because it directly learns the posterior pdf formulating the objective function either using the \(f\)-divergence (Section 3) or with a bottom-up approach (Section 4). When the posterior pdf is learned, the estimation (thus the classification) problem is solved, because the argmax operator in 2 can be easily computed. First, we study the more general problem of estimating the posterior pdf when \(X\) is a continuous random vector. Then, we consider the specific case of \(X\) being discrete. The classification problem is solved choosing the optimal class \(\mathbf{x}\) that, during inference, maximizes the estimate of \(p_{X|Y}(\mathbf{x}|\mathbf{y})\) (i.e., that solves 2 ).

3 Posterior Probability Learning Through the Exploitation of \(f\)-Divergence↩︎

The first approach we propose is to estimate \(p_{X|Y}(\mathbf{x}|\mathbf{y})\) by exploiting the variational representation of the \(f\)-divergence.

3.1 \(f\)-Divergence↩︎

Given two probability distributions \(P\) and \(Q\) admitting, respectively, the absolutely continuous density functions \(p\) and \(q\) with respect to \(d\mathbf{x}\) defined on the domain \(\mathcal{X}\), the \(f\)-divergence (also known as Ali-Silvey distance) is defined as [28], [29] \[D_f(P||Q) = \int_{\mathcal{X}} q(\mathbf{x}) f\left( \frac{p(\mathbf{x})}{q(\mathbf{x})} \right) d\mathbf{x} ,\] where the generator function \(f: \mathbb{R}_+ \longrightarrow \mathbb{R}\) is a convex, lower-semicontinuous function such that \(f(1)=0\). Every generator function has a Fenchel conjugate function \(f^*\), that is defined as \[\label{eq:fenchel95conjugate} f^*(t) = \sup_{u \in dom_f} \left\{ ut - f(u) \right\},\tag{3}\] where \(dom_f\) is the domain of \(f(\cdot)\), \(f^{*}\) is convex and such that \(f^{**}(u) = f(u)\).
Leveraging 3 , the authors in [26] expressed a lower bound on any \(f\)-divergence, that is referred to as variational representation of the \(f\)-divergence: \[\begin{align} \label{eq:variational95representation} D_f(P||Q) &\geq \sup_{T \in \mathcal{T}} \left\{ \mathop{\mathrm{\mathbb{E}}}_{p(\mathbf{x})} \left[ T(\mathbf{x}) \right] - \mathop{\mathrm{\mathbb{E}}}_{q(\mathbf{x})} \left[ f^*(T(\mathbf{x})) \right] \right\} , \end{align}\tag{4}\] where \(T(\mathbf{x})\) is parametrized by an artificial neural network, and the bound in 4 is tight when \(T(\mathbf{x})\) is \[\label{eq:T95hat} T^{\diamond}(\mathbf{x}) = f^{\prime} \left( \frac{p(\mathbf{x})}{q(\mathbf{x})} \right) ,\tag{5}\] where \(f^\prime\) is the first derivative of \(f\).

3.2 Posterior Estimation Through the Variational Representation of the \(f\)-Divergence↩︎

Theorem 1 provides a class of objective functions that, when maximized, leads to the estimation of the posterior probability density.

\begin{table}[t]

-0.1in \end{table}

According to Theorem 1, there exists a class of objective functions to train a discriminator whose output is processed to obtain an estimate of the posterior probability. The choice of the \(f\)-divergence offers a degree of freedom (DOF) in the objective function design. To improve the training convergence of the objective functions formulated as variational representation of \(f\)-divergences, the literature exploits a change of variable \(T(\mathbf{x}, \mathbf{y}) = r(D(\mathbf{x}, \mathbf{y}))\) in 6. With the same goal, we propose to introduce a second DOF in the development of the objective function 6 . Accordingly, it is sufficient to substitute \(D^{\diamond}(\mathbf{x}, \mathbf{y}) = r^{-1}(T^{\diamond}(\mathbf{x}, \mathbf{y}))\) in 7 to attain the corresponding posterior probability estimator. The exploitation of the DOFs uniquely defines the objective function and numerically impacts the discriminator’s parameters convergence during the training phase. We use Theorem 1 to obtain five specific estimators trained with the objective functions reported in Tab. ¿tbl:tab:value32functions?. These objective functions are obtained by first selecting the generator function and then choosing \(r(\cdot)\). The generator functions used to derive Tab. ¿tbl:tab:value32functions? are reported in Tab. ¿tbl:tab:f95divergences95table95unsupervised? of Appendix 9 and are retrieved from a modification of the well known \(f\)-divergences reported for completeness in Tab. ¿tbl:tab:f95divergences95table? of Appendix 9. In particular, the modification applied is attained as \(f_{u}^{*}(t) = |\mathcal{T}_x|f^{*}(t)\), where \(f_u^{*}(t)\) and \(f^{*}(t)\) are referred to as unsupervised and supervised generator functions, respectively. In addition, Tab. ¿tbl:tab:value32functions? comprises the change of variable \(r(D^{\diamond}(\mathbf{x}, \mathbf{y}))\) and the expression of \(D^{\diamond}(\mathbf{x}, \mathbf{y})\) for each objective function.

Lemma 1 proves the convergence property of any posterior probability estimator that is formulated as in Theorem 1.

In addition to proving the convergence property of the posterior estimators in Theorem 1, Lemma 1 provides an intuitive explanation of the posterior probability estimator’s bias’s dependency on \(f\) in 8 .

Theorem 1 provides an effective method to solve classification problems by designing the objective function based on the choice of an \(f\)-divergence. This differs from other methods that leverage \(f\)-divergences and that need a dual optimization strategy [21], [22]. The proposed method relies on a single optimization problem. In fact, it generalizes the cross-entropy minimization approach (see Section 6). In the next section, we propose a novel bottom-up approach that guides the design of new objective functions for the posterior probability estimation problem.

4 Bottom-Up Posterior Probability Learning↩︎

In this section, we propose a bottom-up methodology for developing objective functions that, when maximized, lead to the estimation of the posterior probability. The bottom-up approach reverses the top-down procedure typical of \(f\)-divergence formulations [21], [24]. The main advantage of this new method is that it guides the design of the objective function by starting with the imposition of the optimal convergence condition of the discriminator’s output \(D^{\diamond}(\mathbf{x}, \mathbf{y})\) (see Appendix 11). Theorem 2 presents the class of objective functions that, when maximized, leads to the bottom-up posterior estimator.

The proposed estimator leverages a discriminative formulation to estimate the density ratio corresponding to the posterior probability. Theorem 2 comprises two DOFs in the objective function design. The former is the choice of \(k(\cdot)\), since the discriminator estimates an invertible transformation of the posterior density. We noticed that the classifier performs better when \(k(\cdot)\) is chosen to resemble particular activation functions. The second DOF, represented by \(g_1(\cdot)\), is a rearrangement term that modifies the result of the integration of \(g(\cdot)\) with respect to \(D\). The exploitation of such a DOF allows to attain different objective functions even when imposing the same optimal discriminator’s output. Theorems 1 and 2 have different advantages. The benefit of Theorem 1 is its simple applicability, that must, however, begin with the selection of the generator function \(f\). Theorem 2 has the advantage of guiding the objective function design without relying on existing \(f\)-divergences.
Corollary 4 in Appendix 10.6 exploits the DOFs in the choice of \(k(\cdot)\) and \(g_1(\cdot)\) to obtain the objective functions in Tab. ¿tbl:tab:value32functions?, which were previously derived from Theorem 1. An interesting result highlighted by Corollary 4 is that all the objective functions corresponding to the well-known \(f\)-divergences in Tab. ¿tbl:tab:f95divergences95table95unsupervised? use the same class of functions \(g_1(\cdot) \propto 1/D^{\alpha}\cdot 1/(1-D)^{\beta}\) expressed in ?? in Appendix 10.6. This property, highlighted by the class of bottom-up estimators, shows us that \(\mathcal{J}_{GAN}(D)\) is the only one using \(\beta \neq 0\). Thanks to the observation of this peculiarity and driven by curiosity, we develop a new objective function starting from \(k_{GAN}(\cdot)\) and \(g_{1,GAN}(\cdot)\) but imposing \(\beta=0\).

5 Shifted Log Objective Function and Divergence↩︎

In this section, we present a new objective function for classification problems, that we design by using Theorem 2. Then, we prove that such an objective function corresponds to the variational representation of a novel \(f\)-divergence, called shifted log (SL). We will demonstrate in Section 7 that such a new objective function achieves the best performance in almost all the classification scenarios discussed.

Corollary 1 states that \(\mathcal{J}_{SL}(D)\) in 12 can be obtained from Theorem 1 by using a new \(f\)-divergence referred to as shifted log.

5.1 Remarks on the New Objective Function and \(f\)-Divergence↩︎

Since the proposed objective function in 12 is the variational representation of an \(f\)-divergence (for Corollary 1), Lemma 1 ensures the convergence property of its estimate to the true posterior probability in the nonparametric limit.

The supervised version of the SL divergence is \[\label{eq:general95fSL} f_{SL}(u) = -\log(u + 1) + \log(2) ,\tag{16}\] which is obtained by substituting \(|\mathcal{T}_x|=1\) (see Appendix 9). Although \(f_{SL}\) is obtained in the context of posterior estimation problems, the proposed \(f\)-divergence can be applied to a broader variety of tasks. Since \(D_{SL}\) is upper-bounded (see Corollary 7 in Appendix 10.9), it is a suitable generator function when the optimization problem requires to maximize the variational representation of the \(f\)-divergence (i.e., a max-max game). For instance, for classification tasks as in [21] or for representation learning applications as in [30].

5.2 Comparison Between SL and GAN Divergences↩︎

The GAN divergence is known to be highly-performing in a wide variety of tasks [24], [30]. Moreover, the objective functions corresponding to SL and GAN divergences can be obtained from Theorem 2 by choosing the same \(k(\cdot)\), but different \(g_1(\cdot)\). Corollary 2 compares the concavity of the two objective functions in the neighborhood of the global optimum.

Although the shape of the loss landscape depends on many factors (e.g., the batch size [31], [32]), sharper maxima attain larger test error [33]. The steepness of the concavity of the objective function in the neighborhood of the global optimum provides insights on the basin of attraction of the point of maximum. Intuitively, a flatter landscape corresponds to a larger basin of attraction of the global optimum, rendering training with the SL divergence better than with the GAN divergence. The results in Fig. 5 and Tab. ¿tbl:tab:speed95accuracy? in Appendix 12 validate the findings of Corollary 2.

6 Discriminator Architecture↩︎

In this section, we discuss the appropriate modifications to the discriminator’s architecture to suit our estimators to the classification scenario, where the number of classes is finite. The architecture type used differs depending on the alphabet of \(X\), referred to as \(\mathcal{A}_x\). When \(X\) is continuous, we use a structure referred to as unsupervised architecture. Differently, when \(X\) is discrete, we use a structure referred to as supervised architecture.

6.1 Unsupervised Architecture↩︎

In this setting, the samples \(\mathbf{x}\) and \(\mathbf{y}\) drawn from the empirical probability distributions \(p_{XY}(\mathbf{x}, \mathbf{y})\) and \(p_U(\mathbf{x})p_Y(\mathbf{y})\) are concatenated and fed into the discriminator. The discriminator output is a scalar, since it is the posterior density function estimate corresponding to the pair \((\mathbf{x}, \mathbf{y})\) given as input. The discriminator architecture is represented in Fig. 2 (a), where the concatenation between the \(\mathbf{x}\) and \(\mathbf{y}\) realizations is identified by a dashed rectangle.

6.2 Supervised Architecture↩︎

The supervised architecture (Fig. 2 (b)) introduces in the problem’s formulation the constraint that \(\mathcal{A}_x\) is a set containing a finite number of elements \(\mathcal{A}_x = \left\{ \mathbf{x}_1, \dots, \mathbf{x}_m \right\}\). This constraint is embedded in the architecture (highlighted by a dashed arrow in Fig. 2 (b)), so that the output layer contains one neuron for each sample in \(\mathcal{A}_x\). With this modification, the i-th output neuron returns \(p_{X|Y}(\mathbf{x}_i | \mathbf{y})\). Accordingly, the input layer is fed with only the realizations \(\mathbf{y}\). Notation wise, the discriminator’s output is referred to as \(\boldsymbol{D}(\mathbf{y})\). Theorem 4 shows the modification to the formulation of the objective functions in 6 when using a supervised architecture. First, we define the notation useful for the theorem statement: \(\boldsymbol{D}(\mathbf{y}) = [D(\mathbf{x}_1, \mathbf{y}), \dots, D(\mathbf{x}_m, \mathbf{y})]\) and \(\boldsymbol{1}_m(\mathbf{x}_i)=[0, \dots, 0, \underbrace{1}_{i^{th} \>\> pos.}, 0, \dots, 0]^T\).

The supervised versions of the objective functions utilized in this paper are listed in Section 9 of the Appendix.
For classification problems, the objective function obtained by substituting the KL divergence in Theorem 4 (see 21 in Appendix 9.1) is exactly the cross-entropy loss. When the Softmax function is applied to the discriminator output (because \(D^\diamond(\mathbf{x}, \mathbf{y})=p_{X|Y}(\mathbf{x}|\mathbf{y})\) is a discrete pdf), the expectation over \(p_Y(\mathbf{y})\) becomes a constant always equal to \(1\). Maximizing 21 is equivalent to minimizing the negative of 21 , which is precisely the minimization of the CE.

7 Results↩︎

In this section, we report several numerical results to assess the validity of the methods proposed to estimate the posterior probability density and enable the classification task. The considered scenarios are: classification for image datasets; signal decoding in telecommunications engineering cast into a classification task; posterior probability estimation when \(p_X(\mathbf{x})\) is continuous. The results demonstrate that different \(f\)-divergences attain different performance. We show that the KL divergence (thus the CE) is not necessarily the best choice for classification tasks, and more in general for probability estimation problems. We demonstrate that the SL divergence achieves the best performance in almost all the tested contexts. When referring to the performance of any \(f\)-divergence, we will implicitly imply the performance achieved using the objective function derived using such an \(f\)-divergence.
The first two scenarios are classification problems, therefore we use the supervised formulation of the discriminator architecture. The third scenario comprises two toy cases, where we show that the unsupervised formulation of the proposed estimators works for continuous random vectors \(X\). Before discussing the numerical results, we briefly describe the details of the code implementation¹.

7.1 Implementation Details↩︎

Supervised Architecture: For the first scenario (Section 7.2), we use convolutional neural networks. When referring to tiny network, we use a discriminator comprising a small set of convolutional layers (less than 4) followed by a feedforward fully connected part. Besides the tiny network, in the first scenario we utilize various deep network architectures: VGG [10], ResNet [34], DLA [35], and MobileNetV2 [36]. The discriminator hyper-parameters slightly vary depending on the dataset tested. The network parameters are updated by using SGD with momentum. The activation function of the last layer depends on the objective function optimized during the training phase.
For the second scenario (Section 7.3), we use fully connected feedforward neural networks. The architecture used for the decoding scenario comprises two hidden layers with \(100\) neurons each. The network weights are updated by using the Adam optimizer [37]. The LeakyReLU activation function is utilized in all the layers except the last one, where the activation function is chosen based on the objective function. In some cases, the Dropout technique [38] helps the convergence of the training process.

Unsupervised Architecture: The discriminator architecture utilized for the unsupervised tasks comprises two hidden layers with \(100\) neurons each and the LeakyReLU activation function. The activation function of the output layer depends on the objective function used during training. The network weights and biases are updated by using the Adam optimizer. Dropout is used during training.

7.2 Image Datasets Classification↩︎

The first scenario tackled is the classification of image datasets. The objective functions performance is tested for the MNIST [39], Fashion MNIST [40], CIFAR10, and CIFAR100 [41] datasets. A more detailed description of the datasets can be found in Appendix 12. We compare the classification accuracy of the supervised versions of the objective functions in Tab. ¿tbl:tab:value32functions? and in 12 , which are all reported in Appendix 9. To improve the training procedure, we apply data augmentation on the CIFAR datasets by randomly cropping and flipping the images. The learning rate is initially set to \(0.1\) and then we use a cosine annealing scheduler [42] to modify its value during the \(200\) epochs of training.

We compute the mean accuracy and its standard deviation for each dataset and \(f\)-divergence by running the code over multiple random seeds. The classification accuracy results displayed in Tab. ¿tbl:tab:classification95images? (and in the extended version comprising more network architectures in Tab. ¿tbl:tab:classification95images95appendix? in Appendix 12) confirm that each generator function has a different impact on the neural network training, as also shown in [21], [24]. The SL divergence attains the highest classification accuracy for almost all the architectures tested, showing its effectiveness compared to the other divergences and its stable behavior over different datasets and architectures. In particular, the SL divergence attains better performance than the CE, which corresponds to the state-of-the-art approach for image classification tasks. Furthermore, in the few scenarios where the SL divergence does not achieve the best performance, it achieves the second best (only in one case the third best, in Table ¿tbl:tab:classification95images95appendix? in Appendix 12) performance, with an accuracy close to the optimal one (see Tables ¿tbl:tab:classification95images?, ¿tbl:tab:classification95images95appendix?). The CE is, on average, the second-best objective function. Conversely, the Pearson \(\chi^2\) performs the worst in almost all the scenarios. Moreover, except from the SL divergence and the CE, the performance of the other \(f\)-divergences is more architecture dependent. For instance, as reported in Tab. ¿tbl:tab:classification95images95appendix? in Appendix 12, the PreActResNet [43] attains the highest accuracy when it is trained with the GAN-based objective function. The choice of the architecture often depends on the goal of the classification algorithm. For embedded systems, light architectures are used. Therefore, the MobileNetV2 is an option [44]. In such a case, the SL divergence obtains an accuracy \(1.5/2\%\) higher than the CE, which makes the SL the preferred choice for the network’s training. Additional numerical results on other network architectures and on the speed of convergence of the training phase are reported in Appendix 12 for space limitations. The speed of convergence analysis (Fig. 5) demonstrates that the training with the SL divergence leads to a faster convergence to the optimum discriminator w.r.t. the GAN divergence, as stated in Corollary 2.

7.3 Signal Decoding↩︎

The second scenario is the decoding problem. Decoding a sequence of received bits is crucial in a telecommunication system. In some cases, when the communication channel is known, the optimal decoding technique is also known [27]. However, the communication channel is generally unknown, and DL-based techniques can be used to learn it [45]. By knowing that the optimal decoding criterion is the posterior probability maximization, we demonstrate that the proposed MAP approach solves the decoding problem and that the supervised version of \(\mathcal{J}_{SL}(D)\) (see 31 in Appendix 9.6) achieves optimal performance. We consider a 4-PAM (i.e., pulse amplitude modulation with four symbols) over a nonlinear channel with additive Gaussian noise. In particular, given the symbol at time instant \(t\) (referred to as \(x_t\)), we obtain the channel output as \(y_t = sgn(x_t)\sqrt{|x_t|} + n_t\), where \(n_t\) is the Gaussian noise and \(sgn(\cdot)\) is the sign function. We show the symbol error rate (SER) behavior achieved by the proposed posterior estimators when varying the signal-to-noise ratio (SNR) in Fig. 3. The estimators proposed are compared with the results of the max likelihood estimator (referred to as maxL) and the MAP Genie estimators. The MAP Genie estimator uses the knowledge of the channel nonlinearity to decode the received sequence of bits. The proposed list of posterior probability estimators performs better than the max likelihood estimator, achieving accuracy close to the optimal MAP Genie estimator. In Appendix 12, other scenarios of decoding tasks are reported.

7.4 Continuous Posterior Estimation↩︎

This section considers two toy examples for the case \(|\Omega| \geq |\mathbb{R}|\). The comparison between the closed-form of the posterior distribution and the discriminator estimate is showed in Fig. 4. The discriminator prediction is obtained by training a tiny neural network with the objective functions reported in Tab. ¿tbl:tab:value32functions? and in 12 . The closed-form posterior distribution (referred to as true in Fig. 4) is the first element of each row.

Exponential task. In the first toy example, we define the model \(Y= X+N\), where \(X\) and \(N\) are independent Exponential random variables. Therefore, \(Y\) is a Gamma distribution [46]. The closed-form posterior probability can be expressed as in 47 (see Appendix 10.12 for the formula and proof). Similarly to the previous numerical results, different \(f\)-divergences lead to diverse estimates of the posterior density. The objective functions corresponding to the GAN, SL, and HD divergences attain better estimates w.r.t. the others. For a fixed \(y\), in fact, the posterior density value is constant over \(x\).

Gaussian task. In the second toy example, we consider the model \(Y = X+N\), where \(X\), \(N\) are independent Gaussian random variables. Thus, \(Y\) is a Gaussian distribution. The posterior density expression is reported in Appendix 10.13 (see 48 ). The objective functions corresponding to the SL, KL, and P divergences attain better estimates w.r.t. the others, since the estimate attains the desired saddle shape.

8 Conclusions↩︎

In this paper, we proposed a new MAP perspective for supervised classification problems. We have proposed to use a discriminative formulation to express the posterior probability density, and we have derived two classes of estimators to estimate it. From them, we extracted a list of posterior probability estimators and compared them with the notorious cross-entropy minimization approach. Numerical results on different scenarios demonstrate the effectiveness of the presented estimators and that the proposed SL divergence achieves the highest classification accuracy in almost all the scenarios. Additionally, we show that the proposed posterior probability estimators work for the general case of continuous a priori probabilities, for which we design a specific neural network architecture.

Acknowledgements↩︎

We thank Nunzio Alexandro Letizia for the precious insights and the fruitful discussions. We also thank the anonymous ICML 2024 reviewers for their detailed and helpful feedback.

Impact Statement↩︎

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

9 Appendix: Objective Functions Used in the Experiments↩︎

The unsupervised and supervised versions of the objective functions used to achieve the results showed in Section 7 are reported in this section. The training part consists in the alternation of two phases. In the former phase, the network is fed with \(N\) realizations of the joint distribution \(p_{XY}(\mathbf{x}, \mathbf{y})\) to compute the first term of the objective function. In the latter phase, the type of architecture defines the procedure to compute the second term of the objective function. The unsupervised architecture is fed with \(N\) samples drawn from \(p_U(\mathbf{x})p_Y(\mathbf{y})\). The supervised architecture is fed with \(N\) samples drawn from \(p_Y(\mathbf{y})\) (see Theorem 4).
During the test part, the network is fed with the samples drawn from the joint distribution (\((\mathbf{x}, \mathbf{y}) \sim p_{XY}(\mathbf{x}, \mathbf{y})\)), and the posterior probability density estimate is obtained as in 7 or 9 .
We report the objective functions derived from well-known \(f\)-divergences. Therefore, we first report the notorious \(f\)-divergences in Table ¿tbl:tab:f95divergences95table?, and their unsupervised version in Table ¿tbl:tab:f95divergences95table95unsupervised?. These tables do not contain the constant terms that render \(f_u(1)=0\) or \(f(1)=0\), as their presence do not affect the optimization of the derived objective functions (see the proof of Lemma 3). Let \(f_{u}\) and \(f_{u}^{*}\) the unsupervised generator function and its Fenchel conjugate in Table ¿tbl:tab:f95divergences95table95unsupervised?, respectively. Differently, \(f\) and \(f^{*}\) are the supervised generator function and its Fenchel conjugate in Table ¿tbl:tab:f95divergences95table?, respectively. Then, \[\label{eq:unsupervised95f95def} f_{u}^{*}(t) = |\mathcal{T}_x| f^{*}(t),\tag{19}\] where \(|\mathcal{T}_x|\) is the Lebesgue measure of the support of \(p_X(\mathbf{x})\). Accordingly, \[f_{u}(u) = \sup_{t \in dom_{f_{u}^{*}}} \left\{ ut - f_{u}^{*}(t) \right\} .\] Vice versa, the supervised version of \(f_u\) and \(f_u^*\) can be easily attained by substituting \(|\mathcal{T}_x|=1\) in the unsupervised formulation.
The usage of 19 in Theorem 1 is needed because \(|\mathcal{T}_x|\) counterbalances the effect of \(p_U(\mathbf{x})\) to obtain the pdf ratio equivalent to the posterior density \(p_{X|Y}(\mathbf{x}| \mathbf{y}) = p_{XY}(\mathbf{x}, \mathbf{y})/p_Y(\mathbf{y})\). In fact, \(|\mathcal{T}_x|\) is included in the expectation term computed over \(p_U(\mathbf{x})\). In Lemma 3, we prove that \(f_u\) is the generator function of a valid \(f\)-divergence.

Note: The notations supervised and unsupervised refer to the discriminator architecture. The former one is the typical supervised classification architecture, while the latter one does not use \(X\) as the set of labels, but as an additional input. The terminology we use to distinguish between the two different versions of \(f\)-divergences (\(f\) and \(f_u\)) is a consequence of the architecture’s notation. The unsupervised architecture, in fact, is trained using objective functions defined with \(f_u\) (see Theorem 1 and Corollary 1), while the supervised architecture is trained by using objective functions based on \(f\) (see Theorem 4).

We recall the notation used in the objective functions: \(\boldsymbol{D}(\mathbf{y}) = [D(\mathbf{x}_1, \mathbf{y}), \dots, D(\mathbf{x}_m, \mathbf{y})]\) is a row vector, \(\boldsymbol{1}_m = [1, \dots, 1]^T\) is a column vector and \(\boldsymbol{1}_m(\mathbf{x}_i)=[0, \dots, 0, \underbrace{1}_{i^{th} \>\> pos.}, 0, \dots, 0]^T\) is a column vector.

9.1 KL-Based Objective Functions↩︎

Kullback-Leibler Divergence: The variational representation of the KL divergence is achieved substituting \(f_{u,KL}^{*}\) listed in Table ¿tbl:tab:f95divergences95table95unsupervised? in 6 . This leads to the unsupervised objective function \[\begin{align} \label{eq:kl95value95function} \mathcal{J}_{KL}(D) &= \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_{XY}(\mathbf{x}, \mathbf{y})} \Bigl[ \log(D(\mathbf{x}, \mathbf{y})) \Bigr] - \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_U(\mathbf{x})p_Y(\mathbf{y})} \Bigl[ |\mathcal{T}_x| D(\mathbf{x}, \mathbf{y}) \Bigr] + 1 , \end{align}\tag{20}\] where \(T^{\diamond}(\mathbf{x}, \mathbf{y}) = \log(\frac{p_{XY}(\mathbf{x}, \mathbf{y})}{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y})}) + 1\), \(p_U(\mathbf{x}) = \frac{1}{|\mathcal{T}_x|}\), and \(D^{\diamond}(\mathbf{x}, \mathbf{y})= \frac{p_{XY}(\mathbf{x}, \mathbf{y})}{p_Y(\mathbf{y})}\). The supervised version of the objective function in 20 is attained by using Theorem 4: \[\begin{align} \label{eq:kl95value95function95supervised} \mathcal{J}_{KL}(D) &= \mathop{\mathrm{\mathbb{E}}}_{\mathbf{x} \sim p_{X}(\mathbf{x})} \Biggl[ \mathop{\mathrm{\mathbb{E}}}_{\mathbf{y} \sim p_{Y|X}(\mathbf{y}| \mathbf{x})} \Bigl[ \log(\boldsymbol{D}(\mathbf{y})) \boldsymbol{1}_m(\mathbf{x}) \Bigr] \Biggr] - \mathop{\mathrm{\mathbb{E}}}_{ \mathbf{y} \sim p_Y(\mathbf{y})} \Bigl[ \boldsymbol{D}(\mathbf{y}) \boldsymbol{1}_m \Bigr] . \end{align}\tag{21}\] When the last layer of the supervised discriminator utilizes the softmax activation function (i.e., when the output is normalized to be a discrete probability density function), then the second term in 21 is always equal to 1. Thus, the maximization of 21 exactly corresponds to the minimization of the KL divergence in 1 , and therefore to the minimization of the cross-entropy.
Interestingly, the more general formulation in 21 allows the usage of different activation functions in the last layer, with the only requirement that the discriminator’s output is constrained to assume positive values (e.g., softplus).

9.2 RKL-Based Objective Functions↩︎

Reverse Kullback-Leibler Divergence: Theorem 1 leads to the variational representation of the reverse KL divergence, when substituting \(f_{u,RKL}^{*}\) listed in Table ¿tbl:tab:f95divergences95table95unsupervised? in 6 \[\begin{align} \label{eq:reverseKL95value95function} \mathcal{J}_{RKL}(D) &= \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_{XY}(\mathbf{x}, \mathbf{y})} \Bigl[ - D(\mathbf{x}, \mathbf{y}) \Bigr] + \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_U(\mathbf{x})p_Y(\mathbf{y})} \Bigl[ |\mathcal{T}_x| \log(D(\mathbf{x}, \mathbf{y})) \Bigr] , \end{align}\tag{22}\] where \(T^{\diamond}(\mathbf{x}, \mathbf{y}) = -\frac{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y})}{p_{XY}(\mathbf{x}, \mathbf{y})}\) and \(D^{\diamond}(\mathbf{x}, \mathbf{y}) = \frac{p_Y(\mathbf{y})}{p_{XY}(\mathbf{x}, \mathbf{y})}\). The supervised version of the objective function in 22 is obtained by using Theorem 4. \[\begin{align} \label{eq:supervised95reverseKL95value95fucntion} \mathcal{J}_{RKL}(D) &= \mathop{\mathrm{\mathbb{E}}}_{\mathbf{x} \sim p_{X}(\mathbf{x})} \Biggl[ \mathop{\mathrm{\mathbb{E}}}_{\mathbf{y} \sim p_{Y|X}(\mathbf{y}| \mathbf{x})} \Bigl[ - \boldsymbol{D}(\mathbf{y}) \boldsymbol{1}_m(\mathbf{x}) \Bigr] \Biggr] + \mathop{\mathrm{\mathbb{E}}}_{ \mathbf{y} \sim p_Y(\mathbf{y})} \Bigl[ \log(\boldsymbol{D}(\mathbf{y})) \boldsymbol{1}_m \Bigr] . \\ \notag \end{align}\tag{23}\]

9.3 HD-Based Objective Functions↩︎

Hellinger Squared Distance: Theorem 1 leads to the variational representation of the Hellinger squared distance, when substituting \(f_{u,HD}^{*}\) listed in Table ¿tbl:tab:f95divergences95table95unsupervised? in 6 \[\begin{align} \label{eq:HD95value95function} \mathcal{J}_{HD}(D) &= \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_{XY}(\mathbf{x}, \mathbf{y})} \Bigl[- \sqrt{D(\mathbf{x}, \mathbf{y})} \Bigr] - \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_U(\mathbf{x})p_Y(\mathbf{y})} \Biggl[ |\mathcal{T}_x| \frac{1}{\sqrt{D(\mathbf{x}, \mathbf{y})}} \Biggr] , \end{align}\tag{24}\] where \(T^{\diamond}(\mathbf{x}, \mathbf{y}) = \left( \sqrt{\frac{p_{XY}(\mathbf{x}, \mathbf{y})}{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y})}} - 1 \right) \sqrt{\frac{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y})}{p_{XY}(\mathbf{x}, \mathbf{y})}}\), and \(D^{\diamond}(\mathbf{x}, \mathbf{y}) = \frac{p_Y(\mathbf{y})}{p_{XY}(\mathbf{x}, \mathbf{y})}\). The supervised implementation of the objective function in 24 is achieved by using Theorem 4 \[\begin{align} \label{eq:supervised95HD95value95function} \mathcal{J}_{HD}(D) &= \mathop{\mathrm{\mathbb{E}}}_{\mathbf{x} \sim p_{X}(\mathbf{x})} \Biggl[ \mathop{\mathrm{\mathbb{E}}}_{\mathbf{y} \sim p_{Y|X}(\mathbf{y}| \mathbf{x})} \Bigl[ -\sqrt{\boldsymbol{D}(\mathbf{y})} \boldsymbol{1}_m(\mathbf{x}) \Bigr] \Biggr] - \mathop{\mathrm{\mathbb{E}}}_{ \mathbf{y} \sim p_Y(\mathbf{y})} \Biggl[ \frac{1}{\sqrt{\boldsymbol{D}(\mathbf{y})}} \boldsymbol{1}_m \Biggr] . \\ \notag \end{align}\tag{25}\]

9.4 GAN-Based Objective Functions↩︎

GAN: Theorem 1 leads to the variational representation of the GAN divergence, when substituting \(f_{u,GAN}^{*}\) listed in Table ¿tbl:tab:f95divergences95table95unsupervised? in 6 \[\begin{align} \label{eq:GAN95value95function} \mathcal{J}_{GAN}(D) &= \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_{XY}(\mathbf{x}, \mathbf{y})} \Bigl[\log(1 - D(\mathbf{x}, \mathbf{y}) )\Bigr] + \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_U(\mathbf{x})p_Y(\mathbf{y})} \Bigl[ |\mathcal{T}_x| \log(D(\mathbf{x}, \mathbf{y})) \Bigr] , \end{align}\tag{26}\] where \(T^{\diamond}(\mathbf{x}, \mathbf{y}) = \log{\left( \frac{p_{XY}(\mathbf{x}, \mathbf{y})}{p_{XY}(\mathbf{x}, \mathbf{y}) + |\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y})} \right)}\) and \(D^{\diamond}(\mathbf{x}, \mathbf{y})= \frac{p_Y(\mathbf{y})}{p_Y(\mathbf{y}) + p_{XY}(\mathbf{x}, \mathbf{y})}\). The supervised implementation of the objective function in 26 is \[\begin{align} \label{eq:supervised95GAN95value95function} \mathcal{J}_{GAN}(D) &= \mathop{\mathrm{\mathbb{E}}}_{\mathbf{x} \sim p_{X}(\mathbf{x})} \Biggl[ \mathop{\mathrm{\mathbb{E}}}_{\mathbf{y} \sim p_{Y|X}(\mathbf{y}| \mathbf{x})} \Bigl[ \log(\boldsymbol{1}_m - \boldsymbol{D}(\mathbf{y})) \cdot \boldsymbol{1}_m(\mathbf{x}) \Bigr] \Biggr] \notag + \mathop{\mathrm{\mathbb{E}}}_{ \mathbf{y} \sim p_Y(\mathbf{y})} \Bigl[ \log(\boldsymbol{D}(\mathbf{y})) \boldsymbol{1}_m \Bigr] . \\ \end{align}\tag{27}\]

9.5 P-Based Objective Functions↩︎

Pearson \(\chi^2\): Theorem 1 leads to the variational representation of the Pearson \(\chi^2\) divergence, when substituting \(f_{u,P}^{*}\) listed in Table ¿tbl:tab:f95divergences95table95unsupervised? in 6 \[\begin{align} \label{eq:P95value95function} \mathcal{J}_{P}(D) &= \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_{XY}(\mathbf{x}, \mathbf{y})} \Bigl[ 2 (D(\mathbf{x}, \mathbf{y}) - 1) \Bigr] - \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x}, \mathbf{y}) \sim p_U(\mathbf{x})p_Y(\mathbf{y})} \Biggl[ |\mathcal{T}_x| D^2(\mathbf{x}, \mathbf{y}) \Biggr] , \end{align}\tag{28}\] where \(T^{\diamond}(\mathbf{x}, \mathbf{y}) = 2 \left( \frac{p_{XY}(\mathbf{x}, \mathbf{y})}{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y})} - 1 \right)\), and \(D^{\diamond}(\mathbf{x}, \mathbf{y}) = \frac{p_{XY}(\mathbf{x}, \mathbf{y})}{p_Y(\mathbf{y})}\). The supervised implementation of the objective function in 28 is achieved by using Theorem 4 \[\begin{align} \label{qftcselu} \mathcal{J}_{P}(D) &= \mathop{\mathrm{\mathbb{E}}}_{\mathbf{x} \sim p_{X}(\mathbf{x})} \Biggl[ \mathop{\mathrm{\mathbb{E}}}_{\mathbf{y} \sim p_{Y|X}(\mathbf{y}| \mathbf{x})} \Bigl[ 2(\boldsymbol{D}(\mathbf{y}) - \boldsymbol{1}_m) \boldsymbol{1}_m(\mathbf{x}) \Bigr] \Biggr] - \mathop{\mathrm{\mathbb{E}}}_{ \mathbf{y} \sim p_Y(\mathbf{y})} \Biggl[ \boldsymbol{D}(\mathbf{y})\boldsymbol{D}^T(\mathbf{y}) \Biggr] , \\ \notag \end{align}\tag{29}\] where \(\boldsymbol{D}(\mathbf{y})\boldsymbol{D}^T(\mathbf{y}) = \sum_{i=1}^m D^2(\mathbf{x}_i, \mathbf{y})\).

9.6 SL-Based Objective Functions↩︎

The unsupervised and supervised versions of the objective function corresponding to the shifted log divergence are discussed in this paragraph. For completeness, we report here the generator function and its Fenchel conjugate.
\(f_{u, SL}(u) = - |\mathcal{T}_x| \log(u + |\mathcal{T}_x|)\), \(f_{u, SL}^{*}(t) = -|\mathcal{T}_x|(\log(-t) + t)\).

Shifted log: Theorem 2 leads to the objective function in 12 when \[\begin{align} k(p_{X|Y}(\mathbf{x}|\mathbf{y})) &= \frac{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y})}{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y}) + p_{XY}(\mathbf{x},\mathbf{y})} \\ g_1(D,k; 1, 0) & = - \frac{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y}) + p_{XY}(\mathbf{x}, \mathbf{y})}{D(\mathbf{x}, \mathbf{y})} . \end{align}\] We report here for completeness the unsupervised objective function in 12 \[\begin{align} \label{eq:SL95value95function} \mathcal{J}_{SL}(D) & = - \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x},\mathbf{y}) \sim p_{XY}(\mathbf{x}, \mathbf{y})} \Bigl[ D(\mathbf{x}, \mathbf{y}) \Bigr] + \mathop{\mathrm{\mathbb{E}}}_{(\mathbf{x},\mathbf{y}) \sim p_U(\mathbf{x})p_Y(\mathbf{y})} \Bigl[ |\mathcal{T}_x| \Bigl( \log(D(\mathbf{x}, \mathbf{y})) - D(\mathbf{x}, \mathbf{y}) \Bigr) \Bigr] , \end{align}\tag{30}\] which can be obtained from Theorem 1 with \(T^{\diamond}(\mathbf{x}, \mathbf{y}) = -\frac{|\mathcal{T}_x| p_U(\mathbf{x})p_Y(\mathbf{y})}{|\mathcal{T}_x|p_U(\mathbf{x})p_Y(\mathbf{y}) + p_{XY}(\mathbf{x}, \mathbf{y})}\) and \(D^{\diamond}(\mathbf{x}, \mathbf{y}) = \frac{p_Y(\mathbf{y})}{p_Y(\mathbf{y}) + p_{XY}(\mathbf{x}, \mathbf{y})}\). The supervised implementation of the objective function in 30 is achieved by using Theorem 4 \[\begin{align} \label{eq:supervised95SL95value95function} \mathcal{J}_{SL}(D) & = - \mathop{\mathrm{\mathbb{E}}}_{\mathbf{x} \sim p_{X}(\mathbf{x})} \Biggl[ \mathop{\mathrm{\mathbb{E}}}_{\mathbf{y} \sim p_{Y|X}(\mathbf{y}| \mathbf{x})} \Bigl[ \boldsymbol{D}(\mathbf{y})^T \boldsymbol{1}_m(\mathbf{x})\Bigr] \Biggr] + \mathop{\mathrm{\mathbb{E}}}_{\mathbf{y} \sim p_Y(\mathbf{y})} \Bigl[ \Bigl( \log( \boldsymbol{D}(\mathbf{y})) - \boldsymbol{D}(\mathbf{y}) \Bigr)^T \boldsymbol{1}_m \Bigr] . \end{align}\tag{31}\]

10 Appendix: Proofs↩︎

10.1 Proof of Lemma B.1↩︎

10.2 Proof of Lemma B.2↩︎

10.3 Proof of Theorem 3.1↩︎

10.4 Proof of Lemma 3.2↩︎

10.5 Proof of Theorem 4.1↩︎

10.6 Proof of Corollary 4.2↩︎

10.7 Proof of Theorem 5.1↩︎

10.8 Proof of Corollary 5.2↩︎

10.9 Proof of Corollary 5.3↩︎

10.10 Proof of Corollary 5.4↩︎

10.11 Proof of Theorem 6.1↩︎

10.12 Proof of Closed Form Posterior in the Exponential Case↩︎

Closed Form Posterior Exponential Case: Let define the model \(Y= X+N\), where \(X \sim exp(\lambda)\), \(N \sim exp(\lambda)\). Therefore, \(Y \sim \Gamma(2, \frac{1}{\lambda})\) (i.e., a Gamma distribution with shape \(2\) and rate \(\frac{1}{\lambda}\)). Let define \(Y|X \triangleq Z\), then the cumulative density function (CDF) is \[\begin{align} P\left[ Z \leq z \right] &= P \left[ N+x_c \leq z \right] = P\left[ N \leq z - x_c \right] \notag \\ &= \left( 1-e^{-\lambda(z-x_c)} \right)\mathbb{1}(z-x_c) , \end{align}\] where \(X_c\) is constant. By deriving the CDF w.r.t. \(z\), we compute the likelihood \[p_{Y|X}(y|x) = \lambda e^{-\lambda(y|x - x_c)} \mathbb{1}(y|x - x_c) = p_N(n).\] Then, the posterior probability is computed as \[\begin{align} \label{eq:posterior95closed95form95exponential} p_{X|Y}(x|y) &= \frac{p_N(n)p_X(x)}{p_Y(y)} \notag \\ &= \frac{1}{y} \mathbb{1}(x) \mathbb{1}(y) . \end{align}\tag{47}\]

10.13 Proof of Closed Form Posterior in the Gaussian Case↩︎

Closed Form Posterior Gaussian Case: Let define the model \(Y = X+N\), where \(X \sim \mathcal{N}(0, \Sigma_X)\), \(N \sim \mathcal{N}(0, \Sigma_N)\). Thus, \(Y|X \sim \mathcal{N}(X, \Sigma_N)\), \(Y \sim \mathcal{N}(0, \underbrace{\Sigma_N + \Sigma_X}_{\triangleq \Sigma_Y})\).

The posterior probability is \[\begin{align} p_{X|Y}(x|y) &= \frac{p_{Y|X}(y|x) p_{X}(x)}{p_{Y}(y)} \\ &= \frac{ \frac{1}{\sqrt{(2 \pi)^d |\Sigma_N|}} e^{-\frac{1}{2} (y-x)^T \Sigma_N^{-1} (y - x) } }{\frac{1}{\sqrt{(2 \pi)^d |\Sigma_Y|}} e^{-\frac{1}{2} y^T \Sigma_Y^{-1} y }} \notag \\ & \cdot \frac{1}{\sqrt{(2 \pi)^d |\Sigma_X|}} e^{-\frac{1}{2} x^T \Sigma_X^{-1} x }\\ &= \frac{1}{\sqrt{(2 \pi)^d}} \sqrt{\frac{|\Sigma_Y|}{|\Sigma_N||\Sigma_X|}} \exp\left\{-\frac{1}{2}G\right\} , \end{align}\] where \[\begin{align} G = x^T \Sigma_x^{-1} x + y^T \Sigma_N^{-1} y + x^T \Sigma_N^{-1} x - 2 x^T \Sigma_N^{-1}y - y^T \Sigma_Y^{-1} y . \end{align}\]

In the scalar case, where \(d=1\) and \(\Sigma_N = \sigma_N^2\), \(\Sigma_X = \sigma_X^2\) and \(\Sigma_Y = \sigma_X^2 + \sigma_N^2\), we have: \[\begin{align} G &= \frac{x^2}{\sigma_X^2} + \frac{y^2}{\sigma_N^2} + \frac{x^2}{\sigma_N^2} - 2 \frac{xy}{\sigma_N^2} - \frac{y^2}{\sigma_y^2} \\ &= x^2 \left( \frac{1}{\sigma_X^2} + \frac{1}{\sigma_N^2} \right) + y^2 \left( \frac{1}{\sigma_N^2} - \frac{1}{\sigma_Y^2}\right) - \frac{2xy}{\sigma_N^2} \\ &= x^2 \left( \frac{\sigma_X^2 + \sigma_N^2}{\sigma_X^2\sigma_N^2} \right) + y^2 \left( \frac{\sigma_Y^2 - \sigma_N^2}{\sigma_N^2\sigma_Y^2}\right) - \frac{2xy}{\sigma_N^2} \\ &= \frac{1}{\sigma_N^2} \left[ x^2 \frac{\sigma_N^2 + \sigma_X^2}{\sigma_X^2} + y^2 \frac{\sigma_x^2}{\sigma_X^2 + \sigma_N^2} - 2xy \right] \\ &= \frac{1}{\sigma_N^2} \left[ x^2 k^2 + \frac{y^2}{k^2} - 2xy \right] \\ &= \frac{1}{\sigma_N^2} \left[ kx - \frac{y}{k} \right]^2 , \end{align}\] where \(k \triangleq \frac{\sigma_X^2 + \sigma_N^2}{\sigma_X^2} > 1\).
Then, in the scalar case the posterior density expression becomes \[\begin{align} \label{eq:posterior95closed95form95gaussian} p_{X|Y}(x|y) = \frac{1}{\sqrt{2 \pi}} \sqrt{\frac{\sigma_Y^2}{\sigma_N^2 \sigma_X^2}} e^{-\frac{1}{2 \sigma_N^2} \left[ kx - \frac{y}{k} \right]^2}. \end{align}\tag{48}\]

11 Appendix: Bottom-up Approach↩︎

In this section, we provide additional insights on the bottom-up approach proposed in Theorem 2. The idea is to design the objective function by starting from the desired discriminator’s output. When designing an objective function using the bottom-up approach, the choice of \(k(\cdot)\) is the fundamental starting point of the procedure, which is summarized in the following.

• \(k(\cdot)\) is the first DOF and must be chosen as a deterministic and invertible function in the domain \(x>0\). Some examples are \(k(x)=x\) and \(k(x)=1/(1+x)\). Then the posterior pdf must be expressed as the density ratio \(p_{XY}/p_Y\).

• \(g_1(\cdot)\) is an additional DOF and can be chosen as defined in Corollary 4.

• Substitute \(k(\cdot)\) and \(g_1(\cdot)\) in 11 .

• Compute the integral of 11 w.r.t. \(D\), which leads to \(\tilde{\mathcal{J}}(D)\).

• The integral in 10 represents the expectation computed over the supports \(\mathcal{T}_x\) and \(\mathcal{T}_y\).

This exact procedure is used to prove Theorem 3.
In the case of the KL-based objective function that we proposed, to ensure that 9 is a density for the case of discrete \(X\), we use the softmax as the last activation function of the discriminator. However, \(k(\cdot)\) is not the softmax function because if it were the softmax, then \(D^\diamond(\mathbf{x}, \mathbf{y}) = softmax(p_{X|Y}(\mathbf{x}|\mathbf{y}))\), where \(p_{X|Y}(\mathbf{x}|\mathbf{y})\) is the density (i.e., the output of the softmax will be \(p_{X|Y}(\mathbf{x}|\mathbf{y})\) itself, thus we do not need to apply additional transformations). Therefore, in the case of the KL divergence, \(k^{-1}(x)=x\).

12 Appendix: Additional Numerical Results↩︎

12.1 Image Datasets↩︎

We study the performance of the proposed objective functions for classification (listed in Appendix 9) for four image classification datasets. MNIST [39] comprises 10 classes with 60,000 images for training and 10,000 images for testing. Fashion MNIST has 10 classes with 60,000 images for training and 10,000 images for testing. CIFAR10 [41] has 10 classes with 50,000 images for training and 10,000 images for testing. CIFAR100 [41] has 100 classes with 50,000 images for training and 10,000 images for testing. Table ¿tbl:tab:classification95images95appendix? shows the classification accuracy of different network architectures trained by using the supervised versions of the objective functions listed in Table ¿tbl:tab:value32functions? and the one presented in Theorem 3 (see Appendix 9). The accuracies are in the form \(XX \pm YY\), where \(XX\) and \(YY\) represent the mean and standard deviation, respectively, obtained over multiple runs of the code. The tests are run by using various architectures: VGG [10], DLA² [35], ResNet18 [34], DenseNet [49], PreActResNet [43], MobileNetV2 [36]. Interestingly, from Tab. ¿tbl:tab:classification95images95appendix?, it can be observed that every network architecture suits some objective functions more than others. For instance, the PreActResNet attains the highest accuracy when trained with the GAN-based objective function (even if the difference in performance is just 0.2% different from the second-best one). Differently, the DenseNet performs optimally in CIFAR10 and CIFAR100 when trained with the KL-based objective function, even if the difference in accuracy w.r.t. the SL-based objective function is minimal. The MobileNetV2 obtains the best performance when trained with the SL-based divergence, and the difference with the second-best objective function is around \(2\%\), which is significant. The choice of the architecture often depends on the goal of the classification algorithm. For embedded systems, light architectures are used. Therefore, the MobileNetV2 is an option. In such a case, the SL divergence is the preferred choice for the network’s training.

The SL divergence has been proved to have favourable convergence conditions in Corollary 2. We numerically compare the convergence speed of the GAN and SL divergences on the CIFAR10 dataset in Figure 5, demonstrating the effectiveness of Corollary 2. The accuracy behavior is attained by averaging over multiple runs of the code. In Figure 5, the accuracy over the test dataset is showed, for each training epoch, with a semi-transparent color (blue and orange for the GAN and SL divergences, respectively). The vivid colors represent a moving average over \(5\) epochs, helping the clarity of the visualization. Quantitatively, we report in Tab. ¿tbl:tab:speed95accuracy? the difference between the accuracy obtained by training the network with the SL and GAN divergences. In detail, each column is identified by a number \(k\) and contains, for various discriminator architectures, the quantity \(A^k_{SL} - A^k_{GAN}\), where \(A^k_{f}\) is the average accuracy over multiple runs of the code for the selected \(f\)-divergence, in the epochs interval \([1, k]\). Each cell in Table ¿tbl:tab:speed95accuracy? contains a positive value, which shows the average faster convergence property of the SL divergence. Furthermore, the difference in speed convergence is more significant for the VGG training than for the ResNet18 and DLA architectures.

The speed of convergence and the upper-boundness of \(D_{SL}\) is (see Corollaries 2 and 7 in Appendix 10.10 and 10.9, respectively) confirm the utility of the SL divergence for double optimization problems. For instance, when the objective function of the learning algorithm is formulated as a max-max game. For example, in [21] the \(f\)-mutual information (which is estimated by formulating a maximization problem) is maximized for classification with noisy labels. Similarly, in [30] the authors maximize the JS-mutual information (where JS refers to the Jensen-Shannon divergence, which is equivalently used in this paper as GAN divergence) for representation learning applications (then the model is extended in [50]). In [51], the authors formulate a mutual information maximization algorithm to achieve the channel capacity in a data communication system. In [52], the authors maximize the JS-mutual information for the generation of talking faces.

12.2 Additional Decoding Tasks↩︎

12.2.1 AWGN↩︎

We analyze the decoding task in presence of additive white Gaussian noise (AWGN) in the communication channel. Let \(X\) be a \(d\)-dimensional binary vector, and \(N \sim \mathcal{N}(0,\Sigma_N)\) be Gaussian noise, with \(\Sigma_N\) diagonal. Let \(Y=X+N\) be the output of the communication channel. The SER behavior when varying the SNR is shown in Fig. 6 for each objective function analyzed. To compare the estimated SER, we visualize the SER achieved by the maxL decoder, which corresponds to the optimal decoder for an AWGN channel with uniform distribution \(p_X(\mathbf{x})\) [27]. The proposed SL divergence achieves the best performance and close to the optimal maxL decoder. In general, different objective functions perform better than \(\mathcal{J}_{KL}(D)\).

12.2.2 PAM with Non-Uniform Source↩︎

Similarly to Section 7, we consider a 4-PAM transmission. However, in this case we examine the case where the symbols transmitted do not have a uniform prior probability \(p_X(\mathbf{x})\). We define the alphabet of \(X\) to be \(\mathcal{A}_x = \{ x _1, x_2, x_3, x_4\}\) with probabilities \(P(x_1) = P(x_2) = P/2\) and \(P(x_3) = P(x_4) = (1-P)/2\), where \(P=0.05\). The SER for various values of SNR is reported in Fig. 7, where the discriminators trained with the supervised versions of the objective functions in Tab. ¿tbl:tab:value32functions? and in 12 (see Appendix 9) are compared with the maxL and MAP decoders. Although the maxL decoder is optimal for an AWGN channel, the extreme non-uniformity of the channel significantly impacts its performance. Differently, the optimal MAP decoder knows the distribution of \(p_X(\mathbf{x})\). The discriminator trained with the SL and GAN divergences achieves performance close to the optimal MAP decoder.

References↩︎