Contrastive learning (CL) is a remarkably successful method for learning useful image representations without labeled data. While numerous variants have been suggested, almost all of them follow the recipe suggested by [1]. For each training image, an augmentation is applied, and the goal of learning is to find a representation where two augmentations of the same image are close, while augmentations of different images are far away. Although conceptually simple, when applied to large scale datasets, these approaches have paved the way for image representations that can then be used to solve a large number of computer vision tasks without requiring additional representation learning (e.g. [2]).

In this paper, we seek to understand why contrastive learning works so well when it is applied to natural image datasets. A large number of papers have shown that variants of CL are closely related to spectral methods for unsupervised learning such as Laplacian Eigenmaps [4] and Spectral Clustering [5] (e.g. [6]–[8]). But as pointed out in [9], [10] without additional inductive biases, for finite datasets both CL and spectral methods can lead to trivial solutions that are not useful for any downstream tasks.

Specifically our paper was motivated by two aspects of the success of contrastive learning that we find mysterious: the fact that it works with simple augmentations and the fact that it works with simple images of noise [3].

The first motivation for our work is illustrated by the columns of figure 1 which show the augmentations that are typically used in state-of-the-art CL. These augmentations are extremely low-level. For example, the default configuration of [1] uses only three augmentations: random crop, color jitter and Gaussian blur. These augmentations seem to have nothing to do with object recognition and it is often possible to design handcrafted representations that will be invariant to them (e.g. converting an image to gray level and applying local contrast normalization will make it invariant to color jitter and techniques such as [11], [12] are provably invariant to certain types of random crops). Even though these handcrafted representations will be invariant to the augmentations, they are not particularly useful for object recognition (in fact removing color hurts recognition performance). Nevertheless, CL with these low-level augmentations learns representations where KNN accuracy approaches the performance of supervised learning.

A second motivation is illustrated by the rows of figure 1 which show that CL can learn high-quality visual representations from images of noise [3]. These include fractal noise (i.e. images with a random phase and a power spectrum that fall off like \(1/f^\alpha\)), images generated by StyleGAN [13] with random weights, and images of "dead leaves": superimposed simple shapes with random sizes and colors. Such images look nothing like real images and surely do not capture properties of 3D viewpoint and illuminations, and yet performing contrastive learning with these images yielded representations that gave remarkably high accuracy for recognition with real images. The authors of [3] summarized this intriguing effect by speculating that “vision may be simpler than we thought, and might not require huge data-driven systems to achieve adequate performance."

How, then, can we understand the success of CL in discovering useful image representations for downstream tasks? In this paper, we address this question by analytically computing the optimal representation in terms of the contrastive loss for a range of basic augmentations and any image dataset with stationary statistics. Specifically, we show that:

• For a range of simple augmentations and for any image dataset with stationary statistics, the optimal representation in terms of contrastive loss performs "partial whitening": a dimensionality reduction step that measures a subset of the frequencies, followed by a whitening step that makes the sensitivity to a given frequency inversely proportional to its expected power.

• The optimal representation can be computed by a simple CNN with one hidden layer, a pointwise nonlinearity, followed by global average pooling and a linear projection layer (figure 6).

• The optimal weights in such CNNs for more complicated augmentations are still sinusoids (see figure 2). The frequencies of the sinusoids and their weights can be computed using a simple “waterfilling” algorithm given the dataset’s expected power spectrum.

1 Contrastive Losses and their Optima↩︎

We start by briefly reviewing the InfoNCE loss [14], a popular method for contrastive learning. We denote by \(x\) the input image and by \(y(x)\) the representation of that image. For each image \(x\), we choose a random augmentation \(T\) and want to make the representation of \(x\) and \(Tx\) close to each other while pushing apart the representations of different images. This is done by minimizing the following loss: \[\label{original-loss-eq} L = E_{x,T} \left[ - \log \frac{e^{-t \|y(Tx) - y(x)\|^2}}{e^{-t \|y(x)-y(Tx)\|^2} + \sum_{x'} e^{-t \|y(x)-y(x')\|^2}} \right]\tag{1}\] where \(t\) is the inverse temperature parameter and \(x'\) are other images in the training set. We use here a formulation based on the \(\ell_2\) distance between representations. When the embeddings are constrained to lie on the unit sphere, the measure of similarity between embeddings can equivalently be computed using the dot product [1]. For mathematical convenience, we do not constrain the embeddings to lie exactly on the unit sphere (i.e. \(\|y\|=1\)) but rather constrain the representation so that the average squared \(\ell_2\) norm of the centered embeddings is \(1\): \(E[\|y-\mu_y\|^2]=1\). Note that since the loss in equation 1 depends only on pairwise Euclidean distance, it is invariant to a translation of the embeddings so ideally the constraint should also be invariant to such a translation.

[15] pointed out that the InfoNCE loss can be seen as the sum of two losses: an alignment loss which measures the average distance between an image and its augmentation and a uniformity loss which measures the extent to which the representation of all images is uniformly distributed on the unit sphere.

Alternative approaches to CL replace the uniformity loss by other losses that prevent trivial collapse of the representation without sacrificing performance (e.g. [6], [7], [16]). In this paper, we use an alternative uniformity loss which agrees with the InfoNCE uniformity when \(y\) is Gaussian and is more amenable to analysis.

Previous work has reported that representations learned by CL are approximately Gaussian [17] and we have found that training with the loss \(L_{\mathrm{GUPA}}\) gives almost identical results as training with the InfoNCE loss (appendix section 8). Similar results have been previously reported with other alternatives to InfoNCE (e.g. [7], [16]).

Similar to the InfoNCE loss (equation 1 ), \(L_{\mathrm{GUPA}}\) (equation 2 ) can be written as the sum of an alignment loss and a (Gaussian) uniformity loss. The Gaussian uniformity loss encourages representations whose covariance matrix is white (see section [whitening-sec] in appendix for the proof):

1.1 Optimal Weights↩︎

Having simplified the loss, we show how to find the globally optimal weights when training only the last layer of a neural network. We assume that \(y(x)=W^T \phi(x)\) where \(\phi(x)\) are the neurons in the penultimate layer of a neural network and we assume that \(\phi(x)\) is fixed. Although this scenario is quite far from the use of CL in practice (in fact the whole point of contrastive learning is to learn the representation \(\phi\)), it is easiest to analyze and as we will show, the analysis for this scenario can be applied to more realistic settings.

Proof. We rewrite the loss \(L_{\mathrm{GUPA}}\) (equation 2 ) as a function of \(W\). The alignment term becomes \(\mathop{\mathrm{Tr}}(W^T B W)\) and the covariance of \(y\) becomes \(W^T \Sigma W\). Using Lagrange multipliers, it can be shown that at the optimum \(W\) must satisfy \(B W \Lambda = \Sigma W\) , where \(\Lambda\) is a diagonal matrix, hence the columns of \(W^*\) are generalized eigenvectors of \((B,\Sigma)\) . This means that the optimal \(W\) must satisfy \(W^*=V diag(\alpha)\) where \(V\) is the matrix of generalized eigenvectors of \((B,\Sigma)\) and \(\alpha\) a scaling of the columns. Substituting this form of \(W^*\) into equation 2 gives a loss that only depends on \(P_k=\alpha_k^2\):

\[\label{loss-alpha-paper} L(\{P_k\})= \sum_k P_k \lambda_k - \epsilon \sum_k \log (P_k+\epsilon) - \epsilon^2 \sum_k \frac{P_k}{P_k+\epsilon}\tag{4}\]

This is analogous to power allocation in communication systems. Each channel can only get a positive amount of power allocated (\(P_k \geq 0\)) and since \(E(\|y-\mu_y\|^2)=1\) the total power allocation is constrained \(\sum_k P_k \leq 1\). The benefit of adding power to a particular channel decreases as a function of \(P_k\) and is strictly convex (the second derivative of equation 4 is always positive). This means that the globally optimal solution for \(P_k\) can be found using a variety of “Waterfilling” algorithms [18], including algorithm 3 (see appendix section 10 for detailed proof) ◻

A direct consequence of theorem 3 and algorithm 3 is that the optimal projection matrix \(W^*\) will be low-rank: typically the algorithm will terminate when only a subset of the channels have been allocated nonzero power. This has indeed been previously observed with CL [19].

Theorem 3 is similar to derivations that have appeared in previous papers about contrastive losses (e.g. [6]): once the uniformity loss is simplified, the loss with respect to linear networks can be optimized using spectral methods. However, such theorems by themselves do not tell us what the networks will learn specifically in the case of natural images. We now show how to use the statistics of natural images to predict the optimal weights.

2 Natural Image Statistics↩︎

Perhaps the simplest property that we expect large and varied image datasets to satisfy is that the statistics should be translation invariant. As written in [20] “After all, the covariance of two neighbouring pixels is not likely to be any different depending on whether they are on the left or the right side of the image”.

A second prominent property of natural images is that their expected power spectrum decreases with frequency [21], [22]. The top left plot of figure 4 shows the expected power spectrum for CIFAR10. The expected power spectrum is high at the center of the plot (corresponding to low frequencies) and decreases with distance to the origin (high spatial frequencies).

The property of translation invariance which is often referred to as stationarity leads to highly predictable properties of the Discrete Fourier Transform (DFT) of image datasets. To formalize these properties, we assume that the “image” is one dimensional and denote it by \(x[n]\), while \(x^F[k]\) is the Discrete Fourier Transform of \(x\).

Although theorem 4 only holds asymptotically (as the size of the image goes to infinity), we have found that it holds approximately for real image datasets. Figure 4 shows the histogram of particular DFT coefficients over all \(50,000\) images in the CIFAR10 dataset. For most coefficients, the histogram is indeed highly Gaussian and this implies that most pairs of coefficients are independent. Figure 4 shows the covariance matrix between 32 randomly chosen squared DFT coefficients divided by their expectation. Indeed the covariance matrix is close to diagonal as predicted by the theorem (see also [24] for a direct proof). The bottom of figure 4 shows the same plots for fractal noise. Even though CIFAR10 images and fractal noise images are very different, the covariance of their squared DFT coefficients are very similar due to the property of stationarity. We now show how this allows us to predict optimal representations.

3 Optimal Representations for Contrastive Learning with Natural Images↩︎

Consider the augmentations shown in figure 5. These augmentations are similar (but not identical) to the default augmentations used in [1]. The first augmentation is a random crop with circular boundary conditions: we take a random crop from the periodic extension of the original image. The second augmentation is similar to color jitter: it randomly jitters the brightness and the contrast of the image. The last augmentation applies an ideal low-pass filter to the image (the blur kernel zeros out all frequencies above some cut-off frequency and leaves all other frequencies unchanged).

For these augmentations, we show how to define a representation that is globally optimal in terms of the contrastive loss \(L_{\mathrm{GUPA}}\).

Note that theorem 5 holds for any stationary signal: whether we train the representation with white noise, or filtered noise, or real images, we can find an optimal representation that is the same up to a scalar rescaling of the outputs. Note also that the optimal representation is not unique: the optimality holds for any set of frequencies and multiplying \(y^*\) by an arbitrary orthogonal matrix will also yield an optimal representation.

The optimal representation of theorem 5 can be computed by a very simple neural network architecture (figure 6): a convolution layer with sinusoidal filters, followed by squaring, global average pool, and a linear projection layer that performs the whitening. For this class of augmentations, such a simple architecture is sufficient to compute a globally optimal representation in terms of \(L_{\mathrm{GUPA}}\) for any stationary signal and there is no need for deeper neural networks.

In the appendix (theorem 10) we show that an alternative optimal representation can also be computed when we replace the squaring nonlinearity with a ReLU in the same CNN, yielding:

\[\label{partial-whitening-eq-relu} y^*_i(x) = \frac{1}{\sqrt{g(k_i)}} |x^F[k_i]|\tag{6}\]

Since the optimality of the representations in equations 5 and  6 hold for any choice of frequencies, there is no reason to expect them to be particularly useful for recognition. To learn useful features we need to make the augmentations more difficult.

3.1 Optimal CNN Weights with More Augmentations↩︎

To obtain a set of augmentations that is amenable to analysis but more difficult, we replace the random crop with a circular crop plus noise. In the appendix (section 12), we show that once we add noise to circular crops, their behavior in the Fourier domain is similar to that of standard random crops. We also allow the blur kernel to be any filter.

Unlike the set of augmentations in figure 5, for these augmentations a simple CNN cannot compute a representation that is completely invariant. This is perhaps easiest to see in the case of adding noise: in order to be completely invariant to this augmentation we need to do a perfect denoising which is impossible with a simple CNN with one hidden layer. Although the simple CNN of figure 6 can no longer compute a globally optimal representation, we now show that the optimal weights still perform partial whitening.

Similar to the case of the easy augmentations in figure 5, with the harder augmentations the optimal CNN will still perform partial whitening. The representation will be sensitive to a subset of the spatial frequencies, and the variance of the representation will be the same in all directions. We can make the whitening more explicit by computing the sensitivity of the CNN with optimal weights to a particular frequency \(k\). It is defined as the \(\ell_1\) norm of \(y(x)\) when \(x\) is an image of a pure sinusoid of frequency \(k\). Since the output of the optimal CNN is simply \(W^{eff} \Phi(x)\) the sensitivity can be computed as: \(S[k]= \sum_i |W^{eff}(i,k)|\).

Figure 7 shows examples of such sensitivities for different training sets when the augmentations are cyclic crops plus noise. The sensitivity is shown as a gray level image in frequency space, with the DC frequency corresponding to the central pixel. For all the datasets shown, the sensitivity of the representation is concentrated on mid-range frequencies: the preference towards whitened representations favors frequencies that have low expected power but the robustness to noise favors frequencies that have high expected power. For image datasets whose expected power falls off with distance from the origin, this leads to representations where most of the sensitivity lies on a ring in frequency space while for CIFAR10 the sensitivity lies on a diamond in frequency space. The bottom of figure 7 shows the sensitivity multiplied by the expected power. Indeed it can be seen that the CNNs with optimal weights are performing partial whitening: they are only sensitive to a subset of spatial frequencies and their sensitivity to those frequencies is chosen so that the response will have the same variance in all directions.

4 Experiments↩︎

To see the extent to which this theory can explain what CNNs learn in practice, we trained single-layer CNNs (the architecture in figure 6) by minimizing the \(L_{\mathrm{GUPA}}\) contrastive loss (equation 2 ) using PyTorch.

The CNNs had 256 \(11 \times 11\) filters in the first convolutional layer (with "valid" boundary conditions), followed by a nonlinearity \(\rho\), global average pooling (GAP), and then a linear projection to 32 dimensions. Specifically we trained on six synthetic datasets (fractal noise with different power laws and dead leaves) and three real datasets: CIFAR10, CIFAR100 and ImageNet. For consistency with the theory, most of the experiments were performed after the images were converted to gray scale and only one set of experiments used color images. We repeated the experiments with two nonlinearities \(\rho(z)=z^2\) , \(\rho(z)=ReLU(z)\).

In the first set of experiments, the two augmentations of an image were random crops of equal size (\(84 \%\) of the original size). This means that the two augmentations are related by a crop translation. Figure 2 shows the learned weights for this augmentation in the first layer for CNNs with a quadratic nonlinearity (see figure 17 in the appendix for the weights in all layers). As predicted by our theory, in all cases the learned weights converged to sinusoids. Figure 8 shows similar results when we use a ReLU nonlinearity. For the different datasets, the weights in the first layer converge to sinusoids and the overall sensitivity of the network to a particular frequency is consistent with that predicted by the theory (theorem 10). In particular, the bottom row of figure 8 shows that the networks are learning partial whitening (compare to figure 7).

Figure 9 shows the learned filters with different augmentations. As expected from the proof of theorem 6 the network learns filters that are sinusoids (or sums of a small number of sinusoids) for a wide range of augmentations (including the "no aug." augmentation when the two augmented images are identical and the "mixed" augmentation which combines random cropping, contrast inversion and blur). Since the uniformity loss is always minimized by measuring individual DFT coefficients, as long as the alignment loss is low with simple functions of these coefficients, the network will learn sinusoids. In the case of horizontal flip, the network chooses filters that are localized in frequency that are also horizontally symmetric.

A notable exception is when we use SOTA augmentations taken from [27]. Here the augmentations are horizontal flip, random crop when the size of the crop and aspect ratio are also randomized and nonlinear color jitter. Although these augmentations do not seem very different from the others, they lead to much more localized filters, presumably because such filters are better at minimizing the alignment loss.

In the last set of experiments, we trained the CNNs with color images. Figure 10 shows the filters that are learned with color images for different augmentations. As shown in section 20 in the appendix, for color images the theory still holds if the color channels are decorrelated and for natural images decorrelation yields a new set of three channels that are approximately a gray level channel and two color-opponent channels (red vs. green and blue vs. yellow) [25]. Indeed it can be seen that the CNNs learn sinusoids in their first layer, but each sinusoid is only sensitive to one of the decorrelated channels

We also computed the recognition accuracy on CIFAR10 when a KNN classifier is used based on the representation learned by the network (figure 11). Even with this simple architecture and simple augmentations, CL yields representations that improve the recognition accuracy considerably, relative to the original weights. On the other hand, most of this improvement is due to the partial whitening. To show this, we also computed the accuracy of linear partial whitening [26] (see appendix for details): we compute the PCA of the images, project each image onto the 32 top PCA directions, and then perform partial whitening by dividing each PCA coefficient by the square root of its eigenvalue plus a constant \(\lambda=0.1\). As can be seen in the figure, this simple representation (which is not invariant to most transformations) gives comparable performance as the trained CNN with ReLU for most augmentations. Again a notable exception are the SOTA augmentations where the learned localized filters capture more than just partial whitening.

Figure 11 (left) measures how well synthetic images can be used to learn representations that are useful for CIFAR10. As predicted by our theory, when training on images that are not CIFAR10 and testing on CIFAR10, recognition accuracy improves when the expected power spectrum is closer to that of CIFAR10.

5 Limitations and Extensions↩︎

The main limitation of our work is that the theoretical analysis focuses on very simple CNN architectures and augmentations and the performance of such architectures is far from state-of-the-art. We chose to focus this paper on a basic setting for which analysis is easier, but the theory can be extended to more augmentations and deeper networks.

A second limitation of our work is that the theoretical analysis uses our \(L_{\mathrm{GUPA}}\) contrastive loss rather than the InfoNCE loss and the two losses are only identical when \(y\) is Gaussian. Our theoretical analysis can be straightforwardly extended to other contrastive losses that are based on the covariance of the embedding [7], [28], [29]. Empirically we have found that the same representations are learned with the InfoNCE loss and our \(L_{\mathrm{GUPA}}\) loss (see figure 13 and section [nt-section] in the appendix), consistent with other reports that contrastive learning leads to Gaussian distributions [17], [30].

As discussed after the proof of theorem 5, the minimum of the contrastive loss is not unique. Empirically, we have found that the dynamics of gradient descent influence which minimum is found by a CNN (see section [iterations-sec] in the appendix). It may be possible to relate this effect to theoretical investigations of the implicit bias of gradient descent in deep models (e.g. [31]).

Our theory focused on stationary datasets but many datasets are non-stationary (e.g. images of centered human faces such as CelebA). We have found that when we trained CNNs with GAP on non-stationary datasets, the learned filters are still sinusoids (see appendix). Apparently, the global average pool prevents the CNN from taking advantage of the non-stationarity. We believe that the theory can therefore be extended to deal with non-stationary datasets as long as the architecture is constrained.

6 Conclusion↩︎

The main motivation for our paper was to understand how contrastive learning can learn useful image representations using simple augmentations and simple images. In order to address this question, we have shown how to analytically compute the optimal representation in terms of a particular contrastive loss \(L_{\mathrm{GUPA}}\) for simple augmentations and the optimal weights in a simple CNN under additional augmentations. For this setting, we showed that the assumption of stationarity allowed us to obtain quantitative predictions for the optimal weights given only the expected power spectrum of the dataset.

Perhaps the short answer to the question that we posed in the beginning of this section is that while learning can be successful using simple augmentations and images, the choice of which augmentations and images are used can greatly influence the utility of the representation. While many augmentations will cause the network to learn partial whitening, the details of the augmentations will change which frequencies are the ones that are measured and these small details can have a large influence on recognition performance. Additionally, the power spectrum of the synthetic images should closely match the power spectrum of real images so that the network will learn to measure the same frequencies in both cases.

Partial whitening has previously been suggested as a useful representation of images (e.g. [20], [32], [33]). It reduces the redundancy of the representation while also providing some robustness to noise. Our results suggest that by using particular augmentations in CL we are not necessarily teaching the CNN to be invariant to those transformations but rather guiding the network towards a representation that has been previously suggested by domain experts.

Acknowledgments↩︎

We thank Shaden N. Alshammari for extremely useful comments on an earlier version of this manuscript. YW is supported by the ISF and the Gatsby Foundation. AT is supported by ONR MURI. YW thanks Bill Freeman and Franny Osman for their generous hospitality which was crucial in getting this research started.

Impact Statement↩︎

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Appendix↩︎

7 Comparing the InfoNCE and \(L_{\mathrm{GUPA}}\) contrastive losses↩︎

Figure 12 compares training a ResNet18 with the InfoNCE and \(L_{\mathrm{GUPA}}\) contrastive losses. The two losses are identical when \(y(x)\) is Gaussian. The four subfigures show the recognition accuracy when training on CIFAR10 with the two different losses as a function of epoch. Whether we use KNN or linear readout to do the classification, the performance is nearly identical.

Figure 13 shows the kernels learned when training the one layer convolutional architecture used in this paper (figure 6) using the InfoNCE loss [1]. Figure 14 compares the recognition accuracy for different augmentations, the InfoNCE loss and \(L_{\mathrm{GUPA}}\) when using the representation and projection features.

8 From Uniformity to Cross Entropy↩︎

9 Properties of the \(L_{\mathrm{GUPA}}\) Contrastive Loss↩︎

As is the case for other contrastive losses, \(L_{\mathrm{GUPA}}\) consists of two terms. An alignment term which pushes the embedding of an image and its augmentation towards each other, and a uniformity term which pushes apart embeddings of different images. Explicitly, the second term is a Gaussian uniformity loss: \[\label{Luniformity2-eq-app} L_{GU}= - \frac{1}{2} \log \det (\Sigma_y+ \epsilon I) -\frac{1}{2} \mathop{\mathrm{Tr}}( (\epsilon I + \Sigma_y)^{-1} \Sigma_y).\tag{9}\] To see that this term pushes apart embeddings of different images, we take advantage of the fact that the trace and determinant only depend on the eigenvalues of a matrix and rewrite \(L_{GU}\) as a function of \(\sigma_i^2\), the eigenvalues of \(\Sigma_y\) (these are also the variances of \(y\) along the principal directions of the data): \[L_{GU} = -\frac{1}{2} \sum_i \log (\sigma^2_i+\epsilon) - \frac{1}{2} \sum_i \frac{\sigma^2_i}{\sigma^2_i+\epsilon} \label{simpler-lu-eq}\tag{10}\] We now see that the loss decreases monotonically as we increase the variance along the principal dimensions of \(y\). Thus by pushing the embeddings of different images away from each other, we can arbitrarily decrease \(L_{GU}\). However, when we constrain the expected \(\ell_2\) norm of the centered embeddings, we can no longer arbitrarily decrease \(L_{GU}\). For such a setting, the optimum is obtained when the representation is white.

10 Optimal Weights in the Last Layer↩︎

11 Optimal Representation with ReLU nonlinearity↩︎

Note that when we use ReLU rather than squaring, and the filters are sinusoids, the feature vector \(\phi(x)\) measures the amplitude rather than the power in individual frequencies. To see this, we denote by \(h_c\) the pure sinusoid \(h_c[n]=\frac{1}{2}\cos (2 \pi \frac{k_c}{N} n)\) and by \(r_c = x \star h_c\) (where the convolution is circular) then: \[r_c^F[k]= x^F[k] h_c^F[k]\] with: \[h_c^F[k]= \left\{ \begin{array}{cc} 0 & k \neq k_c, N-k_c \\ 1 & otherwise \end{array} \right.\] and since \(x[n]\) is a real signal, \(|x^F[k_c]|=|x^F[N-k_c]|\) and \(\beta=angle(x^F[k_c])=-angle(x^F[N-k_c])\). So we can write: \[r_c^F[k] = |x^F[k_c]| e^{i \beta k} h_c^F[k]\] and taking the IDFT of both sides gives: \[r_c= |x^F[k_c]| (T h_c)\] where \(T\) is a circular translation operator. This means that after ReLU and global average pooling, the result is proportional to \(|x^F[k_c]|\). This implies that the output of the CNN can be written as: \[y_i \propto |x^F[k_i] |\] and to satisfy whitening we need to divide \(y_i\) by its standard deviation. Since \(x^F[k_i]\) is a complex Gaussian, its absolute value is a Rayleigh random variable whose standard deviation is proportional to the standard deviation of \(x^F[k_i]\) which is in turn equal to \(\sqrt{g(k_i)}\).

12 Cyclic Translations and Crop Translations↩︎

To formalize the effect of crop translations on the DFT of a signal, it is easier to work in the continuous domain. We assume that there are two signals: \[\begin{align} x_1(t) &=& W(t) x(t) \\ x_2(t) &=& W(t) x(t - \Delta) \end{align}\]

Where \(W(t)\) is a window function that defines the crop. We are interested in the difference between the power spectrum of \(x_1\), \(|X_1(\omega)|^2\) and the power spectrum of \(x_2\), \(|X_2(\omega)|^2\).

We can also write: \[\begin{align} x_2(t) &=& W(t-\Delta) x(t - \Delta) + x(t-\Delta) (W(t)-W(t-\Delta)) \\ &=& W(t-\Delta) x(t-\Delta) + \eta(t) \end{align}\] where we have defined \(\eta(t)=x(t-\Delta) (W(t)-W(t-\Delta))\).

Taking the Fourier Transform of both sides gives: \[X_2^F(\omega)= e ^{i \omega \Delta} X_1^F(\omega) + \eta^F(\omega)\] Thus the Fourier transform of the second crop \(x_2\) is given by a phase shift of the Fourier transform of the first crop plus a perturbation \(\eta^F(\omega)\) whose magnitude can be bounded based on the size of the translation. This is similar to the case of cyclic translation plus noise, except the perturbation \(\eta^F(\omega)\) is not independent of \(x\).

Figure 15 (top) shows scatter plots of the squared DFT of two different crop translations for three different CIFAR10 images. The bottom figures show the scatter plots of the squared DFT when the images are related by cyclic translation plus noise. In both cases, power spectra are highly correlated and the differences between the two power spectra is larger for frequencies that have large power.

13 Optimal Weights for simple CNN with cyclic crop plus noise, linear jitter and blur↩︎

The results in figure 7 were computed by running algorithm 3 for the case of cyclic translations plus noise \(\lambda_k = \frac{8 \sigma^2 g[k] + 4 \sigma^4}{g^2[k]}\). The expected power spectrum \(g[k]\) was computed numerically for CIFAR10 using a Hann window and for the fractal images, we used the analytical form of the expected power spectrum (e.g. \(g[k] \propto \frac{1}{\\k\|^{\alpha}}\)). The value of \(\sigma\) and the temperature were the same for all experiments (\(\sigma=0.0014, t=1\)). Different values give qualitatively similar results but the distance of the rings from the origin changes.

14 Computing sensitivities of CNNs↩︎

We define the sensitivity of a representation \(y\) to frequency \(k\) as the \(\ell_1\) norm of \(y(x)\) when \(x\) is an image of a pure sinusoid of frequency \(k\). For the simple CNN we are considering this can be computed directly from the weights. Recall that: \[\begin{align} y_i(x) &=& \sum_k |x^F[k]|^2 \left( \frac{1}{N} \sum_c w_{ic} |h^F_c[k]|^2 \right) \\ &=& \sum_k W^{eff}(i,k) |x^F[k]|^2 \end{align}\] We define the sensitivity as \(S(k)=\sum_i |W^{eff}(i,k)|\). All the results shown in the paper use this definition of sensitivity, but we have found the results to be nearly identical when we replace the \(\ell_1\) norm with the \(\ell_2\) norm, or \(\tilde{S}(k)= \sqrt{\sum_i |W^{eff}(i,k)|^2}\).

15 Partial Whitening using PCA↩︎

We follow the code of [26]. Let \(C,\mu\) be the covariance and mean of the dataset, and let \(V,\Lambda\) be the eigenvectors and eigenvalues of \(C\). Let \(V_K,\Lambda_K\) be the first \(K\) columns of \(V,\Sigma\) and let \(I_K\) be the first \(K\) columns of the identity matrix. We define the matrix \(W\) using: \[W= V_K \left( \sqrt{\Lambda_K+\lambda I_K}\right) ^{-1}.\]

The representation of an input \(x\) is now given by \(y(x)=W^T(x-\mu)\). It is easy to see that when \(\lambda<<\Lambda(j,j)\) the representation \(y\) has unit variance in direction \(v_j\) and when \(\lambda >> \Lambda(j,j)\) the variance of \(y\) in direction \(v_j\) goes to zero. Thus this linear mapping performs partial whitening.

When performing KNN classification, we followed the code of [26] in augmenting each training example with its horizontal flip. This gave an accuracy of 47%. Without the horizontal flip, the accuracy was 45%.

16 Experiments with different augmentations↩︎

Figure 18 shows the learned filters with different augmentations as well as the frequency sensitivity of the learned networks. As expected from the proof of theorem 6 the network learns filters that are highly localized in frequency space for a wide range of augmentations. Since the uniformity loss is always minimized by measuring individual DFT coefficients, as long as the alignment loss is low with simple functions of these coefficients, the network will learn sinusoids. In the case of horizontal flip, the network chooses filters that are localized in frequency that are also horizontally symmetric. The sensitivity profiles reveal that for all the augmentations the network is learning to perform partial whitening: the sensitivity increases with frequency but frequencies that have low expected power are ignored. For the squaring nonlinearity, the sensitivity profiles look like diamonds, but the size of the diamonds depends on the augmentation as well as details of the learning procedure (e.g. number of iterations, weight decay). For ReLU nonlinearity the pictures are noisier, but agree with the general trend.

17 Experiments with nonstationary datasets↩︎

Figures 19,20 show experiments with a highly nonstationary dataset, CelebA. Even though the statistics are highly location dependent (see left column of figure 19), the DFT coefficients are still approximately Gaussian and the covariance of squared DFT coefficients is still approximately diagonal. As a result, the CNN still learns sinusoidal filters.

18 Visualizing all learned filters↩︎

In the main paper we display the 30 filters with highest variance. Here we show all 256 filters, both in the spatial domain and the Fourier domain. We also show the best fitting sinusoid for each filter and the difference between them. Figures 21,22 show all 256 kernels when trained on ImageNet with cyclic translations (figure 21) and crop translations (figure 22). Almost all filters are sinusoids (two peaks in the Fourier domain) and the difference between the best fitting sinusoid and the learned filter is mostly noise.

Figures 23,24 show all 256 filters when the augmentations are horizontal flip. Now the kernels are still highly localized in Fourier space but sometimes they are a combination of two sinusoids (four peaks in Fourier space) rather than a single sinusoid. In these cases, the difference to the sinusoidal fit (bottom), is another sinusoid but with a horizontally flipped frequency.

19 Non-Uniqueness of the Optimal Solution↩︎

As we discussed after the proof of theorem 5, the minimum of the contrastive loss is not unique. For the augmentations in figure 5, measuring any set of \(K\) frequencies and then whitening will yield an optimal representation. More generally, if \(y^*(x)\) is an optimal representation, then \(Q y^*(x)\) is also an optimal representation for any orthogonal matrix \(Q\).

Empirically, we have found that the dynamics of gradient descent influence which minimum is found by a CNN. In the case of theorem 5, where the optimal representation is to measure a set of \(K\) frequencies, the chosen frequencies depend on the number of iterations. As we increase the number of iterations, the network learns higher-frequency sinusoids. Figure 25 shows the filters as a function of iterations when training using the crop translation augmentation, and figure 26 shows the evolution of the loss and the recognition performance. Note that even though the loss continues to decrease, recognition accuracy peaks at a small number of iterations for this augmentation.

A second source of non-uniqueness comes from the architecture in figure 6. The exact same representation can be computed using many different settings of the weights.

Recall that for this architecture, the output can be written as \(y(x)=W^{eff}\Phi(x)\) where the set of features \(\Phi(x)\) is the squared DFT of \(x\), \(|x^F[k]|^2\), and by equation 7 , \(W^{eff}\) is given by: \[\label{equation:matrix-product} W^{eff}= W H\tag{17}\] Here \(W\) is a \(K \times C\) matrix that represents the projection layer weights and \(H\) is a \(C \times N\) matrix where each row gives the squared DFT of filter \(h_c\). Note that \(H\) is by definition non-negative.

Thus the output is given by \(y(x)=W H \Phi(x)\) and there are many different settings of \((W,H)\) that would give the same \(W^{eff}\) and the same representation. In particular, when the number of channels is large we can compute the optimal representation by freezing the projection layer weights and only optimizing the filters, or by freezing the filters and only optimizing the projection layer weights. We have found that when the variance of the random initialization of the projection layer weights is large, gradient descent will barely change the projection weights and will minimize the loss by changing the filters in the first layer. For such a setting, when the projection matrix is an orthogonal matrix, we can prove that the optimal loss can only be obtained when the filters are sinusoids.

The difference between theorem 6 and theorem 12 is in the uniqueness of the solution. Theorem 6 shows that the global minimum can be obtained when the filters in the first layer are sinusoids but the solution is not unique. On the other hand, theorem 12 shows that when the weights in the last layer are frozen and equal to an orthogonal matrix, then the minimum can only be obtained when the filters are sinusoids.

20 Extensions to Color Images↩︎

The theoretical results in the paper are for single channel signals (i.e. gray level images). The results can be straightforwardly extended to multichannel signals (i.e. color images) in which each channel is stationary and independent of the other channels. For such signals, the DFT coefficients of each channel are asymptotically Gaussian and independent so that CNNs trained with contrastive loss will learn to measure sinusoids in each channel.

For natural images, the three standard color channels (R,G,B) are not independent but it is known that one can apply a \(3 \times 3\) linear transformation that makes the channels uncorrelated. Specifically these uncorrelated channels are "one corresponding to simple changes in radiance and two that are reminiscent of the blue–yellow and red–green chromatic-opponent mechanisms found in the primate visual system." [25]. Once we decorrelate the color channels, the DFT coefficients of each color channel will be uncorrelated for any stationary signal, and asymptotically Gaussian (by theorem 4). This implies that DFT coefficients of these decorrelated channels will also be asymptotically pairwise independent. Thus the theory predicts that if we train CNNs with contrastive losses on color images, they should learn sinusoids in one of these three uncorrelated channels. Figure 10 shows that this is indeed the case for ImageNet and figure 27 shows the same for CIFAR10.

References↩︎