1 INTRODUCTION↩︎

Identifying explanatory factors from raw sensory data is a fundamental challenge in machine learning. Originated from the problem of blind source separation, independent component analysis (ICA) and its nonlinear variations [1]–[3] aim to identify independent latent factors that improve downstream interpretability, controllability, fairness, and sample efficiency [4]–[6]. Unfortunately, directly learning these factors from nonlinear mixtures is unidentifiable without additional structural constraints or weak supervision [6]–[8].

Canonical Correlation Analysis (CCA) [9] is another classic method, which aims to learn representations by maximizing the cross-correlation between paired views. While nonlinear CCA and its extensions [10]–[16] demonstrate empirical performance comparable to strong contrastive learning [17], [18] baselines, the theoretical understanding of nonlinear CCA identifiability lags behind. Contrastive learning now offers robust block- and factor-level identifiability guarantees by leveraging specific data or model assumptions [19]–[26]. In contrast, existing analyses for nonlinear CCA either assume restricted post-nonlinear mixing [27] or only guarantee recovery up to broad, arbitrary invertible transformations [28], [29]. This discrepancy raises a fundamental open question: Under what conditions can nonlinear CCA provide strictly stronger identifiability guarantees, up to simpler transformations that are comparable to those of contrastive learning?

This work answers the above question by revisiting the classical theory of bivariate distributions [30], [31]. We establish the sufficient conditions under which nonlinear CCA provably recovers ground-truth latent variables up to an affine transformation. Crucially, we formalize the role of representation whitening in ensuring the boundedness and well-conditioning strictly necessary for this identifiability. We empirically validate these theoretical guarantees and systematically ablate our core assumptions to characterize failure modes when these conditions are violated. We summarize our contributions as follows:

1. We characterize the invariance of the population CCA objective and its maximizers under reparameterization via composition with the generators, enabling direct analysis in the latent source space (1).

2. We extend the classical theory of bivariate distributions to representation learning and derive affine identifiability results for nonlinear CCA in the population setting across a broad class of latent distributional priors (1).

3. We analyze ridge-regularized empirical CCA and show its convergence to the population counterpart, linking the identifiability results to the finite-sample regime (2).

4. We assess these theoretical predictions across candidate latent distributions using both a fully controlled synthetic dataset and a rendered 3D image dataset.

5. We study the failure modes of nonlinear CCA under partial violations of the core assumptions through systematic ablations.

2 RELATED WORK↩︎

2.0.0.1 Disentangled Representation Learning.

Disentangled representation learning aims to isolate independent explanatory factors of variation, originating from blind source separation [1], [32]. Because disentanglement is provably impossible for i.i.d.nonlinear mixtures without structural assumptions [6], [7], research has focused on exploiting suitable inductive biases to invert nonlinear generative processes. One prevalent approach leverages shared or co-observed variables, such as temporal dynamics [24], nonstationarity [33], auxiliary labels [3], [34], weak supervision [8], [35], and multi-view observations [19], [22], [27]. A complementary direction restricts the model class or source distribution, enforcing structural constraints like mechanism sparsity [26], conformal mappings [36], or specific latent priors (e.g., exponential families [3], causal structures [37], or energy-based models [38]). Our work aligns with the latter by imposing latent distributional priors within a two-view setting.

2.0.0.2 Nonlinear CCA.

Nonlinear CCA extends classical CCA to arbitrary nonlinear mappings via kernels [11] or deep neural networks [12], [39]. While these methods excel at maximizing cross-view correlations, achieving strictly identifiable representations was not their original objective. Recent efforts have explored the identifiability of nonlinear CCA-based models [27]–[29], [40]. However, existing theoretical guarantees of factor-level identifiability rely heavily on restrictive post-nonlinear mixing assumptions [27] or only guarantee recovery up to broad equivalence classes, such as arbitrary invertible transformations [28], [29]. In contrast, modern contrastive learning frameworks successfully invert the data-generating process to recover latent variables up to strict affine or permutation ambiguities [21]–[23]. A detailed theoretical comparison between the identifiability frameworks of nonlinear CCA and contrastive learning is provided in 7. An equivalently rigorous, affine identifiability theory for nonlinear CCA remains absent.

We bridge this gap by revisiting classical bivariate distribution theory, which establishes that linear mappings uniquely maximize canonical correlations under Gaussian priors [30], a result later generalized to broader distribution families [31]. Building on these foundations, we extend this classical theory to modern representation learning. By proving the affine identifiability of nonlinear CCA under a family of latent distributional priors, we cement CCA as a theoretically principled alternative to contrastive learning for exact latent recovery, moving beyond mere view alignment.

3 CANONICAL CORRELATION ANALYSIS FOR DISENTANGLED REPRESENTATION↩︎

3.0.0.1 Notations.

Throughout this work, scalars are denoted by plain letters, (random) vectors or vector-valued functions by bold lowercase, matrices by bold uppercase symbols, and sets/spaces by calligraphic letters.

3.1 Data Generating Process↩︎

We consider a latent pair \((\mathbf{s},\mathbf{s}') \in \mathcal{S}\times\mathcal{S}\), where \(\mathcal{S}\subseteq\mathbb{R}^{d_{\mathcal{S}}}\) and \(d_{\mathcal{S}}\ge 2\). For the main population identifiability theorem, we assume that \((\mathbf{s},\mathbf{s}')\) is a non-degenerate jointly Gaussian pair and parameterize its dependence through its canonical correlations, which are the natural quantities for CCA. Our primary analysis focuses on the Gaussian case with extensions to other latent distribution families admitting orthogonal polynomial expansions. Formal proofs for these Lancaster-type distributions [30], namely the negative binomial, Gamma, Poisson, and hypergeometric cases, are deferred to 11.

The spectral bounds on \(\rho_i\) guarantee a full-rank correlated subspace while excluding deterministic equivalence between the views. We then give the practical implication of this assumption and present the corresponding latent model for better structural clarity.

To construct the joint distributions in our experiments, we employ the explicit additive generative model as follows: \[\label{equ:additive} \mathbf{s} = \mathbf{a}+\mathbf{c}, \qquad \mathbf{s}' = \mathbf{b}+\mathbf{c},\tag{1}\] where \(\mathbf{a}, \mathbf{b}, \mathbf{c}\) are mutually independent latent vectors. We emphasize that 1 serves as an experimental device rather than a prerequisite for our population identifiability theorem.

Let \(\mathbf{g}\colon \mathcal{S} \to \mathcal{X} \subset \mathbb{R}^{d_\mathcal{X}}\) and \(\mathbf{g}'\colon \mathcal{S} \to \mathcal{X}' \subset \mathbb{R}^{d_{\mathcal{X}'}}\) be injective and Borel-measurable mappings that generate the high-dimensional observed pairs \((\mathbf{x}, \mathbf{x}') = (\mathbf{g}(\mathbf{s}), \mathbf{g}'(\mathbf{s}'))\), where \(\mathcal{X}, \mathcal{X}'\) denote the observation spaces of two modalities with \(d_{\mathcal{S}} \ll d_{\mathcal{X}}, d_{\mathcal{X}'}\). Under 1, our objective is to learn encoders \(\mathbf{f}\colon \mathcal{X} \to \mathcal{Z}\) and \(\mathbf{f}'\colon \mathcal{X}' \to \mathcal{Z}\) that invert the generative process to recover the true latent sources \((\mathbf{s}, \mathbf{s}')\). To establish baseline affine identifiability, we assume a dimension-matched latent space (\(d_{\mathcal{Z}} = d_{\mathcal{S}}\)), deferring the analysis of mismatched regimes (\(d_{\mathcal{Z}} \neq d_{\mathcal{S}}\)) to the ablation study.

3.2 Canonical Correlation Analysis↩︎

Nonlinear CCA aims to learn a pair of nonlinear encoders \(\mathbf{f}\colon \mathcal{X}\to \mathcal{Z}\) and \(\mathbf{f}'\colon \mathcal{X}'\to \mathcal{Z}\) that maximize the sum of the singular values of their normalized cross-covariance. To ensure well-conditioned representations, we restrict the search space to zero-mean, identity-covariance encoders:

3.2.0.1 CCA Population Objective.

Given these feasible encoder classes, nonlinear CCA maximizes the following population objective: \[\label{equ:CCA95objective} \max_{\tilde{\mathbf{f}} \in \tilde{\mathcal{F}}_{\mathcal{X}}, \, \tilde{\mathbf{f}}' \in \tilde{\mathcal{F}}'_{\mathcal{X}'}} J(\tilde{\mathbf{f}}, \tilde{\mathbf{f}}') = \sum_{i=1}^{d_{\mathcal{Z}}} \sigma_i\big(\mathrm{Cov}(\tilde{\mathbf{f}}(\mathbf{x}), \tilde{\mathbf{f}}'(\mathbf{x}'))\big),\tag{2}\] where \(\sigma_i(\cdot)\) denotes the \(i\)-th largest singular value and \(\mathrm{Cov}(\cdot)\) the population covariance. Due to the orthogonal invariance of the objective and the post-orthogonal closure of the encoder classes, every population maximizer generates an equivalent \(O(d_{\mathcal{Z}})\times O(d_{\mathcal{Z}})\)-orbit of maximizers. In general, this does not imply that all maximizers lie in a single orbit. Stronger single-orbit statement is obtained later under the additional assumptions of 1 as shown in Appendix 9.2 and 4.

3.3 Population Affine Identifiability↩︎

In this section, we derive precise conditions under which solving nonlinear CCA leads to provable affine identification of the ground-truth latent factors at the population level. To leverage the distributional priors of the source, we begin by transporting the CCA problem from the observation domain to the underlying source domain. By composing the observation space encoders with the generators, \(\mathbf{h}=\mathbf{f}\!\circ\!\mathbf{g}\) and \(\mathbf{h}'=\mathbf{f}'\!\circ\!\mathbf{g}'\), we re-express the learned latent via representation maps \(\mathbf{h}, \mathbf{h}'\) and ground-truth latent pair. This change of variables preserves the first- and second-order structure of the CCA problem, so the whitening constraints and the CCA objective are unaffected. We formally justify the invariance of transporting CCA problems from observation spaces to source space.

By 1, the observation-space CCA problem is isometrically equivalent to the representable source-space problem, and this representable problem attains the same optimal value as the full feasible source-space problem. To establish affine identifiability, it suffices to characterize the maximizers of \(J_{\mathcal{S}}\) over \(\hat{\mathcal{F}}_{\mathcal{S}}\times\hat{\mathcal{F}}'_{\mathcal{S}}\), any observation-space maximizer induces one such source-space maximizer.

We then state our central population identifiability result as follows:

At the population optimum under Gaussian priors with \(d_{\mathcal{Z}} = d_{\mathcal{S}}\), the learned encoders successfully invert the data-generating process up to an affine ambiguity, i.e., centering, scaling, and an orthogonal rotation. In the proof of 1, we formally prove that explicit whitening or unit-variance regularization, as employed by non-contrastive methods like Barlow Twins, is strictly necessary to ensure the boundedness and well-conditioning of the learned mappings. This provides a partial but rigorous explanation for the effectiveness of whitening-based non-contrastive methods. While our theoretical guarantees require matched dimensions (\(d_{\mathcal{S}} = d_{\mathcal{Z}}\)), we extend our analysis to mismatched regimes, i.e., \(d_{\mathcal{S}} \neq d_{\mathcal{Z}}\) and finite-sample convergence in the subsequent empirical sections. A detailed discussion of the role of canonical-correlation separation and the consequences of relaxing this condition is provided in 8.

3.4 Empirical Estimation and Statistical Consistency↩︎

While population identifiability characterizes the infinite-sample optimum of 2 , practical representation learning requires statistical consistency with finite data. To bridge this gap, we introduce a regularized empirical CCA estimator and analyze its statistical consistency in the finite-sample regime. We prove that as the regularization parameter decays asymptotically, the empirical maximizers strictly converge to the population solution, formally linking theoretical identifiability to finite-sample learnability.

To construct this empirical objective, we parameterize the nonlinear encoders \(\mathbf{f}_{\boldsymbol{\theta}}\colon \mathbb{R}^{d_\mathcal{X}} \to \mathbb{R}^{d_\mathcal{Z}}\) and \(\mathbf{f}'_{\boldsymbol{\theta}'}\colon \mathbb{R}^{d_{\mathcal{X}'}} \to \mathbb{R}^{d_\mathcal{Z}}\) using neural networks with sufficient capacity to act as universal function approximators. Let \(\boldsymbol{\theta}\) and \(\boldsymbol{\theta}'\) denote their respective weights. The regularized finite-sample objective is formulated as follows:

3.4.0.1 Empirical Objective.

Let \(\mathbf{X} \in \mathbb{R}^{n \times d_{\mathcal{X}}}\) and \(\mathbf{X}' \in \mathbb{R}^{n \times d_{\mathcal{X}'}}\) denote observation matrices comprising \(n\) mutually independent samples drawn from \(P_{\mathbf{x}\mathbf{x}'}\). We define the learned representations as \(\mathbf{Z} = \mathbf{f}_{\boldsymbol{\theta}}(\mathbf{X})\) and \(\mathbf{Z}' = \mathbf{f}'_{\boldsymbol{\theta}'}(\mathbf{X}') \in \mathbb{R}^{n \times d_{\mathcal{Z}}}\). Let \(\mathbf{Z}_c\) and \(\mathbf{Z}'_c\) denote their respective mean-centered counterparts. The empirical auto- and cross-covariance matrices are given by \(\hat{\mathbf{\Sigma}}_{\mathbf{z}\mathbf{z}} := \frac{1}{n-1} \mathbf{Z}_c^\top \mathbf{Z}_c\), \(\hat{\mathbf{\Sigma}}_{\mathbf{z}'\mathbf{z}'} := \frac{1}{n-1}{\mathbf{Z}'_c}^\top \mathbf{Z}'_c\), and \(\hat{\mathbf{\Sigma}}_{\mathbf{z}\mathbf{z}'} := \frac{1}{n-1}\mathbf{Z}_c^\top \mathbf{Z}'_c\). To ensure well-conditioning in the finite-sample regime, we apply a ridge penalty \(\epsilon > 0\) to construct the regularized whitening matrices: \[\hat{\mathbf{W}}_{\mathbf{z}} := (\hat{\mathbf{\Sigma}}_{\mathbf{z}\mathbf{z}} + \epsilon\mathbf{I})^{-1/2}, \quad \hat{\mathbf{W}}_{\mathbf{z}'} := (\hat{\mathbf{\Sigma}}_{\mathbf{z}'\mathbf{z}'} + \epsilon\mathbf{I})^{-1/2}.\] The empirical CCA objective maximizes the nuclear norm, i.e., the sum of singular values, of the normalized cross-covariance \(\hat{\mathbf{K}} := \hat{\mathbf{W}}_{\mathbf{z}} \hat{\mathbf{\Sigma}}_{\mathbf{z}\mathbf{z}'} \hat{\mathbf{W}}_{\mathbf{z}'}\), formulated via the variational trace characterization: \[\begin{align} \label{equ:empirical95CCA} \max_{\boldsymbol{\theta}, \boldsymbol{\theta}'} \hat{J}(\boldsymbol{\theta}, \boldsymbol{\theta}') &= \max_{\boldsymbol{\theta}, \boldsymbol{\theta}'} \sum_{k=1}^{d_{\mathcal{Z}}} \sigma_k(\hat{\mathbf{K}}) \nonumber \\ &= \max_{\substack{\boldsymbol{\theta}, \boldsymbol{\theta}' \\ \mathbf{U}^\top \mathbf{U} = \mathbf{I} \\ \mathbf{V}^\top \mathbf{V} = \mathbf{I}}} \operatorname{tr}\big(\mathbf{U}^\top \hat{\mathbf{K}}\mathbf{V}\big). \end{align}\tag{3}\] The following theorem establishes our central statistical consistency result. By demonstrating the uniform convergence of the empirical covariance operators and the stability of both the regularized whitening and spectral mappings, we prove the asymptotic consistency of the empirical maximizers.

An interpretation of Assumptions A3–A4 and their role in the consistency argument is provided in 8.

4 EXPERIMENTS↩︎

    \centering
    \resizebox{\textwidth}{!}{
    \setlength{\tabcolsep}{3pt}
    \large
    \begin{tabular}{l*{10}{c}}
        Methods   & \multicolumn{2}{c}{Gaussian}  & \multicolumn{2}{c}{Negative Binomial}  & \multicolumn{2}{c}{Gamma}    & \multicolumn{2}{c}{Poisson}    & \multicolumn{2}{c}{Hypergeometric} \\
        \cmidrule(r){2-3}
        \cmidrule(r){4-5}
        \cmidrule(r){6-7}
        \cmidrule(r){8-9}
        \cmidrule(r){10-11}
            & $\mathbf f$ & $\mathbf f'$   & $\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$  \\
        \midrule
        SwAV & 32.96 $\pm$4.05 & 37.88 $\pm$0.45 & 33.17 $\pm$3.67 & 28.85 $\pm$2.62 & 29.37 $\pm$4.85 & 28.38 $\pm$2.74 & 36.43 $\pm$0.88 & 35.23 $\pm$1.69 & 12.88 $\pm$8.25 & 12.84 $\pm$8.66\\
        BarlowTwins    & 79.44$\pm$0.26 &79.51 $\pm$0.3             & 67.22 $\pm$5.67 & 68.33 $\pm$4.11 & 61.99 $\pm$2.4 & 61.98 $\pm$2.43 & 55.77 $\pm$5.82 & 55.29 $\pm$4.59 &25.47 $\pm$0.49 &23.9 $\pm$1.53\\
        VICReg  &  14.54 $\pm$1.14 & 13.05 $\pm$1.1 & 13.83 $\pm$1.82 & 12.82 $\pm$1.08 & 12.27 $\pm$0.56 & 13.61 $\pm$0.82 & 11.88 $\pm$0.48 & 11.37 $\pm$0.82 & 18.94 $\pm$2.37 & 16.7 $\pm$0.64 \\
        W-MSE & 99.17 $\pm$0.21 & 99.4 $\pm$0.04 & \textbf{99.07 $\pm$0.04} & 99.23 $\pm$0.04 & 99.11 $\pm$0.15 & 99.34 $\pm$0.05 & \textbf{99.18 $\pm$0.08} & \textbf{99.4 $\pm$0.04} & \textbf{98.97 $\pm$0.04} & \textbf{99.02 $\pm$0.32} \\
        DGCCA & 21.9 $\pm$0.79 & 18.17 $\pm$1.4 & 21.45 $\pm$1.52 & 23.9 $\pm$0.86 & 22.0 $\pm$0.73 & 21.17 $\pm$1.65 & 22.16 $\pm$1.43 & 18.94 $\pm$1.2 & 23.81 $\pm$1.09 & 23.95 $\pm$1.25\\
        rDGCCA & 90.57 $\pm$2.89 & 82.48 $\pm$3.7 & 90.62 $\pm$3.33 & 78.51 $\pm$2.73 & 90.35 $\pm$3.23 & 83.35 $\pm$4.16 & 92.97 $\pm$2.26 & 83.33 $\pm$1.99 & 91.74 $\pm$3.04 & 86.93 $\pm$2.89\\
        DeepCCA   & \textbf{99.4 $\pm$0.05} & \textbf{99.41 $\pm$0.03} &  99.05 $\pm$0.04 & \textbf{99.23 $\pm$0.03}  & \textbf{99.12 $\pm$0.14} & \textbf{99.35 $\pm$0.03} & 99.18 $\pm$0.09& 99.36 $\pm$0.03 &98.91 $\pm$0.15&98.37 $\pm$0.1\\
        \bottomrule
    \end{tabular}}
        \caption{Comparison of the  coefficient of determination $R^2$ $\uparrow$(\%) of both encoders $\mathbf f, \mathbf f'$ on synthetic data ($d_\mathcal{S}=d_\mathcal{Z}=10$).}
        \label{tab:synthetic95identifiability}

In this section, we empirically validate our theoretical guarantees for nonlinear CCA. We evaluate affine identifiability across diverse latent distributions, verify reparameterization invariance and finite-sample consistency, and systematically ablate our core assumptions. We adopt the experimental setup from previous works [21], [23] on disentangled representations and extend it to a multi-modal setting. The code is available at https://github.com/ZhiweiHan9277/AISTATS2026_Provable_Affine_Identifiability.

4.1 Experimental Setup↩︎

4.1.0.1 Datasets.

We evaluate our theoretical claims on a synthetic dataset and the 3DIdent image dataset [21]. For the synthetic setup, we adapt the online sampling mechanism from prior work [21], [23], generalizing their single-generator framework to two independent generators to simulate multi-modal observations. 3DIdent comprises images of a 3D object rendered from 11 latent factors, e.g., position, rotation and illumination, yielding (image, latent) tuples.

To map our continuous generative process to the finite 3DIdent dataset, we employ a nearest-neighbor matching strategy. We first independently sample \(\mathbf{a}, \mathbf{b}, \mathbf{c}\) to construct continuous latent pairs \((\mathbf{s}, \mathbf{s}')\) as in 1 . We then retrieve the corresponding rendered images \((\mathbf{x}, \mathbf{x}')\) by finding the Euclidean nearest neighbors of \(\mathbf{s}\) and \(\mathbf{s}'\) in the 3DIdent ground-truth latent space. To emulate distinct multi-view modalities, we retain the original color image for \(\mathbf{x}\) and use a grayscale conversion for \(\mathbf{x}'\), preserving shared generative factors while marginalizing out color.

4.1.0.2 Baselines.

We benchmark the CCA-like objectives and strong non-contrastive SSL baselines to test affine identifiability under a unified encoder architecture and a family of distributions aligned with our theory.

• SwAV [43] learns view-invariant features via online clustering with prototype assignments.

• BarlowTwins [15] enforces invariance and minimizes redundancy through cross-correlation diagonalization.

• VICReg [16] encourages invariance while regularizing variance and covariance to decorrelate neural representations.

• W-MSE [14] aligns whitened multi-view neural representations by minimizing mean-squared error.

• DGCCA [39] is a nonlinear extension of generalized CCA, optimizing a shared latent that reconstructs all view projections.

• rDGCCA [29] improves DGCCA by extracting shared representations under conditionally independent private latents, offering theoretical recovery guarantees.

• DeepCCA [12] maximizes the CCA objective with whitened multi-view representations.

We exclude Kernel CCA due to poor high-dimensional scalability. Among the evaluated baselines, only DeepCCA faithfully enforces exact whitening and maximizes canonical correlations. We therefore use its converged solution as a proxy for population CCA with the data generated from online sampling of the underlying distributions. All baselines are implemented using their official repositories when available, otherwise we reimplement by following their paper specifications.

4.1.0.3 Metrics.

Following prior work [21], [23], [24], we assess affine recoverability by regressing ground-truth latents onto learned latents and reporting the coefficient of determination (\(R^2\)) averaged across dimensions. For settings with partial distributional violations or dimensional mismatch, we evaluate subspace error using the mean (\(PA_{\text{mean}}\)) and maximum (\(PA_{\text{max}}\)) principal angles between the learned span and the ground-truth canonical subspace, in line with subspace identification studies [44]–[46]. To quantify the alignment between two encoders up to orthogonal transformations, we measure the orbit distance defined as \(\min_{\mathbf{Q},\mathbf{Q}'\in O(d_\mathcal{Z})} \; \tfrac{1}{n}\Big( \|\hat{\mathbf{Z}}-\mathbf{Z} \mathbf{Q}\|_F^2 \;+\; \|\hat{\mathbf{Z}}'-\mathbf{Z}' \mathbf{Q}'\|_F^2 \Big),\) where \(n\) is the batch size, \(\mathbf{Z}, \mathbf{Z}'\) and \(\hat{\mathbf{Z}}, \hat{\mathbf{Z}}'\) are the learned latents of two sets of encoders.

4.2 Validation of Theoretical Findings↩︎

4.2.0.1 Affine Identifiability.

For the synthetic experiments (\(d_{\mathcal{S}}=10\)), latent pairs \((\mathbf{s}, \mathbf{s}')\) are generated via 1 by sampling \(\mathbf{a}, \mathbf{b}\), and \(\mathbf{c}\) independently from five candidate distributions, and observations \((\mathbf{x}, \mathbf{x}')\) are produced by independent decoders \(\mathbf{g}\) and \(\mathbf{g}'\). The \(R^2\) scores in [tab:synthetic95identifiability] support the predicted affine identifiability under this additive model: DeepCCA and W-MSE consistently achieve near-perfect recovery across all distributions. rDGCCA forms a weaker second tier with noticeable asymmetry across the two views, while Barlow Twins deteriorates substantially on the hypergeometric setting. SwAV, VICReg, and DGCCA remain far from exact recovery. Detailed distribution settings and subspace recovery errors are deferred to 13.

On 3DIdent (1), DeepCCA again performs best, with W-MSE close behind in both \(R^2\) and principal-angle errors. rDGCCA and Barlow Twins provide partial recovery but remain clearly below these two methods, whereas DGCCA fails outright and SwAV and VICReg also perform poorly. Because 3DIdent is constructed by nearest-neighbor matching in latent space rather than from exact generative pairs, exact recovery is inherently harder. We therefore also report subspace recovery errors as a more robust measure of representation alignment under finite-sample and matching noise.

2pt

1 visualizes the 2D whitened representations learned by DeepCCA across four distributions, omitting Hypergeometric due to representation collapse. The source coordinates manifest as smoothly varying, mutually orthogonal color gradients. As theoretically expected, these gradients are rotated relative to the canonical axes, empirically validating that whitening identifies the true subspace only up to an orthogonal transformation.

4.2.0.2 Reparameterization Invariance.

This experiment is not a downstream-performance benchmark and therefore does not admit a meaningful chance-level baseline. The relevant reference point is exact agreement, i.e., zero singular-value gap and zero orbit distance before the logarithmic transform. Our goal is only to test Proposition 1, namely whether CCA trained in observation space and in source space converges to the same \(O(d_{\mathcal{Z}})\)-orbit and canonical spectrum. To this end, we independently optimize two CCA encoders under identical synthetic configurations: one operating on the observation space and another directly on the latent source space. 2 tracks the \(l_\infty\)-norm singular-value gap and the orbit distance between the two learned encoders for a given view. Smaller values signify closer alignment, exact invariance implies a zero singular value gap and zero orbit distance prior to logarithmic transformation. Across five random seeds, both metrics rapidly converge and remain consistently small throughout training. The median orbit distance remains below 0.2, while the \(l_\infty\) singular-value gap stays below \(0.01\). Crucially, this demonstrates that the observation-space maximizer of the CCA objective tightly aligns with its source-space counterpart, providing strong empirical corroboration for 1.

4.2.0.3 Finite-sample Consistency.

We evaluated the finite-sample consistency of CCA by varying the sample size \(n\) while keeping the remaining synthetic setup fixed, utilizing \(\epsilon=0.01\cdot n^{-1/4}\) balancing bias and stability. We fix the source dimension to \(d_{\mathcal{S}}=10\) and report both the \(R^2\) score and the orbit distance between the empirical and population CCA maximizers for a single view in 3. As the sample size increases, the \(R^2\) improves steadily across all distributions, exceeding 90% at roughly \(4\cdot 10^5\) samples, while orbit distance rapidly decays to near zero, indicating convergence of the learned representation to the population solution up to an orthogonal transformation. These results demonstrate that empirical CCA recovers the affine structure of the latent sources as the sample size grows as stated in 2.

4.3 Ablation Study↩︎

We provide an ablation study to isolate the effects of source dimensionality, first-order canonical dominance and latent–dimension mismatch on affine identifiability. Following the synthetic experimental setup, we vary only one parameter at a time while keeping all others fixed.

4.3.0.1 Source Dimension.

CCA consistently achieves affine identifiability in low-dimensional settings (\(d_\mathcal{S}=2,10,20\)), as evidenced by small principal angles between the recovered and ground-truth subspaces and high \(R^2\) scores, as shown in the left panel of 4. As the source dimension increases (\(d_{\mathcal{S}}=30,40\)), CCA continues to recover a substantial portion of the correlated subspace, but the recovery quality gradually deteriorates. Both the mean and variance of the principal angles increase, indicating reduced estimation stability in higher dimensions, as illustrated in the right panel of 4.

4.3.0.2 First-order Canonical Dominance.

We introduce the first-order canonical dominance ratio, \(\rho_{d_{\mathcal{S}}} / \rho_1^2\), to characterize the weakest canonical correlation’s strength relative to the squared strongest correlation. Assuming source canonical correlations are uniformly sampled from \([\rho_{d_{\mathcal{S}}}, \rho_1]\), 5 illustrates the impact of this ratio on affine identifiability. Increasing the dominance ratio consistently improves \(R^2\) scores across latent dimensions, indicating enhanced source recoverability. Notably, identifiability exhibits a threshold effect at \(1\). For \(\rho_{d_{\mathcal{S}}} / \rho_1^2 < 1\), elevated maximum principal angles (\(PA_{\max}\)) reveal that at least one canonical direction remains poorly identifiable. Once the ratio exceeds \(1\), \(PA_{\max}\) rapidly converges to zero, confirming full recovery of all canonical directions and strict alignment with the ground-truth source subspace.

4.3.0.3 Dimension Mismatch.

We investigate both under- and overcomplete learning regimes. In the overcomplete case (\(d_{\mathcal{S}} < d_{\mathcal{Z}}\)), consistently high \(R^2\) scores (2) demonstrate that the true source subspace is recoverable despite redundant latent coordinates, though the stability of these redundancies remains unclear. Conversely, in the undercomplete regime (\(d_{\mathcal{S}} > d_{\mathcal{Z}}\)), insufficient capacity precludes full \(d_{\mathcal{S}}\)-dimensional recovery. Although four canonical directions are recovered with minimal principal angle deviation (6), affine identifiability cannot be broadly guaranteed.

4.3.0.4 Discussion on Whitening.

Whitening, enforcing unit variance and decorrelation across latent dimensions for self-supervised learning w [47]–[49], is critical for affine identifiability as it ensures well-conditioned representations and stabilizes canonical direction estimation. Empirically, methods explicitly enforcing whitening, e.g., DeepCCA and W-MSE, achieve superior affine identifiability. Furthermore, even relaxed decorrelation constraints, as in Barlow Twins, yield substantial improvements over unregularized baselines. These empirical results corroborate the theoretical analysis detailed in 10.

5 CONCLUSION↩︎

We characterized the sufficient conditions under which nonlinear CCA achieves affine identifiability, advancing the theoretical understanding of correlation-based non-contrastive learning. Crucially, our framework provides a principled explanation for the recent remarkable empirical performance of non-contrastive methods: it proves that explicit whitening and unit-variance regularizers, e.g., in Barlow Twins, natively drive the recovery of ground-truth latent factors under (near-)Gaussian priors. Although the affine guarantee is naturally weaker than the permutation-level identifiability of contrastive methods, CCA provides practical advantages by circumventing negative sampling and exhibiting robustness to small batch sizes. Finally, our proof of finite-sample consistency guarantees that scaling dataset size directly improves empirical source recovery and downstream disentanglement.

Checklist↩︎

The checklist follows the references. For each question, choose your answer from the three possible options: Yes, No, Not Applicable. You are encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description (1-2 sentences). Please do not modify the questions. Note that the Checklist section does not count towards the page limit. Not including the checklist in the first submission won’t result in desk rejection, although in such case we will ask you to upload it during the author response period and include it in camera ready (if accepted).

In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

1. For all models and algorithms presented, check if you include:

   1. A clear description of the mathematical setting, assumptions, algorithm, and/or model. [Yes] The mathematical setting (data generating process, CCA objective, whitened encoder classes) and all assumptions (Assumptions 1–3) are formally stated in Section 3.

   2. An analysis of the properties and complexity (time, space, sample size) of any algorithm. [Yes] Population identifiability properties are analyzed in Theorem 1, and finite-sample convergence is analyzed in Theorem 2. Training times and computational costs are reported in 12.

   3. (Optional) Anonymized source code, with specification of all dependencies, including external libraries. [Yes] Source code is available on GitHub.

2. For any theoretical claim, check if you include:

   1. Statements of the full set of assumptions of all theoretical results. [Yes] All assumptions (Assumptions 1–3 and conditions A1–A4 in Theorem 2) are explicitly stated before each result.

   2. Complete proofs of all theoretical results. [Yes] Proof sketches are provided in the main text; complete proofs are given in the supplementary material (9).

   3. Clear explanations of any assumptions. [Yes] Each assumption is followed by a remark explaining its role, e.g., the spectral gap condition in Assumption 3 is explained via the Hermite expansion.

3. For all figures and tables that present empirical results, check if you include:

   1. The code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL). [Yes] Source code is available on GitHub, and all datasets used are publicly available.

   2. All the training details (e.g., data splits, hyperparameters, how they were chosen). [Yes] Training configurations (optimizer, learning rate, batch size, iterations, encoder architectures, distribution setups) are detailed in 12.

   3. A clear definition of the specific measure or statistics and error bars (e.g., with respect to the random seed after running experiments multiple times). [Yes] All metrics (\(R^2\), \(PA_{\text{mean}}\), \(PA_{\max}\), orbit distance) are defined in Section 4.1. Error bars report standard deviations over five independent random seeds.

   4. A description of the computing infrastructure used. (e.g., type of GPUs, internal cluster, or cloud provider). [Yes] Experiments were run on NVIDIA RTX A5000 GPUs, as described in 12.

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets, check if you include:

   1. Citations of the creator If your work uses existing assets. [Yes] The 3DIdent dataset [21] and all baseline methods are properly cited.

   2. The license information of the assets, if applicable. [Yes] The 3DIdent dataset is released under the Creative Commons Attribution 4.0 International (CC-BY-4.0) license.

   3. New assets either in the supplemental material or as a URL, if applicable. [Yes] Source code is available on GitHub.

   4. Information about consent from data providers/curators. [Not Applicable] We use only publicly available datasets and open-source code.

   5. Discussion of sensible content if applicable, e.g., personally identifiable information or offensive content. [Not Applicable] The datasets used (synthetic data and rendered 3D object images) do not contain personally identifiable information or offensive content.

5. If you used crowdsourcing or conducted research with human subjects, check if you include:

   1. The full text of instructions given to participants and screenshots. [Not Applicable] No crowdsourcing or human subjects research was conducted.

   2. Descriptions of potential participant risks, with links to Institutional Review Board (IRB) approvals if applicable. [Not Applicable] No crowdsourcing or human subjects research was conducted.

   3. The estimated hourly wage paid to participants and the total amount spent on participant compensation. [Not Applicable] No crowdsourcing or human subjects research was conducted.

6 OVERVIEW OF THE SUPPLEMENTARY MATERIAL↩︎

This supplementary material is organized as follows. 7 compares the identifiability frameworks of nonlinear CCA and contrastive learning. 9 contains complete proofs of the main theoretical results and 10 isolates the role of whitening in the identifiability argument. 11 provides proof sketches for additional latent distribution families. 12 describes implementation details and 13 reports additional experimental results. Accompanying source code is provided to support reproducibility and independent verification.

7 EXTENDED RELATED WORK: COMPARISON OF IDENTIFIABILITY FRAMEWORKS: NONLINEAR CCA VERSUS CONTRASTIVE LEARNING↩︎

While both nonlinear Canonical Correlation Analysis (CCA) and Contrastive Learning (CL), e.g., InfoNCE, serve as foundational paradigms for multi-view representation learning, their underlying theoretical mechanisms for achieving identifiability differ fundamentally. Recent works have established strong identifiability guarantees for contrastive methods [3], [21], [23]. However, unifying CL and CCA under a single theoretical framework remains challenging due to divergent structural assumptions, regularization mechanisms, and proof techniques. We systematically compare these two theoretical frameworks below.

7.0.0.1 1. Distributional Assumptions

• Contrastive Learning: CL identifiability typically assumes the ground-truth latents are sampled from an exponential family distribution. Crucially, it relies on non-Gaussianity (often requiring that at most one latent dimension is Gaussian, following classical ICA theory) or imposes no constraints on the marginal distributions [3]. CL models remain identifiable even when the latent spaces are strictly bounded or distributed on a hypersphere (e.g., von Mises-Fisher distributions).

• Nonlinear CCA: In contrast, our nonlinear CCA framework requires strong joint distributional assumptions across all latent dimensions, specifically relying on families that admit an orthogonal polynomial expansion (such as the joint Gaussian priors driving the Mehler expansion). Consequently, nonlinear CCA becomes theoretically non-identifiable under bounded or hyperspherical latent spaces—precisely the regimes where CL theories thrive.

7.0.0.2 2. Regularization Mechanisms

• Contrastive Learning: CL objectives inherently regularize the geometry of the representation space. This is typically achieved via unit-norm constraints on the embeddings (e.g., temperature-scaled InfoNCE) or by assuming bounded latent spaces, which prevents the representations from diverging.

• Nonlinear CCA: CCA relies strictly on representation whitening to achieve well-conditioning. As demonstrated in our theoretical analysis, unit variance ensures the compactness of the representation functions, while zero cross-correlation guarantees orthogonality between the components in the function space. Without whitening, the orthogonal polynomial expansion collapses, and the CCA objective cannot reliably separate first-order components from higher-order artifacts.

7.0.0.3 3. Proof Techniques

• Contrastive Learning: The standard proof mechanism for CL demonstrates that, under exponential family generative models, the contrastive loss forces the learned representation to act as an isometry. By preserving the underlying metric structure of the data-generating process, the learned features asymptotically align with the true latents [21].

• Nonlinear CCA: Our analysis of CCA operates via function expansion in an infinitesimal function space. The proof mechanism relies on decomposing the representation mapping into an orthogonal polynomial basis (e.g., normalized Hermite polynomials). Identifiability is established by proving that the CCA objective uniquely isolates the first-order basis components that maximize canonical correlations, provided that structural conditions—such as the First-Order Canonical Dominance (3)—hold.

7.0.0.4 4. Identifiability Guarantees

• Contrastive Learning: Because CL forces metric preservation, it yields exceptionally strong global identifiability conclusions. When the order of the sufficient statistics is known, CL can achieve permutation or block-permutation disentanglement (up to sign flips) [23].

• Nonlinear CCA: CCA achieves strictly affine identifiability; the ground-truth representation is recovered up to a linear shift and an orthogonal rotation (after whitening). While this represents a broader equivalence class than permutation disentanglement, our theory directly explains the empirical success of purely non-contrastive, correlation-based methods (e.g., Barlow Twins [15], W-MSE [14]). It proves that exact latent recovery is possible under minimal architectural complexity, entirely circumventing the need for negative sampling.

8 Further Remarks on Theoretical Results↩︎

9 Proofs of Main Theoretical Results↩︎

9.1 Proof of 1↩︎

9.2 Proof of the rotation-invariance of CCA↩︎

In this appendix, we formalize the rotational symmetry underlying the population CCA objective in 2 . Throughout, for any feasible pair \((\tilde{\mathbf{f}},\tilde{\mathbf{f}}') \in \tilde{\mathcal{F}}_{\mathcal{X}}\times \tilde{\mathcal{F}}'_{\mathcal{X}'}\), we write \[\mathbf{C}_{\tilde{\mathbf{f}},\tilde{\mathbf{f}}'} := \mathrm{Cov}\!\big(\tilde{\mathbf{f}}(\mathbf{x}),\tilde{\mathbf{f}}'(\mathbf{x}')\big) \in \mathbb{R}^{d_{\mathcal{Z}}\times d_{\mathcal{Z}}}.\]

9.3 Proof of 1↩︎

Throughout this subsection, write \[d:=d_{\mathcal{S}}=d_{\mathcal{Z}}.\]

9.3.0.1 Why the strict gap is needed.

The argument above uses the strict inequality \[\rho_d>\rho_1^2\] exactly once: it ensures that the first-order block \(\{\mathbf{e}_1,\dots,\mathbf{e}_d\}\) is separated from every higher-order multi-index. If this fails, then a second-order term such as \(2\mathbf{e}_1\) can have coefficient \(\rho_1^2\ge \rho_d\), and a population maximizer need not be supported entirely on the first-order block. In that regime, nonlinear CCA can mix linear and higher-order terms, and strict affine identifiability is no longer guaranteed.

9.4 Preservation of Higher-Order Nonlinearities Beyond First-Order Dominance↩︎

In the main text, 3 requires first-order canonical dominance, specifically \(\rho_{d_{\mathcal{S}}}>\rho_1^2\), to guarantee strictly affine identifiability. This section provides a self-contained illustration of the representation learning dynamics when this assumption is relaxed. Specifically, we demonstrate how the canonical spectrum interacts with the representation capacity \(d_{\mathcal{Z}}\) to determine the survival of higher-order nonlinearities.

9.4.0.1 Theoretical Mechanism.

As established in the proof of 1, the normalized orthogonal polynomial expansion (e.g., the Mehler expansion for Gaussian priors) diagonalizes the CCA objective. Under this expansion, a multivariate polynomial term of multi-index degree \(\mathbf{n}=(n_1,\dots,n_{d_{\mathcal{S}}})\) corresponds to a canonical correlation given by \(\prod_{i=1}^{d_{\mathcal{S}}}\rho_i^{n_i}\). Because nonlinear CCA strictly prioritizes orthogonal components that maximize the sum of inter-view correlations, it greedily selects the \(d_{\mathcal{Z}}\) terms with the largest corresponding values of \(\prod_{i=1}^{d_{\mathcal{S}}}\rho_i^{n_i}\), regardless of their polynomial degree. Consequently, higher-order nonlinear terms of strongly correlated source components can overshadow the first-order (linear) terms of more weakly correlated components.

9.4.0.2 Infinite Capacity Guarantee.

Because the true canonical correlations satisfy \(0<\rho_i<1\) for all \(i\in\{1,\dots,d_{\mathcal{S}}\}\), the correlation of any higher-order term diminishes exponentially as its total polynomial degree \(|\mathbf{n}|\to\infty\). Therefore, while a limited capacity \(d_{\mathcal{Z}}\) can lead to the truncation of weakly correlated linear terms, the asymptotic limit of infinite representation capacity (\(d_{\mathcal{Z}}\to\infty\)) guarantees the eventual recovery of all first-order linear terms. However, in this unconstrained regime, the desired linear features are inherently entangled with a multitude of higher-order artifacts.

9.4.0.3 A Concrete Example.

To rigorously illustrate this phenomenon, consider a two-dimensional latent space (\(d_{\mathcal{S}}=2\)) with ground-truth canonical correlations \(\rho_1=0.9\) and \(\rho_2=0.5\). Notice that this violates the first-order dominance condition since \(\rho_2<\rho_1^2\) (\(0.5<0.81\)).

Evaluating the canonical correlations for the lowest-degree polynomial terms yields:

• First-order terms: \(s_1\) yields \(\rho_1=0.9\), and \(s_2\) yields \(\rho_2=0.5\).

• Higher-order terms of \(s_1\): \(s_1^2\) yields \(\rho_1^2=0.81\), \(s_1^3\) yields \(\rho_1^3=0.729\), \(s_1^4\) yields \(\rho_1^4=0.6561\), \(s_1^5\) yields \(\rho_1^5=0.59049\) and \(s_1^6\) yields \(\rho_1^6=0.531441\).

Sorting these available orthogonal terms in descending order of their canonical correlations generates the sequence: \(\{0.9, 0.81, 0.729, 0.6561, 0.59049, 0.531441,0.5, \dots\}\).

If we constrain the representation capacity to match the source dimension (\(d_{\mathcal{Z}}=2\)), nonlinear CCA greedily selects the top two components. In this case, it captures \(\{s_1,s_1^2\}\), introducing a second-order nonlinearity of \(s_1\) and entirely failing to recover the linear term \(s_2\).

If the representation capacity is expanded to \(d_{\mathcal{Z}}=7\), the algorithm selects the top seven components: \(\{s_1, s_1^2, s_1^3, s_1^4, s_1^5, s_1^6, s_2\}\). Here, the linear term \(s_2\) is successfully recovered, consistent with the infinite capacity guarantee, but the learned representation is now dominated by higher-order nonlinearities up to the sixth degree. This example clearly demonstrates why the strict spectral gap \(\rho_{d_{\mathcal{S}}}>\rho_1^2\) is mathematically necessary to isolate purely affine representations under finite capacity.

9.5 Proof of 2↩︎

We write \(\|\cdot\|\) for the spectral norm and \(\|\cdot\|_*\) for the nuclear norm.

10 Necessity of whitening↩︎

For precision, the necessity claim has to be stated carefully. Centering and unit covariance play different roles, and only the latter is the genuinely restrictive whitening condition. What is mathematically indispensable is a scale-fixing normalization equivalent to whitening. In our formulation this normalization is imposed by restricting the encoder classes to zero-mean, identity-covariance maps. The proof of 1 uses this normalization in two logically distinct ways: centering removes the constant polynomial mode, while unit covariance yields the orthonormal coefficient geometry required by the diagonal selection argument.

We first record that normalized CCA can always be represented on whitened encoders.

The next proposition shows why some such normalization is necessary.

The preceding propositions isolate the formal necessity of whitening. In the proof of 1, whitening enters at three specific places.

1. Basis alignment and exclusion of the constant mode. The Hermite-Mehler expansions in 2 and 4 are written in the standardized source coordinates introduced there. This basis alignment is determined by the latent Gaussian law after source standardization and is independent of the encoder parameterization. For the encoder expansions, the only degree-zero condition needed is centering. Indeed, if \[\tilde{\mathbf{h}}_r(\mathbf{s}) = \sum_{\mathbf{n}\in\mathbb{N}^{d_{\mathcal{S}}}} \alpha_{r,\mathbf{n}}\Psi_{\mathbf{n}}(\mathbf{u}), \qquad \tilde{\mathbf{h}}'_q(\mathbf{s}') = \sum_{\mathbf{n}\in\mathbb{N}^{d_{\mathcal{S}}}} \beta_{q,\mathbf{n}}\Psi_{\mathbf{n}}(\mathbf{v}),\] then \(\Psi_{\mathbf{0}}\) is the unique constant basis element and \(\mathbb{E}[\Psi_{\mathbf{n}}]=0\) for every \(\mathbf{n}\neq \mathbf{0}\). Hence \[\mathbb{E}[\tilde{\mathbf{h}}_r(\mathbf{s})]=0 \iff \alpha_{r,\mathbf{0}}=0, \qquad \mathbb{E}[\tilde{\mathbf{h}}'_q(\mathbf{s}')]=0 \iff \beta_{q,\mathbf{0}}=0.\] Thus the constant mode disappears because the encoders are centered. The unit-covariance part of whitening is used later.

2. Feasible-set geometry for the CCA objective. After applying 5, we may work entirely with whitened encoders. Writing \[\tilde{\mathbf{h}}_r(\mathbf{s}) = \sum_{\mathbf{n}\neq \mathbf{0}}\alpha_{r,\mathbf{n}}\Psi_{\mathbf{n}}(\mathbf{u}), \qquad \tilde{\mathbf{h}}'_q(\mathbf{s}') = \sum_{\mathbf{n}\neq \mathbf{0}}\beta_{q,\mathbf{n}}\Psi_{\mathbf{n}}(\mathbf{v}),\] and collecting coefficients into matrices \(\mathbf{A}=(\alpha_{r,\mathbf{n}})\) and \(\mathbf{B}=(\beta_{q,\mathbf{n}}),\) Corollary 2 gives \[\mathrm{Cov}\big(\tilde{\mathbf{h}}(\mathbf{s})\big)=\mathbf{A}\mathbf{A}^\top, \qquad \mathrm{Cov}\big(\tilde{\mathbf{h}}'(\mathbf{s}')\big)=\mathbf{B}\mathbf{B}^\top.\] Hence whitening is exactly the constraint \[\mathbf{A}\mathbf{A}^\top=\mathbf{I}_{d_{\mathcal{Z}}}, \qquad \mathbf{B}\mathbf{B}^\top=\mathbf{I}_{d_{\mathcal{Z}}}.\] Under these constraints, the cross-covariance diagonalizes as \[\mathrm{Cov}\big(\tilde{\mathbf{h}}(\mathbf{s}),\tilde{\mathbf{h}}'(\mathbf{s}')\big) = c\,\mathbf{A}\,\mathrm{diag}(t_{\mathbf{n}})\,\mathbf{B}^\top,\] and 3 applies directly. Without the unit-covariance constraints, the coefficient matrices are no longer row-orthonormal, and the proof no longer reduces to selecting the largest diagonal entries \(\{|t_{\mathbf{n}}|\}\). This is the precise point where whitening enters the identifiability argument.

3. Scale fixing and collapse avoidance. 6 already shows that without any scale-fixing normalization the raw covariance objective is unbounded. After passing to the whitened representation, the remaining first-order block is also well-conditioned. Once 3 excludes all higher-order indices, the optimizer is confined to the first-order block \(\{\mathbf{e}_1,\dots,\mathbf{e}_{d_{\mathcal{S}}}\}\). Let \(\mathbf{A}_{\mathrm{lin}}\) and \(\mathbf{B}_{\mathrm{lin}}\) denote the corresponding \(d_{\mathcal{Z}}\times d_{\mathcal{S}}\) coefficient matrices. Because the full coefficient matrices are whitened, we still have \[\mathbf{A}_{\mathrm{lin}}\mathbf{A}_{\mathrm{lin}}^\top = \mathbf{I}_{d_{\mathcal{Z}}}, \qquad \mathbf{B}_{\mathrm{lin}}\mathbf{B}_{\mathrm{lin}}^\top = \mathbf{I}_{d_{\mathcal{Z}}}.\] When \(d_{\mathcal{Z}}=d_{\mathcal{S}}\), this forces \(\mathbf{A}_{\mathrm{lin}},\mathbf{B}_{\mathrm{lin}}\in O(d_{\mathcal{Z}})\). Therefore the recovered linear maps are full-rank and perfectly conditioned in the whitened coordinates: no output coordinate can vanish, duplicate another one, or absorb an arbitrary scale. This is exactly what yields the orthogonal ambiguities in 1. Without whitening, even after higher-order terms are removed, the remaining linear coefficient matrices need only be invertible (if at all) and may be arbitrarily ill-conditioned.

11 Proof Sketches for the Remaining Candidate Distributions↩︎

The Gaussian proof of 1 relies on three Gaussian-specific ingredients: the scalar Hermite–Mehler expansion in 2, its tensor-product extension in 4, and the resulting orthogonality statement in 2. For the remaining candidate priors, the same optimization argument carries over once these ingredients are replaced by the corresponding scalar Lancaster expansion and the associated orthonormal polynomial basis [30], [31]. The only family-dependent objects are therefore the one-dimensional orthogonal polynomials and the associated diagonal coefficient sequence.

11.1 Unified Proof Template↩︎

We state the common reduction at the level needed by the CCA proof. Throughout this subsection, let \(I_i=\mathbb{N}_0\) for Poisson, negative binomial, and gamma marginals, and let \(I_i=\{0,\dots,m_i\}\) for hypergeometric marginals, where \(m_i\) is the maximal polynomial degree permitted by the finite support of the \(i\)-th coordinate. In the hypergeometric case, all sums below are therefore finite.

11.2 Instantiations for the Four Families↩︎

For each family below, the first-degree orthonormal polynomial is precisely the standardized coordinate, \[\psi_1(x)=\frac{x-\mu}{\sigma},\] with \(\mu\) and \(\sigma^2\) the marginal mean and variance. In particular, \[\lambda_{i,1} = \mathbb{E}\!\left[\psi_{i,1}(s_i)\psi_{i,1}(s_i')\right] = \mathrm{Corr}(s_i,s_i').\] Therefore, once higher-degree terms are excluded by ?? , 7 immediately yields affine recovery.

11.2.0.1 Poisson \((\lambda)\).

The marginal law is \[\nu(x)=e^{-\lambda}\frac{\lambda^x}{x!}, \qquad x\in \mathbb{N}_0.\] The associated orthogonal family is the Charlier family \(C_m(x;\lambda)\). After normalization, the first-degree term is \[\psi_1(x)=\frac{x-\lambda}{\sqrt{\lambda}}.\]

11.2.0.2 Negative Binomial \((r,p)\).

We use the convention \[\nu(x)=\binom{r+x-1}{x}p^r(1-p)^x, \qquad x\in \mathbb{N}_0,\quad 0<p<1.\] The associated orthogonal family is the Meixner family \(M_m(x;\beta,c)\) with \(\beta=r\) and \(c=1-p\). Its mean and variance are \[\mu=\frac{r(1-p)}{p}, \qquad \sigma^2=\frac{r(1-p)}{p^2},\] hence \[\psi_1(x)=\frac{x-\mu}{\sigma}.\]

11.2.0.3 Hypergeometric \((N,K,n)\).

The marginal law is \[\nu(x)=\frac{\binom{K}{x}\binom{N-K}{n-x}}{\binom{N}{n}}, \qquad x=\max\{0,n-(N-K)\},\dots,\min\{n,K\}.\] The associated orthogonal family is the Hahn family associated with the hypergeometric weight; its degree is finite because the support is finite. Its mean and variance are \[\mu=n\frac{K}{N}, \qquad \sigma^2=n\frac{K}{N}\left(1-\frac{K}{N}\right)\frac{N-n}{N-1},\] so the first-degree normalized polynomial is \[\psi_1(x)=\frac{x-\mu}{\sigma}.\]

11.2.0.4 Gamma \((k,\theta)\).

The marginal law is \[d\nu(x)=\frac{1}{\Gamma(k)\theta^k}x^{k-1}e^{-x/\theta}\mathbf{1}_{(0,\infty)}(x)\,dx.\] The associated orthogonal family is the generalized Laguerre family \(L_m^{(k-1)}(x/\theta)\). Its mean and variance are \[\mu=k\theta, \qquad \sigma^2=k\theta^2,\] hence \[\psi_1(x)=\frac{x-k\theta}{\sqrt{k}\,\theta}.\]

11.3 Connection to the Gaussian Proof↩︎

The relationship to the Gaussian case is now transparent.

• Only the basis changes. Replace the Hermite–Mehler system by the appropriate Lancaster system: Charlier for Poisson, Meixner for negative binomial, Hahn for hypergeometric, and generalized Laguerre for gamma.

• The diagonalization step is identical. Once the joint law is expanded in an orthonormal polynomial basis, the cross-covariance of the encoder coefficients is diagonal, and the CCA objective reduces to a diagonal selection problem exactly as in the Gaussian proof.

• The optimization step is unchanged. 3 still implies that the maximizer selects the \(d_{\mathcal{Z}}\) largest absolute Lancaster coefficients.

• Affine recovery again follows from first-degree dominance. If the degree-one coefficients dominate all higher-degree coefficients, only first-degree basis elements survive at the optimum. Since those first-degree basis elements are affine in the latent coordinates, the conclusion of 1 carries over verbatim.

12 Implementation Details↩︎

12.0.0.1 Training Configurations.

We adapt the experimental protocols of [21], [23] to our dual-encoder framework. All models are optimized via Adam [50] with a constant learning rate of \(10^{-4}\). For the synthetic dataset, we train for \(10^5\) iterations with a batch size of \(1024\), requiring approximately \(1.5\) hours for \(d_{\mathcal{S}}=40\) on a single NVIDIA RTX A5000 GPU. For 3DIdent, we train for \(10^4\) iterations with a batch size of \(512\), requiring roughly \(4\) hours. Empirically, we observe that CCA-based objectives converge \(2\)–\(3\times\) faster than standard contrastive losses. With the exception of qualitative visualizations, all reported metrics are averaged across five independent random seeds. Comprehensive implementation details and extended results are provided in 13.

For encoder architectures, we adopt the same design as [21] for both synthetic and 3DIdent datasets, except that for synthetic data we employ a lighter residual network. This network comprises two hidden layers of sizes \(10\cdot d_\mathcal{S}\) and \(20\cdot d_\mathcal{S}\), followed by three residual blocks and an output layer. Each residual block contains two layers of width \(20\cdot d_\mathcal{S}\). We apply leaky-ReLU activations and batch normalization to all hidden layers.

The decoders are implemented as approximately invertible multi-layer perceptrons. The encoders \(\mathbf{f}\) and \(\mathbf{f}'\) are parameterized independently using residual connections and batch normalization for optimization stability.

Training the synthetic model with \(d_\mathcal{S}=10\) on a single RTX A5000 GPU takes approximately one hour, whereas experiments on 3DIdent require roughly four hours using eight RTX A5000 GPUs.

12.0.0.2 Candidate Distribution Setups.

For each latent dimension, we generate a paired source \((\mathbf{s}, \mathbf{s}') \in \mathbb{R}^{d_\mathcal{S}} \times \mathbb{R}^{d_\mathcal{S}}\) according to the additive latent model \(\mathbf{s} = \mathbf{a} + \mathbf{c}\), \(\mathbf{s}' = \mathbf{b} + \mathbf{c}\), where \(\mathbf{a}, \mathbf{b}, \mathbf{c}\) are independent random vectors. Unless otherwise stated, the coordinates are independent across dimensions except for Gaussian case as stated in 1.

Joint Gaussian. We sample \((\mathbf{s}, \mathbf{s}') \sim \mathcal{N}(\mathbf{0}, \boldsymbol{\Sigma})\) with block covariance \[\boldsymbol{\Sigma} = \begin{pmatrix} \mathbf{I}_n & \mathbf{A}\,\mathrm{diag}(\boldsymbol{\rho})\,\mathbf{B}^\top \\ \mathbf{B}\,\mathrm{diag}(\boldsymbol{\rho})\,\mathbf{A}^\top & \mathbf{I}_n \end{pmatrix},\] where \(\mathbf{A}, \mathbf{B}\) are independent random orthogonal matrices (constructed via QR decomposition of Gaussian matrices with \(\det=+1\)). The canonical correlations \(\boldsymbol{\rho} = (\rho_1, \ldots, \rho_n)\) are linearly spaced in \([0.3,\,0.5]\). This yields standard-normal marginals and the desired cross-covariance structure \(\mathbf{A}\,\mathrm{diag}(\boldsymbol{\rho})\,\mathbf{B}^\top\).

Joint Gamma. For each coordinate \(j\), we draw three independent random variables \(c_j \sim \mathrm{Gamma}(k=1, \mathrm{rate}=1)\) and \(a_j, b_j \sim \mathrm{Gamma}(k=2, \mathrm{rate}=1)\), and set \[s_j = a_j + c_j, \qquad s'_j = b_j + c_j.\] Then \(\mathbb{E}[s_j] = \mathbb{E}[s'_j] = 3\), \(\mathrm{Var}(s_j) = \mathrm{Var}(s'_j) = 3\), and \(\mathrm{Cov}(s_j, s'_j) = 1\).

Joint Poisson. For each coordinate \(j\), \(c_j \sim \mathrm{Poisson}(1)\) and \(a_j, b_j \sim \mathrm{Poisson}(2)\) independently, and \[s_j = a_j + c_j, \qquad s'_j = b_j + c_j.\] This yields \(\mathbb{E}[s_j] = \mathbb{E}[s'_j] = 3\), \(\mathrm{Var}(s_j) = \mathrm{Var}(s'_j) = 3\), and \(\mathrm{Cov}(s_j, s'_j) = 1\).

Joint Negative Binomial. Each coordinate is generated as \(c_j \sim \mathrm{NegBin}(r=1, p=0.5)\) and \(a_j, b_j \sim \mathrm{NegBin}(r=2, p=0.5)\), and we set \[s_j = a_j + c_j, \qquad s'_j = b_j + c_j.\] Under the PyTorch parameterization, \(\mathbb{E}[\mathrm{NegBin}(r, p)] = r(1-p)/p\) and \(\mathrm{Var} = r(1-p)/p^2\). Hence \(\mathbb{E}[s_j] = 3\), \(\mathrm{Var}(s_j) = 6\), and \(\mathrm{Cov}(s_j, s'_j) = 2\). Note that this construction uses the negative binomial distribution (rather than the bounded binomial).

Joint Hypergeometric. For each dimension \(j\), we first sample population and success counts \(N_j \sim \text{Uniform}\{10, \ldots, 20\}\) and \(M_j \sim \text{Uniform}\{1, \ldots, N_j - 1\}\), and fix \(n_{1j} = n_{2j} = 1\). We draw without replacement \(n_{1j} + n_{2j} = 2\) items from a population of \(N_j\) containing \(M_j\) successes, and define \[s_j = \text{\#successes in first draw}, \qquad s'_j = \text{\#successes in second draw}.\] The marginals are \(\mathrm{Hypergeometric}(N_j, M_j, n=1)\), equivalently \(\mathrm{Bernoulli}(p_j = M_j/N_j)\), and the two draw indicators are negatively correlated with \[\mathrm{Cov}(s_j, s'_j) = -\frac{M_j (N_j - M_j)}{N_j^2 (N_j - 1)}.\]

12.0.0.3 Implementation Notes.

Each sampling routine draws \(\mathbf{s}\) and caches the paired \(\mathbf{s}'\) for retrieval during conditional evaluation; it thus implements a paired sampler rather than a conditional generator. All coordinates are sampled independently across \(j\).

13 Further Experimental Details and Additional Experimental Results↩︎

[tab:synthetic95identifiability95pa95max] and [tab:synthetic95identifiability95pa95mean] gives the subspace errors measured by maximal and mean principal angels on synthetic data across five candidate distributions.

[ht!]
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tabular}{l*{10}{c}}
        \toprule
        Methods   & \multicolumn{2}{c}{Gaussian}  & \multicolumn{2}{c}{Negative Binomial}  & \multicolumn{2}{c}{Gamma}    & \multicolumn{2}{c}{Poisson}    & \multicolumn{2}{c}{Hypergeometric} \\
        \cmidrule(r){2-3}
        \cmidrule(r){4-5}
        \cmidrule(r){6-7}
        \cmidrule(r){8-9}
        \cmidrule(r){10-11}
            & $\mathbf f$ & $\mathbf f'$   & $\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$  \\
        \midrule
        SwAV & 89.26 $\pm$0.55 & 88.88 $\pm$0.45 & 89.64 $\pm$0.2 & 89.65 $\pm$ 0.26 & 89.49 $\pm$0.25 & 89.32 $\pm$0.46 & 89.67 $\pm$0.27 & 89.5 $\pm$ 0.33 & 86.48 $\pm$0.27 & 89.5 $\pm$0.33\\
        BarlowTwins    & 89.44$\pm$0.86 &89.51 $\pm$0.63             & 85.53 $\pm$ 0.86 & 85.36 $\pm$ 0.59 & 86.16 $\pm$ 1.15 & 88.6 $\pm$ 0.58 & 87.73 $\pm$0.86 & 87.43 $\pm$0.91 &88.93 $\pm$0.51 &89.59 $\pm$0.58\\
        VICReg  &  88.99 $\pm$0.54 & 89.07 $\pm$0.61 & 89.36 $\pm$0.48 & 89.16 $\pm$0.73 & 89.6 $\pm$0.24 & 89.03 $\pm$0.59 & 89.66 $\pm$0.41 & 88.38 $\pm$0.4 & 88.95 $\pm$0.43 & 89.29 $\pm$0.22 \\
        W-MSE & \textbf{7.83 $\pm$0.31} & \textbf{8.11 $\pm$0.24} & \textbf{8.5 $\pm$0.18} & \textbf{8.17 $\pm$0.19} & 8.4 $\pm$0.37 & \textbf{8.01 $\pm$0.32} & \textbf{7.63 $\pm$ 0.42} & 7.61 $\pm$ 0.2 & 8.72 $\pm$0.22 & 8.5 $\pm$ 0.18\\
        DGCCA & 89.33 $\pm$ 1.19 & 88.17 $\pm$0.76 & 89.12 $\pm$0.51 & 89.38 $\pm$0.68 & 88.60 $\pm$0.61 & 89.06 $\pm$0.62 & 89.21 $\pm$0.61 & 89.3 $\pm$ 0.62 & 89.21 $\pm$0.7 & 88.39 $\pm$0.59\\
        DeepCCA   & 8.88 $\pm$0.28 & 8.22 $\pm$0.23 &  8.79 $\pm$0.66 & 9.65 $\pm$0.3  & \textbf{7.93 $\pm$0.4} & 8.81 $\pm$0.45 & 11.43 $\pm$0.59& \textbf{7.37 $\pm$0.45} &\textbf{8.61 $\pm$0.24}&\textbf{7.39 $\pm$0.45}\\
        \bottomrule
     \end{tabular}}
        \caption{Comparison of the maximal principal angles $PA_{max}$ $\downarrow$(°) of both encoders $\mathbf f, \mathbf f'$ on synthetic data ($d_\mathcal{S}=d_\mathcal{Z}=10$).}
        \label{tab:synthetic95identifiability95pa95max}

[ht!]
    \centering
    \resizebox{\textwidth}{!}{
    \begin{tabular}{l*{10}{c}}
        \toprule
        Methods   & \multicolumn{2}{c}{Gaussian}  & \multicolumn{2}{c}{Negative Binomial}  & \multicolumn{2}{c}{Gamma}    & \multicolumn{2}{c}{Poisson}    & \multicolumn{2}{c}{Hypergeometric} \\
        \cmidrule(r){2-3}
        \cmidrule(r){4-5}
        \cmidrule(r){6-7}
        \cmidrule(r){8-9}
        \cmidrule(r){10-11}
            & $\mathbf f$ & $\mathbf f'$   & $\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$   &$\mathbf f$ & $\mathbf f'$  \\
        \midrule
        SwAV & 54.95 $\pm$1.75 & 55.18 $\pm$0.95 & 59.13 $\pm$10.12 & 62.48 $\pm$ 9.98 & 56.68 $\pm$0.59 & 55.17 $\pm$ 0.6 & 57.57 $\pm$0.46 & 56.39 $\pm$0.33 &63.21 $\pm$9.22 & 68.92 $\pm$8.69 \\
        BarlowTwins    & 33.91$\pm$0.16 &32.78 $\pm$0.23             & 30.27 $\pm$0.18 & 28.66 $\pm$0.15 & 33.6 $\pm$ 0.22 & 32.29 $\pm$ 0.17 & 42.18 $\pm$ 0.15 & 40.58 $\pm$ 0.22 &60.77 $\pm$ 0.29 & 60.72 $\pm$ 0.23\\
        VICReg  &  68.47 $\pm$0.44 & 67.89 $\pm$0.63 & 67.56 $\pm$0.45 & 68.91 $\pm$0.32 & 74.27 $\pm$0.18 & 70.39 $\pm$0.25 & 75.46 $\pm$0.27 & 72.95 $\pm$0.31 & 65.8 $\pm$0.43 & 64.75 $\pm$0.29 \\
        W-MSE & 5.12 $\pm$0.09 & 5.15 $\pm$0.03 & 5.21 $\pm$ 0.09 & \textbf{4.74 $\pm$0.07} & 4.75 $\pm$0.06 & \textbf{4.57 $\pm$0.1} & \textbf{4.69 $\pm$ 0.1} & 4.37 $\pm$0.04 & 5.38 $\pm$0.05 & \textbf{5.13 $\pm$0.02}\\
        DGCCA & 67.22 $\pm$ 0.38 & 65.91 $\pm$0.71 & 65.65 $\pm$0.23 & 64.89 $\pm$0.17 & 67.27 $\pm$0.11 & 66.12 $\pm$0.32 & 68.24 $\pm$ 0.25 & 68.82 $\pm$0.33 & 63.12 $\pm$ 0.23 & 63.58 $\pm$0.17\\
        DeepCCA   & \textbf{5.06 $\pm$0.08} & \textbf{4.92 $\pm$0.09} &  \textbf{5.00 $\pm$0.11} & 5.16 $\pm$0.05 & \textbf{4.65 $\pm$0.1} & 4.86 $\pm$0.08 & 5.18 $\pm$0.07& \textbf{4.06 $\pm$0.08} &\textbf{5.37 $\pm$0.07}&6.17 $\pm$0.07\\
        \bottomrule
    \end{tabular}}
        \caption{Comparison of the mean principal angles $PA_{\text{mean}}$ $\downarrow$(°) of both encoders $\mathbf f, \mathbf f'$ on synthetic data ($d_\mathcal{S}=d_\mathcal{Z}=10$).}
        \label{tab:synthetic95identifiability95pa95mean}

References↩︎