0.0.1 Introduction↩︎

Causal representation learning (CRL) seeks to describe high-dimensional, low-level observations through a small number of interpretable, causally-related latent variables [1], [2]. In doing so, its goal is to combine the strengths of classical causal inference with those of modern machine learning. A causal model represents an entire family of distributions arising from interventions on a system of variables [3]–[5]. This provides a principled way for reasoning about distribution shifts, which facilitates out-of-distribution generalization and planning [6]–[9]. However, causal models require that most (or at least some) relevant causal variables are directly observed. While reasonable in domains such as economics [10], social [11] or biomedical science [12], [13], this assumption has challenged the application of causal methodology to complex and high-dimensional data [14]. Machine learning, on the other hand, has proven successful at learning useful “representations”—latent vectors generating the observables via some nonlinear map—of high-dimensional data such as images, video, or text [15]–[18]. However, most methods rely on independent and identically distributed (i.i.d.) data and only extract associational information. As a consequence, they often fail under distribution shifts and do not generalize beyond the training distribution [19], as exhibited by their reliance on spurious correlations [20], [21] or their vulnerability to adversarial examples [22], [23].

Identifiability. It has been argued that to address these shortcomings, we ought to learn representations endowed with causal model semantics. However, a major challenge to this goal is that different representations can explain the same data equally well. Strictly simpler representation learning tasks, such as disentanglement [24] or independent component analysis (ICA) [25], are already non-identifiable in general [26], [27]. Identifiability studies are thus required to characterize additional assumptions under which the desired latent variables can be provably recovered [28]–[51]. In CRL, we need not only identify the latents, but also the causal graph encoding their relations. Even in the fully observed case, this task of causal discovery, or structure learning, is very challenging: the graph can only be recovered up to Markov equivalence [52] based only on (observational) i.i.d.data [53], meaning that the direction of some edges cannot be determined. For CRL, the task gets strictly harder. For instance, if the causal latents \(V_i\) in 1 (Left) form a valid representation, then replacing them by the independent exogeneous \(U_i\) might be considered an equally valid alternative.

What sets causal representations apart? A crucial feature of causal variables is that they are the ones on which interventions are defined and whose relations we are interested in [3]. Causal discovery and CRL thus often rely on non-i.i.d.data linked to interventions on the underlying causal variables [54], [55]. Unless all variables are subject to intervention, however, some fundamental differences between the fully observed and representation learning settings in the level of ambiguity in the graph remain [56], as illustrated in 1 (Right). In a sense, the non-identifiabilities of representation and structure learning combine, and both need to be addressed in conjunction.

Problem Setting. We study the general nonparametric CRL problem (0.0.2.1) in which both the causal mechanisms and the mixing function are completely unconstrained. Our goal ([subsec:learning95target]) is to the latent causal variables up to element-wise nonlinear rescaling and their graph up to isomorphism ([def:identifiability]). As motivated above, doing so without further supervision requires access to interventional data. To this end, we consider learning from heterogeneous data from multiple related domains, or environments, that arise from interventions in a shared underlying causal model (0.0.2.3).

Contributions. Our main contributions are theoretical in nature (0.0.3). First, we establish the minimality of the targeted equivalence class ([prop:equivalence]) in the sense that its ambiguities cannot be resolved from interventional data. We then present our main identifiability results. For the case of two latent causal variables, we show that an observational environment and one for each perfect intervention on either variable suffice ([thm:bivariate])—provided that the intervened and unintervened mechanisms are not “fine-tuned” to each other, which we formalize in the form of a genericity condition 7 . For any number of latent causal variables, we prove that access to pairs of environments corresponding to two distinct perfect interventions on each node guarantees identifiability ([thm:general]). We then question how to use or interpret causal representations (0.0.4), and show that certain quantities, such as the strengths of causal influences among variables, are preserved by all equivalent solutions ([prop:influence]). We sketch possible learning objectives (0.0.5), and empirically investigate training different generative models (0.0.6), finding that only those based on the correct causal structure attain the best fit and identify the ground truth. We conclude by discussing limitations and extensions of our work (0.0.7).

Related Work on Multi-Environment CRL. Most closely related are recent identifiability studies which also leverage multiple environments arising from single node interventions [56], [58]–[60], thus mirroring in different interventional setups the result of [61] based on counterfactual, multi-view data, which is harder to obtain. [56] provide results for linear causal models and linear mixing; [58] consider nonlinear causal models and polynomial mixings, subject to additional constraints on the latent support [62]; and [59] employ a score-based approach for nonlinear causal models and linear mixing. The concurrent work of [60] extends the results of [56] to general nonlinear mixings and linear Gaussian causal models. [63] leverage recent advances in nonlinear ICA [35], [64] to identify a linear Gaussian causal model with context-dependent weights and nonlinear mixing from sufficiently diverse environments. A more extensive discussion of other related works and a detailed comparison of existing results with the present study are provided in 1.0.1 and 1.

0.0.2 Nonparametric Causal Representation Learning↩︎

In this section, we describe the considered problem setting and state our main assumptions. First, we specify the assumed data generating process (0.0.2.1) and learning task in the form of a target identifiability class ([subsec:learning95target]). We then demonstrate the hardness of our task from i.i.d.data or imperfect interventions and use this to motivate a multi-environment approach with perfect interventions (0.0.2.3).

Notation. We use upper-case \(X\) for random variables and lower-case \(x\) for their realizations. Bold uppercase \({\boldsymbol{X}}\) denotes random vectors and lowercase \({\boldsymbol{x}}\) their realizations. We assume throughout that all distributions \(P_X\) possess densities \(p(x)\) with respect to (w.r.t.) the Lebesgue measure. We denote the pushforward of \(P_X\) through a measurable function \(f\) by \(f_*(P_X)\), and write \([n]=\{1, ..., n\}\).

0.0.2.1 Data Generating Process

The assumed data generating process consists of a latent causal model and a mixing function, see 1 (left). For the former, we build on the structural causal model (SCM) framework [3], [4].

By acyclicity, recursive substitution of the assignments in 1 yields the reduced form \({\boldsymbol{V}}{=}f_\mathrm{\small rf}({\boldsymbol{U}})\). The SCM thus induces a unique distribution \(P_{\boldsymbol{V}}\) over the endogenous variables, given by the pushforward of \(P_{\boldsymbol{U}}\) via 1 , that is, \(P_{\boldsymbol{V}}=f_\mathrm{\small rf*}(P_{\boldsymbol{U}})\). By construction, \(P_{\boldsymbol{V}}\) is Markovian w.r.t.the causal graph \(G\) [3], [4], meaning that its density obeys the causal Markov factorization: \[\textstyle \label{eq:causal95factorisation} p(v_1,\dots,v_n) = \prod_{i=1}^n p_i(v_i \mid {\boldsymbol{v}}_{\mathrm{pa}(i)}).\tag{2}\] We place the following additional assumption on the distribution \(P_{\boldsymbol{V}}\) induced by the SCM.

1 ensures a one-to-one correspondence between (conditional) independence in \(P_{\boldsymbol{V}}\) and graphical separation in \(G\) and is a standard assumption in causal discovery [52]. Faithfulness rules out cancellations of influences along different paths, which occurs with probability zero for random path-coefficients [65]. It can thus also be viewed as a minimality or genericity assumption.

In contrast to classical causal inference, we assume that both the exogenous variables \({\boldsymbol{U}}\) and the endogenous causal variables \({\boldsymbol{V}}\) are unobserved. Instead, we will only have access to \(d\)-dimensional nonlinear mixtures \({\boldsymbol{X}}\) of \({\boldsymbol{V}}\). We therefore make the following additional assumption.

Next, we specify the relationship between the unobserved causal variables \({\boldsymbol{V}}\) and the observed \({\boldsymbol{X}}\).

The terminology and setting of a deterministic mixing is rooted in the nonlinear ICA literature [36]. In deep generative models, it is commonly relaxed by considering additive noise in 2  [44], [64]. For representation learning scenarios, we are particularly interested in the case \(n\ll d\). To allow for recovery of \({\boldsymbol{V}}\) from \({\boldsymbol{X}}\), we assume that \(f\) is invertible, which is a standard assumption for identifiability.

0.0.2.2 Learning Target: The CRL Identifiability Class \(\sim_\mathrm{\small crl}\)

Our goal is to infer the underlying latent causal variables \({\boldsymbol{V}}=f^{-1}({\boldsymbol{X}})\) and their causal relations. We therefore consider the true unmixing function \(f^{-1}\) and the causal graph \(G\) our joint learning target: \(f^{-1}\) informs us how to map observations \({\boldsymbol{X}}\) to causal variables \({\boldsymbol{V}}\), and \(G\) tells us how to factorise the implied joint \(p({\boldsymbol{v}})\) into the causal mechanisms \(p_i(v_i~|~{\boldsymbol{v}}_{\mathrm{pa}(i)})\) from 2 . Given only observations of \({\boldsymbol{X}}\), this is a challenging task since neither \({\boldsymbol{V}}\) nor \(G\) are directly observed or known a priori.

When is a candidate solution \((h,G')\) that satisfies a given learning objective (such as maximizing the likelihood, possibly subject to additional constraints) guaranteed to match the ground truth \((f^{-1},G)\)? This is the subject of identifiability studies and the main focus of our work. A statistical model \(\mathcal{P}=\{p_\theta: \theta\in\Theta \}\) with parameter space \(\Theta\) is identifiable if the mapping \(\theta\mapsto p_\theta\) is injective [66]. For representation learning tasks, full identifiability is often not attainable, as there are some fundamental ambiguities that cannot be resolved. One therefore typically instead considers identifiability up to an appropriately chosen equivalence class in the model space [48], [64], [67].

For the assumed data generating process (0.0.2.1), the order of the causal variables is arbitrary, since \({\boldsymbol{V}}\) is unobserved. We can therefore assume without loss of generality (w.l.o.g.) that the \(V_i\)’s are partially ordered w.r.t.\(G\), that is, \(V_i\to V_j \implies i<j\).² Learning \(G\) thus reduces to inferring whether the edges \(\{V_1, \dots, V_{i-1}\}\to V_i\) exist for \(i=2,\dots,n\). The only remaining permutation ambiguity arises from permutations \(\pi\) that preserve the partial order: for example, if \(G\) is given by \(V_1\to V_3 \leftarrow V_2\), the order of \(V_1\) and \(V_2\) cannot be uniquely determined without further assumptions. Moreover, the scaling of the causal variables is also arbitrary: any invertible element-wise transformation can be undone as part of \(f\). We therefore define the desired identifiability class through the following equivalence relation.³

The ground truth \((f^{-1}, G)\) is identified up to \(\sim_\mathrm{\small crl}\) by a given learning objective if any candidate solution \((h,G')\) satisfies \((h,G')\sim_\mathrm{\small crl} (f^{-1}, G)\). We seek to discover suitable conditions that ensure this.

0.0.2.3 Multi-Environment Data

Given only a single dataset of i.i.d.observations from \(P_{\boldsymbol{X}}\), there is no hope for \(\sim_\mathrm{\small crl}\)-identifiability. Even for observed \({\boldsymbol{V}}\) (i.e., with \(n=d\) and known \(f=\mathrm{id}\)), \(G\) can only be identified up to Markov equivalence [52]. With unknown mixing \(f\), the degree of observational non-identifiability gets even worse: for example, by using the reduced form of the SCM (0.0.2.1) we can express \({\boldsymbol{X}}\) in terms of the latent exogenous variables \({\boldsymbol{U}}\) via \({\boldsymbol{X}}=f({\boldsymbol{V}})=f\circ f_\mathrm{\small rf}({\boldsymbol{U}})\) [2]. This gives rise to a “spurious ICA solution” \((f_\mathrm{\small rf}^{-1}\circ f^{-1}, G_\mathrm{\small ica})\) where \(G_\mathrm{\small ica}\) denotes the empty graph with independent components. Due to the non-identifiability of nonlinear ICA [26], [27], however, we cannot even learn the composition \(f\circ f_\mathrm{\small rf}\), let alone separate it into its constituents \(f\) and \(f_\mathrm{\small rf}\) to isolate the intermediate causal variables \({\boldsymbol{V}}\).

Motivated by these challenges to identifiability from i.i.d.data, we instead consider learning from multiple environments \(e\in \mathcal{E}\). That is, we assume access to heterogenous data from multiple distinct distributions \(P^e_{\boldsymbol{X}}\). Environments can arise, for example, from different experimental settings or correspond to broader contexts such as climate or time. Previous work has shown that this setting can, in principle, provide useful causal learning signals [6], [54]–[56], [58], [59], [63], [68]–[81]. However, multi-environment data is not necessarily useful if the corresponding distributions \(P^e_{\boldsymbol{X}}\) are allowed to differ in arbitrary ways. What makes this setting interesting is the assumption that certain parts of the causal generative process are shared across environments.

Here, we assume that all environments share the same invariant mixing function and underlying SCM, and that any distribution shifts arise from interventions on some of the causal mechanisms.⁴ General interventions can be modelled in SCMs by replacing some of the equations in 1 by new assignments \(V_i:=\tilde{f}_i({\boldsymbol{V}}_{\mathrm{pa}(i)}, \tilde{U}_i)\), resulting in “intervened” mechanisms \(\tilde{p}_i(v_i~|~{\boldsymbol{v}}_{\mathrm{pa}(i)})\) replacing the corresponding conditionals \(p_i(v_i~|~{\boldsymbol{v}}_{\mathrm{pa}(i)})\) in 2  [3], [82], [83]. We summarise this as follows.

An intervention on some \(V_i\) that fully removes the influence from its parents \({\boldsymbol{V}}_{\mathrm{pa}(i)}\) is referred to as perfect, otherwise as imperfect. It has been shown that imperfect interventions are generally insufficient for full identifiability [61], even in the linear case [56]. This is intuitive: if arbitrary imperfect interventions were allowed, including ones which preserve \(f_i({\boldsymbol{V}}_{\mathrm{pa}(i)}, \cdot)\) and only replace \(U_i\) with some new \(\tilde{U}_i\), then the spurious ICA solution \((f_\mathrm{\small rf}^{-1}\circ f^{-1}, G_\mathrm{\small ica})\) should be indistinguishable from the ground truth. In line with prior work [56], [58], [59], [61], we therefore assume perfect interventions.

Based on the principle of independent causal mechanisms [4], [70], the sparse mechanism shift hypothesis [1], [80] posits that distribution changes tend to manifest themselves in a sparse or local way in the causal factorization. In this spirit, we will assume single-node (“atomic”) interventions, \(|\mathcal{I}^e|=1\), for our main results, as also required for existing results [56], [58], [59], [61].

As motivated in [subsec:learning95target], given data from \(\{P^e_{\boldsymbol{X}}\}_{e\in\mathcal{E}}\) we consider candidate solutions of the form \((h,G')\) where \(h\) is an unmixing function, or encoder, which maps observations \({\boldsymbol{X}}\) to the inferred latents \({\boldsymbol{Z}}=h({\boldsymbol{X}})\), and \(G'\) is causal graph capturing the relations among the \(Z_i\). The corresponding distributions of the inferred latents are thus given by the push-forward \[\begin{align} \label{eq:QeZ} Q_{\boldsymbol{Z}}^e=h_*(P^e_{\boldsymbol{X}})=(h\circ f)_*P^e_{\boldsymbol{V}} \end{align}\tag{5}\] The multi-environment setup with unknown single-node perfect interventions is illustrated in 2.

0.0.3 Identifiability Theory↩︎

We start by showing that identifiability up to \(\sim_\mathrm{\small crl}\) is, in fact, the best we can hope for when learning from interventional data, without any more direct forms of supervision. To this end, we state the following result, which is presented more formally and proven in 1.0.2.

We now present our identifiability results. We first study the most fundamental bivariate case with two latent causal variables \(V_1\) and \(V_2\). This can loosely be seen as the CRL analogue of the widely studied cause-effect problem (\(X{\to}Y\) or \(Y{\to}X\)?) in classical structure learning [4], [84].

[Proof sketch (full proof in 1.0.3.2).] Consider \(V_1\to V_2\) (the proof for \(V_1\not\to V_2\) is similar). Let \(\psi=f^{-1}\circ h^{-1}:\mathbb{R}^n\to\mathbb{R}^n\) such that \({\boldsymbol{V}}=\psi({\boldsymbol{Z}})\).⁵ By the change of variable formula, for all \({\boldsymbol{z}}\) \[\label{eq:change95of95variable} \textstyle q^e({\boldsymbol{z}})=p^e(\psi({\boldsymbol{z}}))\left|\det {\boldsymbol{J}}_\psi({\boldsymbol{z}})\right|\tag{8}\] where \(({\boldsymbol{J}}_\psi({\boldsymbol{z}}))_{ij}=\frac{\partial \psi_i}{\partial z_j}({\boldsymbol{z}})\) denotes the Jacobian of \(\psi\). We consider two separate cases, depending on whether the intervention targets in \(q^{e_i}\) for \(e_i\in\{e_1,e_2\}\) match those in \(p^{e_i}\) (Case 1) or not (Case 2).

Case 1: Aligned Intervention Targets. According to 4 and (A2), 8 applied to the known observational environment \(e_0\) and the interventional environments \(e_1,e_2\) leads to the system of equations: \[\begin{align} \tag{9} q_1(z_1)q_2(z_2~|~z_{\mathrm{pa}(2;G')})&= p_1\left(\psi_1({\boldsymbol{z}})\right)p_2\left(\psi_2({\boldsymbol{z}})~|~\psi_1({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \\ \tag{10} \tilde{q}_1(z_1)q_2(z_2~|~z_{\mathrm{pa}(2;G')})&= \tilde{p}_1\left(\psi_1({\boldsymbol{z}})\right) p_2\left(\psi_2({\boldsymbol{z}})~|~\psi_1({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \\ \tag{11} q_1(z_1)\tilde{q}_2(z_2)&= p_1\left(\psi_1({\boldsymbol{z}})\right)\tilde{p}_2\left(\psi_2({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \end{align}\] where \(z_{\mathrm{pa}(2;G')}\) denotes the parents of \(z_2\) in \(G'\). Taking quotients of 10 and 9 yields \[\label{eq:triangularJ} \frac{\tilde{q}_1}{q_1}(z_1)=\frac{\tilde{p}_{1}}{p_{1}}(\psi_1({\boldsymbol{z}})) \quad \overset{\frac{\partial}{\partial z_2}}{\implies} \quad 0=\left(\frac{\tilde{p}_{1}}{p_{1}}\right)'\left(\psi_1({\boldsymbol{z}})\right)\frac{\partial \psi_1}{\partial z_2}({\boldsymbol{z}}) \quad \overset{(A3)}{\implies} \quad \frac{\partial \psi_1}{\partial z_2}({\boldsymbol{z}})=0 \,.\tag{12}\] Thus \(V_1=\psi_1(Z_1)\) and \(q_1(z_1)=p_1(\psi_1(z_1))\left|\frac{\partial \psi_1}{\partial z_1}(z_1)\right|\). Substitution into 11 yields \[\begin{align} \label{eq:measure95preserving} \tilde{q}_2(z_2)&= \tilde{p}_2\left(\psi_2(z_1,z_2)\right) \left|\frac{\partial \psi_2}{\partial z_2}(z_1,z_2)\right| \end{align}\tag{13}\] where we have used that, according to 12 , \({\boldsymbol{J}}_\psi\) is triangular. According to 13 , for all \(z_1\), the mapping \(z_2\mapsto \psi_2(z_1,z_2)\) is measure preserving for \(\tilde{q}_2\) and \(\tilde{p}_2\). By 1 [61], it follows that \(\psi_2\) must be constant in \(z_1\).⁶ Hence, \(\psi\) is an element-wise function. Finally, \(G=G'\) follows from faithfulness (1), for otherwise \((V_1,V_2)=(\psi_1(Z_1), \psi_2(Z_2))\) would be independent.

Case 2: Misaligned Intervention Targets. If \(G'\neq G\), a similar argument to Case 1 (with the roles of \(z_1\) and \(z_2\) interchanged) also yields a contradiction to faithfulness. This leaves \(G=G'\). Writing down the system of equations similar to 9 –11 , and then taking ratios of \(e_1\) and \(e_2\) with \(e_0\) yields \[\begin{align} \label{eq:misaligned95ratios} \frac{\tilde{q}_1}{q_{1}}(z_1) =\frac{\tilde{p}_2\left(\psi_2({\boldsymbol{z}})\right)}{p_{2}\left(\psi_2({\boldsymbol{z}})~|~\psi_1({\boldsymbol{z}})\right)} \qquad and \qquad \frac{\tilde{q}_2(z_2)}{q_{2}(z_2~|~z_1)} =\frac{\tilde{p}_1}{p_1}\left(\psi_1({\boldsymbol{z}})\right)\,. \end{align}\tag{14}\] These conditions highlight the misalignment in intervention targets (see 2). Unlike in Case 1, they do not directly imply that some elements of \({\boldsymbol{J}}_\psi\) need to be zero, that is \({\boldsymbol{Z}}\) can be arbitrarily mixed w.r.t. \({\boldsymbol{V}}\). However, 14 imposes constraints on the form of \(\psi\) that, by exploiting the invariance of \(q_1\) across \(e_0\) and \(e_1\) while \(p_1\) changes to \(\tilde{p}_1\), can ultimately be shown to only be satisfied if the two expectations in 7 are equal for all continuous \(\varphi\). However, such fine-tuning is ruled out by (A4).

---

On the Genericity Condition (A4). The condition in 7 contrasts expectations of the same quantity w.r.t. the observational distribution \(P_{\boldsymbol{V}}^{e_0}\) and the interventional distribution \(P_{\boldsymbol{V}}^{e_1}\) corresponding to an intervention on \(V_1\) that turns \(p_1\) into \(\tilde{p}_1\). The shared argument, on the other hand, is a function of the ratio between the intervened mechanism \(\tilde{p}_2\) and its base mechanism \(p_2\). While the two expectations are always equal for linear \(\varphi\), other choices imply non-trivial constraints. For instance, \(\varphi(y)=y^2\) yields \[\begin{align} \label{eq:genericity95example} \int \left(\tilde{p}_1(v_1)-p_1(v_1)\right) \int \frac{\tilde{p}_2(v_2)^2}{p_2(v_2\mid v_1)} \mathop{}\!\mathrm{d}v_2\mathop{}\!\mathrm{d}v_1 \neq 0\,. \end{align}\tag{15}\] Since \(p_1 \neq \tilde{p}_1\) by assumption (A3), we argue that (A4) should generally hold for randomly chosen \((p_1, p_2, \tilde{p}_1, \tilde{p}_2)\) and can only be violated if they are fine-tuned to each other. It can thus be viewed as encoding some notion of genericity—in line with the principle of independent causal mechanisms [4], [46], [70], [89], [90], but also involving the intervened mechanisms. Interestingly, related genericity conditions also arise in the study of nonlinear cause-effect inference from observational data, where identifiability is often obtained up to a set of pathological (“fine-tuned”) spurious solutions satisfying a partial differential equation involving the original mechanisms [91]–[94]. Further, we note that \(\varphi\) in 7 can also be thought of as a witness function of genericity, similar to witness functions in kernel-based two sample and independence testing [95].

Next, we provide our identifiability result for an arbitrary number of causal variables.

[Proof sketch (full proof in 1.0.3.3).] By considering ratios between \(e_j\) and \(e_j'\), taking partial derivatives w.r.t.\(z_l\), and using assumptions (A3’), we can identify a subset \(\mathcal{E}_n\subseteq \mathcal{E}\) of exactly \(n\) pairs of environments corresponding to distinct intervention targets in \(p\) (for otherwise \(\psi\) cannot be invertible). For \((e_i,e_i')\in\mathcal{E}_n\), w.l.o.g.fix the intervention targets in \(p\) to \(\mathcal{I}^{e_i}=\mathcal{I}^{e_i'}=\{i\}\) and let \(\pi\) be a permutation of \([n]\) such that \(\pi(i)\) denotes the inferred intervention target in \(q\) that by (A2’) is shared across \((e_i,e_i')\). By the same argument as before, we must have \(V_i=\psi_i(Z_{\pi(i)})\), that is, \(\psi\) is a permutation composed with an element-wise function. It remains to show that \(\pi\) is a graph isomorphism, that is, \(V_i\to V_j\) in \(G\) \(\iff\) \(Z_{\pi(i)}\to Z_{\pi(j)}\) in \(G'\). We prove \(\implies\); the other direction is analogous. Suppose for a contradiction that there exist \((i,j)\) such that \(V_i\to V_j\) in \(G\), but \(Z_{\pi(i)}\not\to Z_{\pi(j)}\) in \(G'\). Consider \(e_i\) in which there are perfect interventions on \(Z_{\pi(i)}\) and \(V_i\). For \(Z_{\pi(k)}\in{\boldsymbol{Z}}_{\mathrm{pa}(\pi(j);G')}\), let \(\tilde{V}_k=\psi_{k}(Z_{\pi(k)})\) and denote \(\tilde{\boldsymbol{V}}=\cup_k \tilde{V}_k\subset {\boldsymbol{V}}\setminus \{V_i,V_j\}\). Since \(Z_{\pi(i)}\) and \(Z_{\pi(j)}\) are d-separated by \({\boldsymbol{Z}}_{\mathrm{pa}(\pi(j);G')}\) in the post-intervention graph \(G'_{\overline{Z}_{\pi(i)}}\) with arrows pointing into \(Z_{\pi(i)}\) removed [3], it follows by Markovianity of \(q\) w.r.t.\(G'\) that \(Z_{\pi(i)}\mathpalette{\independenT}{\perp}Z_{\pi(j)}~|~{\boldsymbol{Z}}_{\mathrm{pa}(\pi(j);G')}\) in \(Q_{\boldsymbol{Z}}^{e_i}\). By applying the corresponding diffeomorphic functions \(\psi_i\), it follows from 2 in 1.0.3.1 that \(V_i\mathpalette{\independenT}{\perp}V_j~|~\tilde{\boldsymbol{V}}\) in \(P^{e_i}_{\boldsymbol{V}}\). This violates faithfulness (1) of \(P_{\boldsymbol{V}}\) to \(G\) since \(V_i\) and \(V_j\) are d-connected in \(G_{\overline{V}_i}\).

---

(A2’) states that we know that a pair of datasets corresponds to two distinct interventions on the same underlying variable, even though we may not know the exact target of the intervention. This situation could arise, for example, if both datasets are collected under the same experimental setup but with varying experimental parameters. We stress that this is different from data consisting of pairs of views \(({\boldsymbol{x}},\tilde{\boldsymbol{x}})\) sharing the values of some variables, which is counterfactual in nature [32], [61], [96]. One of the main challenges for our analysis (compared to a counterfactual multi-view setting) thus stems from the lack of correspondences across observations from different datasets. We also stress that a purely observational environment is not needed in this case, cf.(A2) in [thm:bivariate].

(A3’) states that the intervention mechanisms are distinct in that their ratio is strictly monotonic, similar to 6 in (A3). This is a slightly stronger version of the assumption of interventional discrepancy proposed by [87],⁷ which has been shown to be necessary for identifiability even if the graph \(G\) is known. For Gaussian \(\tilde{p}_i, \tilde{\raisebox{0pt}[0.85\height]{\tilde{p}}}_i\), (A3’) is satisfied, for example, by a shift in mean. In the proofs, this is used to show that \(\psi\) must be an element-wise function, see 12 . Intuitively, if \({\tilde{p}_i=\tilde{\raisebox{0pt}[0.85\height]{\tilde{p}}}_i}\) in some open set for more than one \(i\), then the underlying variables can be nonlinearly mixed by a measure-preserving automorphism within this set without affecting the observed distributions [87].

0.0.4 Interpreting Causal Representations↩︎

Suppose that we succeed in identifying \({\boldsymbol{V}}\) and \(G\) up to \(\sim_\mathrm{\small crl}\) ([def:identifiability]). How can we use or interpret such a causal representation? Since the scale of the variables is arbitrary ([subsec:learning95target]), we clearly cannot predict the exact outcomes of interventions. We therefore seek causal quantities that are preserved by the irresolvable ambiguities of \(\sim_\mathrm{\small crl}\). A prime candidate for this are interventional causal notions defined in terms of information theoretic quantities [97] and in particular the KL divergence \(D_\mathrm{\small kl}\).

The following result, proven in 1.0.3.4, states that the causal influences are invariant to reparametrisation and equivariant to permutations, the two irresolvable ambiguities of the \(\sim_\mathrm{\small crl}\) equivalence class.

[prop:influence] implies that the strength of causal relations among variables in the inferred graph carry meaning. They can thus be used to uncover changes to the latent causal mechanisms underlying different experimental datasets, for example, to gain scientific insights when combined with domain knowledge.

0.0.5 Learning Objectives↩︎

While our main focus is on studying identifiability, our theoretical insights also suggest approaches to learning causal representations from finite interventional datasets sampled from \(\{P^e_{\boldsymbol{X}}\}_{e\in\mathcal{E}}\). The main idea is to fit the data in a way that preserves the sparsity of interventions [1], [80] by employing the same (un)mixing function and sharing mechanisms across environments (4). We sketch two approaches, which, according to our theory, should both asymptotically identify the ground truth up to \(\sim_\mathrm{\small crl}\) if the set of available environments \(\mathcal{E}\) is sufficiently diverse (and the other assumptions hold).

• Autoencoder Framework: Jointly learn an encoder \(h\), a graph \(G'\), and intervention targets \(\mathcal{I}^e\) such that the encoded latents \({\boldsymbol{Z}}=h({\boldsymbol{X}})\) can be used to reconstruct the observed \({\boldsymbol{X}}\) across all environments, while ensuring all but the intervened mechanisms are shared. Using \(E\) as an environment indicator, the latter corresponds to the constraint \(Z_i \mathpalette{\independenT}{\perp}E~|~{\boldsymbol{Z}}_{\mathrm{pa}(i;G')}\) for \(i\not\in\mathcal{I}^e\), implementable, for example, through a suitable conditional independence test [99]–[101].

• Generative Modelling Approach: Fit a base generative model \((G',p,f)\) and intervention models \((\mathcal{I}^e,\{p^e_i\}_{i\in\mathcal{I}^e})_{e\in \mathcal{E}}\) by maximizing the likelihood of the multi-environment data. For example, given a candidate graph \(G'\) and candidate intervention targets \(\{\mathcal{I}^e\}_{e\in \mathcal{E}}\), learn the base and intervened mechanisms and mixing; then pick the graph and intervention targets that achieve the best fit.

0.0.6 Experiments↩︎

Setup. We experimentally pursue the second, generative approach. Specifically, we model the mixing function generating \({\boldsymbol{X}}\) from \({\boldsymbol{Z}}\) as a normalizing flow [102], [103] with different environment-specific base distributions, determined by the underlying causal graph, intervention targets, and (learnt) base and intervened mechanisms. Here, we focus on the bivariate case with two ground-truth causal latent variables \(V_1\to V_2\). According to [thm:bivariate], this setting should be identifiable from three environments: an observational one and one perfect intervention on each of \(V_1\) and \(V_2\). Our goal is to verify this empirically in light of finite data and optimization issues. We fix the observation dimension to \(d=2\) to facilitate exact likelihood training of the normalizing flows, and fit a separate generative model for each choice of graph and intervention targets.⁹ The base and intervened mechanisms are linear Gaussian and the mixing function a three-layer MLP, see 1.0.4.1 for implementation details.

Results. The results are summarized in 3. Our main findings are two-fold: First, we observe that, in the majority of cases, the well-specified model attains the highest held-out log-likelihood, as shown in 3 (Right). This suggests that the likelihood of otherwise comparable generative models can act as a reliable criterion to select the correct causal graph and intervention targets. Second, we find that the ground truth latent causal variables are approximately identified up to element-wise rescaling (MCC values close to one) by the correctly specified model and not by any other model, as shown in 3 (Left). This indicates that recovering the correct graph and targets is not only sufficient but also necessary for reliable identification of the causal representation. Taken together, these findings are consistent with our notion of \(\sim_\mathrm{\small crl}\)-identifiability ([def:identifiability]) and [thm:bivariate].

In 1.0.4.2, we perform an additional experiment with \(n=3\) variables, nonlinear latent SCMs, fixed causal order without graph search, and one interventional environment per node, thus assaying violations of assumption (A2’) in the context of [thm:general]. Generally, we find that a correct choice of intervention targets can be selected based on model likelihood, leading to approximate identification of the causal variables, even when only the causal order is given, see 4 and 1.0.4.2 for details.

0.0.7 Conclusions and Discussion↩︎

The world is full of domain shifts and different environments. Often, we cannot pin down what exactly differs between two domains, but it may reasonably be modelled as a change in some causal mechanisms. While prior work relied on harder-to-obtain counterfactual data or parametric constraints for identifiability, our study demonstrates that interventional data can be sufficient—even in the nonparametric case where the spaces of mixing functions and mechanisms are infinite dimensional. Our results can be considered a step towards justifying the use of expressive machine learning methods for learning interpretable causal representations from high-dimensional experimental data in situations where parametric assumptions are undesirable, e.g., for complex systems in physics, biology, or medicine.

Weaker Notions of Identifiability. In the present work, we have focused on the strongest notion of identifiability that is achievable in a nonparametric setting ([def:identifiability]). However, subject to the available data and assumptions, identifiability up to \(\sim_\mathrm{\small crl}\) will not always be possible. In this case, weaker notions of identifiability are of interest. For example, we may not be able to uniquely recover variables that are not targeted by interventions [58], [85], [96], or only recover groups of variables up to (non-)linear mixing [58], [59], [96] and the graph up to transitive closure [56] if interventions are imperfect, or soft. A precise characterization of weaker notions of nonparametric identifiability from different types of interventions (cf. [104] for a temporal, semi-parametric setting) is an interesting direction for future work.

Known vs.Unknown Intervention Targets. When intervention targets can be considered known appears to be a more nuanced concept in CRL than in a fully observed setting, see also [87] for an extended discussion. Recall that we assume w.l.o.g.that \(V_1\preceq...\preceq V_n\) and only consider graphs respecting this ordering ([subsec:learning95target]), see also [56]. The intervention targets are then unknown w.r.t.the pre-imposed causal ordering. This is a key aspect that makes our setting more realistic, but also substantially complicates the analysis (see 14 and 2). Similar to [59], for [thm:bivariate] we require a set of exactly \(n\) environments (one intervention on each node).¹⁰ However, we relax this requirement to mere coverage (“at least \(1\)”) in [thm:general] as shown for linear causal models in [56], [60].

Identifiability From One Intervention per Node for Any \(n\). We conjecture that [thm:bivariate] can be generalized to \(n>2\), subject to a suitably adjusted set of genericity conditions involving several intervened and base mechanisms, akin to 7 in the bivariate case. The main challenge to such a generalization appears to be combinatorial, as there are \(n!-1\) ways of misaligning intervention targets across \(p\) and \(q\). In [thm:general], we sidestep this issue by assuming pairs of environments. Thus, while two single-node perfect interventions are sufficient, we do not believe this to be necessary.

On the Assumption of Known \(n\) and Its Relation to Markovianity. 2 relates to the notions of causal sufficiency or Markovianity in classical causal inference, which correspond to the assumption of independent \(U_i\) in 1, implying the causal Markov factorization 2  [3], [4], [52]. With unobserved \({\boldsymbol{V}}\) and unknown \(n\), the notion of “unobserved confounders” gets blurred, since one can, in principle, always construct a causally sufficient system by increasing \(n\) and adding any causes of two or more endogenous variables to \({\boldsymbol{V}}\). 2 then states that the minimum number of variables required to do so is known.¹¹ However, this can lead to very large systems, which may in turn challenge the assumption of an invertible mixing (3). Extensions of our analysis to unknown \(n\), non-Markovian, or non-invertible models constitute an interesting direction for future investigations.

Practicality. The method explored in 0.0.6 requires searching over graphs and intervention targets, which gets intractable even for moderate \(n\). Simultaneously learning an (un)mixing function, causal graph, intervention targets, and mechanisms is challenging. Further work is needed to make methods for nonparametric CRL from multi-environment data more practical, e.g., by exploring the proposed autoencoder framework with a continuous parametrisation of graph [105], [106] and targets [107].

1 Appendices↩︎

1.0.1 Extended Discussion of Related Work↩︎

Prior work on causal representation learning with general nonlinear relationships (both among latents and between latents and observations) and without an explicit task or label typically relies on some form of weak supervision. One example of weak supervision is “multi-view” data consisting of tuples of related observations. [96] consider counterfactual pairs of observations arising from imperfect interventions through data augmentation, and prove identifiability for the invariant non-descendants of intervened-upon variables. [61] also use counterfactual pre- and post-intervention views and show that the latent SCM can be identified given all single-node perfect stochastic interventions. Another type of weak-supervision is temporal structure [67], possibly combined with nonstationarity [108], [109], interventions on known targets [85], [86], or observed actions inducing sparse mechanism shifts [1], [49], [104]. Other works use more explicit supervision in the form of annotations of the ground truth causal variables or a known causal graph [87], [110], [111].

A different line of work instead approaches causal representation learning from the perspective of causal discovery in the presence of latent variables [52]. Doing so from purely observational i.i.d.data requires additional constraints on the generative process, such as restrictions on the graph structure or particular parametric and distributional assumptions, and typically leverages the identifiability of linear ICA [112]–[114]. For linear, non-Gaussian models, [115] show that the causal DAG can be recovered up to Markov equivalence if all observed variables are “pure” in that they have a unique latent causal parent. [116] and [117], [118] extend this result to identify the full graph given two pure observed children per latent, and [119] provide sufficient and necessary graphical conditions for full identification. For discrete causal variables, [120] introduce a similar “no twins” condition to reduce the task of learning the number and cardinality of latents and the Markov-equivalence class of the graph to mixture identifiability.

Other lines of work have investigated the relationship between causal models at different levels of coarse-graining or abstraction [121]–[130], or learning invariant representations in a supervised setting [62], [77]–[79], [88], [131]–[134], often for domain generalization.

1.0.1.1 Comparison of Related Identifiability Results

To complement the presentation of related multienvironment CRL works in [sec:related95work], we provide a structured overview of and comparison with existing identifiability results for causal representation learning in 1. The table categorizes work along different dimensions. First, we make a distinction based on the type of data (observational, interventional, or counterfactual) results rely on (colour coded in green, yellow, and red, respectively). These are also referred to as different rungs in the “ladder of causation” [135] or layers in the Pearl Causal Hierarchy [5]. Second, we categorize work depending on the assumptions placed on the latent causal model and the mixing function. As can be seen, works relying solely on observational data often require restrictive graphical assumptions on the mixing function. On the other hand, access to much more informative counterfactual data has allowed identification even for nonparametric causal models and mixing functions. Our work can be viewed as a step towards addressing the lack of nonparametric identifiability results in the interventional regime.

We emphasize that 1 is not exhaustive: certain relevant works did not easily fit into our categorization or the causal representation learning framework adopted in the present work. For example, not listed are works that leverage temporal structure [49], [67], [85], [86], [104], rely on heterogenous data and distribution shifts which are not directly expressed in terms of or linked to interventions [63], [108], [109], allow for edges from observed to latent variables [119], or require more direct supervision [110], [111].

1.0.2 Proof of Minimality of the CRL Equivalence Class \(\sim_\mathrm{\small crl}\)↩︎

First, let us recall the main statements from [subsec:learning95target].

We now restate this result more formally.

Since \((h,G')\sim_\mathrm{\small crl}(f^{-1},G)\), by [def:identifiability] we have \[\begin{align} \label{eq:Z95in95CRL} {\boldsymbol{Z}}={\boldsymbol{P}}_{\pi^{-1}}\circ \phi ({\boldsymbol{V}}) \end{align}\tag{17}\] for some element-wise diffeomorphism \(\phi\) with inverse \(\psi=\phi^{-1}\). Then 17 implies that for all \(i\in[n]\) \[\begin{align} \label{eq:element95wise95change95of95var} V_i=\psi_i(Z_{\pi(i)}) \end{align}\tag{18}\]

According to 18 , each conditional in the Markov factorization of \(Q_{\boldsymbol{Z}}\) is given in terms of \(p\) by \[\begin{align} \label{eq:transformed95mechanisms} q_{\pi(i)}\left(z_{\pi(i)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right) =p_i\left(\psi_i\left(z_{\pi(i)}\right)~|~\psi_{\mathrm{pa}(i)}\left({\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right)\right) \left|\frac{\mathop{}\!\mathrm{d}\psi_i}{\mathop{}\!\mathrm{d}z_{\pi(i)}}\left(z_{\pi(i)}\right)\right| \end{align}\tag{19}\] where we have used the change of variables in 18 and the fact that \(\pi(\mathrm{pa}(i))=\mathrm{pa}(\pi(i);G')\) since \(\pi:G\mapsto G'\) is a graph isomorphism.

Consider an intervention that changes \(p_i(v_i~|~{\boldsymbol{v}}_{\mathrm{pa}(i)})\) to some intervened mechanism \(\tilde{p}_i(v_i~|~{\boldsymbol{v}}_{\mathrm{pa}(i)})\) for all \(i\in \mathcal{I}^e\). Denote the corresponding intervened joint distribution by \(P^e_{\boldsymbol{V}}\) with joint density \(p^e\) given by \[\label{eq:soft95intervention} p^e({\boldsymbol{v}})= \prod_{i\in \mathcal{I}^e} \tilde{p}_i\left(v_i~|~{\boldsymbol{v}}_{\mathrm{pa}(i)}\right) \, \prod_{j\in[n]\setminus \mathcal{I}^e}p_j\left(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)}\right)\,.\tag{20}\]

Denote by \(Q^e_{\boldsymbol{Z}}=({\boldsymbol{P}}_{\pi^{-1}}\circ \phi)_*(P^e_{\boldsymbol{V}})\) the corresponding distribution over \({\boldsymbol{Z}}\) with joint density given by \(q^e\) \[\begin{align} q^e({\boldsymbol{z}})&=p^e(\psi\circ{\boldsymbol{P}}_{\pi}({\boldsymbol{z}}))\left|\det{\boldsymbol{J}}_{\psi\circ{\boldsymbol{P}}_{\pi}}({\boldsymbol{z}})\right| \\ &= \prod_{i\in \mathcal{I}^e} \tilde{p}_i\left(\psi_i\left(z_{\pi(i)}\right)~|~\psi_{\mathrm{pa}(i)}\left({\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right)\right) \left|\frac{\mathop{}\!\mathrm{d}\psi_i}{\mathop{}\!\mathrm{d}z_{\pi(i)}}\left(z_{\pi(i)}\right)\right| \, \prod_{j\in[n]\setminus \mathcal{I}^e}q_{\pi(j)}\left(z_{\pi(j)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(j);G')}\right)\,, \end{align}\] where we have used 19 , 20 , and the fact that \({\boldsymbol{J}}_\psi\) is diagonal.

By defining \[\begin{align} \label{eq:induced95intervened95mechanism95q} \tilde{q}_{\pi(i)}\left(z_{\pi(i)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right) := \tilde{p}_i\left(\psi_i\left(z_{\pi(i)}\right)~|~\psi_{\mathrm{pa}(i)}\left({\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right)\right) \left|\frac{\mathop{}\!\mathrm{d}\psi_i}{\mathop{}\!\mathrm{d}z_{\pi(i)}}\left(z_{\pi(i)}\right)\right| \end{align}\tag{21}\] we finally arrive at \[\begin{align} q^e({\boldsymbol{z}})= \prod_{i\in \mathcal{I}^e} \tilde{q}_{\pi(i)}\left(z_{\pi(i)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right) \, \prod_{j\in[n]\setminus \mathcal{I}^e}q_{\pi(j)}\left(z_{\pi(j)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(j);G')}\right)\,. \end{align}\]

This shows that any intervention on \(\{V_i\}_{i\in\mathcal{I}^e}\subseteq {\boldsymbol{V}}\) which replaces \[\left\{p_i(v_i~|~{\boldsymbol{v}}_{\mathrm{pa}(i)}) \right\}_{i\in\mathcal{I}^e} \mapsto \left\{\tilde{p}_i(v_i~|~{\boldsymbol{v}}_{\mathrm{pa}(i)})\right\}_{i\in\mathcal{I}^e}\,,\] can equivalently be captured by an intervention on \(\{Z_{\pi(i)}\}_{i\in\mathcal{I}^e}\subseteq{\boldsymbol{Z}}\) which replaces \[\left\{q_{\pi(i)}\left(z_{\pi(i)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right)\right\}_{i\in\mathcal{I}^e} \mapsto \left\{\tilde{q}_{\pi(i)}\left(z_{\pi(i)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right)\right\}_{i\in\mathcal{I}^e}\,.\] with \(\tilde{q}_i\) defined according to  21 .

---

1.0.3 Identifiability Proofs↩︎

1.0.3.1 Auxiliary Lemmata

1.0.3.2 Proof of [thm:bivariate]

From the assumption of a shared mixing \(f\) and shared encoder \(h\) across all environments, we have that \[{\boldsymbol{Z}}=h({\boldsymbol{X}})=h(f({\boldsymbol{V}}))=h\circ f({\boldsymbol{V}})\,.\] Let \(\psi=f^{-1}\circ h^{-1}:\mathbb{R}^n\to\mathbb{R}^n\) such that \[{\boldsymbol{V}}=\psi({\boldsymbol{Z}})\,.\] By 3, \(f\), \(h\), and thus also \(h\circ f\) are diffeomorphisms. Hence, \(\psi\) is well-defined and also diffeomorphic.

By the change of variable formula, for all \(e\) and all \({\boldsymbol{z}}\) the density \(q^e({\boldsymbol{z}})\) is given by \[\begin{align} \label{eq:app95change95of95variable} q^e({\boldsymbol{z}})=p^e(\psi({\boldsymbol{z}}))\left|\det {\boldsymbol{J}}_\psi({\boldsymbol{z}})\right| \end{align}\tag{25}\] where \(({\boldsymbol{J}}_\psi({\boldsymbol{z}}))_{ij}=\frac{\partial \psi_i}{\partial z_j}({\boldsymbol{z}})\) denotes the Jacobian of \(\psi\).

We now consider two separate cases, depending on whether the intervention targets in \(q^{e_i}\) for \(e_i\in\{e_1,e_2\}\) match those in \(p^{e_i}\) (Case 1) or not (Case 2).

Case 1: Aligned Intervention Targets. According to 4 and (A2), 25 applied to the known observational environment \(e_0\) and the interventional environments \(e_1,e_2\) leads to the system of equations: \[\begin{align} \tag{26} q_1(z_1)q_2(z_2~|~z_{\mathrm{pa}(2;G')})&= p_1\left(\psi_1({\boldsymbol{z}})\right)p_2\left(\psi_2({\boldsymbol{z}})~|~\psi_{\mathrm{pa}(2)}({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \\ \tag{27} \tilde{q}_1(z_1)q_2(z_2~|~z_{\mathrm{pa}(2;G')})&= \tilde{p}_1\left(\psi_1({\boldsymbol{z}})\right) p_2\left(\psi_2({\boldsymbol{z}})~|~\psi_{\mathrm{pa}(2)}({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \\ \tag{28} q_1(z_1)\tilde{q}_2(z_2)&= p_1\left(\psi_1({\boldsymbol{z}})\right)\tilde{p}_2\left(\psi_2({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \end{align}\] where \(z_{\mathrm{pa}(2;G')}\) denotes the parents of \(z_2\) in \(G'\), and \(\mathrm{pa}(2)\) denotes the parents of \(V_2\) in \(G\).

Note that neither side of the previous equations can be zero due to the full support assumption¹² (A1) and \(\psi\) being diffeomorphic implying the determinant is non-zero.

Taking quotients of 27 and 26 , yields \[\frac{\tilde{q}_1}{q_1}(z_1)=\frac{\tilde{p}_{1}}{p_{1}}(\psi_1({\boldsymbol{z}}))\,.\] Next, we take the partial derivative w.r.t.\(z_2\) on both sides and use the chain rule to obtain: \[\label{eq:app95zero95product} 0=\left(\frac{\tilde{p}_{1}}{p_{1}}\right)'\left(\psi_1({\boldsymbol{z}})\right)\frac{\partial \psi_1}{\partial z_2}({\boldsymbol{z}})\,.\tag{29}\] Now, by assumption (A3), the first term on the RHS of 29 is non-zero everywhere. Hence, \[\label{eq:app95triangularJ} \forall {\boldsymbol{z}}: \qquad \frac{\partial \psi_1}{\partial z_2}({\boldsymbol{z}})=0 \,.\tag{30}\] It follows that \(\psi_1\) is, in fact, a scalar function, and \[\label{eq:app95reparam95V1} V_1=\psi_1(Z_1)\,.\tag{31}\] Since \(\psi\) is a diffeomorphism, \(\psi_1\) must also be diffeomorphic. Hence, by the change of variable formula applied to 31 , the marginal density \(q_1(z_1)\) is given by \[\label{eq:app95marginal95density} q_1(z_1)=p_1(\psi_1(z_1))\left|\frac{\partial \psi_1}{\partial z_1}(z_1)\right|\,.\tag{32}\]

Further, since \({\boldsymbol{J}}_\psi\) is triangular due to 30 , its determinant is given by \[\label{eq:app95det95triangluar} \left|\det{\boldsymbol{J}}_\psi({\boldsymbol{z}})\right|=\left|\frac{\partial \psi_1}{\partial z_1}(z_1)\, \frac{\partial \psi_2}{\partial z_2}(z_1,z_2)\right|\,.\tag{33}\]

Substituting 32 and  33 into 28 yields (after cancellation of equal terms): \[\begin{align} \label{eq:app95measure95preserving} \tilde{q}_2(z_2)&= \tilde{p}_2\left(\psi_2(z_1,z_2)\right) \left|\frac{\partial \psi_2}{\partial z_2}(z_1,z_2)\right|\,. \end{align}\tag{34}\] The expression in 34 implies that, for all \(z_1\), the mapping \(\psi_2(z_1,\cdot):\mathbb{R}\to\mathbb{R}\) is measure preserving for the differentiable \(\tilde{q}_2\) and \(\tilde{p}_2\). By 1 (Lemma 2 of [61]), it then follows that \(\psi_2\) must, in fact, be constant in \(z_1\), that is \[\label{eq:app95psi95diagonal} \forall {\boldsymbol{z}}: \qquad \frac{\partial \psi_2}{\partial z_1}({\boldsymbol{z}})=0 \,.\tag{35}\]

Note that this last step is where the assumption of perfect interventions (5) is leveraged: the conclusion would not hold for arbitrary imperfect interventions for which 13 would involve \(\tilde{q}_2(z_2~|~z_1)\) and \(p_2\left(\psi_2(z_1,z_2)~|~\psi_{1}(z_1)\right)\).

Hence, we have shown that \(\psi\) is an element-wise function: \[\label{eq:app95element95wise95psi} {\boldsymbol{V}}=(V_1,V_2)=\psi({\boldsymbol{Z}})=(\psi_1(Z_1), \psi_2(Z_2))\,.\tag{36}\]

Finally, since \(\psi\) is a diffeomorphism, 36 implies that \[V_1 \mathpalette{\independenT}{\perp}V_2 \iff Z_1\mathpalette{\independenT}{\perp}Z_2\,.\] It then follows from the faithfulness assumption (1) that we also must have \(G=G'\).

This concludes the proof of Case 1, as we have shown that \((h,G')\sim_\mathrm{\small crl}(f^{-1},G)\) with \(G'=\pi(G)=G\) (\(\pi\) being the identity permutation) and \(h\circ f=\psi^{-1}=:\phi\) an element-wise diffeomorphism.

Case 2: Misaligned Intervention Targets. Writing down the system of equations similar to 26 –28 , but for the case with misaligned intervention targets across \(p\) and \(q\) yields: \[\begin{align} \tag{37} q_1(z_1)q_2(z_2~|~z_{\mathrm{pa}(2;G')})&= p_1\left(\psi_1({\boldsymbol{z}})\right)p_2\left(\psi_2({\boldsymbol{z}})~|~\psi_{\mathrm{pa}(2)}({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \\ \tag{38} \tilde{q}_1(z_1)q_2(z_2~|~z_{\mathrm{pa}(2;G')})&= p_1\left(\psi_1({\boldsymbol{z}})\right) \tilde{p}_2\left(\psi_2({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \\ \tag{39} q_1(z_1)\tilde{q}_2(z_2)&= \tilde{p}_1\left(\psi_1({\boldsymbol{z}})\right)p_2\left(\psi_2({\boldsymbol{z}})~|~\psi_{\mathrm{pa}(2)}({\boldsymbol{z}})\right) \left|\det{\boldsymbol{J}}_{\psi}({\boldsymbol{z}})\right| \,. \end{align}\]

Taking ratios of \(e_1\) and \(e_2\) with \(e_0\) yields \[\begin{align} \tag{40} \frac{\tilde{q}_1}{q_{1}}(z_1) &=\frac{\tilde{p}_2\left(\psi_2({\boldsymbol{z}})\right)}{p_{2}\left(\psi_2({\boldsymbol{z}})~|~\psi_{\mathrm{pa}(2)}({\boldsymbol{z}})\right)} \\ \tag{41} \frac{\tilde{q}_2(z_2)}{q_{2}(z_2~|~z_{\mathrm{pa}(2;G')})} &=\frac{\tilde{p}_1}{p_1}\left(\psi_1({\boldsymbol{z}})\right)\,. \end{align}\]

We separate the remainder of the proof of Case 2 into different subcases depending on \(G\) and \(G'\): as we will see, we can use a similar reasoning as in Case 1, except for the case where both \(G\) and \(G'\) are missing no edge.

Case 2a: \(V_1\not\to V_2\) in \(G\), that is \(\mathrm{pa}(2)=\varnothing\). Then we can proceed similarly to Case 1. First, we take the partial derivative of 40 w.r.t.\(z_2\) to arrive at: \[\label{eq:app952zero95product} 0=\left(\frac{\tilde{p}_{2}}{p_{2}}\right)'\left(\psi_2({\boldsymbol{z}})\right)\frac{\partial \psi_2}{\partial z_2}({\boldsymbol{z}})\,.\tag{42}\] Using (A3), this implies that \(\psi_2\) does not depend on \(Z_2\), that is, \(V_2=\psi_2(Z_1)\).

Next, we again write \(q(z_1)\) in terms of \(p_2(\psi_2(z_1))\) using the univariate change of variable formula, substitute into 39 , cancel the corresponding terms, and arrive at: \[\tilde{q}_2(z_2)= \tilde{p}_1\left(\psi_1(z_1,z_2)\right) \left|\frac{\partial \psi_1}{\partial z_2}(z_1,z_2)\right|\] 1 applied to \(\psi_1(z_1,\cdot)\) which preserves \(\tilde{q}_2\) and \(\tilde{p}_1\) for all \(z_1\) shows that \(\psi_1\) is constant in \(Z_1\), that is \[\label{eq:app95permutation} {\boldsymbol{V}}=(V_1,V_2)=\psi({\boldsymbol{Z}})=(\psi_1(Z_2),\psi_2(Z_1))\,.\tag{43}\] Since \(V_1\mathpalette{\independenT}{\perp}V_2\) by the assumption of Case 2a, it follows from the invertible element-wise reparametrisation above that \(Z_1 \mathpalette{\independenT}{\perp}Z_2\) and hence, by faithfulness, \(Z_1\not\to Z_2\) in \(G'\).

Finally, note that there is no partial order on the empty graph and so \(G'=\pi(G)=G\) and \({\boldsymbol{Z}}={\boldsymbol{P}}_{\pi^{-1}}\cdot \psi^{-1}({\boldsymbol{V}})\) where \(\pi\) is the nontrivial permutation of \(\{1,2\}\).

Case 2b: \(V_1\to V_2\) in \(G\), that is \(\mathrm{pa}(2)=\{1\}\). If \(G'\neq G\), that is \(Z_1\not\to Z_2\) in \(G'\), then the same argument as in Case 2a, this time starting by taking the partial derivative of 41 w.r.t.\(z_1\), can be used to reach the same conclusion in 43 . However, this contradicts faithfulness since \(V_1\not\mathpalette{\independenT}{\perp}V_2\) in \(G\).

Hence, we must have \(G'=G\), and the following two equations must hold for all \({\boldsymbol{z}}\): \[\begin{align} \frac{\tilde{q}_1(z_1)}{q_{1}(z_1)} &=\frac{\tilde{p}_2\left(\psi_2({\boldsymbol{z}})\right)}{p_{2}\left(\psi_2({\boldsymbol{z}})~|~\psi_1({\boldsymbol{z}})\right)} \tag{44} \\ \frac{\tilde{q}_2(z_2)}{q_{2}(z_2~|~z_1)} &=\frac{\tilde{p}_1\left(\psi_1({\boldsymbol{z}})\right)}{p_1\left(\psi_1({\boldsymbol{z}})\right)} \tag{45} \end{align}\]

The remainder of the proof consists of exploring the implications of 45 and  44 , ultimately resulting in a violation of the genericity condition (A4).

To ease notation, define the following auxiliary functions: \[\begin{align} a(z_1)&:=\frac{\tilde{q}_1(z_1)}{q_{1}(z_1)}\,, \\ b({\boldsymbol{v}})&:=\frac{\tilde{p}_2(v_2)}{p_2(v_2~|~v_1)}\,, \\ c({\boldsymbol{z}})&:=\frac{\tilde{q}_2(z_2)}{q_{2|1}(z_2~|~z_1)}\,, \\ d(v_1)&:=\frac{\tilde{p}_1\left(v_1\right)}{p_1\left(v_1\right)}\,. \end{align}\]

With this, 44 and 45 take the following form: \[\begin{align} \tag{46} a(z_1)&=b(\psi({\boldsymbol{z}}))\,. \\ \tag{47} c({\boldsymbol{z}})&=d(\psi_1({\boldsymbol{z}}))\,, \end{align}\]

Next, define the following maps: \[\begin{align} \tag{48} \kappa&: {\boldsymbol{z}}\mapsto \begin{bmatrix} a(z_1) \\ c({\boldsymbol{z}}) \end{bmatrix} \\ \tag{49} \rho&:{\boldsymbol{v}}\mapsto \begin{bmatrix} b({\boldsymbol{v}}) \\ d(v_1) \end{bmatrix} \end{align}\]

Then, 46 and 47 together imply that \[\label{eq:identity95kappa95rho95psi} \kappa=\rho \circ \psi\,.\tag{50}\]

Recalling that by (A1) all densities are continuously differentiable, the Jacobians of \(\kappa\) and \(\rho\) are given by: \[\begin{align} {\boldsymbol{J}}_\kappa({\boldsymbol{z}}) &= \begin{bmatrix} a'(z_1) & 0\\ \frac{\partial c}{\partial z_1}({\boldsymbol{z}}) & \frac{\partial c}{\partial z_2}({\boldsymbol{z}}) \end{bmatrix}\,, \\ {\boldsymbol{J}}_\rho({\boldsymbol{v}}) &= \begin{bmatrix} \frac{\partial b}{\partial v_1}({\boldsymbol{v}}) & \frac{\partial b}{\partial v_2}({\boldsymbol{v}})\\ d'(v_1) & 0 \end{bmatrix}\,, \end{align}\] and the corresponding determinants are given by \[\begin{align} \left|\det {\boldsymbol{J}}_\kappa({\boldsymbol{z}})\right|&=\left|a'(z_1)\frac{\partial c}{\partial z_2}({\boldsymbol{z}})\right|\neq 0 \\ \left|\det {\boldsymbol{J}}_\rho({\boldsymbol{v}})\right|&=\left|d'(v_1)\frac{\partial b}{\partial v_2}({\boldsymbol{v}})\right| \neq 0 \end{align}\] where the inequalities for all \({\boldsymbol{z}}\) follow since, by assumption (A3), the derivatives of ratios of intervened and original mechanisms are non-vanishing everywhere: \[\begin{align} \label{eq:non95vanishing95derivatives} a'(z_1)\neq 0 \neq \frac{\partial c}{\partial z_2}({\boldsymbol{z}}) \qquad and \qquad d'(v_1)\neq 0 \neq \frac{\partial b}{\partial v_2}({\boldsymbol{v}})\,, \end{align}\tag{51}\] This implies that the following families of maps are continuously differentiable, monotonic, and invertible, \[\begin{align} a&: z_1\mapsto a(z_1)\,,\\ b_{v_1}&: v_2 \mapsto b(v_1,v_2)\,,\\ c_{z_1}&: z_2 \mapsto c(z_1,z_2)\,,\\ d&: v_1 \mapsto d(v_1)\,, \end{align}\] with continuously differentiable inverses \[\begin{align} a^{-1}&: w_1\mapsto a^{-1}(w_1)\,,\\ b_{v_1}^{-1}&: w_1 \mapsto b^{-1}_{v_1}(w_1)\,,\\ c_{z_1}^{-1}&: w_2 \mapsto c^{-1}_{z_1}(w_2)\,,\\ d^{-1}&: w_2 \mapsto d^{-1}(w_2)\,. \end{align}\]

This implies that \(\rho\) and \(\kappa\) are valid diffeomorphisms onto their image and their inverses are given by: \[\begin{align} \kappa^{-1}&: {\boldsymbol{w}}\mapsto \begin{bmatrix} a^{-1}(w_1) \\ c^{-1}_{a^{-1}(w_1)}(w_2) \end{bmatrix}\,, \\ \rho^{-1}&:{\boldsymbol{w}}\mapsto \begin{bmatrix} d^{-1}(w_2)\\ b^{-1}_{d^{-1}(w_2)}(w_1) \end{bmatrix}\,. \end{align}\]

Since \({\boldsymbol{V}}=\psi({\boldsymbol{Z}})\), by 50 we have \[\label{eq:W95def} {\boldsymbol{W}}:=\rho({\boldsymbol{V}})=\rho\circ \psi({\boldsymbol{Z}})=\kappa({\boldsymbol{Z}})\,.\tag{52}\]

Denote the distributions of \({\boldsymbol{W}}\) by \(R_{\boldsymbol{W}}\) and its density by \(r({\boldsymbol{w}})\). Since for all \(e\), we have \[P^e_{\boldsymbol{V}}=\psi_*(Q^e_{\boldsymbol{Z}})\] it follows from 52 that \[R_{\boldsymbol{W}}^e=\rho_*(P^e_{\boldsymbol{V}})=\kappa_*(Q^e_{\boldsymbol{Z}})\,.\]

This provides two different ways of applying the change of variable formula to compute \(r({\boldsymbol{w}})\).

First, we consider the pushforward of \(Q^{e_0}_{\boldsymbol{Z}}\) by \(\kappa\): \[\begin{align} r({\boldsymbol{w}}) &= q\left(\kappa^{-1}({\boldsymbol{w}})\right) \left|\det {\boldsymbol{J}}_{\kappa^{-1}}({\boldsymbol{w}})\right| \\ &= q_1\left(a^{-1}(w_1)\right) q_2\left(c^{-1}_{a^{-1}(w_1)}\left(w_2\right)~\mid~a^{-1}(w_1)\right)\left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1}a^{-1}(w_1)\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_2}c^{-1}_{a^{-1}(w_1)}(w_2)\right|\, \label{eq:r95w95q} \end{align}\tag{53}\] By integrating this joint density with respect to \(w_2\), we obtain the following expression for the marginal \(r_1(w_1)\): \[\begin{align} r_1(w_1) &= \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1}a^{-1}(w_1)\right|q_1\left(a^{-1}(w_1)\right)\int q_2\left(c^{-1}_{a^{-1}(w_1)}\left(w_2\right)~\mid~a^{-1}(w_1)\right)\left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_2}c^{-1}_{a^{-1}(w_1)}(w_2)\right|\mathop{}\!\mathrm{d}w_2\,. \end{align}\]

By the diffeomorphic change of variable \(z_2=c^{-1}_{a^{-1}(w_1)}\left(w_2\right)\), ¹³ this can be written as \[\begin{align} r_1(w_1) &= \left|\frac{d}{dw_1}a^{-1}(w_1)\right|q_1\left(a^{-1}(w_1)\right)\int q_2\left(z_2~\mid~a^{-1}(w_1)\right)dz_2\,\\ &=\left|\frac{d}{dw_1}a^{-1}(w_1)\right|q_1\left(a^{-1}(w_1)\right)\label{eq:r1e0q} \end{align}\tag{54}\]

Next, we carry out the same calculation for the pushforward of \(P^{e_0}_{\boldsymbol{V}}\) by \(\rho\): \[\begin{align} r({\boldsymbol{w}}) &= p\left(\rho^{-1}({\boldsymbol{w}})\right) \left|\det {\boldsymbol{J}}_{\rho^{-1}}({\boldsymbol{w}})\right| \\ &=p_1\left( d^{-1}(w_2)\right) p_2\left(b^{-1}_{d^{-1}(w_2)}(w_1)~\mid~d^{-1}(w_2)\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_2} d^{-1}(w_2)\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{d^{-1}(w_2)}(w_1)\right|\,, \end{align}\] leading to the marginal \[\begin{align} r_1(w_1) &= \int p_1\left( d^{-1}(w_2)\right)p_2\left(b^{-1}_{d^{-1}(w_2)}(w_1)~\mid~d^{-1}(w_2)\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{d^{-1}(w_2)}(w_1)\right|\left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_2} d^{-1}(w_2)\right|\mathop{}\!\mathrm{d}w_2\\ &= \int p_1(v_1)p_2\left(b^{-1}_{v_1}(w_1)~\mid~v_1\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{v_1}(w_1)\right|\mathop{}\!\mathrm{d}v_1\label{eq:r1e0p}\,, \end{align}\tag{55}\] where the second line is obtained by the diffeomorphic change of variable \(v_1=d^{-1}(w_2)\).

Equating the two expressions for \(r(w_1)\) in \(e_0\) in 55 and 54 , we obtain for all \(w_1\): \[\left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1}a^{-1}(w_1)\right|q_1\left(a^{-1}(w_1)\right)=\int p_1(v_1)p_2\left(b^{-1}_{v_1}(w_1)~\mid~v_1\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{v_1}(w_1)\right|\mathop{}\!\mathrm{d}v_1\label{eq:r1e0}\,.\tag{56}\]

Applying the same approach to the environment in which \(V_1\) is intervened upon changing \(p_1\) to \(\tilde{p}_1\) while \(Z_2\) is intervened upon leaving \(q_1\) invariant, yields for all \(w_1\): \[\left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1}a^{-1}(w_1)\right|q_1\left(a^{-1}(w_1)\right)=\int \tilde{p}_1(v_1)p_2\left(b^{-1}_{v_1}(w_1)~\mid~v_1\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{v_1}(w_1)\right|\mathop{}\!\mathrm{d}v_1\label{eq:r1e1}\,.\tag{57}\]

Finally, by equating 56 and 57 , we arrive at the following expression which must hold for all \(w_1\): \[\int p_1(v_1)p_2\left(b^{-1}_{v_1}(w_1)~\mid~v_1\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{v_1}(w_1)\right|\mathop{}\!\mathrm{d}v_1=\int \tilde{p}_1(v_1)p_2\left(b^{-1}_{v_1}(w_1)~\mid~v_1\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{v_1}(w_1)\right|\mathop{}\!\mathrm{d}v_1\label{eq:r1diff}\tag{58}\] which we can rewrite as \[\int \left(\tilde{p}_1(v_1)-p_1(v_1)\right)p_2\left(b^{-1}_{v_1}(w_1)~\mid~v_1\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{v_1}(w_1)\right|\mathop{}\!\mathrm{d}v_1=0 \,.\]

Multiplying by any continuous function \(\varphi(w_1)\), integrating w.r.t.\(w_1\) and applying the diffeomorphic change of variable \(v_2=b^{-1}_{v_1}(w_1)\), this can be expressed as: \[\begin{align} 0 &= \int \varphi(w_1) \int \left(\tilde{p}_1(v_1)-p_1(v_1)\right)p_2\left(b^{-1}_{v_1}(w_1)~\mid~v_1\right) \left|\frac{\mathop{}\!\mathrm{d}}{\mathop{}\!\mathrm{d}w_1} b^{-1}_{v_1}(w_1)\right|\mathop{}\!\mathrm{d}v_1 \mathop{}\!\mathrm{d}w_1\label{eq:r1int} \\ &= \int\int \varphi\left(b_{v_1}(v_2)\right) \left(\tilde{p}_1(v_1)-p_1(v_1)\right)p_2(v_2~\mid~v_1) \mathop{}\!\mathrm{d}v_2 \mathop{}\!\mathrm{d}v_1\\ &= \int\int \varphi\left(\frac{\tilde{p}_2(v_2)}{p_2(v_2\mid v_1)}\right) \left(\tilde{p}_1(v_1)-p_1(v_1)\right)p_2(v_2~\mid~v_1) \mathop{}\!\mathrm{d}v_2 \mathop{}\!\mathrm{d}v_1 \end{align}\tag{59}\] where we have resubstituted the expression for \(b_{v_1}(v_2)\) in the last line.

Equivalently, this can be written as: for any continuous function \(\varphi\), \[\mathbb{E}_{{\boldsymbol{v}}\sim P^{e_0}_{\boldsymbol{V}}}\left[\varphi\left(\frac{\tilde{p}_2(v_2)}{p_2(v_2~|~v_1)}\right)\right] = \mathbb{E}_{{\boldsymbol{v}}\sim P^{e_1}_{\boldsymbol{V}}}\left[\varphi\left(\frac{\tilde{p}_2(v_2)}{p_2(v_2~|~v_1)}\right)\right] \, .\]

However, the genericity condition (A4) precisely rules this out, since the above equality must be violated for at least one \(\varphi\), concluding this last case.

To sum up, all cases either lead to a contradiction, or imply the conclusion that \((f^{-1},G)\sim_\mathrm{\small crl}(h,G')\), concluding the proof.

---

1.0.3.3 Proof of [thm:general]

First, we show that we can extract from the \(m\geq n\) available pairs of environments a suitable subset \(\mathcal{E}_n\) of exactly \(n\) pairs, containing one pair of interventional environments for each node.

Let \(\mathcal{E}_n\subseteq \mathcal{E}\) be a subset of \(n\) pairs of environments which are assumed to correspond to distinct targets in the model \(q\), and suppose for a contradiction that this is not actually the case for the ground truth \(p\) (i.e., there are duplicate and missing interventions w.r.t.\(p\)). Then there must be two pairs of environments \((e_a, e_a'),(e_b, e_b')\in \mathcal{E}_n\), both corresponding to interventions on some \(V_i\) in \(p\), but which are modelled as interventions on distinct nodes \(Z_j\) and \(Z_k\) with \(j\neq k\) in \(q\). We show that this implies that \(V_i\) must simultaneously be a deterministic function of only \(Z_j\) and only \(Z_k\). Similar to the proof of [thm:bivariate], we obtain the following equations, \[\begin{align} \frac{\tilde{\raisebox{0pt}[0.85\height]{\tilde{q}}}_{j}}{\tilde{q}_{j}}\left(z_{j}\right)&=\frac{\tilde{\raisebox{0pt}[0.85\height]{\tilde{p}}}_i}{\tilde{p}_i}\left(\psi_i({\boldsymbol{z}})\right)\,,\\ \frac{\tilde{\raisebox{0pt}[0.85\height]{\tilde{q}}}_{k}}{\tilde{q}_{k}}\left(z_{k}\right)&=\frac{\hat{\hat{p_i}}}{\hat{p}_i}\left(\psi_i({\boldsymbol{z}})\right)\,. \end{align}\] By taking partial derivatives w.r.t.\(z_l\) and applying assumption (A3’), we find that \[\begin{align} \frac{\partial \psi_i}{\partial z_l}=0 \qquad \forall l\neq j\,,\\ \frac{\partial \psi_i}{\partial z_l}=0 \qquad \forall l\neq k\,. \end{align}\] Since \(j\neq k\), this implies that \(\partial \psi_i / \partial z_l =0\) for all \(l\) which contradicts invertibility of \(\psi\). Thus, by contradiction, we find that \(\mathcal{E}_n\) must contain exactly one pair of intervention per node also w.r.t.\(p\). For the remainder of the proof, we only consider \(\mathcal{E}_n\).

W.l.o.g., for any \((e_i,e_i')\in\mathcal{E}_n\) we now fix the intervention targets in \(p\) to \(\mathcal{I}^{e_i}=\mathcal{I}^{e_i'}=\{i\}\) and let \(\pi\) be a permutation of \([n]\) such that \(\pi(i)\) denotes the inferred intervention target in \(q\) that by (A2’) is shared across \((e_i,e_i')\). (We will show later that not all permutations are admissible, but only ones that preserve the partial order of \(G\).)

The first part of the proof is similar to Case 1 in the proof of [thm:bivariate]. Consider the densities in environments \(e_i\) and \(e_i'\), which are related through the change of variable formula by: \[\begin{align} \tilde{q}_{\pi(i)}\left(z_{\pi(i)}\right) \prod_{j\in [n]\setminus\{\pi(i)\}} q_j\left(z_j~|~{\boldsymbol{z}}_{\mathrm{pa}(j; G')}\right) &= \tilde{p}_i\left(\psi_i({\boldsymbol{z}})\right) \prod_{j\in [n]\setminus\{i\}} p_j\left(\psi_j({\boldsymbol{z}})~|~\psi_{\mathrm{pa}(j)}({\boldsymbol{z}})\right) \left|\det {\boldsymbol{J}}_\psi({\boldsymbol{z}})\right|\,, \\ \tilde{\raisebox{0pt}[0.85\height]{\tilde{q}}}_{\pi(i)}\left(z_{\pi(i)}\right) \prod_{j\in [n]\setminus\{\pi(i)\}} q_j\left(z_j~|~{\boldsymbol{z}}_{\mathrm{pa}(j; G')}\right) &= \tilde{\raisebox{0pt}[0.85\height]{\tilde{p}}}_i\left(\psi_i({\boldsymbol{z}})\right) \prod_{j\in [n]\setminus\{i\}} p_j\left(\psi_j({\boldsymbol{z}})~|~\psi_{\mathrm{pa}(j)}({\boldsymbol{z}})\right) \left|\det {\boldsymbol{J}}_\psi({\boldsymbol{z}})\right|\,, \end{align}\] where \({\boldsymbol{Z}}_{\mathrm{pa}(j;G')}\subseteq {\boldsymbol{Z}}\setminus \{Z_j\}\) denotes the parents of \(Z_j\) in \(G'\).

Taking the quotient of the two equations yields \[\begin{align} \frac{\tilde{\raisebox{0pt}[0.85\height]{\tilde{q}}}_{\pi(i)}}{\tilde{q}_{\pi(i)}}\left(z_{\pi(i)}\right)&=\frac{\tilde{\raisebox{0pt}[0.85\height]{\tilde{p}}}_i}{\tilde{p}_i}\left(\psi_i({\boldsymbol{z}})\right)\,. \end{align}\]

Next, for any \(j\neq \pi(i)\), taking partial derivatives w.r.t.\(z_j\) on both sides yields \[\begin{align} \label{eq:psi95permutation} 0&=\left(\frac{\tilde{\raisebox{0pt}[0.85\height]{\tilde{p}}}_i}{\tilde{p}_i}\right)'\left(\psi_i({\boldsymbol{z}})\right)\frac{\partial \psi_i}{\partial z_j}({\boldsymbol{z}})\,. \end{align}\tag{60}\]

By assumption (A3’), the first term on the RHS is non-zero everywhere. Hence, 60 implies \[\begin{align} \forall j\neq \pi(i), \, \forall {\boldsymbol{z}}: \quad \frac{\partial \psi_i}{\partial z_j}({\boldsymbol{z}})=0 \end{align}\] from which we can conclude that \[\begin{align} \label{eq:ViZpii} V_i=\psi_i\left(Z_{\pi(i)}\right) \end{align}\tag{61}\] for all \(i\in[n]\). That is, \(\psi\) is the composition of the permutation \(\pi\) with an element-wise reparametrisation.

It remains to show that \(\pi\) must, in fact, be a graph isomorphism, which is equivalent to the statement \[V_i\to V_j \quad in \quad G \quad \iff \quad Z_{\pi(i)}\to Z_{\pi(j)}\quad in \quad G'.\] (\(\implies\)) Suppose for a contradiction that there exist \((i,j)\) such that \(V_i\to V_j\) in \(G\), but \(Z_{\pi(i)}\not\to Z_{\pi(j)}\) in \(G'\).

The main idea is to demonstrate that the lack of such direct arrow implies a certain conditional independence which, by faithfulness, would contradict the unconditional dependence of \(V_i\) and \(V_j\).

Consider environment \(e_i\) in which there are perfect interventions on \(Z_{\pi(i)}\) and \(V_i\), which has the effect of removing all incoming arrows to \(Z_{\pi(i)}\) and \(V_i\) in the respective post-intervention graphs \(G'_{\overline{Z}_{\pi(i)}}\) and \(G_{\overline{V}_i}\).

As a result of this and the lack of direct arrow by assumption, any d-connecting path between \(Z_{\pi(i)}\) and \(Z_{\pi(j)}\) must enter the latter via \({\boldsymbol{Z}}_{\mathrm{pa}(\pi(j);G')}\) [3].

It then follows from Markovianity of \(q\) w.r.t.\(G'\) that the following holds in \(Q_{\boldsymbol{Z}}^{e_i}\): \[\label{eq:app95cond95indep95Z} Z_{\pi(i)}\mathpalette{\independenT}{\perp}Z_{\pi(j)}~|~{\boldsymbol{Z}}_{\mathrm{pa}(\pi(j);G')}\,.\tag{62}\]

We now consider the corresponding implication for \(P^{e_i}_{\boldsymbol{V}}\). Define \[\begin{align} \tilde{\boldsymbol{V}}=\left\{V_k=\psi_k\left(Z_{\pi(k)}\right):Z_{\pi(k)}\in{\boldsymbol{Z}}_{\mathrm{pa}(\pi(j);G')}\right\}\subseteq{\boldsymbol{V}}\setminus\{V_i,V_j\}\,, \end{align}\] and note that by assumption, \(Z_{\pi(i)}\not\in {\boldsymbol{Z}}_{\mathrm{pa}(\pi(j);G')}\) and hence \(V_i\not\in\tilde{\boldsymbol{V}}\).

By applying the corresponding diffeomorphic functions \(\psi_i\) from 61 to 62 , it follows from 2 that \[V_i\mathpalette{\independenT}{\perp}V_j~|~\tilde{\boldsymbol{V}}\] in \(P^{e_i}_{\boldsymbol{V}}\). However, this violates faithfulness (1) of \(P_{\boldsymbol{V}}\) to \(G\) since \(V_i\) and \(V_j\) are d-connected in \(G_{\overline{V}_i}\).

Thus, by contradiction, we must have \(Z_{\pi(i)}\to Z_{\pi(j)}\) in \(G'\).

(\(\Longleftarrow\)) Now, suppose for a contradiction that there exist \((i,j)\) such that \(Z_{\pi(i)}\to Z_{\pi(j)}\) in \(G'\), but \(V_i\not \to V_j\) in \(G\).

By the same argument as before, we find that \[V_i\mathpalette{\independenT}{\perp}V_j ~|~ {\boldsymbol{V}}_{\mathrm{pa}(j)}\] in \(P^{e_i}_{\boldsymbol{V}}\), and thus by 2 \[Z_{\pi(i)}\mathpalette{\independenT}{\perp}Z_{\pi(j)}~|~\tilde{\boldsymbol{Z}}\] in \(Q_{\boldsymbol{Z}}^{e_i}\) where \[\tilde{\boldsymbol{Z}}=\left\{Z_{\pi(k)}:V_k\in{\boldsymbol{V}}_{\mathrm{pa}(j)}\right\}\subseteq{\boldsymbol{Z}}\setminus \{Z_{\pi(i)}, Z_{\pi(j)}\}\,.\] However, this contradicts faithfulness of \(Q_{\boldsymbol{Z}}\) to \(G'\). Hence, we must have that \(V_i \to V_j\) in \(G\).

This shows that \(\pi\) must be a graph isomorphism, thus concluding the proof.

---

1.0.3.4 Proof of [prop:influence]

First, recall that according to 3, \[\mathfrak{C}^{P_{\boldsymbol{V}}}_{i\to j}:=D_\mathrm{\small kl}\big(P_{\boldsymbol{V}}~\big\|~ P^{i\to j}_{\boldsymbol{V}}\big),\] where \(P^{i\to j}_{\boldsymbol{V}}\) denotes the interventional distribution obtained by replacing \(p_j\left(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)}\right)\) with \[\quad p^{i\to j}_j\big(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)\setminus \{i\}}\big) = \int_{\mathcal{V}_i} p_j\big(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)}\big) p_i(v_i) \mathop{}\!\mathrm{d}v_i\,.\] Writing out the KL divergence and noting that all terms except the interved mechanism \(j\) cancel inside the \(\log\), we obtain \[\mathfrak{C}^{P_{\boldsymbol{V}}}_{i\to j} = \int_{\mathcal{V}} \log \left(\frac{p_j\left(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)}\right)}{\int_{\mathcal{V}_i} p_j\big(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)}\big) p_i(v_i) \mathop{}\!\mathrm{d}v_i}\right)p({\boldsymbol{v}})\mathop{}\!\mathrm{d}{\boldsymbol{v}}\,.\] and similarly \[\mathfrak{C}^{Q_{\boldsymbol{Z}}}_{\pi(i)\to \pi(j)} = \int_{\mathcal{Z}} \log \left(\frac{q_{\pi(j)}\left(z_{\pi(j)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(j);G')}\right)}{\int_{\mathcal{Z}_{\pi(i)}} q_{\pi(j)}\left(z_{\pi(j)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(j);G')}\right) q_{\pi(i)}(z_{\pi(i)}) \mathop{}\!\mathrm{d}z_{\pi(i)}}\right)q({\boldsymbol{z}})\mathop{}\!\mathrm{d}{\boldsymbol{z}}\,.\]

Since \({\boldsymbol{Z}}={\boldsymbol{P}}_{\pi^{-1}}\circ \phi({\boldsymbol{V}})\), we have \(V_{i}=\psi_i(Z_{\pi(i)})\) for all \(i\in[n]\) where \(\psi=\phi^{-1}\).

Thus, by the change of variable formula, and using the fact that \(\pi(\mathrm{pa}(i))=\mathrm{pa}(\pi(i);G')\) since \(\pi:G\mapsto G'\) is a graph isomorphism, we have for all \(i\in[n]\): \[\begin{align} \label{eq:app95transformed95mechanisms} q_{\pi(i)}\left(z_{\pi(i)}~|~{\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right) =p_i\left(\psi_i\left(z_{\pi(i)}\right)~|~\psi_{\mathrm{pa}(i)}\left({\boldsymbol{z}}_{\mathrm{pa}(\pi(i);G')}\right)\right) \left|\frac{\mathop{}\!\mathrm{d}\psi_i}{\mathop{}\!\mathrm{d}z_{\pi(i)}}\left(z_{\pi(i)}\right)\right|\,, \end{align}\tag{63}\] as well as for the marginal density \[\begin{align} \label{eq:app95transformed95mechanisms95marginal} q_{\pi(i)}\left(z_{\pi(i)}\right) =p_i\left(\psi_i\left(z_{\pi(i)}\right)\right) \left|\frac{\mathop{}\!\mathrm{d}\psi_i}{\mathop{}\!\mathrm{d}z_{\pi(i)}}\left(z_{\pi(i)}\right)\right|\,, \end{align}\tag{64}\] and \[\begin{align} q({\boldsymbol{z}})=p(\psi\circ{\boldsymbol{P}}_{\pi}({\boldsymbol{z}}))\left|\det {\boldsymbol{J}}_\psi({\boldsymbol{z}})\right|\,. \end{align}\]

Substitution into the expression for \(\mathfrak{C}^{Q_{\boldsymbol{Z}}}_{\pi(i)\to \pi(j)}\) yields: \[\begin{align} \mathfrak{C}^{Q_{\boldsymbol{Z}}}_{\pi(i)\to \pi(j)} &= \int_{\mathcal{Z}} \log \left(\frac{p_{j}\left(\psi_j(z_{\pi(j)})~|~\psi_{\mathrm{pa}(j)}({\boldsymbol{z}}_{\mathrm{pa}(\pi(j);G')})\right)}{\int_{\mathcal{Z}_{\pi(i)}} p_{j}\left(\psi_j(z_{\pi(j)})~|~\psi_{\mathrm{pa}(j)}({\boldsymbol{z}}_{\mathrm{pa}(\pi(j);G')})\right) p_{i}(\psi_i(z_{\pi(i)}))\left|\frac{\mathop{}\!\mathrm{d}\psi_i}{\mathop{}\!\mathrm{d}z_{\pi(i)}}(z_{\pi(i)})\right| \mathop{}\!\mathrm{d}z_{\pi(i)}}\right) \\ & \qquad p(\psi\circ{\boldsymbol{P}}_{\pi}({\boldsymbol{z}}))\left|\det {\boldsymbol{J}}_\psi({\boldsymbol{z}})\right|\mathop{}\!\mathrm{d}{\boldsymbol{z}}\,. \\ &=\int_{\mathcal{V}} \log \left(\frac{p_j\left(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)}\right)}{\int_{\mathcal{V}_i} p_j\big(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)}\big) p_i(v_i) \mathop{}\!\mathrm{d}v_i}\right)p({\boldsymbol{v}})\mathop{}\!\mathrm{d}{\boldsymbol{v}} \\ &=\mathfrak{C}^{P_{\boldsymbol{V}}}_{i\to j}\,. \end{align}\] where the second to last line follows by integration by substitution, applied to both integrals.

---

1.0.4 Experimental Details and Additional Results↩︎

In this appemndix, we describe the experiments presented in 0.0.6 in more details (1.0.4.1), and present additional results (1.0.4.2).

1.0.4.1 Experimental Details for 0.0.6

1.0.4.1.0.1 Synthetic Data Generating Process.

We consider linear Gaussian latent SCMs of the form \[\label{eq:app95exp95scm} V_1 := U_1, \qquad \qquad V_2 := \alpha V_1 + U_2, \quad\tag{65}\] with standard normal \(U_1\) and \(U_2\). As a mixing function, we use a three-layer multilayer perceptron (MLP), \[f = \sigma \circ {\boldsymbol{W}}_3 \circ \sigma \circ {\boldsymbol{W}}_2 \circ \sigma \circ {\boldsymbol{W}}_1 \label{eqn:random95mlp}\tag{66}\] where \({\boldsymbol{W}}_1, {\boldsymbol{W}}_2, {\boldsymbol{W}}_3 \in \mathbb{R}^{2 \times 2}\) are invertible weight matrices, and \(\sigma\) is an element-wise invertible nonlinear leaky-tanh activation function used in [137]: \[\sigma(x) = \tanh(x) + 0.1 x \, .\]

To compute averages of our results over multiple runs, we construct different ground truth data generating processes as follows. We generate different latent SCMs by drawing \(\alpha\) uniformly from \([-10, -2] \cup [2, 10]\). (We exclude \((-2,2)\) to avoid sampling near unfaithful models.) We generate the corresponding mixing functions by uniformly sampling each element of the weight matrices, \(({\boldsymbol{W}}_k)_{ij}\sim U(0,1)\). (To avoid the sampled weight matrices being too close to singular, we reject and resample if \(\left|\det {\boldsymbol{W}}_k\right| < 0.1\).)

1.0.4.1.0.2 Interventional Environments.

In line with [thm:bivariate], for each choice of latent SCM and mixing function, we generate three environments: one observational environment and one interventional environment for each perfect single-node intervention. For \(i=1,2\), we model a perfect intervention on \(V_i\) by removing the influence of the parent variables and changing the exogenous noise by shifting its mean up or down. Specifically, we replace the corresponding assignment in 65 by \[V_i := \tilde{U}_i\, , \quad \text{where} \quad \tilde{U}_i \sim \mathcal{N}(m_i, 1) \,\] where the mean \(m_i\) of the shifted Gaussian noise is fixed per environment and sampled uniformly from \(\{\pm 2\}\).

We label the observational environment as \(e=0\) and the environment arising from intervention on \(V_i\) by \(e=i\) for \(i=1,2\). Samples from \(p^e\) are then generated by sampling latents \({\boldsymbol{v}}\) from the respective (un)intervened SCM and then applying the mixing function.

1.0.4.1.0.3 Model Architecture.

We use normalizing flows [102] to model observations \({\boldsymbol{x}}\) as the result of an invertible, differentiable transformation \(g\) of some latent (noise) variable \({\boldsymbol{z}}\), \[{\boldsymbol{x}}= g ({\boldsymbol{z}})\, .\] We apply a series of \(L\) such transformations \(g^l:\mathbb{R}^2 \rightarrow \mathbb{R}^2\) such that \(g= g^L \circ \ldots \circ g^1\) which we refer to as flow layers. We use Neural Spline Flows [103] for the invertible transformation, with a 3-layer feedforward neural network with hidden dimension \(128\) and permutation in each flow layer and \(L=12\) layers. The transformations \(g, g^1, \ldots, g^L\) have learnable parameters (the weights and biases of the neural networks), which we omit to simplify notation.

Typically, simple distributions such as a uniform or isotropic Gaussian are used as base distribution \(q({\boldsymbol{z}})\) in normalizing flows. Here, we instead choose a base distribution that encodes information about the latent SCM. Specifically, we model the base mechanism as \[\label{eq:app95model95obs} q_1(z_1) = \mathcal{N} \left( {\mu}_1, {\sigma}_1^2 \right)\,, \qquad q_2(z_2 \mid z_1) = \mathcal{N} \left( \hat{\alpha} z_1, {\sigma}_2^2 \right)\, ,\qquad q_2(z_2)=\mathcal{N}\left(\mu_2,\hat{\sigma}_2^2\right)\tag{67}\] and the intervened mechanism as \[\tilde{q}_1(z_1) = \mathcal{N} ( {\tilde{\mu}}_1, {\tilde{\sigma}}_1^2)\,, \qquad \tilde{q}_2(z_2) = \mathcal{N} ( {\tilde{\mu}}_2, {\tilde{\sigma}}_2^2)\,.\]

1.0.4.1.0.4 Candidate Graphs and Intervention Targets.

We train a separate normalizing-flow based model for each choice of candidate graph \(G'\) and inferred intervention targets. For the bivariate case with \(n=2\), this gives rise to four models, depending on whether \(G'\) matches \(G\) or not, and whether the intervention targets are aligned or misaligned w.r.t.the ground truth intervention targets. To model the setting \(G'\neq G\) in which \(Z_1\) and \(Z_2\) are assumed independent, we use \(q_2(z_2)\) in place of \(q_2(z_2~|~z_1)\) in 67 . If the intervention targets are aligned, we use \(\tilde{q}_i\) instead of \(q_i\) in \(e=i\) for \(i=1,2\). Else, if they are misaligned, we use \(\tilde{q}_2\) instead of \(q_2\) in \(e=1\) and \(\tilde{q}_1\) instead of \(q_1\) in \(e=2\). By multiplying the respective mechanisms, we thus obtain three environment-specific joint base distributions \(q^e({\boldsymbol{z}})\) for \(e=0,1,2\).

1.0.4.1.0.5 Learning Objective.

Given multi-environment data, the parameters \({\mu}_1\), \({\sigma}_1\), \(\hat{\alpha}\), \({\sigma}_2\), \(\mu_2\), \(\hat{\sigma}_2\), \({\tilde{\mu}}_2\), \({\tilde{\sigma}}_2\), \({\tilde{\mu}}_1\) and \({\tilde{\sigma}}_1\) are jointly learned with the parameters of the invertible transformations \(g^l\) by maximising the log-likelihood of the data under our model, which is given by: \[\sum_{e\in\mathcal{E}}\mathbb{E}_{{\boldsymbol{x}}\sim p^e({\boldsymbol{x}})}\left[ \log p^e_\text{model}({\boldsymbol{x}})\right]= \sum_{e\in\mathcal{E}}\mathbb{E}_{{\boldsymbol{x}}\sim p^e({\boldsymbol{x}})}\left[ \log q^e(h({\boldsymbol{x}}))+\log \left|\det {\boldsymbol{J}}_h({\boldsymbol{x}})\right|\right]\] where the encoder \(h:=g^{-1}\) is the inverse of the normalizing flow which is readily available by construction; and where the expectations are empirical averages over the respective datasets in practice.

1.0.4.1.0.6 Training and Model Selection Details.

Each environment comprises a total of \(200\)k data points. We use the ADAM optimizer [138] with cosine annealing learning rate scheduling, starting with a learning rate of \(5\times 10^{-3}\) and ending with \(1\times 10^{-7}\). We train the model for \(200\) epochs with a batch size of \(4096\). We split the dataset into \(70\%\) for training, and \(15\%\) for validation and held-out test data, each sampled randomly across all environments. For each drawn data generating process, we train three versions of each model with different random initializations and select the one with the highest validation log likelihood at the end of training for evaluation.

1.0.4.1.0.7 Evaluation Metrics.

We evaluate the trained models w.r.t.mean correlation coefficient (MCC) on held-out data and log-likelihood on validation data (for model selection).

• The MCC measures the extent to which there is a one-to-one correspondence between the ground truth latents \(V_i\) and (a permuted version of) the inferred latents \(Z_i=h_i({\boldsymbol{X}})\). Its maximum value of one indicates a perfect correlation between the two. MCC is thus a proxy measure for the level of identifiability up to permutation and invertible reparametrisation. We report MCC based on Pearson (linear) correlation, though we found the results based on Spearman (nonlinear monotonic) correlation to be almost identical.

• The log-likelihood, on the other hand, measures how well a model explains or fits the data. Since the ground truth is typically unknown, a reasonable procedure when training multiple models is to select the one that attains the highest likelihood. For this reason, we report the difference in log-likelihood between misspecified models (ones assuming a wrong graph or intervention targets) to the correctly specified model. Whenever this difference is larger than zero, the correct model fits the data better and would thus be selected.

1.0.4.2 Additional Results: Learning Nonlinear Latent SCMs from Partial Causal Order

In this subsection, we present an additional experiment, in which we extend the setting investigated in 0.0.6 and 1.0.4.1 along the following axes.

• We fit generative models over three instead of two variables, corresponding to the setting of [thm:general].

• The ground-truth SCM is now given by nonlinear mechanisms with non-additive, non-Gaussian noise.

• The generative model, including the learnt mechanisms, is now fully nonlinear.

• Despite [thm:general] formally requiring two environments per single-node intervention, we only provide one interventional environment per node.

• Rather than searching over candidate graphs, we only fix the causal order and fit the reduced form of the SCM (see 0.0.2.1) with a second normalizing flow.

Below, we describe these differences in more detail.

1.0.4.2.0.1 Three-Variable Graph.

The unknown ground truth graph is given by \[V_2 \leftarrow V_1 \rightarrow V_3 \,. \label{eqn:gt95dag95nonparam}\tag{68}\] This is consistent with the partial ordering \(V_1\preceq V_2 \preceq V_3\), which is assumed for all models a priori w.l.o.g., see [subsec:learning95target]. Note that, due to the encoding of causal structure in the nonparametric model explained below, we only iterate over different permutations of the intervention targets and not over latent graph configurations. Due to the causal order implied by the graph 68 , the permutations \((1, 2, 3)\) (no permutaion) and \((1, 3, 2)\) (permutation of the two effects) are equivalent since the latter also implies the correct causal ordering.

1.0.4.2.0.2 Nonlinear, Non-Gaussian SCM.

The mechanisms in the ground-truth SCM are now given by \[V_i := \beta f_i^\mathrm{loc}({\boldsymbol{V}}_{\mathrm{pa}(i)}) + f_i^\mathrm{scale}({\boldsymbol{V}}_{\mathrm{pa}(i)}) U_i\] for all \(i\), where the location and scale functions \(f_i^\mathrm{loc}\), \(f_i^\mathrm{scale} : \mathbb{R}^{|\mathrm{pa}(i)|} \to \mathbb{R}\) are parameterized by random 3-layer neural networks (sampled as the random mixing function in 66 ) and the noise variables are Gaussian, \(U_i \sim \mathcal{N}(0, 1)\). The factor \(\beta\) controls the influence of the parent variables relative to the exogenous noise. As \(\beta\) increases, variables tend to become more dependent, as also the mean shifts as a function of the parent variables. We set \(\beta=10\) for the experiments shown in 4.

1.0.4.2.0.3 Nonparametric Latent SCM.

We use a second normalizing flow to learn a reduced form of the latent SCM via the transformation \(g^\mathrm{SCM}: \mathbb{R}^3 \to \mathbb{R}^3\) mapping an exogenous noise variable \(\boldsymbol{\epsilon}\) to the latent variable \({\boldsymbol{z}}\), \[\label{eq:reduced95form95flow} {\boldsymbol{z}}= g^\mathrm{SCM}(\boldsymbol{\epsilon})\,.\tag{69}\] The distribution of the exogenous noise variable \(\boldsymbol{\epsilon}\) as well as the distribution of the intervened mechanisms \(\tilde{q}_i(z_i)\) for \(i=1,2,3\) is fixed and standard (isotropic) Gaussian. The flow layers in \(g^\mathrm{SCM}\) have an upper triangular Jacobian and thus allow us to encode assumptions about the causal graph: by passing the variables in topological order, which we can assume w.l.o.g., we ensure that an exogenous noise variable \(\epsilon_i\) can only influence endogenous variables in \({\boldsymbol{z}}\) that are descendants of \(z_i\). The learned weights of the flow layers then implicitly encode which endogenous variables are connected. Therefore, only different choices of the permutations of the intervention targets need to be considered as candidate models. We use a similar architecture based on Neural Spline Flows. However, we omit permutation layers, which would violate the topological order of the variables.

1.0.4.2.0.4 Results.

In 4, we present identifiability scores and model fits for both well-specified and misspecified models (corresponding to different intervention target choices). Notably, we observe that the well-specified model (in blue) or its equivalent (in yellow) yield the highest log-likelihood in the majority of cases, as depicted in 4 (Right). This demonstrates that, even in this nonparametric setting without fully specified graph, the log-likelihood remains a reliable criterion for selecting the correct intervention targets. 4 (Left) shows that the selected models (blue or yellow) approximately identify the ground-truth latent variables up to element-wise rescaling, whereas other choices lead to much lower MCCs.

It is worth noting that, compared to the parametric setting investigated in 0.0.6 and 3, the nonparametric setting appears to be more challenging (as expected), as there is a less pronounced distinction between well-specified and misspecified models, both in terms of identifiability scores and model fits. Moreover, future work is needed to parse the implicitly learned causal relationships in the transformation \(g^\mathrm{SCM}\) in 69 : since only the (pre-imposed) causal order is specified, in practice, \(g^\mathrm{SCM}\) may learn to use additional or fewer edges than in the true graph \(G\).

1.0.5 Discussion of the Role of Our Assumptions↩︎

Below, we summarize the rationale and intuition behind each assumption:

• 1 is required to rule out degenerate cases (cancellation along different paths) in which variables are (conditionally) independent despite being causally related. It is a standard assumption in classical causal discovery, and therefore also needed in CRL to recover the causal graph.

• 2 is required to know how many latent variables we are looking for. It is a standard assumption in identifiable representation learning (that is often made implicitly). However, it may be dropped when suitable techniques for estimating the intrinsic dimensionality of \(\mathcal{X}\) can be employed.

• 3 is needed for the mapping between latents and observations to be invertible in the first place. Without it, full recovery of the causal variables (up to CRL equivalence) is infeasible. This assumption is also standard for the simpler problem of nonlinear ICA.

• 4 is a characterisation of our generative setup. Sharing of some mechanisms and the mixing function is needed for the multi-environment setting to provide useful additional information: if everything may change across environments, the datasets can only be analysed in isolation, running into the non-identifiability of CRL from iid data.

• 5 and (A2) / (A2’) are needed since with imperfect interventions or interventions not on all nodes, identifiability is not achievable even in the linear setting as shown by [56].

• Asm.(A1) is a technical assumption needed for our analysis. It is not strictly necessary (it can also be relaxed to fully supported on a Cartesian product of intervals) but substantially eases the readability and accessibility of the proof, without a major impact on the main causal aspects of the problem setup.

• Asm.(A3) / (A3’) is needed to avoid spurious solutions based on applying a measure preserving transformation on a part of the domain unaffected by the intervention.

• Asm.(A4) is needed to rule out a fine-tuning of the ground-truth generating process that are possible due to fully non-parametric nature of the setup, see also Remark 4.2 and the following paragraph.

References↩︎

---

1. A diffeomorphism is a bijective function \(f\) such that both \(f\) and \(f^{-1}\) are continuously differentiable.↩︎

2. If they were not in such order to begin with, we could apply an appropriate permutation \(\tilde{\pi}\) to \({\boldsymbol{V}}\) and incorporate the inverse permutation into the unknown mixing \(f\) without affecting \({\boldsymbol{X}}=f({\boldsymbol{V}})=(f\circ \tilde{\pi}^{-1})(\tilde{\pi}({\boldsymbol{V}}))\) [56].↩︎

3. \(\sim_\mathrm{\small crl}\) satisfies symmetry and transitivity because permutations and element-wise functions commute.↩︎

4. One could also say that there exists only one environment and each dataset corresponds to an intervention.↩︎

5. By 3, \(f\), \(h\), and thus also \(h\circ f\) are diffeomorphisms. Hence, \(\psi\) is well-defined and also diffeomorphic.↩︎

6. This step is where the assumption of perfect interventions (5) is leveraged: the conclusion would not hold for arbitrary imperfect interventions for which 13 would involve \(\tilde{q}_2(z_2~|~z_1)\) and \(p_2\left(\psi_2(z_1,z_2)~|~\psi_1(z_1)\right)\).↩︎

7. Cf.the interventional regularity assumption of [59] which instead considers partial derivatives w.r.t.the parents and is related to c-faithfulness [55].↩︎

8. Intuitively, this intervention captures the process of removing the arrow \(V_i\to V_j\) in \(G\) and “feeding” the conditional \(p(v_j~|~{\boldsymbol{v}}_{\mathrm{pa}(j)})\) with an independent copy of \(V_i\), distributed according to its marginal, see [98] for details.↩︎

9. This can be viewed as a nested approach in which the inner loop corresponds to the method for Causal Component Analysis (CauCA) of [87], and the outer loop to a search over \(G'\) and \(\{\mathcal{I}^e\}_{e\in\mathcal{E}}\). Code to reproduce our experiments is available at: https://github.com/akekic/causal-component-analysis.↩︎

10. By dropping the assumption of a fixed ordering and considering all DAGs, in this case one could also call \(V_i\) the variable intervened upon in environment \(i\), and then consider the targets “known” in this sense.↩︎

11. Techniques for estimating the intrinsic dimensionality \(n\) of the observation manifold \(\mathcal{X}\subseteq \mathbb{R}^d\) or methods rooted in Bayesian nonparametrics could provide a means of relaxing this assumption.↩︎

12. This can also be relaxed to fully supported on a Cartesian product of intervals.↩︎

13. Note that: \(\int q_2(z_2(w_2))\left|\frac{\mathop{}\!\mathrm{d}z_2}{\mathop{}\!\mathrm{d}w_2}\right|\mathop{}\!\mathrm{d}w_2=\int q_2(z_2)\mathop{}\!\mathrm{d}z_2\) .↩︎