Identifiability of latent-variable and structural-equation models: from linear to nonlinear

Abstract

An old problem in multivariate statistics is that linear Gaussian models are often unidentifiable. In factor analysis, an orthogonal rotation of the factors is unidentifiable, while in linear regression, the direction of effect cannot be identified. For such linear models, non-Gaussianity of the (latent) variables has been shown to provide identifiability. In the case of factor analysis, this leads to independent component analysis, while in the case of the direction of effect, non-Gaussian versions of structural equation modeling solve the problem. More recently, we have shown how even general nonparametric nonlinear versions of such models can be estimated. Non-Gaussianity is not enough in this case, but assuming we have time series, or that the distributions are suitably modulated by observed auxiliary variables, the models are identifiable. This paper reviews the identifiability theory for the linear and nonlinear cases, considering both factor analytic and structural equation models.

A. Liouville’s theorem and nonlinear ICA

A possible line of research for making nonlinear ICA identifiable is imposing suitable conditions on the Jacobian of the transformation. The conditions are typically related to orthogonality of the Jacobian, also called local isometry (Gresele et al. 2021; Zimmermann et al. 2021; Buchholz et al. 2022). In this appendix, we consider the set of invertible mappings within a real space, i.e., such that the dimension is not changed, as is typical in ICA theory. We show that functions which are locally isometric and $\mathbb {R}^n \rightarrow \mathbb {R}^n$ are necessarily affine, and thus do not provide a meaningful basis for nonlinear ICA theory. We do this by proving a variant of the theorem on conformal mappings by Liouville. While the result can be considered well known, we provide a very simple proof for this variant, which is difficult to find in the literature.

1.1 A.1 Definitions

We start by some definitions:

Definition 1

Let $\textbf{f}$ be a differentiable mapping from an open subset U of $\mathbb {R}^n$ to an open subset V of $\mathbb {R}^n$. The mapping called conformal if

$$\begin{aligned} \textbf{J}\textbf{f}(\textbf{x})^T \textbf{J}\textbf{f}(\textbf{x})=c(\textbf{x}) \textbf{I} \end{aligned}$$

(27)

where $\textbf{J}\textbf{f}$ is the Jacobian matrix (of partial derivatives) of $\textbf{f}$, and $c(\textbf{x})$ takes positive scalar values.

The geometrical meaning of the definition is that the mapping preserves angles, but the scaling can be changed by $c(\textbf{x})$. An important special case is obtained when the Jacobian is orthogonal:

Definition 2

If in Def. 1, $c(\textbf{x})\equiv 1$, the mapping is called locally isometric.

In a geometrical interpretation, the mapping is required to preserve both angles and the scaling. This is related to the definition used by Donoho and Grimes (2003), and based on them, by Horan et al. (2021). However, those authors consider the case where the dimension is reduced by the mapping $\textbf{f}$, and thus our results are rather different. In fact, local isometry is typically defined in the context of manifolds which are of a lower dimension than the space itself, and therefore, our definition is a special case of the more conventional one. The orthogonality of the Jacobian has also been used, more heuristically, as a regularizer (Wei et al. 2021; Kumar and Poole 2020).

Merely for simplicity, we further introduce the following terminology:

Definition 3

A function $\textbf{f}$ is called orthogonally affine if it is of the form

$$\begin{aligned} \textbf{f}(\textbf{x})=\textbf{U}\textbf{x}+\textbf{b}\end{aligned}$$

(28)

for some constant vector $\textbf{b}$ and an orthogonal matrix $\textbf{U}$.

Such a function is also called “rigid motion” in some contexts.

1.2 A.2 Locally isometric functions are orthogonally affine

Our analysis is based on a well-known theorem on conformal mappings by Liouville from 1850. To keep the presentation short, we simply provide our variant of the theorem, and refer the reader to the literature for the original theorem (Flanders 1966; Nevanlinna 1960). Our variant of the theorem is as follows:

Theorem 2

Assume $\textbf{f}$ is locally isometric (Def. 2), and in $\mathcal {C}^2$ (i.e., with two continuous derivatives), in a real space of dimension $n\ge 2$. Then, $\textbf{f}$ is orthogonally affine (Def. 3).

Proof

This proof is closely related to the first half of the proof of Liouville’s theorem by Flanders (1966); see Nevanlinna (1960) for another well-known proof.

We drop the argument $\textbf{x}$ for notational simplicity, and denote by $\textbf{w}\cdot \textbf{v}$ the dot-product. Denote by $\textbf{J}_i$ the i-th column of the Jacobian of $\textbf{f}$, i.e., the vector of the partial derivatives with respect to $x_i$. Consider the i, j-th element of the matrix equation defining isometry by orthogonality of the Jacobian:

$$\begin{aligned} \textbf{J}_i \cdot \textbf{J}_j=\delta _{i=j} \end{aligned}$$

(29)

Take the derivative with respect to $x_k$ of both sides:

$$\begin{aligned} \textbf{J}_i\cdot \textbf{J}_j^k + \textbf{J}_j\cdot \textbf{J}_i^k= 0 \end{aligned}$$

(30)

where $\textbf{J}_i^k$ is the vector of the partial derivatives with respect to $x_k$ of the entries of $\textbf{J}_i$. Thus, we get the following skew-symmetricity condition:

$$\begin{aligned} \textbf{J}_i\cdot \textbf{J}_j^k = - \textbf{J}_j\cdot \textbf{J}_i^k .\end{aligned}$$

(31)

On the other hand, by the fact that the order of derivation can be changed, we have the symmetricity condition:

$$\begin{aligned} \textbf{J}_i\cdot \textbf{J}_j^k = \textbf{J}_i\cdot \textbf{J}_k^j .\end{aligned}$$

(32)

Obviously, a matrix which is both symmetric and skew-symmetric is necessarily zero. Here, we actually have a tensor but we show next that the same principle applies to such a tensor; this result is sometimes known as the Braid Lemma. Take any indices i, j, k, and apply (31) and (32) in alternation, three times. We get

$$\begin{aligned} \textbf{J}_i\cdot \textbf{J}_j^k = - \textbf{J}_j\cdot \textbf{J}_i^k = - \textbf{J}_j\cdot \textbf{J}_k^i = \textbf{J}_k\cdot \textbf{J}_j^i =\textbf{J}_k\cdot \textbf{J}_i^j = - \textbf{J}_i\cdot \textbf{J}_k^j = - \textbf{J}_i\cdot \textbf{J}_j^k .\end{aligned}$$

(33)

The equality of the first and last term shows that they must be zero. This holds for all i, and the $\textbf{J}_i$ form an orthogonal basis, which implies that $\textbf{J}_j^k$ is zero. Thus, all second-order partial derivatives vanish at every point, and $\textbf{f}$ must be affine. By definition of isometry, it must further be orthogonally affine. $\square$

Our theorem extends the Liouville theory in the sense that our theorem applies to $n\ge 2$ while Liouville’s theorem assumes $n\ge 3$. On the other hand, ours is a special case since Liouville assumes conformality while we assume local isometry. Given Liouville’s theorem, it would be very easy to prove a corollary which gives the desired result for $n\ge 3$ but we prefer to prove a theorem which is not a strict corollary but an extension at the same time, without any help from such advanced theory. It was in fact possible to give a very simple proof in our case, while only the very complicated proofs seem to be available for Liouville’s theorem.

The implication is that a nonlinear ICA model (or any deep latent variable model) where the (dimension-preserving) mixing is assumed locally isometric trivially reduces to a linear ICA model. (Note that we did not use any statistical properties of any components here.) The assumption of local isometry is far too strong from this viewpoint.