Data Analysis from Empirical Moments and the Christoffel Function

Abstract

Spectral features of the empirical moment matrix constitute a resourceful tool for unveiling properties of a cloud of points, among which, density, support and latent structures. This matrix is readily computed from an input dataset, and its eigen decomposition can then be used to identify algebraic properties of the support or density/support estimates with the Christoffel function. It is already well known that the empirical moment matrix encodes a great deal of subtle attributes of the underlying measure. Starting from this object as base of observations, we combine ideas from statistics, real algebraic geometry, orthogonal polynomials and approximation theory for opening new insights relevant for machine learning problems with data supported on algebraic sets. Refined concepts and results from real algebraic geometry and approximation theory are empowering a simple tool (the empirical moment matrix) for the task of solving non-trivial questions in data analysis. We provide (1) theoretical support, (2) numerical experiments and (3) connections to real-world data as a validation of the stamina of the empirical moment matrix approach.

Appendices

Introduction

We collect below some supplementary information to the main body of the present article. Section B provides additional details regarding the notations and constructions employed in the main text. More insights about the numerical simulations are given in Sect. C. Technical Lemmas are presented in Sect. D.

Polynomials and the Moment Matrix

In this section, $\mu$ denotes a positive Borel measure on ${\mathbb {R}}^p$ with finite moments, and for any $d \in {\mathbb {N}}$, ${\mathbf {M}}_{\mu ,d}$ denotes its moment matrix with moments of degree up to 2d. As a matter of fact, we will soon restrict our attention to probability measures, which is a minor constraint to impose on the constructs below.

Henceforth, we fix the dimension of the ambient euclidean space to be p, for example vectors in ${\mathbb {R}}^p$ as well as p-variate polynomials with real coefficients. We denote by X the tuple of p variables $X_1, \ldots , X_p$ which appear in mathematical expressions involving polynomials. Monomials from the canonical basis of p-variate polynomials are identified with their exponents in ${\mathbb {N}}^p$: Specifically, $\alpha = (\alpha _i)_{i =1 \ldots p} \in {\mathbb {N}}^p$ is associated with the monomial $X^\alpha := X_1^{\alpha _1} X_2^{\alpha _2} \ldots X_p^{\alpha _p}$ of degree $\deg (\alpha ) := \sum _{i=1}^p \alpha _i=\vert \alpha \vert$. The notations $<_{gl}$ and $\le _{gl}$ stand for the graded lexicographic order, a well ordering over p-variate monomials. This amounts to, first, use the canonical order on the degree and, second, break ties in monomials with the same degree using the lexicographic order with $X_1 =a, X_2 = b \ldots$. For example, the monomials in two variables $X_1, X_2$ of degree less than or equal to 3 listed in this order are given by: $1,\,X_1, \,X_2, \,X_1^2, \,X_1X_2, \,X_2^2,\, X_1^3,\, X_1^2X_2,\, X_1X_2^2,\, X_2^3$. We focus here on the graded lexicographic order to provide a concrete example, but any ordering compatible with the degree would work similarly.

By definition, ${\mathbb {N}}^p_d$ is the set $\left\{ \alpha \in {\mathbb {N}}^p;\; \deg (\alpha ) \le d \right\}$, while ${\mathbb {R}}[X]$ is the algebra of p-variate polynomials with real coefficients. The degree of a polynomial is the highest of the degrees of its monomials with nonzero coefficients². The notation $\deg (\cdot )$ applies a polynomial as well as to an element of ${\mathbb {N}}^p$. For $d \in {\mathbb {N}}$, ${\mathbb {R}}_d[X]$ stands for the set of p-variate polynomials of degree less than or equal to d. We set $s(d) = {p+d \atopwithdelims ()d} = \dim {\mathbb {R}}_d[X]$; this is of course the number of monomials of degree less than or equal to d.

From now on, ${\mathbf {v}}_d(X)$ denotes the vector of monomials of degree less or equal to d (sorted using $\le _{gl}$), i.e., ${\mathbf {v}}_d(X) := \left( X^\alpha \right) _{\alpha \in {\mathbb {N}}^p_d} \in {\mathbb {R}}_d[X]^{s(d)}$. With this notation, one can write a polynomial $P\in {\mathbb {R}}_d[X]$ as $P(X) = \left\langle {\mathbf {p}}, {\mathbf {v}}_d(X)\right\rangle$ for some real vector of coefficients ${\mathbf {p}}= \left( p_{\alpha } \right) _{\alpha \in {\mathbb {N}}_d^p} \in {\mathbb {R}}^{s(d)}$ ordered using $\le _{gl}$. Given ${\mathbf {x}}= (x_i)_{i = 1 \ldots p} \in {\mathbb {R}}^p$, $P({\mathbf {x}})$ denotes the evaluation of P with respect to the assignments $X_1 = x_1, X_2 = x_2, \ldots X_p = x_{p}$. Given a Borel probability measure $\mu$ and $\alpha \in {\mathbb {N}}^p$, $y_{\alpha }(\mu )$ denotes the moment $\alpha$ of $\mu$, i.e., $y_{\alpha }(\mu ) = \int _{{\mathbb {R}}^p} {\mathbf {x}}^{\alpha } \mathrm{d}\mu ({\mathbf {x}})$. Throughout the paper, we will only consider rapidly decaying measures, that is, measure whose all moments are finite. For a positive Borel measure $\mu$ on ${\mathbb {R}}^p$, denote by $\mathrm{supp}(\mu )$ its support, i.e., the smallest closed set ${{\varvec{\Omega }}}\subset {\mathbb {R}}^p$ such that $\mu ({\mathbb {R}}^p{\setminus }{{\varvec{\Omega }}})=0$.

Moment matrix The moment matrix of $\mu$, ${\mathbf {M}}_{\mu ,d}$, is a matrix indexed by monomials of degree at most d ordered with respect to $\le _{gl}$. For $\alpha ,\beta \in {\mathbb {N}}^p_d$, the corresponding entry in ${\mathbf {M}}_{\mu ,d}$ is defined by $[{\mathbf {M}}_{\mu ,d}]_{\alpha ,\beta } := y_{\alpha +\beta }(\mu )$, the moment $\int {\mathbf {x}}^{\alpha +\beta }\mathrm{d}\mu$ of $\mu$. For example, in the case $p = 2$, letting $y_{\alpha } = y_{\alpha } (\mu )$ for $\alpha \in {\mathbb {N}}_4^2$, one finds:

$$\begin{aligned} {\mathbf {M}}_{\mu ,2}: \quad \begin{array}{rccccccccc} &{} &{} &{}1&{} X_1 &{} X_2 &{} X_1^2 &{} X_1 X_2 &{} X_2^2 \\ &{} &{} &{} \\ 1 &{} \quad &{} &{}1 &{} y_{10} &{} y_{01} &{} y_{20}&{} y_{11}&{} y_{02} \\ X_1&{} \quad &{}&{}y_{10} &{} y_{20} &{} y_{11}&{}y_{30} &{} y_{21}&{} y_{12} \\ X_2&{} \quad &{} &{}y_{01} &{} y_{11} &{} y_{02}&{} y_{21} &{}y_{12}&{} y_{03} \\ X_1^2&{} \quad &{} &{}y_{20} &{} y_{30} &{} y_{21}&{} y_{40} &{} y_{31}&{} y_{22} \\ X_1X_2&{} \quad &{} &{}y_{11} &{} y_{21} &{} y_{12}&{} y_{31} &{} y_{22}&{} y_{13}\\ X_2^2&{} \quad &{} &{}y_{02} &{} y_{12} &{}y_{03}&{} y_{22} &{} y_{13}&{} y_{04}\\ \end{array}. \end{aligned}$$

The matrix ${\mathbf {M}}_{\mu ,d}$ is positive semidefinite for all $d \in {\mathbb {N}}$. Indeed, for any $p \in {\mathbb {R}}^{s(d)}$, let $P \in {\mathbb {R}}_d[X]$ be the polynomial with vector of coefficients ${\mathbf {p}}$; then, ${\mathbf {p}}^T{\mathbf {M}}_{\mu ,d}{\mathbf {p}}= \int _{{\mathbb {R}}^p} P({\mathbf {x}})^2 \mathrm{d}\mu ({\mathbf {x}}) \ge 0$. We also have the identity ${\mathbf {M}}_{\mu ,d} = \int _{{\mathbb {R}}^p} {\mathbf {v}}_d({\mathbf {x}}) {\mathbf {v}}_d({\mathbf {x}})^T \mathrm{d}\mu \mathrm{d}\mu ({\mathbf {x}})$ where the integral is understood element-wise. Actually, it is useful to interpret the moment matrix as representing the bilinear form

$$\begin{aligned} \left\langle \cdot , \cdot \right\rangle _\mu :{\mathbb {R}}[X] \times {\mathbb {R}}[X]&\mapsto {\mathbb {R}}\\ (P,Q)&\mapsto \int _{{\mathbb {R}}^p} P({\mathbf {x}})Q({\mathbf {x}})\mathrm{d}\mu ({\mathbf {x}}), \end{aligned}$$

restricted to polynomials of degree up to d. Indeed, if ${\mathbf {p}},{\mathbf {q}}\in {\mathbb {R}}^{s(d)}$ are the vectors of coefficients of any two polynomials P and Q of degree up to d, one has ${\mathbf {p}}^T {\mathbf {M}}_{\mu ,d} {\mathbf {q}}= \left\langle P, Q\right\rangle _\mu$

Numerical Experiments

This section provides additional details regarding numerical experiments.

1.1 Rank of the Moment Matrix

We choose four different subsets of ${\mathbb {R}}^3$:

• The unit cube $\left\{ x,y,z, |x| \le 1, |y| \le 1, |z|\le 1 \right\}$.

• The three-dimensional unit sphere $\left\{ x,y,z, x^2 + y^2 + z^2 = 1 \right\}$.

• The three-dimensional TV screen $\left\{ x,y,z, x^6 + y^6 + z^6 - 2x^2y^2z^2 = 1 \right\}$.

• The three-dimensional torus $\Big \{ x,y,z, \left( x^2 + y^2 + z^2 + \frac{9}{16} - \frac{1}{16}\right) ^2 - \frac{9}{16} (x^2 + y^2) = 0 \Big \}$.

Among the above sets, the first one is three-dimensional, while all the others are two-dimensional. The two-dimensional sets are displayed in Fig. 1. For each set, we sample 20000 points on it and compute the rank of the empirical moment matrix for different values of the degree. To perform this computation, we threshold the singular values of the design matrix consisting in the expansion of each data point in the multivariate Tchebychev polynomial basis.

1.2 Density Estimation on Algebraic Sets

The following quantities are used in the literature as divergences combined with density estimation techniques for the examples treated in the main article.

• The quantity $\cos (\theta _1 - \theta _2)$ where $\theta _1$ and $\theta _2$ are angular coordinates of two points on the circle.

• The dot product on the sphere which generalizes the previous situation to larger dimensions, used in [17].

• The quantity $\cos \left( \sqrt{(\phi _1 - \phi _2)^2+(\psi _1 - \psi _2)^2 } \right)$ where $\phi _1, \phi _2, \psi _1, \psi _2$ are angles which correspond to points on the torus in ${\mathbb {R}}^4$, used in [38].

Technical Lemmas

We begin with the following simple Lemma:

Lemma 9

Let $\mu$ be a Borel probability measure on ${\mathbb {R}}^p$ and S its support which is assumed to be a bounded subset of ${\mathbb {R}}^p$. Then for any $d \in {\mathbb {N}}^*$ and for any $P \in {\mathbb {R}}_d[X]$:

$$\begin{aligned} \sup _{{\mathbf {x}}\in S} |P({\mathbf {x}})|^2 \le \sup _{{\mathbf {z}}\in S} \varLambda _{\mu ,d}^{-1}({\mathbf {z}}) \int (P({\mathbf {x}}))^2 \mathrm{d}\mu ({\mathbf {x}}). \end{aligned}$$

Proof

For any ${\mathbf {x}}\in S$ and $P \in {\mathbb {R}}_d[X]$, one finds

$$\begin{aligned} \varLambda _{\mu ,d}({\mathbf {x}}) \le \frac{\int (P({\mathbf {z}}))^2 d \mu ({\mathbf {z}})}{P({\mathbf {x}})^2} \end{aligned}$$

and

$$\begin{aligned} P({\mathbf {x}})^2 \le \frac{\int (P({\mathbf {z}}))^2 d \mu ({\mathbf {z}})}{\varLambda _{\mu ,d}({\mathbf {x}})}. \end{aligned}$$

The result follows by considering the supremum over S on both sides. $\square$

Finally, the following Lemma is a quantitative adaptation of [35, Lemma 2.1]:

Lemma 10

For any $d \in {\mathbb {N}}^*$ and any $\delta \in (0,1)$, there exists a p-variate polynomial of degree 2d, Q, such that

$$\begin{aligned} Q(0) \,=\, 1\,;\quad -1\,\le \,Q \,\le \, 1,\text { on } {\mathbf {B}}\,;\quad \vert Q\vert \,\le \, 2^{1-\delta d} \text { on } {\mathbf {B}}{\setminus } {\mathbf {B}}_{\delta }(0). \end{aligned}$$

Proof

Let R be the univariate polynomial of degree 2d, defined by

$$\begin{aligned} R:t \rightarrow \frac{T_d(1+\delta ^2 - t^2)}{T_d(1+\delta ^2)}, \end{aligned}$$

where $T_d$ is the Chebyshev polynomial of the first kind. We obtain

$$\begin{aligned} R(0) = 1. \end{aligned}$$

(18)

Furthermore, for $t \in [-1,1]$, we have $0 \le 1+\delta ^2 - t^2 \le 1+\delta ^2$. $T_d$ has absolute value less than 1 on $[-1,1]$ and is increasing on $[1, \infty )$ with $T_d(1) = 1$, so for $t \in [-1,1]$:

$$\begin{aligned} -1 \le R(t) \le 1. \end{aligned}$$

(19)

For $|t| \in [\delta , 1]$, we have $\delta ^2 \le 1+\delta ^2-t^2 \le 1$, so

$$\begin{aligned} |R(t)| \le \frac{1}{T_d(1+\delta ^2)}. \end{aligned}$$

(20)

Let us bound the last quantity. Recall that for $t \ge 1$, we have the following explicit expression:

$$\begin{aligned} T_d(t) = \frac{1}{2}\left( \left( t + \sqrt{t^2-1} \right) ^d + \left( t + \sqrt{t^2-1} \right) ^{-d}\right) . \end{aligned}$$

We have $1 + \delta ^2 +\sqrt{(1+\delta ^2)^2 - 1} \ge 1 + \sqrt{2} \delta$, which leads to

$$\begin{aligned} T_d(1+\delta ^2)&\ge \frac{1}{2}\left( 1 + \sqrt{2} \delta \right) ^d\nonumber \\&= \frac{1}{2} \exp \left( \log \left( 1+\sqrt{2}\delta \right) d \right) \nonumber \\&\ge \frac{1}{2} \exp \left( \log (1+\sqrt{2}) \delta d \right) \nonumber \\&\ge 2^{\delta d - 1}, \end{aligned}$$

(21)

where we have used concavity of the $\log$ and the fact that $1+\sqrt{2} \ge 2$. It follows by combining (18), (19), (20) and (21) that $Q :{\mathbf {y}}\rightarrow R(\Vert {\mathbf {y}}\Vert _2)$ satisfies the claimed properties. $\square$

1.1 Proof of Lemma 8

Proof

The implication (ii) to (i) is trivial since $\sigma _W$ is supported on W. Let us assume that (i) is true and deduce (ii). This is classically formulated in the language of sheaves; we adopt a more elementary language and describe details for completeness. The main idea of the proof is to reduce the statement to Rückert’s complex analytic Nullstellensatz [32, Proposition 1.1.29] which characterizes the class of analytic functions vanishing locally on the zero set of a family of analytic functions. The first point is precisely of local nature, and the Nullstellensatz combined with properties of analytic functions allows to deduce properties of P which extend globally [49]. The irreducibility hypothesis and the definition of the area measure allow precisely to switch from real to complex variables.

First, let $P_1, \ldots , P_k$ generate the ideal I of polynomials vanishing on W. Since W is irreducible, I is a prime ideal and $W = \left\{ {\mathbf {x}}\in {\mathbb {R}}^p,\, P_i({\mathbf {x}}) = 0,\, i =1,\ldots ,k \right\}$ (see Propositions 3.3.14 and 3.3.16 in [8]). Point (i) entails that there exists ${\mathbf {x}}_0 \in W$ and $U_1$ an euclidean neighborhood of ${\mathbf {x}}_0$ in ${\mathbb {R}}^p$ such that $W \cap U_1$ is an analytic submanifold, or more precisely, the Jacobian matrix $\left( \frac{\partial P_i}{\partial X_j} \right) _{i=1\ldots k,\, j=1\ldots p}$ has rank k on $U_1$, and furthermore, $P({\mathbf {x}}) = 0$ for all ${\mathbf {x}}\in W \cap U_1$.

Consider the complex analytic manifold $Z = \left\{ {\mathbf {z}}\in {\mathbb {C}}^p,\, P_i({\mathbf {z}}) = 0,\, i =1,\ldots ,k \right\}$ and the polynomial map

$$\begin{aligned} G:(X_1,\ldots ,X_p) \mapsto (P_1(X_1,\ldots ,X_p),\ldots ,P_k(X_1,\ldots ,X_p),X_{k+1},\ldots , X_p). \end{aligned}$$

This map is locally invertible around ${\mathbf {x}}_0$ in ${\mathbb {C}}^p$, and its inverse is analytic. The function $(H:X_{k+1},\ldots ,X_p)\mapsto P(G^{-1}(0,\ldots ,0,X_{k+1},\ldots ,X_p)$ is analytic and vanishes in a euclidean neighborhood, of $(x_{0,k+1},\ldots ,x_{0,p})$, the last k coordinates of ${\mathbf {x}}_0$, in ${\mathbb {R}}^k$. Hence, it can be identified with the constant null function on $U_1$. This shows that H vanishes in a euclidean neighborhood of $(x_{0,k+1},\ldots ,x_{0,p})$ in ${\mathbb {C}}^k$.

This proves that there exists a euclidean neighborhood of ${\mathbf {x}}_0$ in ${\mathbb {C}}^p$, $U_2$, such that P vanishes on $Z \cap U_2$. At this point, we can invoke the complex analytic Nullstellensatz [32, Proposition 1.1.29], to obtain k analytic functions, $O_1,\ldots ,O_k$, on $U_2$, and an integer $m \ge 1$, such that, for all ${\mathbf {z}}\in U_2$, we have

$$\begin{aligned} (P({\mathbf {z}}))^m = \sum _{i=1}^k P_i({\mathbf {z}}) O_i({\mathbf {z}}). \end{aligned}$$

(22)

We can now use powerful result related to completion of local rings. Combining Corollary 1 of Proposition 4 and Proposition 22 in [49], we obtain that identity (22) still holds with the constraint that $O_i$, $i=1,\ldots , k$, are rational functions. In other words, reducing to common denominator, there exists a euclidean neighborhood of ${\mathbf {x}}_0$, $U_3$, and k complex polynomials $Q_1,\ldots ,Q_k$ and a complex polynomial Q such that Q does not vanish on $U_3$, and

$$\begin{aligned} Q({\mathbf {z}})(P({\mathbf {z}}))^m = \sum _{i=1}^k P_i({\mathbf {z}}) Q_i({\mathbf {z}}). \end{aligned}$$

(23)

From this, we deduce that identity (23) still holds by restricting to real variables and real polynomials. Now since the ideal I is prime and $Q \not \in I$ (since $Q({\mathbf {x}}_0) \ne 0$), we deduce that $P \in I$, that is, P vanishes on W. This is what we wanted to prove. $\square$