Modelling persistence diagrams with planar point processes, and revealing topology with bagplots

Abstract

We introduce a new model for planar point processes, with the aim of capturing the structure of point interaction and spread in persistence diagrams. Persistence diagrams themselves are a key tool of topological data analysis (TDA), crucial for the delineation and estimation of global topological structure in large data sets. To a large extent, the statistical analysis of persistence diagrams has been hindered by difficulties in providing replications, a problem that was addressed in an earlier paper, which introduced a procedure called replicating statistical topology (RST). Here we significantly improve on the power of RST via the introduction of a more realistic class of models for the persistence diagrams. In addition, we introduce to TDA the idea of bagplotting, a powerful technique from non-parametric statistics well adapted for differentiating between topologically significant points, and noise, in persistence diagrams. Outside the setting of TDA, our model provides a setting for fashioning point processes, in any dimension, in which both local interactions between the points, along with global restraints on the overall, global, shape of the point cloud, are important and perhaps competing.

Appendices

Appendices

1.1 Simulating from ${\bar{f}}^G$

While the MCMC calculations of the paper are well summarised in Algorithm 1 of Sect. 5.4, sampling from the distribution ${\bar{f}}^G(x)$ of (12) itself requires some care. While reasonably straightforward, there are numerical subtleties, and so for completeness we describe the full procedure here.

Recall that ${\bar{f}}^G(x)$ is just the kernel density estimate ${\hat{f}}^G$, restricted to $\mathbb {R}\times \mathbb {R}_+$, and normalised. To sample from it, we first denote by R the smallest rectangular subset of the half plane which includes the set $\{x\in \mathbb {R}\times \mathbb {R}_+: {\bar{f}}^G(x)>\varepsilon \}$, for some $\varepsilon >0$ that is case specific. Then divide R into $I_1\times I_2$ equal sized rectangles $I_{ij}$, where $I_1$ and $I_2$ are typically of the order of 100, but, again, case specific.

The second step involves assigning probabilities to these rectangles, which, a prior, could be done by integrating ${\bar{f}}^G$ over each one. However, noting the original empirical density ${\hat{f}}^G$ comes from a Gaussian kernel, considerable computational time is saved by first defining its integrated version

$$\begin{aligned} {\hat{F}}^G (x) \ {\mathop {=}\limits ^{\Delta }}\ \frac{1}{n} \sum _{i=1}^{n}\Phi _{\Sigma } (x-x_{i}), \end{aligned}$$

where $\Phi _\Sigma$ is the Gaussian (cumulative) distribution function corresponding to the Gaussian kernel in the definition of ${\hat{f}}^G$. Extend ${\hat{F}}^G$ to a measure on rectangles in the usual way, and define the probabilities (which now sum to 1)

$$\begin{aligned} p_{ij} \ = \ \frac{{\hat{F}}^G(I_{ij})}{\sum _{i=1}^{I_1}\sum _{j=1}^{I_2}{\hat{F}}^G(I_{ij})}. \end{aligned}$$

By taking any linear enumeration of the indices (i, j) it is now trivial to chose a rectangle at random, according to these probabilities, by the inverse transform method [e.g. Robert and Casella (2004), Brooks et al. (2011)].

Having chosen a rectangle, we now chose a point uniformly, at random, from it. This is the value $x^*$ taken for Step 3 of Algorithm 1.

1.2 The model of Adler et al. (2017)

The model originally developed in Adler et al. (2017), as with the one used in the current paper, was a Gibbs distribution, and so can be described through its Hamiltonian, as below, retaining the notation of Sect. 4. we shall do this only for projected persistence diagrams, so that each point x in the diagram is of the form $x=(x^{(1)},x^{(2)})\in \mathbb {R}\times \mathbb {R}_+$.

Define

$$\begin{aligned} \sigma _H^2 = \sum _{x\in {{\tilde{x}}}_N} \big (x^{(1)} - {\bar{x}}^{(1)}\big )^2,\ \ \ \sigma _V^2 = \sum _{x\in {{\tilde{x}}}_N} \big (x^{(2)} \big )^2, \end{aligned}$$

where ${\bar{x}}^{(1)}=N^{-1} \sum _{i=1}^N x_i^{(1)}$, so $\sigma _H^2$ is the variance of the horizontal points. On the other hand, $\sigma _V^2$ is square of the $L_2$ norm of the vertical points, rather than the centred variance (because of the non-negativeness of the $x^{(2)}$).

For integral $K> 0$, a collection $\Theta = (\theta _H,\theta _V,\theta _1,\ldots \theta _K)$ of $\mathbb {R}$-valued parameters, and a $\delta >0$, define the Hamiltonian

$$\begin{aligned} H_{\delta ,\Theta }^K({{\tilde{x}}}_N)= & {} \theta _H \sigma ^2_H +\theta _V \sigma ^2_V + \delta ^{-2} \sum _{k=1}^K \theta _k \sum _{i=1}^N \nonumber \\&\quad \sum _{z\in {\mathcal N}_k(x_i)} \Vert z-x_i\Vert \, \mathbb {1}_{\{\Vert z-x \Vert \le \delta \}}. \end{aligned}$$

(15)

With this Hamiltonian replacing the one defined by (8) the remaining development in Adler et al. (2017)—in particular that of an appropriate pseudo-likelihood model—is parallel to that in Sect. 4.

We note though the main differences between the models. The first is the parameter $\delta$, which limits nearest neighbour interactions only to those neighbours that are closer than $\delta$. This had a mild numerically stabilising effect in the model defined by (15), that, for reasons that are not entirely clear, disappeared in the model of the current paper. Consequently, we no longer use it. The first two terms in the Hamiltonian, involving second moments, were intended to play the role that the empirical density ${\bar{f}}^G$ plays in the current paper; viz. they controlled the overall shape of the random diagrams, and worked “against” the control resulting from the nearest neighbour interactions. However, as shown by most of the examples in Sect. 6—in particular the Gaussian excursion set and non-concentric circles examples—these terms were not able to capture many of the subtleties found in persistence diagrams. Furthermore, as the MCMC simulations progressed, the simulated diagrams had a tendency to move towards the diagonal, in a fashion that was inconsistent with their overall use.