Eigen-stratified models

Abstract

Stratified models depend in an arbitrary way on a selected categorical feature that takes K values, and depend linearly on the other n features. Laplacian regularization with respect to a graph on the feature values can greatly improve the performance of a stratified model, especially in the low-data regime. A significant issue with Laplacian-regularized stratified models is that the model is K times the size of the base model, which can be quite large. We address this issue by formulating eigen-stratified models, which are stratified models with an additional constraint that the model parameters are linear combinations of some modest number m of bottom eigenvectors of the graph Laplacian, i.e., those associated with the m smallest eigenvalues. With eigen-stratified models, we only need to store the m bottom eigenvectors and the corresponding coefficients as the stratified model parameters. This leads to a reduction, sometimes large, of model size when $m\le n$ and $m \ll K$. In some cases, the additional regularization implicit in eigen-stratified models can improve out-of-sample performance over standard Laplacian regularized stratified models.

Appendix: Eigenvectors and eigenvalues of common graphs

The direct relation of a graph’s structure to the eigenvalues and eigenvectors of its corresponding graph Laplacian is well-known (Jr and Morley 1985). In some cases, mentioned below, we can find them analytically, especially when the graph has many symmetries. The eigenvectors are given in normalized form (i.e., $\Vert q_k\Vert _2 = 1$.) Outside of these common graphs, many other simple graphs can be analyzed analytically; see, e.g., (Brouwer and Haemers 2012).

A note on complex graphs. If a graph is complex, i.e., there is no analytical form for its graph Laplacian’s eigenvalues and eigenvectors, the bottom eigenvalues and eigenvectors of the Laplacian of a graph can be computed extremely efficiently by, e.g., the Lanczos algorithm, variants of power iteration, the Jacobi eigenvalue algorithm, the folded spectrum method, and other exotic methods. We refer the reader to (Jacobi 1846; Mises and Pollaczek-Geiringer 1929; MacDonald 1934; Lanczos 1950; Ojalvo and Newman 1970; Paige 1971) for these methods.

Path graph. A path or linear/chain graph is a graph whose vertices can be listed in order, with edges between adjacent vertices in that order. The first and last vertices only have one edge, whereas the other vertices have two edges. Figure 8 shows a path graph with 8 vertices and unit weights.

Fig. 8

A path graph with 8 vertices and unit weights

Eigenvectors $q_1, \ldots , q_{K}$ of a path graph Laplacian with K nodes and unit edge weights are given by

$$\begin{aligned} q_k = \cos (\pi k v / K - \pi k / 2K)/\Vert \cos (\pi k v / K - \pi k / 2K)\Vert _2 \quad k = 0, \ldots , K-1, \end{aligned}$$

where $v = (0, \ldots , K-1)$ and $\cos (\cdot )$ is applied elementwise. The eigenvalues are $2-2\cos (\pi k/K), k = 0, \ldots , K-1$.

Cycle graph. A cycle graph or circular graph is a graph where the vertices are connected in a closed chain. Every node in a cycle graph has two edges. Figure 9 shows a cycle graph with 10 vertices and unit weights.

Fig. 9

A cycle graph with 10 vertices and unit weights

Eigenvectors of a cycle graph Laplacian with K nodes and unit weights are given by

$$\begin{aligned}&\frac{1}{\sqrt{K}}{\mathbf {1}}, k=0\\&\cos (2 \pi k v / K)/\Vert \cos (2 \pi k v / K)\Vert _2 \ \mathrm {and} \ \sin (2 \pi k v / K)/\Vert \sin (2 \pi k v / K)\Vert _2, \quad k = 1, \ldots , K/2, \end{aligned}$$

where $v = (0, \ldots , K-1)$ and $\cos (\cdot )$ and $\sin (\cdot )$ are applied elementwise. The eigenvalues are $2-2\cos (2\pi k/K), k = 0, \ldots , K-1$.

Star graph. A star graph is a graph where all of the vertices are only connected to one central vertex. Figure 10 shows an example of a star graph with 10 vertices (9 outer vertices) and unit weights.

Fig. 10

A star graph with 10 vertices (9 outer vertices) and unit weights

Eigenvectors of a star graph with K vertices (i.e., $K-1$ outer vertices) and unit edge weights are given by

$$\begin{aligned}&q_0 = \frac{1}{\sqrt{K}}{\mathbf {1}}\\&q_k = \frac{1}{\sqrt{2}} (e_i - e_{i+1}), \quad 1 \le i \le K-2\\&q_{K-1} = \frac{1}{\sqrt{K(K-1)}}(K-1, -1, -1, \ldots , -1, -1), \end{aligned}$$

where $e_i$ is the ith basis vector in ${\mathbf{R}}^K$. The smallest eigenvalue of this graph is zero, the largest eigenvalue is K, and all other eigenvalues are 1.

Wheel graph. A wheel graph with K nodes consists of a center (hub) vertex and a ring of $K-1$ peripheral vertices, each connected to the hub (Boyd et al. 2009). Figure 11 shows a wheel graph with 11 vertices (10 peripheral vertices) and unit weights.

Fig. 11

A wheel graph with 11 vertices (10 peripheral vertices) and unit weights

Eigenvectors of a wheel graph with K vertices (i.e., $K-1$ peripheral vertices) are given by (Zhang et al. 2009)

$$\begin{aligned}&q_0 = \frac{1}{\sqrt{K}}{\mathbf {1}}\\&q_k = \sin (2 \pi k v / K) / \Vert \sin (2 \pi k v / K)\Vert _2, \quad 1 \le i \le K-2, i \text { odd}\\&q_k = \cos (2 \pi k v / K) / \Vert \cos (2 \pi k v / K)\Vert _2, \quad 1 \le i \le K-2, i \text { even}\\&q_{K-1} = \frac{1}{\sqrt{K(K-1)}}(K-1, -1, -1, \ldots , -1, -1), \end{aligned}$$

where $v = (0, \ldots , K-1)$ and $\cos (\cdot )$ and $\sin (\cdot )$ are applied elementwise. The smallest eigenvalue of the graph is zero, the largest eigenvalue is K, and the middle eigenvalues are given by $3 - 2\cos (2 \pi i/(K-1)), i = 1, \ldots , (K-2)/2$, with multiplicity 2 (Butler 2008).

Complete graph A complete graph contains every possible edge; we assume here the edge weights are all one. The first eigenvector of a complete graph Laplacian with K nodes is $\frac{1}{\sqrt{K}} {\mathbf {1}}$, and the other $K-1$ eigenvectors are any orthonormal vectors that complete the basis. The eigenvalues are 0 with multiplicity 1, and K with multiplicity $K-1$.

Figure 12 shows an example of a complete graph with 8 vertices and unit weights.

Fig. 12

A complete graph with 8 vertices and unit weights

Complete bipartite graph A bipartite graph is a graph whose vertices can be decomposed into two disjoint sets such that no two vertices share an edge within a set. A complete bipartite graph is a bipartite graph such that every pair of vertices in the two sets share an edge. We denote a complete bipartite graph with $\alpha$ vertices on the first set and $\beta$ vertices on the second set as an $(\alpha , \beta )$-complete bipartite graph. We have that $\alpha +\beta =K$, and use the convention that $\alpha \le \beta$. Figure 13 illustrates an example of a complete bipartite graph with $(\alpha , \beta )=(3,6)$ and unit weights.

Fig. 13

A (3,6)-complete bipartite graph with unit weights

Eigenvectors of an $(\alpha , \beta )$-complete bipartite graph with unit edge weights are given by (Merris 1998):

$$\begin{aligned}&q_0 = \frac{1}{\sqrt{K}}{\mathbf {1}}\\&q_k = \frac{1}{\sqrt{2}}(e_k - e_{k+1}), \quad 1 \le k \le \alpha -1\\&q_k = \frac{1}{\sqrt{2}}(e_k - e_{k+1}), \quad \alpha \le k \le K-1\\&(q_{K-1})_i = {\left\{ \begin{array}{ll} \frac{-\beta }{\sqrt{\alpha ^2\beta + \beta ^2\alpha }} & 1 \le i \le \alpha \\ \frac{\alpha }{\sqrt{\alpha ^2\beta + \beta ^2\alpha }} & \alpha < i \le K \\ \end{array}\right. }. \end{aligned}$$

The eigenvalues are zero (multiplicity 1), $\alpha$ (multiplicity $\beta -1$), $\beta$ (multiplicity $\alpha -1$), and $K=\alpha +\beta$ (multiplicity 1).

Scaling and products of graphs We can find the eigenvectors and eigenvalues of the graph Laplacian of more complex graphs using some simple relationships. First, if we scale the edge weights of a graph by $\alpha \ge 0$, the eigenvectors remain the same, and the eigenvalues are scaled by $\alpha$. Second, the eigenvectors of a Cartesian product of graph Laplacians are given by the Kronecker products between the eigenvectors of each of the individual graph Laplacians; the eigenvalues consist of the sums of one eigenvalue from one graph and one from the other. This can be seen by noting that the Laplacian matrix of the Cartesian product of two graphs with graph Laplacians $L_1 \in {\mathbf{R}}^{P \times P}$ and $L_2 \in {\mathbf{R}}^{Q \times Q}$ is given by

$$\begin{aligned} L = (L_1 \otimes I) + (I \otimes L_2), \end{aligned}$$

where L is the Laplacian matrix of the Cartesian product of the two graphs. With Cartesian products of graphs, we find it convienent to index the eigenvalues and eigenvectors of the Laplacian by two indices, i.e., the eigenvalues may be denoted as $\lambda _{i,j}$ with corresponding eigenvector $q_{i,j}$ for $i = 0, \ldots , P-1$ and $j = 0, \ldots , Q-1$. (The eigenvalues will need to be sorted, as explained below.)

As an example, consider a graph which is the product of a chain graph with P vertices, edge weights $\alpha ^\text {ch}$ and eigenvalues $\lambda ^{\text {ch}} \in {\mathbf{R}}^P$; and a cycle graph with Q vertices, edge weights $\alpha ^\text {cy}$, and eigenvalues $\lambda ^{\text {ch}} \in {\mathbf{R}}^P$. The eigenvalues have the form

$$\begin{aligned} \lambda ^{\text {ch}}_i + \lambda ^{\text {cy}}_j, \quad i=0\ldots , P-1, \quad j=0, \ldots , Q-1, \end{aligned}$$

To find the m smallest of these, we sort them. The order depends on the ratio of the edge weights, $\alpha ^\text {ch}/\alpha ^\text {cy}$.

As a very specific example, take $P=4$ and $Q=5$, $\alpha ^\text {ch} = 1$, and $\alpha ^\text {cy} = 2$. The eigenvalues of the chain and cycle graphs are

$$\begin{aligned} \lambda ^{\text {ch}} = (0, 0.586, 2, 3.414), \quad \lambda ^{\text {cy}} = (0, 2.764, 2.764, 7.236, 7.236). \end{aligned}$$

The bottom six eigenvalues of the Cartesian product of these two graphs are then

$$\begin{aligned} 0, 0.586, 2, 2.764, 2.764, 3.350. \end{aligned}$$