Parametrisation Independence of the Natural Gradient in Overparametrised Systems

Abstract

In this paper we study the natural gradient method for overparametrised systems. This method is based on the natural gradient field which is invariant with respect to coordinate transformations. One calculates the natural gradient of a function on the manifold by multiplying the ordinary gradient of the function by the inverse of the Fisher Information Matrix (FIM). In overparametrised models, the FIM is degenerate and therefore one needs to use a generalised inverse. We show explicitly that using a generalised inverse, and in particular the Moore-Penrose inverse, does not affect the parametrisation independence of the natural gradient. Furthermore, we show that for singular points on the manifold the parametrisation independence is not even guaranteed for non-overparametrised models.

4Appendix

Example 1

Let us consider the specific example where:

$$\begin{aligned} \mathcal {Z}&= \varXi = H = \mathbb {R}^2, \end{aligned}$$

(32)

$$\begin{aligned} \phi (\xi ^1, \xi ^2)&= (\xi ^1 + \xi ^2, 0)\end{aligned}$$

(33)

$$\begin{aligned} f(\eta ^1, \eta ^2)&= (2\eta ^1, \eta ^2)\end{aligned}$$

(34)

$$\begin{aligned} \mathcal {L}(x,y)&= x^2 \end{aligned}$$

(35)

Plugging this into the expressions derived above gives:

$$\begin{aligned} \tilde{\phi }(\eta ^1, \eta ^2)&= (2\eta ^1 + \eta ^2, 0)\end{aligned}$$

(36)

$$\begin{aligned} \partial _1(\xi )&= \partial _2(\xi ) = \frac{\partial {}}{\partial {x}}|_{\phi (\xi )}&F(\eta ) = \begin{bmatrix} 2 &{} 0 \\ 0 &{} 1 \end{bmatrix}\end{aligned}$$

(37)

$$\begin{aligned} \tilde{\partial }_1(\eta )&= 2\frac{\partial {}}{\partial {x}}|_{\tilde{\phi }(\eta )}&G(\xi ) = \begin{bmatrix} 1 &{} 1 \\ 1 &{} 1 \end{bmatrix}\end{aligned}$$

(38)

$$\begin{aligned} \tilde{\partial }_2(\eta )&= \frac{\partial {}}{\partial {x}}|_{\tilde{\phi }(\eta )}&\widetilde{G}(\eta ) = \begin{bmatrix} 4 &{} 2 \\ 2 &{} 1 \end{bmatrix}\end{aligned}$$

(39)

$$\begin{aligned} \nabla _{\partial (\xi )} \mathcal {L}&= (2(\xi ^1 + \xi ^2), 2(\xi ^1 + \xi ^2)) \end{aligned}$$

(40)

$$\begin{aligned} \nabla _{\tilde{\partial }(\eta )} \mathcal {L}&= (4(2\eta ^1 + \eta ^2), 2(2\eta ^1 + \eta ^2)) \end{aligned}$$

(41)

Now we fix $\eta = (1,1)$ and $\xi = f(\eta ) = (2,1)$. We start by computing $y_\varXi$. From the above we know that:

$$\begin{aligned} y_\varXi = \mathop {\mathrm {arg\,min}}\limits _y\{ ||y|| : G(\xi ) y = \nabla _{\partial (\xi )} \mathcal {L}\}. \end{aligned}$$

(42)

It can be easily verified that this gives $y_\varXi = (3,3)$. For $y_H$ we get:

$$\begin{aligned} y_H = \mathop {\mathrm {arg\,min}}\limits _y\{ ||\left( F^T\right) ^{-1} y|| : G(\xi ) y = \nabla _{\partial (\xi )} \mathcal {L}\} \end{aligned}$$

(43)

which gives: $y_H = (4 \frac{4}{5}, 1\frac{1}{5})$. Evidently we have $y_\varXi \ne y_H$. Note however that when we map the difference of the two gradient vectors from $T_{(2,1)} \varXi$ to $T_{(3,0)}\mathcal {M}$ through $d\phi _{(2,1)}$ we get:

$$\begin{aligned} d\phi _\xi \left( df_\eta \mathrm {grad}^H_\eta \mathcal {L}- \mathrm {grad}_\xi ^\varXi \mathcal {L}\right)&= \left( y_H - y_\varXi \right) ^i \partial _i(\xi )\end{aligned}$$

(44)

$$\begin{aligned}&= (4\frac{4}{5} - 3) \frac{\partial {}}{\partial {x}}|_{(3,0)} + (1\frac{1}{5} - 3) \frac{\partial {}}{\partial {x}}|_{(3,0)} \end{aligned}$$

(45)

$$\begin{aligned}&= 0 \end{aligned}$$

(46)

which shows that although the gradient vectors can be dependent on the parametrisation or inner product on the parameter space, when mapped to the manifold $\mathcal {M}$ they are invariant.

Example 2

We illustrate the discussion in Sect. 2.2 with a specific example. Let us consider the following parametrisation:

$$\begin{aligned} \phi : (-1/8\pi , 5/8\pi )&\rightarrow (\mathbb {R}^2, \bar{g}) \end{aligned}$$

(47)

$$\begin{aligned} t&\mapsto (x,y) = (\sin (2t), \sin (t)) \end{aligned}$$

(48)

This gives $\xi _1 = 0, \xi _2 = \frac{1}{2} \pi$ in the above discussion. We get the following calculation for $\overline{\mathrm {grad}}_p^{\partial (0)} \mathcal {L}$:

$$\begin{aligned} \partial ^{(0)}&= d\phi _t \left( \frac{\partial {}}{\partial {t}}|_t\right) \end{aligned}$$

(49)

$$\begin{aligned}&= 2\cos (2t) \frac{\partial {}}{\partial {x}}|_{\phi (t)} + \cos (t)) \frac{\partial {}}{\partial {y}}|_{\phi (t)}\end{aligned}$$

(50)

$$\begin{aligned} \partial ^{(0)}&= 2 \frac{\partial {}}{\partial {x}}|_{(0,0)} + \frac{\partial {}}{\partial {y}}|_{(0,0)}\end{aligned}$$

(51)

$$\begin{aligned} G(0)&= 2^2 + 1^1 = 5 \end{aligned}$$

(52)

$$\begin{aligned} \overline{\mathrm {grad}}_p^{\partial (0)} \mathcal {L}&= \frac{1}{5} \frac{\partial {}}{\partial {t}}|_{t=0} \bigg (\mathcal {L}(\sin (2t), \sin (t))\bigg ) \left( 2 \frac{\partial {}}{\partial {x}}|_{(0,0)} + \frac{\partial {}}{\partial {y}}|_{(0,0)} \right) \end{aligned}$$

(53)

Now let:

$$\begin{aligned} f: (-1/8\pi , 5/8\pi )&\rightarrow (-1/8\pi , 5/8\pi )\end{aligned}$$

(54)

$$\begin{aligned} t&\mapsto -(t-\frac{1}{4}\pi ). \end{aligned}$$

(55)

We define the alternative parametrisation $\tilde{\phi } = \phi \circ f$. Note that we have $\tilde{\phi }(t) = (-\sin (2t), \sin (t))$ and thus $\tilde{\phi }(0) = \phi (0) = (0,0)$. A similar calculation as before gives:

$$\begin{aligned} \tilde{\partial }^{(0)}&= -2 \frac{\partial {}}{\partial {x}}|_{(0,0)} + \frac{\partial {}}{\partial {y}}|_{(0,0)}\end{aligned}$$

(56)

$$\begin{aligned} \overline{\mathrm {grad}}_p^{\tilde{\partial }(0)} \mathcal {L}&= \frac{1}{5} \frac{\partial {}}{\partial {t}}|_{t=0} \bigg (\mathcal {L}(-\sin (2t), \sin (t))\bigg ) \left( -2 \frac{\partial {}}{\partial {x}}|_{(0,0)} + \frac{\partial {}}{\partial {y}}|_{(0,0)} \right) . \end{aligned}$$

(57)

Note that because $\partial ^{(0)} \ne \tilde{\partial }^{(0)}$ (53) and (57) are not equal to each other. We can therefore conclude that in this case the expression for the gradient is parametrisation-dependent.