Cramér–Rao lower bounds arising from generalized Csiszár divergences

Abstract

We study the geometry of probability distributions with respect to a generalized family of Csiszár f-divergences. A member of this family is the relative $\alpha$-entropy which is also a Rényi analog of relative entropy in information theory and known as logarithmic or projective power divergence in statistics. We apply Eguchi’s theory to derive the Fisher information metric and the dual affine connections arising from these generalized divergence functions. This enables us to arrive at a more widely applicable version of the Cramér–Rao inequality, which provides a lower bound for the variance of an estimator for an escort of the underlying parametric probability distribution. We then extend the Amari–Nagaoka’s dually flat structure of the exponential and mixer models to other distributions with respect to the aforementioned generalized metric. We show that these formulations lead us to find unbiased and efficient estimators for the escort model. Finally, we compare our work with prior results on generalized Cramér–Rao inequalities that were derived from non-information-geometric frameworks.

A Proof of Theorem 3

Taking log on both sides of (48),

$$\begin{aligned} \log p_{\theta }^{(\alpha )}(x)&= -\log M(\theta ) + {\frac{1}{1-\frac{1}{\alpha }}}\log \Big [ {q^{(\alpha )}(x)}^{1-\frac{1}{\alpha }} + \sum \limits _{i=1}^k \theta _i h_i(x) \Big ]. \end{aligned}$$

(62)

Partial derivative produces

$$\begin{aligned} \partial _i \log p_{\theta }^{(\alpha )}(x)&= -\partial _i\log M(\theta ) + {\frac{1}{1-\frac{1}{\alpha }}}\frac{h_i(x)}{ \Big [ {q^{(\alpha )}(x)}^{1-\frac{1}{\alpha }} +\sum \limits _{i=1}^k \theta _i h_i(x) \Big ]}, \end{aligned}$$

(63)

or

$$\begin{aligned} \partial _i \log p_{\theta }^{(\alpha )}(x)&= -\partial _i\log M(\theta ) + \frac{\alpha }{\alpha -1} \frac{f_i(x)}{ \Big [q(x)^{\alpha -1} + \sum \limits _{i=1}^k\theta _i f_i(x) \Big ]}. \end{aligned}$$

(64)

Taking expectation on both sides of (64), we obtain

$$\begin{aligned} E_{\theta ^{(\alpha )}} \left[ \partial _i \log p_{\theta }^{(\alpha )}(x)\right]&= -\partial _i\log M(\theta ) + \frac{\alpha }{\alpha -1} E_{\theta ^{(\alpha )}} \Bigg [\frac{f_i(X)}{ q(X)^{\alpha -1} + \sum \limits _{i=1}^k\theta _i f_i(X)}\Bigg ]. \end{aligned}$$

(65)

Since the expected value of the score function vanishes (left hand side of (65)), we have

$$\begin{aligned} \partial _i\log M(\theta ) = \frac{\alpha }{\alpha -1} E_{\theta ^{(\alpha )}} \Bigg [\frac{f_i(X)}{ q(X)^{\alpha -1} + \sum \limits _{i=1}^k\theta _i f_i(X)}\Bigg ]. \end{aligned}$$

(66)

Substituting (66) into (64), we get

$$\begin{aligned} \partial _i \log p_{\theta }^{(\alpha )}(x)= & {} \frac{\alpha }{\alpha -1} \frac{f_i(x)}{ \Big [ q(x)^{\alpha -1} + \sum \limits _{i=1}^k\theta _i f_i(x) \Big ]} \nonumber \\&- \frac{\alpha }{\alpha -1} E_{\theta ^{(\alpha )}} \Bigg [\frac{f_i(X)}{ q(X)^{\alpha -1} + \sum \limits _{i=1}^k\theta _i f_i(X) }\Bigg ] \nonumber \\=: & {} \widehat{\eta }_i(x) - \eta _i, \end{aligned}$$

(67)

where

$$\begin{aligned} \widehat{\eta }_i(x) := \frac{\alpha }{\alpha -1}\frac{ f_i(x)}{ \Big [q(x)^{\alpha -1} + \sum \limits _{i=1}^k\theta _i f_i(x) \Big ]} \text { and } \eta _i := E_{\theta ^{(\alpha )}}[\widehat{\eta }_i(X)] \end{aligned}$$

Moreover, (66) implies that $\log M(\theta )$ should be the potential (if exists).

The Riemmanian metric becomes

$$\begin{aligned} g_{ij}^{(\alpha )}(\theta ) = \frac{1}{\alpha ^2}E_{\theta ^{(\alpha )}}[(\widehat{\eta }_i(X) - \eta _i)(\widehat{\eta }_j(X) - \eta _j)]. \end{aligned}$$

(68)

This further strengthens our expectation that the $\eta _i$’s are dual parameters to $\theta _i$’s. However, it is surprising that it is not so as we shall see now. We have

$$\begin{aligned} \eta _j= & {} \frac{\alpha }{\alpha -1}E_{\theta ^{(\alpha )}}\Bigg [ \frac{f_j(X)}{ q(X)^{\alpha -1} + \sum \limits _{j=1}^k\theta _j f_j(X) }\Bigg ]\nonumber \\= & {} \frac{\alpha }{\alpha -1}\sum \limits _x \Bigg [\frac{f_j(x)}{ q(x)^{\alpha -1} + \sum \limits _{j=1}^k\theta _j f_j(x)}\Bigg ] p_{\theta }^{(\alpha )}(x). \end{aligned}$$

(69)

Let $R_{\theta }(x) = q(x)^{\alpha -1} + \sum \limits _{j=1}^k\theta _j f_j(x)$. Partial differentiation produces

$$\begin{aligned} \frac{\partial \eta _j}{\partial \theta _i}= & {} \frac{\alpha }{\alpha -1}\frac{\partial }{\partial \theta _i}\left( \sum _x \frac{f_j(x)p_{\theta }^{(\alpha )}(x)}{R_{\theta }(x)}\right) \nonumber \\= & {} \frac{\alpha }{\alpha -1}\sum _x \frac{R_{\theta }(x)f_j(x) \partial _i p_{\theta }^{(\alpha )}(x) - f_j(x) p_{\theta }^{(\alpha )}(x) \partial _i(R_{\theta }(x))}{(R_{\theta }(x))^2}\nonumber \\= & {} \frac{\alpha }{\alpha -1}\sum _x \left[ \frac{f_j(x)}{R_{\theta }(x)}\partial _i p_{\theta }^{(\alpha )}(x) - p_{\theta }^{(\alpha )}(x)\frac{f_j(x)}{R_{\theta }(x)} \frac{f_i(x)}{R_{\theta }(x)} \right] . \end{aligned}$$

(70)

From (64) - (67), we have

$$\begin{aligned} \frac{\alpha }{\alpha -1}\frac{f_i(x)}{ R_{\theta }}&= \partial _i \log p_{\theta }^{(\alpha )}(x) + \eta _i. \end{aligned}$$

(71)

Substituting (71) into (70) gives

$$\begin{aligned} \frac{\partial \eta _j}{\partial \theta _i}&= \frac{\alpha }{\alpha -1}\sum _x \bigg [ \partial _i p_{\theta }^{(\alpha )}(x) \frac{\alpha -1}{\alpha } (\partial _i \log p_{\theta }^{(\alpha )}(x) + \eta _i) \nonumber \\&- p_{\theta }^{(\alpha )}(x) \left( \frac{\alpha -1}{\alpha }\right) ^2 (\partial _j \log p_{\theta }^{(\alpha )}(x) + \eta _j) (\partial _i \log p_{\theta }^{(\alpha )}(x) + \eta _i) \bigg ]\nonumber \\&= \sum _x \bigg [\partial _i p_{\theta }^{(\alpha )}(x) (\partial _i \log p_{\theta }^{(\alpha )}(x) - \eta _i)\nonumber \\&- p_{\theta }^{(\alpha )}(x) \left( \frac{\alpha -1}{\alpha }\right) (\partial _j \log p_{\theta }^{(\alpha )}(x) + \eta _j) (\partial _i \log p_{\theta }^{(\alpha )}(x) + \eta _i)\bigg ]\nonumber \\&= \sum _x \bigg [\partial _i p_{\theta }^{(\alpha )}(x) \partial _i \log p_{\theta }^{(\alpha )}(x) + \eta _i \partial _i p_{\theta }^{(\alpha )}(x)\nonumber \\&- \left( \frac{\alpha -1}{\alpha }\right) p_{\theta }^{(\alpha )}(x) \bigg (\partial _j \log p_{\theta }^{(\alpha )}(x) \partial _i \log p_{\theta }^{(\alpha )}(x) + \eta _i \partial _i \log p_{\theta }^{(\alpha )}(x) \nonumber \\&+\eta _i \partial _j \log p_{\theta }^{(\alpha )}(x) + \eta _i \eta _j \bigg )\bigg ]\nonumber \end{aligned}$$

$$\begin{aligned}&= \sum _x \bigg [\partial _i p_{\theta }^{(\alpha )}(x) \partial _i \log p_{\theta }^{(\alpha )}(x) - p_{\theta }^{(\alpha )}(x) \frac{1}{p_{\theta }^{(\alpha )}(x)} \partial _j p_{\theta }^{(\alpha )}(x) \partial _i \log p_{\theta }^{(\alpha )}(x) \nonumber \\&+\frac{1}{\alpha }\bigg (p_{\theta }^{(\alpha )}(x)\partial _j \log p_{\theta }^{(\alpha )}(x) \partial _i \log p_{\theta }^{(\alpha )}(x)\bigg ) - \left( \frac{\alpha -1}{\alpha }\right) \eta _i \eta _j p_{\theta }^{(\alpha )}(x) \bigg ) \bigg ]\nonumber \\&= \sum _x \bigg [\frac{1}{\alpha }\bigg (p_{\theta }^{(\alpha )}(x)\partial _j \log p_{\theta }^{(\alpha )}(x) \partial _i \log p_{\theta }^{(\alpha )}(x)\bigg ) - \left( \frac{\alpha -1}{\alpha }\right) \eta _i \eta _j p_{\theta }^{(\alpha )}(x) \bigg ) \bigg ]\nonumber \\&= \alpha g_{ij}^{(\alpha )}(\theta ) - \left( \frac{\alpha -1}{\alpha }\right) \eta _i \eta _j.\end{aligned}$$

(72)

This shows that $\eta _i$ cannot be the dual parameters of $\theta _i$ for the statistical model $\mathbb {M}^{(\alpha )}$. This completes the proof.