Exact Maximum-Entropy Estimation with Feynman Diagrams

Abstract

A longstanding open problem in statistics is finding an explicit expression for the probability measure which maximizes entropy with respect to given constraints. In this paper a solution to this problem is found, using perturbative Feynman calculus. The explicit expression is given as a sum over weighted trees.

Appendix A: Proof of Lemma 3

Recall that by Lemma 24

$$\begin{aligned} \log \left( \sum _{\sigma }q_{\sigma }\exp \left( -\sum _{i=1}^{k}\lambda _{i}r_{i}(\sigma )\right) \right) =1+\lambda _{0}=1+\lambda _{0}(\lambda _{1},\ldots ,\lambda _{k}). \end{aligned}$$

Hence $\lambda _{0}$ is an analytic function of $\lambda _{1},\ldots ,\lambda _{k}$ around $\lambda _{1}=\dots =\lambda _{k}=0$. Now,

$$\begin{aligned}&\rho _{i}(\lambda _{1},\ldots ,\lambda _{k})=\sum _{\sigma }r_{i}(\sigma )q_{\sigma }\exp \left( -1-\lambda _{0}(\lambda _{1},\ldots ,\lambda _{k})-\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) \\&\quad =\frac{\sum _{\sigma }r_{i}(\sigma )q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) }{\exp (1+\lambda _{0}(\lambda _{1},\ldots ,\lambda _{k}))}=\frac{\sum _{\sigma }r_{i}(\sigma )q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) }{\sum _{\sigma }q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) } \end{aligned}$$

So that $\rho _{i}(\lambda _{1},\ldots ,\lambda _{k})$ is an analytic function of $\lambda _{1},\ldots ,\lambda _{k}$

The proof that $\lambda _{i}=\lambda _{i}(\rho _{1},\ldots ,\rho _{k})$ is an analytic function of $\rho _{1},\ldots ,\rho _{k}$ around

$$\begin{aligned} \lambda _{1}=\dots =\lambda _{k}=\rho _{1}=\dots =\rho _{k}=0, \end{aligned}$$

uses the analytic inverse function theorem. It is enough to show that the Jacobian $\frac{\partial (\rho _{1},\ldots ,\rho _{k})}{\partial (\lambda _{1},\ldots ,\lambda _{k})}$ is invertible for$\lambda _{1}=\dots =\lambda _{k}=0.$

But

$$\begin{aligned} \frac{\partial \rho _{i}}{\partial \lambda _{j}}&= \frac{\left( \sum _{\sigma }r_{i}(\sigma )r_{j}(\sigma )q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) \right) \left( \sum _{\sigma }q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) \right) }{\left( \sum _{\sigma }q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) \right) ^{2}}\nonumber \\&\quad -\frac{\left( \sum _{\sigma }r_{l}(\sigma )q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) \right) \left( \sum _{\sigma }r_{j}(\sigma )q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) \right) }{{\left( \sum _{\sigma }q_{\sigma }\exp \left( -\sum _{l=1}^{k}\lambda _{l}r_{l}(\sigma )\right) \right) ^{2}}}. \end{aligned}$$

(35)

Evaluation at $\lambda _{1}=\dots =\lambda _{k}=0$ gives

$$\begin{aligned} \frac{\partial \rho _{i}}{\partial \lambda _{j}}\left| _{\lambda _{1}=\cdots =\lambda _{k}=0}\right.= & {} \frac{\left( \sum _{\sigma }r_{i}(\sigma )r_{j}(\sigma )q_{\sigma }\right) \left( \sum _{\sigma }q_{\sigma }\right) -\left( \sum _{\sigma }r_{i}(\sigma )q_{\sigma }\right) \left( \sum _{\sigma }r_{j}(\sigma )q_{\sigma }\right) }{\left( \sum _{\sigma }q_{\sigma }\right) ^{2}} \\= & {} \frac{\mathbb {E}_{Q}(r_{i}r_{j})\cdot 1-\mathbb {E}_{Q}(r_{i})\mathbb {E}_{Q}(r_{j})}{1^{2}}=\mathrm {Cov}_{Q}(r_{i},r_{j}). \end{aligned}$$

By assumption the KL constraint problem is normalized, hence

$$\begin{aligned} \frac{\partial \rho _{i}}{\partial \lambda _{j}}\left| _{\lambda _{1}=\cdots =\lambda _{k}=0}\right. =\mathrm {Cov}_{Q}(r_{i},r_{j})=\delta _{i,j}. \end{aligned}$$

The Jacobian $\frac{\partial (\rho _{1},\ldots ,\rho _{k})}{\partial (\lambda _{1},\ldots ,\lambda _{k})}=I_{k}$ is thus invertible. $\square$