Sum-of-Squares Relaxations for Information Theory and Variational Inference

Abstract

We consider extensions of the Shannon relative entropy, referred to as f-divergences. Three classical related computational problems are typically associated with these divergences: (a) estimation from moments, (b) computing normalizing integrals, and (c) variational inference in probabilistic models. These problems are related to one another through convex duality, and for all of them, there are many applications throughout data science, and we aim for computationally tractable approximation algorithms that preserve properties of the original problem such as potential convexity or monotonicity. In order to achieve this, we derive a sequence of convex relaxations for computing these divergences from non-centered covariance matrices associated with a given feature vector: starting from the typically non-tractable optimal lower-bound, we consider an additional relaxation based on “sums-of-squares”, which is is now computable in polynomial time as a semidefinite program. We also provide computationally more efficient relaxations based on spectral information divergences from quantum information theory. For all of the tasks above, beyond proposing new relaxations, we derive tractable convex optimization algorithms, and we present illustrations on multivariate trigonometric polynomials and functions on the Boolean hypercube.

Appendices

Appendix A: Newton Method for Computing F

In order to compute $F(v,w) = \sup _{r \geqslant 0} \frac{ r v + w - f(r)}{r+1}$ in Sect. 2.3, we assume $(v,w)\in \mathbb {{R}}^2$ fixed and define $\varphi (r) = \frac{ r v + w - f(r)}{r+1}$, which is twice differentiable, and so that, $\varphi '(r) = \frac{1}{(r+1)^2} \big [ v-w - \big ( (r+1)f'(r) - f(r) \big ) \big ]$. Since the function $r \mapsto (r+1)f'(r) - f(r)$ has derivative $r \mapsto (r+1)f''(r)$, it is strictly positive. Thus the derivative $\varphi '$ thus has at most one zero, and we aim to solve the equation $(r+1)f'(r) - f(r) = v - w$.

We now focus on the special case $f(t) = t \log t - t + 1$, where $\psi (r) = (r+1)f'(r) - f(r) = r - 1 + \log r$, and has full range on $\mathbb {{R}}_+$ so that the equation above has a unique solution. In order to compute its solution, for $v-w \geqslant 0$, we iterate Newton’s method [24, Section 4.8] $r \leftarrow r + \frac{v-w - \psi (r)}{\psi '(r) } = \frac{2 - \log (r) + v-w}{ 1 + 1/r}$ for 5 iterations to reach machine precision, while for $v-w \leqslant 0$ we iterate Newton’s method on the logarithm of r $\log r \leftarrow \log r + \frac{v-w - \psi (r)}{r \psi '(r) } = \frac{ 1 + v-w + ( \log r -1) r}{ 1 + r}$ for 5 iterations.

Appendix B: Decomposition of Operator-Convex Functions

We have the following particular cases from Sect. 2.

• $\alpha$-divergences: $f(t) =\frac{1}{\alpha (\alpha -1)} \big [t^\alpha - \alpha t + (\alpha -1) \big ] = \frac{1}{\alpha } \frac{\sin (\alpha -1)\pi }{(\alpha -1)\pi } (t-1)^2 \int _0^{+\infty } \frac{1}{t+\lambda } \frac{ \lambda ^\alpha d\lambda }{(1+\lambda )^2}$ for $\alpha \in (-1,2)$. Other representations exist for $\alpha =-1$ and $\alpha =2$ (see below), but other cases are not operator-convex.

• KL divergence ($\alpha =1$): $f (t) = t \log t -t + 1 = \int _0^{+\infty } \frac{(t-1)^2}{t+\lambda } \frac{ \lambda d\lambda }{(\lambda +1)^2}$.

• Rerverse KL divergence ($\alpha =0)$: $f(t) = - \log t + t - 1= \int _0^{+\infty } \frac{(t-1)^2}{t+\lambda } \frac{ d\lambda }{(\lambda +1)^2}$.

• Pearson $\chi ^2$ divergence ($\alpha =2$): $f(t) = \frac{1}{2} (t-1)^2$ is operator convex.

• Reverse Pearson $\chi ^2$ divergence ($\alpha =-1$): $f(t) = \frac{1}{2} \big ( \frac{1}{t} + t \big ) - 1 = \frac{1}{2} \frac{(t-1)^2}{t}$ is operator convex, with $d\nu (\lambda )$ proportional to a Dirac at $\lambda = 0$.

• Le Cam distance: $f(t) = \frac{(t-1)^2}{t+1}$ is operator convex with $d\nu (\lambda )$ proportional to a Dirac at $\lambda = 1$.

• Jensen–Shannon divergence: $f(t) =2 t \log \frac{2t}{t+1} + 2 \log \frac{2}{t+1} =2 t \log t - 2(t+1) \log (t+1) + 2(t+1) \log 2 = 2t \log t - 4 \frac{t+1}{2}\log \frac{t+1}{2}$ is operator convex, as it can be written $f(t) = 2(t-1)^2 \int _0^{+\infty } \big ( \frac{1}{t+\lambda } - \frac{1}{t+1+2\lambda } \big ) \frac{\lambda d\lambda }{(1+\lambda )^2}$, which leads to $f(t) =2 (t-1)^2 \int _0^{+\infty } \frac{1}{t+\lambda } \frac{\lambda d\lambda }{(1+\lambda )^2} - 2 (t-1)^2 \int _1^{+\infty } \frac{1}{t+\lambda } \frac{(\lambda -1) d\lambda }{(1+\lambda )^2}$.

Appendix C: Proofs of Lemmas 1 and 2

In this section, we prove Lemma 1 from Sect. 7, taken from [40, Section 9.1] and shown in here for completeness and the expression of the maximizers. We also prove the extension Lemma 2.

We start by the dual formulation:

$$\begin{aligned}{} & {} \sup _{M,N \in {\mathbb {H}}_d } \ \mathop { \mathrm tr}[ MA ] + \mathop { \mathrm tr}[ NB] \quad \text{ such } \text{ that }\quad \forall r \geqslant 0, \ r M + N \preccurlyeq f(r) I\\{} & {} \quad = \inf _{\Lambda \ {\mathbb {H}_d^+}-\mathrm{valued \ measure \ on}\ \mathbb {{R}}_+} \int _0^{+\infty }\! f(r) \mathop { \mathrm tr}\big [ d\Lambda (r) ] \quad \text{ such } \text{ that }\quad \int _0^{+\infty }\! d\Lambda (r) \\{} & {} \quad = B \quad \text{ and }\quad \int _0^{+\infty } \! r d\Lambda (r) = A, \end{aligned}$$

which can be solved in closed form.

Indeed, given the eigendecomposition $B^{-1/2} A B^{-1/2} = \sum _{i =1}^d \lambda _i u_i u_i^*$, we consider $\Lambda = \sum _{i =1}^d B^{1/2} u_i u_i^*B^{1/2} \delta _{\lambda _i}$, where $\delta _{\lambda _i}$ is the Dirac measure at $\lambda _i$, so that we get a feasible measure $\Lambda$, and an objective equal to $\sum _{i=1}^d f(\lambda _i) \mathop { \mathrm tr}\big [B^{1/2} V B^{1/2} u_i u_i^*\big ] = \mathop { \mathrm tr}\big [ B^{1/2} V B^{1/2} f \big ( B^{-1/2}AB^{-1/2} \big ) \big ]$. Thus the infimum is less than $\mathop { \mathrm tr}\big [ B^{1/2} V B^{1/2} f \big ( B^{-1/2}A B^{-1/2} \big ) \big ]$.

The other direction is a direct consequence of the operator Jensen’s inequality [28]: for any feasible measure $\Lambda$ approached by an empirical measure $\sum _{i =1}^m M_i \delta _{r_i}$, with $M_i \succcurlyeq 0$, we have $\sum _{i =1}^n( M_i^{1/2} B^{-1/2} )^*( M_i^{1/2} B^{-1/2} ) = I$, and thus

$$\begin{aligned} \int _{\mathbb {{R}}_+} \! f(r) d\Lambda (r)= & {} B^{1/2} \Big (\sum _{i =1}^m ( M_i^{1/2} B^{-1/2} )^*f(r_iI) ( M_i^{1/2} B^{-1/2} ) \Big ) B^{1/2} \\&\succcurlyeq&B^{1/2} f \Big (\sum _{i =1}^m( M_i^{1/2} B^{-1/2} )^*(r_i I) ( M_i^{1/2} B^{-1/2} ) \Big ) B^{1/2} \\= & {} B^{1/2} f \big ( B^{-1/2}A B^{-1/2} \big ) B^{1/2}. \end{aligned}$$

The lower bound follows by letting the number m of Diracs go to infinity to tightly approximate any feasible matrix $\Lambda$.

In order to obtain the minimizers M and N, we simply notice that they are the gradients of the function $(A,B) \mapsto \mathop { \mathrm tr}\big [ B f \big ( B^{-1/2}A B^{-1/2} \big ) \big ]$ with respect to A and B. We thus get, using gradients of spectral functions

$$\begin{aligned} M^*= \sum _{i,j=1}^d \frac{f(\lambda _i)-f(\lambda _j)}{\lambda _i - \lambda _j} u_i^\top B u_j \cdot B^{-1/2} u_i u_j^*B^{-1/2}, \end{aligned}$$

with the convention that for $\lambda _i = \lambda _j$, $\frac{f(\lambda _i)-f(\lambda _j)}{\lambda _i - \lambda _j} = f'(\lambda _i)$. To obtain $N^*$, we simply consider the identity $\mathop { \mathrm tr}\big [ B f \big ( B^{-1/2}A B^{-1/2} \big ) \big ] = \mathop { \mathrm tr}\big [ A g \big ( A^{-1/2}B A^{-1/2} \big ) \big ]$, for $g(t) = t f(1/t)$, and use the corresponding formula.

Proof of Lemma 2

We simply need to change slightly the proof of the lemma above, by replacing $\int _0^{+\infty }\! f(r) \mathop { \mathrm tr}\big [ d\Lambda (r) \big ]$ by $\int _0^{+\infty }\! f(r) \mathop { \mathrm tr}\big [ V d\Lambda (r) \big ]$ with all inequalities preserved because $V \succcurlyeq 0$.

Appendix D: Computing Integrals

A third task is related to f-divergences beyond computing the divergences themselves and the associated log-partition functions. In this section, we consider the task of computing $\int _{\mathcal {X}} f^*( h(x)) dq(x)$, where q is a finite positive measure on $\mathcal {X}$, $f^*$ is the Fenchel conjugate of f, and $h: \mathcal {X}\rightarrow \mathbb {{R}}$ an arbitrary function (such that the integral is finite). The difference with computing log-partition functions in Eq. (24) is minor and we thus extend only a few results from Sect. 8 and Sect. 9, most without proofs as they follow the same lines as results for f-partition functions.

For $f(t) = t \log t - t + 1$, we have $f^*(u) =e^u - 1$, and we there aim at estimating integrals of exponential functions, a classical task in probabilistic modelling (see [43, 62] and references therein), which up to a logarithm is the same as computing the log-partition function; however, they are different for other functions f.

This computational task can be classically related to f-divergences by Fenchel duality as we have:

$$\begin{aligned} \int _{\mathcal {X}} f^*( h(x)) dq(x) = \sup _{p\ \mathrm{positive \ measure \ on}\ \mathcal {X}}\ \int _\mathcal {X}h(x) dp(x) - D(p\Vert q), \end{aligned}$$

where the only difference with Eq. (23) is that p is not assumed to sum to one. Below, we show that for functions h(x) which are quadratic forms in $\varphi (x)$, we can replace $D(p\Vert q)$ by the lower-bound we just defined above, and obtain a computable upper bound of the integral.

We also have the representation corresponding to Eq. (25), that will be useful later:

$$\begin{aligned} \int _\mathcal {X}f^*(h(x)) dq(x)= & {} \inf _{ w: \mathcal {X}\rightarrow \mathbb {{R}}} - \int _\mathcal {X}w(x) dq(x) \quad \text{ such } \text{ that }\quad \forall x \in \mathcal {X},\nonumber \\{} & {} \forall r \geqslant 0, \quad r h(x) + w(x) \leqslant f(r). \end{aligned}$$

(33)

1.1 Related Work

There exist many ways of estimating integrals, in particular in compact sets in small dimensions, where various quadrature rules, such as the trapezoidal or Simpson’s rule, can be applied to compute integrals based on function evaluations, with well-defined convergence rates [18]. In higher dimensions, still based on function evaluations, Bayes–Hermite quadrature rules [48], and the related kernel quadrature rules [5, 16] come with precise convergence rates linking approximation error and number of function evaluations [3]. An alternative in our context is Monte-Carlo integration from samples from q [52], with convergence rate in $O(1/\sqrt{n})$ from n function evaluations.

In this paper, we follow [8] and consider computing integrals given a specific knowledge of the integrand, here of the form $f^*(h(x))$, where h is a known quadratic form in a feature vector $\varphi (x)$. While we also use a sum-of-squares approach as in [8], we rely on different tools (link with f-divergences and partition functions rather than integration by parts).

1.2 Relaxations

In order to compute integrals, we simply use the same technique but without the constraint that measures sum to one, that is, without the constraint that $\mathop { \mathrm tr}[A]=1$. Starting from Eq. (33), we get, with $B = \Sigma _q$:

$$\begin{aligned} \widetilde{C}_q(H)= & {} \int _\mathcal {X}f^*\big ( \varphi (x)^*H \varphi (x) \big ) dq(x) = \sup _{ p\in {\mathcal {M}}_+(\mathcal {X})} \ \int _\mathcal {X}\varphi (x)^*H \varphi (x) dp(x) - D(p\Vert q)\\\leqslant & {} \sup _{ A \in \mathcal {C} } \ \mathop { \mathrm tr}[ HA] - D^{\textrm{OPT}}(A\Vert B) = \widetilde{C}_q^{\textrm{OPT}} (H) \\= & {} \inf _{N \in {\mathbb {H}}_d } - \mathop { \mathrm tr}[N B ] \text{ such } \text{ that } \forall r \geqslant 0, \quad \Gamma ( rH + N) \leqslant f(r). \end{aligned}$$

Note that we only have an inequality here because we are not optimizing over q. We then get two computable relaxations by considering $D^{\textrm{SOS}}(A\Vert B)$ and $D^{\textrm{QT}}(A\Vert B)$ instead of $D^{\textrm{OPT}}(A\Vert B)$, with the respective formulations:

$$\begin{aligned} \widetilde{C}_q^{\textrm{SOS}} (H)= & {} \inf _{N \in {\mathbb {H}}_d } \ \ - \mathop { \mathrm tr}[N B ] \text{ such } \text{ that } \forall r \geqslant 0, \ f(r) U - rH - N \in \widehat{\mathcal {C}}^*\\ \widetilde{C}_q^{\textrm{QT}} (H)= & {} \sup _{A \in {\mathbb {H}}_d } \ \inf _{V \succcurlyeq 0, \ V - I\in {\mathcal {V}}^\perp } \ \mathop { \mathrm tr}[A H ] - \mathop { \mathrm tr}\big [ B^{1/2} V B^{1/2} f \big (B^{-1/2}A B^{-1/2} \big ) \big ]. \end{aligned}$$

Dual formulations and algorithms can then easily be derived.

Appendix E: Dual Formulations

In this appendix we present dual variational formulations to most of the formulations proposed in the main paper. We only consider the relaxations of f-divergences. Formulations for f-partition functions can be derived similarly.

1.1 f-Divergences

We can consider the Lagrangian dual of Eq. (3), by introducing a Lagrange multiplier $\lambda$ for the infinite-dimensional constraint $\forall x \in \mathcal {X}, \forall r \geqslant 0, \ r v(x)+w(x) \leqslant f(r)$ in the form of a positive finite measure $\lambda$ on $\mathcal {X}\times \mathbb {{R}}_+$ [30]. We then obtain, using strong duality for equality constraints [39, Section 8.6]:

$$\begin{aligned} D(p\Vert q)= & {} \inf _{ \lambda \in {\mathcal {M}}_+( \mathcal {X}\times \mathbb {{R}}_+) } \sup _{ v,w: \mathcal {X}\rightarrow \mathbb {{R}}} \int _\mathcal {X}v(x) dp(x) + \int _\mathcal {X}w(x) dq(x)\nonumber \\{} & {} + \int _\mathcal {X}\int _{\mathbb {{R}}_+} \big [ f(r) - rv(x) - w(x) \big ] d\lambda (x,r)\nonumber \\= & {} \inf _{\lambda \in {\mathcal {M}}_+( \mathcal {X}\times \mathbb {{R}}_+) } \ \int _\mathcal {X}\int _{\mathbb {{R}}_+} f(r) d\lambda (x,r) \\{} & {} \text{ such } \text{ that }\quad \int _{\mathbb {{R}}_+} \!d\lambda (\cdot ,r)=dq(\cdot )\quad \text{ and }\quad \int _{\mathbb {{R}}_+} \! r d\lambda (\cdot ,r)=dp(\cdot ),\nonumber \end{aligned}$$

(34)

with the two constraints resulting from the maximization with respect to v and w.

1.2 Optimal Relaxation of f-Divergences ($D^{\textrm{OPT}}$)

We can also formulate Eq. (15) as a minimization problem; we we can use Lagrangian duality, akin to Eq. (34). This requires to introduce a Lagrange multiplier for the constraint $\forall r \geqslant 0, \ \Gamma (rM+N) \leqslant f(r)$, which is equivalent to, $\forall r \geqslant 0, \ f(r) - rM-N \in \widehat{\mathcal {C}}^*$, which leads to a ${\mathcal {C}}$-valued finite measure on $\mathbb {{R}}_+$ [30], to get:

$$\begin{aligned} D^{\textrm{OPT}} ( A \Vert B )= & {} \inf _{\Lambda \ { \mathcal {C}}-\mathrm{valued \ measure \ on}\ \mathbb {{R}}_+} \ \ \sup _{M,N \in \mathbb {H}_d} \ \ \mathop { \mathrm tr}[ AM ] + \mathop { \mathrm tr}[BN] \nonumber \\{} & {} + \int _0^{+\infty }\! \mathop { \mathrm tr}\big [ d\Lambda (r) ( f(r) - rM - N ) \big ] \nonumber \\= & {} \inf _{\Lambda \ { \mathcal {C}}-\mathrm{valued \ measure \ on}\ \mathbb {{R}}_+} \int _0^{+\infty }\! f(r) \mathop { \mathrm tr}\big [ d\Lambda (r)] \quad \text{ such } \text{ that }\nonumber \\{} & {} \int _0^{+\infty }\! d\Lambda (r)= B \quad \text{ and }\quad \int _0^{+\infty } \! r d\Lambda (r) = A. \end{aligned}$$

(35)

1.3 SOS Relaxation of f-Divergences ($D^{\textrm{SOS}}$)

We can also get a formulation for Eq. (17), akin to Eqs. (34) and (35). This requires to introduce a Lagrange multiplier for the constraint $\forall r \geqslant 0, \ f(r) U - rM-N \in \widehat{\mathcal {C}}^*$, which is a $\widehat{\mathcal {C}}$-valued finite measure on $\mathbb {{R}}_+$ [30], to get:

$$\begin{aligned} D^{\textrm{SOS}} ( A \Vert B )= & {} \inf _{\Lambda \ {\widehat{\mathcal {C}}}-\mathrm{valued \ measure \ on}\ \mathbb {{R}}_+} \ \ \sup _{M,N \in \mathbb {H}_d} \ \ \mathop { \mathrm tr}[ AM ] + \mathop { \mathrm tr}[ BN] \nonumber \\{} & {} +\int _0^{+\infty }\! \mathop { \mathrm tr}\big [ d\Lambda (r) ( f(r) - rM - N ) \big ] \nonumber \\= & {} \inf _{\Lambda \ {\widehat{\mathcal {C}}}-\mathrm{valued \ measure \ on}\ \mathbb {{R}}_+} \int _0^{+\infty }\! f(r) \mathop { \mathrm tr}\big [ d\Lambda (r) ] \ \text{ such } \text{ that } \nonumber \\{} & {} \int _0^{+\infty }\! d\Lambda (r) = B\quad \text{ and }\quad \int _0^{+\infty } \! r d\Lambda (r) = A, \end{aligned}$$

(36)

as maximizing out $M,N \in \mathbb {H}_d$ introduces linear constraints.

1.4 Quantum Relaxations of f-Divergences ($D^{\textrm{SOS}}$)

Equation (21) also has a primal–dual form as

$$\begin{aligned} D^{\textrm{QT}}(A\Vert B)= & {} \sup _{V \succcurlyeq 0, \ V - I\in {\mathcal {V}}^\perp } \ \ \inf _{\Lambda \ {\mathbb {H}_d^+}-\mathrm{valued \ measure \ on}\ \mathbb {{R}}_+} \int _0^{+\infty }\! f(r) \mathop { \mathrm tr}\big [ d\Lambda (r) V ] \nonumber \\{} & {} \text{ such } \text{ that } \int _0^{+\infty }\! d\Lambda (r) = B \quad \text{ and }\quad \int _0^{+\infty } \! r d\Lambda (r) = A. \end{aligned}$$

(37)

We also have a formulation akin to Eq. (36), that is, adding a Lagrange multiplier $\Sigma \in {\mathcal {V}}$ in Eq. (37) for the constraint $I-V \in {\mathcal {V}}^\perp$:

$$\begin{aligned} D^{\textrm{QT}}(A\Vert B)= & {} \inf _{\Sigma \in {\mathcal {V}}, \ \Lambda \ {{\mathbb {H}}_d^+}-\mathrm{valued \ measure \ on}\ \mathbb {{R}}_+} \int _0^{+\infty }\! f(r) \mathop { \mathrm tr}\big [ d\Lambda (r) ] + \mathop { \mathrm tr}[ \Sigma ] \\{} & {} \text{ such } \text{ that } \int _0^{+\infty }\! d\Lambda (r) = B, \int _0^{+\infty } r d\Lambda (r) = A, \text{ and } \\{} & {} \int _0^{+\infty }\! f(r) d\Lambda (r) \preccurlyeq \Sigma , \end{aligned}$$

which shows the additional relaxation compared to $D^{\textrm{SOS}}(A\Vert B)$, for which $\Lambda$ is a measure (almost everywhere) valued in ${\mathcal {V}}$, while here it is only in $\mathbb {H}_d^+$.