Shannon Entropy Versus Renyi Entropy from a Cryptographic Viewpoint

Abstract

We provide a new inequality that links two important entropy notions: Shannon Entropy $H_1$ and collision entropy $H_2$. Our formula gives the worst possible amount of collision entropy in a probability distribution, when its Shannon Entropy is fixed. While in practice it is easier to evaluate Shannon entropy than other entropy notions, it is well known in folklore that it does not provide a good estimate of randomness quality from a cryptographic viewpoint, except very special settings. Our results and techniques put this in a quantitative form, allowing us to precisely answer the following questions:

• (a) How accurately does Shannon entropy estimate uniformity? Concretely, if the Shannon entropy of an n-bit source X is $n-\epsilon$, where $\epsilon$ is a small number, can we conclude that X is close to uniform? This question is motivated by uniformity tests based on entropy estimators, like Maurer’s Universal Test.

• (b) How much randomness can we extract having high Shannon entropy? That is, if the Shannon entropy of an n-bit source X is $n-O(1)$, how many almost uniform bits can we retrieve, at least? This question is motivated by the folklore upper bound $O(\log (n))$.

• (c) Can we use high Shannon entropy for key derivation? More precisely, if we have an n-bit source X of Shannon entropy $n-O(1)$, can we use it as a secure key for some applications, such as square-secure applications? This is motivated by recent improvements in key derivation obtained by Barak et al. (CRYPTO’11) and Dodis et al. (TCC’14), which consider keys with some entropy deficiency.

Our approach involves convex optimization techniques, which yield the shape of the “worst” distribution, and the use of the Lambert W function, by which we resolve equations coming from Shannon Entropy constraints. We believe that it may be useful and of independent interests elsewhere, particularly for studying Shannon Entropy with constraints.

A full version of this paper is available at https://eprint.iacr.org/2014/967.pdf.

M. Skórski — This work was partly supported by the WELCOME/2010-4/2 grant founded within the framework of the EU Innovative Economy Operational Programme.

Appendices

A Proof of Theorem 2

Proof

(Proof of Theorem 2 ). Consider the corresponding optimization problem

(15)

The Lagrangian associated to (15) is given by

$$\begin{aligned} L(\varvec{p},(\lambda _1,\lambda _2)) = -\log _2\left( \sum _{i=1}^{d} \varvec{p}_i^2 \right) - \lambda _1\left( \sum _{i=1}^{d}\varvec{p}_i-1\right) - \lambda _2\left( -\sum _{i=1}^{d} \varvec{p}_i\log _2 \varvec{p}_i-k\right) \end{aligned}$$

(16)

Claim

The first and second derivative of the Lagrangian (16) are given by

$$\begin{aligned} \frac{\partial L}{\partial \varvec{p}_i}&= -2\log _2\mathrm {e}\cdot \frac{\varvec{p}_i}{\varvec{p}^2} - \lambda _1 + \lambda _2\log _2\mathrm {e}+ \lambda _2\log _2 \varvec{p}_i \end{aligned}$$

(17)

$$\begin{aligned} \frac{\partial ^2 L}{\partial \varvec{p}_i\varvec{p}_j}&=4\log _2\mathrm {e}\cdot \frac{\varvec{p}_i\varvec{p}_j}{(\varvec{p}^2)^2}+[i=j]\cdot \left( -\frac{2\log _2\mathrm {e}}{\varvec{p}^2} + \frac{\lambda _2\log _2\mathrm {e}}{\varvec{p}_i} \right) \end{aligned}$$

(18)

Claim

Let $\varvec{p}^{*}$ be a non-uniform optimal point to 15. Then it satisfies $\varvec{p}^{*}_i \in \{a,b\}$ for every i, where a, b are some constant such that

$$\begin{aligned} -\frac{2\log _2\mathrm {e}}{{\varvec{p}^{*}}^2} + \frac{\lambda _2\log _2\mathrm {e}}{a} > 0 > -\frac{2\log _2\mathrm {e}}{{\varvec{p}^{*}}^2} + \frac{\lambda _2\log _2\mathrm {e}}{b} \end{aligned}$$

(19)

Proof

(Proof of Appendix A ). At the optimal point $\varvec{p}$ we have $\frac{\partial L}{\partial \varvec{p}_i} = 0$ which means

$$\begin{aligned} -2\log _2\mathrm {e}\cdot \frac{\varvec{p}_i}{\varvec{p}^2} - \lambda _1 + \lambda _2\log _2\mathrm {e}+ \lambda _2\log _2 \varvec{p}_i = 0,\quad i=1,\ldots ,d. \end{aligned}$$

(20)

Think of $\varvec{p}^2$ as a constant, for a moment. Then the left-hand side of Eq. (20) is of the form $-c_1\varvec{p}_i +c_2\log _2\varvec{p}_i +c_3$ with some positive constant $c_1$ and real constants $c_2,c_3$. Since the derivative of this function equals $-c_1 + \frac{c_2}{\varvec{p}_i}$, the left-hand side is either decreasing (when $c_2 \leqslant 0$) or concave (when $c_2> 0$). For the non-uniform solution the latter must be true (because otherwise $\varvec{p}_i$ for $i=1,\ldots ,d$ are equal). Hence the Eq. (20) has at most two solutions $\{a,b\}$, where $a<b$ and both are not dependent on i. Moreover, its left-hand side has the maximum between a and b, thus we must have $-c_1 + \frac{c_2}{a} > 0 > -c_1 + \frac{c_2}{b}$. Expressing this in terms of $\lambda _1,\lambda _2$ we get Eq. (19).

Claim

Let $\varvec{p}^{*}$ and a, b be as in Appendix A. Then $\varvec{p}_i = a$ for all but one index i.

Proof

(Proof of Appendix A ). The tangent space of the constraints $\sum _{i=1}^{d}\varvec{p}_i-1=~0$ and $-\sum _{i=1}^{d} \varvec{p}_i\log _2 \varvec{p}_i-k=0$ at the point $\varvec{p}$ is the set of all vectors $h\in \mathbb {R}^d$ satisfying the following conditions

$$\begin{aligned} \begin{array}{rl} \sum _{i=1}^{d} h_i &{} = 0 \\ \sum _{i=1}^{d}-(\log _2\varvec{p}_i+\log _2\mathrm {e})h_i &{} = 0 \end{array} \end{aligned}$$

(21)

Intuitively, the tangent space includes all infinitesimally small movements that are consistent with the constraints. Let $D^{2} L = \left( \frac{\partial ^2 L}{\partial \varvec{p}_i\varvec{p}_j}\right) _{i,j}$ be the second derivative of L. It is well known that the necessary second order condition for the minimizer $\varvec{p}$ is $h^{T}(D^{2}) L h \geqslant 0$ for all vectors in the tangent space (21). We have

$$\begin{aligned} {h}^T \cdot (D^{2} L) \cdot {h} = 4\log _2\mathrm {e}\cdot \frac{\left( \sum _{i=}^{d}\varvec{p}_i {h}_i\right) ^2}{(\varvec{p}^2)^2}+\sum _{i=1}^{d}\left( -\frac{2\log _2\mathrm {e}}{\varvec{p}^2} + \frac{\lambda _2\log _2\mathrm {e}}{\varvec{p}_i} \right) h_i^2. \end{aligned}$$

Now, if the are two different indexes $i_1,i_2$ such that $\varvec{p}^{*}_{i_1} = \varvec{p}^{*}_{i_2} = b$, we can define $h_{i_1} = -\delta$, $h_{i_2}=\delta$ and $h_i = 0$ for $i\not \in \{i_1,i_2\}$. Then we get

$$\begin{aligned} {h}^T \cdot (D^{2} L) \cdot {h} = 2 \left( -\frac{2\log _2\mathrm {e}}{\varvec{p}^2} + \frac{\lambda _2\log _2\mathrm {e}}{b} \right) \delta ^2 \end{aligned}$$

which is negative according to Eq. (19). Thus we have reached a contradiction.

Finally, taking into account the case of possibly uniform $\varvec{p}^{*}$ and combining it with the last claim we get

Claim

The optimal point $\varvec{p}^{*}$ satisfies $\varvec{p}^{*}_{i_0} = b$ and $\varvec{p}^{*}_i =\frac{1-b}{d-1}$ for $i\not =i_0$, for some $b \geqslant \frac{1}{d}$. Then we have $\mathrm {H}(\varvec{p}^{*}) = \mathrm {H}(b) +(1-b)\log _2(d-1)$ and $H_2(\varvec{p}^{*})=~-\log _2\left( b^2 + \frac{(1-b)^2}{d-1} \right)$.

It remains to take a closer look at Eq. (7). It defines b as an implicit function of k and d. Its uniqueness is a consequence of the following claim

Claim

The function $f(b)= \mathrm {H}(b) +(1-b)\log _2(d-1)$ is strictly decreasing and concave for $b\geqslant \frac{1}{d}$.

Proof

(Proof of Appendix A ). The derivative equals $\frac{\partial f}{\partial b}=-\log _2\frac{b}{1-b}-\log _2 (d-1)$ and hence, for $\frac{1}{d}< b < 1$, is at most $-\log _2\frac{\frac{1}{d}}{1-\frac{1}{d}}-\log _2 (d-1) = 0$. The second derivative is $\frac{\partial ^2 f}{\partial b^2}= -\frac{\log _2\mathrm {e}}{b(1-b)}$. Thus, the claim follows.

The statement follows now by Appendices A and B.

B Proof of Lemma 2

Proof

Let $\varDelta = \log _2 d - k$ be the gap in the Shannon Entropy. Note that from Eq. (7) and the inequality $-2 \leqslant d( \log _2 (d-1) - \log _2 d) \leqslant -\log _2\mathrm {e}$ it follows that

$$\begin{aligned} -b\log _2 b -(1-b)\log _2 (1-b) -b\log _2 d= -\varDelta + C_1(d)\cdot d^{-1} \end{aligned}$$

where $\log _2\mathrm {e} \leqslant C_1 \leqslant 2$. Note that $f\left( \frac{1}{2}\right) = -1+\frac{1}{2}\log _2(d-1) < \log _2 d -1$. Since $\varDelta \leqslant 1$ implies $f(b) \geqslant \log _2 d-1$, by Appendix A we conclude that $b<\frac{1}{2}$. Next, observe that $1 \leqslant \frac{-(1-b)\log _2(1-b)}{b} \leqslant \log _2\mathrm {e}$, for $0 < b<\frac{1}{2}$. This means that $-(1-b)\log _2(1-b) = -b\log _2 C_2(d)$ where $\frac{1}{\mathrm {e}}\leqslant C_2(d) \leqslant \frac{1}{2}$. Now we have

$$\begin{aligned} -b\log _2 ( C_2(d)\cdot d\cdot b ) = -\varDelta + C_1(d)\cdot d^{-1}. \end{aligned}$$

Let $y = C_2(d)\cdot d\cdot b$. Our equation is equivalent to $y\ln y = C_3(d)\cdot d\cdot \varDelta -C_1(d)C_3(d)$. where $C_3=C_2/\log _2\mathrm {e}$. Using the Lambert-W function, which is defined as $W(x)\cdot \mathrm {e}^{W(x)}=x$, we can solve this equations as

$$\begin{aligned} b = \frac{\mathrm {e}^{W\left( C_3(d) d \varDelta - C_3(d)C_1(d) \right) }}{C_2(d) d}. \end{aligned}$$

(22)

For $x\geqslant \mathrm {e}$ we have the well-known approximation for the Lambert W function [HH08]

$$\begin{aligned} \ln x- \ln \ln x < W(x)\leqslant \ln x- \ln \ln x + \ln (1+\mathrm {e}^{-1}). \end{aligned}$$

(23)

Provided that $C_3(d) d \varDelta - C_3(d)C_1(d) \geqslant 1$, which is satisfied if $d \varDelta \geqslant 6$, we obtain

$$\begin{aligned} b=\frac{C_3(d) d \varDelta - C_3(d)C_1(d) }{C_3(d) d \cdot \log _2\left( C_3(d) d \varDelta - C_3(d)C_1(d)\right) } \cdot C_4(d) \end{aligned}$$

(24)

where $1 \leqslant C_4(d) \leqslant 1+\mathrm {e}^{-1}$. Since the function $x\rightarrow \frac{x}{\log _2 x}$ is increasing for $x\geqslant \mathrm {e}$ and since for $d \varDelta \geqslant 13$ we have $C_3(d) d \varDelta - C_3(d)C_1(d) \geqslant \mathrm {e}$, from Eq. (24) we get

$$\begin{aligned} b&\leqslant \frac{C_3(d) d \varDelta }{C_3(d) d \cdot \log _2\left( C_3(d) d \varDelta \right) } \cdot {C_4(d)} = \frac{C_4(d)\varDelta }{\log _2\left( C_3(d) d \varDelta \right) } \end{aligned}$$

(25)

from which the right part of Eq. (8) follows by the inequalities on $C_3$ and $C_4$. For the lower bound, note that for $d \varDelta \geqslant 13$ we have $C_3(d) d \varDelta - C_3(d)C_1(d) \geqslant C_3(d) d \varDelta \cdot \frac{11}{13}$ because it reduces to $C_1(d)\leqslant 2$, and that $C_3(d) d \varDelta \cdot \frac{11}{13} \geqslant 13\cdot \frac{1}{\mathrm {e}\log _2\mathrm {e}}\cdot \frac{11}{13} >\mathrm {e}$. Therefore, by Eq. (24) and the mononicity of $\frac{x}{\log _2 x}$ we get

$$\begin{aligned} b\geqslant \frac{\frac{11}{13}C_3(d) d \varDelta }{C_3(d) d \cdot \log _2\left( \frac{11}{13}C_3(d) d \varDelta \right) } \cdot {C_4(d)} = \frac{\frac{11}{13}C_4(d) \varDelta }{ \log _2\left( \frac{11}{13}C_3(d) d \varDelta \right) }, \end{aligned}$$

(26)

from which the left part of Eq. (8) follows by the inequalities on $C_3$ and $C_4$.