Shannon Entropy Versus Renyi Entropy from a Cryptographic Viewpoint

Maciej Skórski

doi:10.1007/978-3-319-27239-9_16

IMA International Conference on Cryptography and Coding

IMACC 2015: Cryptography and Coding pp 257-274 | Cite as

Shannon Entropy Versus Renyi Entropy from a Cryptographic Viewpoint

Authors
Authors and affiliations

Maciej Skórski

Conference paper

First Online: 08 December 2015

Part of the Lecture Notes in Computer Science book series (LNCS, volume 9496)

Abstract

We provide a new inequality that links two important entropy notions: Shannon Entropy $H_1$ and collision entropy $H_2$. Our formula gives the worst possible amount of collision entropy in a probability distribution, when its Shannon Entropy is fixed. While in practice it is easier to evaluate Shannon entropy than other entropy notions, it is well known in folklore that it does not provide a good estimate of randomness quality from a cryptographic viewpoint, except very special settings. Our results and techniques put this in a quantitative form, allowing us to precisely answer the following questions:

(a)

How accurately does Shannon entropy estimate uniformity? Concretely, if the Shannon entropy of an n-bit source X is $n-\epsilon $, where $\epsilon $ is a small number, can we conclude that X is close to uniform? This question is motivated by uniformity tests based on entropy estimators, like Maurer’s Universal Test.
(b)

How much randomness can we extract having high Shannon entropy? That is, if the Shannon entropy of an n-bit source X is $n-O(1)$, how many almost uniform bits can we retrieve, at least? This question is motivated by the folklore upper bound $O(\log (n))$.
(c)

Can we use high Shannon entropy for key derivation? More precisely, if we have an n-bit source X of Shannon entropy $n-O(1)$, can we use it as a secure key for some applications, such as square-secure applications? This is motivated by recent improvements in key derivation obtained by Barak et al. (CRYPTO’11) and Dodis et al. (TCC’14), which consider keys with some entropy deficiency.

Our approach involves convex optimization techniques, which yield the shape of the “worst” distribution, and the use of the Lambert W function, by which we resolve equations coming from Shannon Entropy constraints. We believe that it may be useful and of independent interests elsewhere, particularly for studying Shannon Entropy with constraints.

Keywords

Shannon entropy Renyi entropy Smooth renyi entropy Min-entropy Lambda w function

This is a preview of subscription content, to check access.

Notes

Acknowledgment

The author thanks anonymous reviewers for their valuable comments.

A Proof of Theorem 2

Proof

(Proof of Theorem 2 ). Consider the corresponding optimization problem

Open image in new window

(15)

The Lagrangian associated to (15) is given by

$$\begin{aligned} L(\varvec{p},(\lambda _1,\lambda _2)) = -\log _2\left( \sum _{i=1}^{d} \varvec{p}_i^2 \right) - \lambda _1\left( \sum _{i=1}^{d}\varvec{p}_i-1\right) - \lambda _2\left( -\sum _{i=1}^{d} \varvec{p}_i\log _2 \varvec{p}_i-k\right) \end{aligned}$$

(16)

Claim

The first and second derivative of the Lagrangian (16) are given by

$$\begin{aligned} \frac{\partial L}{\partial \varvec{p}_i}&= -2\log _2\mathrm {e}\cdot \frac{\varvec{p}_i}{\varvec{p}^2} - \lambda _1 + \lambda _2\log _2\mathrm {e}+ \lambda _2\log _2 \varvec{p}_i \end{aligned}$$

(17)

$$\begin{aligned} \frac{\partial ^2 L}{\partial \varvec{p}_i\varvec{p}_j}&=4\log _2\mathrm {e}\cdot \frac{\varvec{p}_i\varvec{p}_j}{(\varvec{p}^2)^2}+[i=j]\cdot \left( -\frac{2\log _2\mathrm {e}}{\varvec{p}^2} + \frac{\lambda _2\log _2\mathrm {e}}{\varvec{p}_i} \right) \end{aligned}$$

(18)

Claim

Let $\varvec{p}^{*}$ be a non-uniform optimal point to 15. Then it satisfies $\varvec{p}^{*}_i \in \{a,b\}$ for every i, where a, b are some constant such that

$$\begin{aligned} -\frac{2\log _2\mathrm {e}}{{\varvec{p}^{*}}^2} + \frac{\lambda _2\log _2\mathrm {e}}{a} > 0 > -\frac{2\log _2\mathrm {e}}{{\varvec{p}^{*}}^2} + \frac{\lambda _2\log _2\mathrm {e}}{b} \end{aligned}$$

(19)

Proof

(Proof of Appendix A ). At the optimal point $\varvec{p}$ we have $\frac{\partial L}{\partial \varvec{p}_i} = 0$ which means

$$\begin{aligned} -2\log _2\mathrm {e}\cdot \frac{\varvec{p}_i}{\varvec{p}^2} - \lambda _1 + \lambda _2\log _2\mathrm {e}+ \lambda _2\log _2 \varvec{p}_i = 0,\quad i=1,\ldots ,d. \end{aligned}$$

(20)

Think of $\varvec{p}^2$ as a constant, for a moment. Then the left-hand side of Eq. (20) is of the form $-c_1\varvec{p}_i +c_2\log _2\varvec{p}_i +c_3$ with some positive constant $c_1$ and real constants $c_2,c_3$. Since the derivative of this function equals $-c_1 + \frac{c_2}{\varvec{p}_i}$, the left-hand side is either decreasing (when $c_2 \leqslant 0$) or concave (when $c_2> 0$). For the non-uniform solution the latter must be true (because otherwise $\varvec{p}_i$ for $i=1,\ldots ,d$ are equal). Hence the Eq. (20) has at most two solutions $\{a,b\}$, where $a<b$ and both are not dependent on i. Moreover, its left-hand side has the maximum between a and b, thus we must have $-c_1 + \frac{c_2}{a} > 0 > -c_1 + \frac{c_2}{b}$. Expressing this in terms of $\lambda _1,\lambda _2$ we get Eq. (19).

Claim

Let $\varvec{p}^{*}$ and a, b be as in Appendix A. Then $\varvec{p}_i = a$ for all but one index i.

Proof

(Proof of Appendix A ). The tangent space of the constraints $\sum _{i=1}^{d}\varvec{p}_i-1=~0$ and $-\sum _{i=1}^{d} \varvec{p}_i\log _2 \varvec{p}_i-k=0$ at the point $\varvec{p}$ is the set of all vectors $h\in \mathbb {R}^d$ satisfying the following conditions

$$\begin{aligned} \begin{array}{rl} \sum _{i=1}^{d} h_i &{} = 0 \\ \sum _{i=1}^{d}-(\log _2\varvec{p}_i+\log _2\mathrm {e})h_i &{} = 0 \end{array} \end{aligned}$$

(21)

Intuitively, the tangent space includes all infinitesimally small movements that are consistent with the constraints. Let $D^{2} L = \left( \frac{\partial ^2 L}{\partial \varvec{p}_i\varvec{p}_j}\right) _{i,j}$ be the second derivative of L. It is well known that the necessary second order condition for the minimizer $\varvec{p}$ is $h^{T}(D^{2}) L h \geqslant 0$ for all vectors in the tangent space (21). We have

$$\begin{aligned} {h}^T \cdot (D^{2} L) \cdot {h} = 4\log _2\mathrm {e}\cdot \frac{\left( \sum _{i=}^{d}\varvec{p}_i {h}_i\right) ^2}{(\varvec{p}^2)^2}+\sum _{i=1}^{d}\left( -\frac{2\log _2\mathrm {e}}{\varvec{p}^2} + \frac{\lambda _2\log _2\mathrm {e}}{\varvec{p}_i} \right) h_i^2. \end{aligned}$$

Now, if the are two different indexes $i_1,i_2$ such that $\varvec{p}^{*}_{i_1} = \varvec{p}^{*}_{i_2} = b$, we can define $h_{i_1} = -\delta $, $h_{i_2}=\delta $ and $h_i = 0$ for $i\not \in \{i_1,i_2\}$. Then we get

$$\begin{aligned} {h}^T \cdot (D^{2} L) \cdot {h} = 2 \left( -\frac{2\log _2\mathrm {e}}{\varvec{p}^2} + \frac{\lambda _2\log _2\mathrm {e}}{b} \right) \delta ^2 \end{aligned}$$

which is negative according to Eq. (19). Thus we have reached a contradiction.

Finally, taking into account the case of possibly uniform $\varvec{p}^{*}$ and combining it with the last claim we get

Claim

The optimal point $\varvec{p}^{*}$ satisfies $\varvec{p}^{*}_{i_0} = b$ and $\varvec{p}^{*}_i =\frac{1-b}{d-1}$ for $i\not =i_0$, for some $b \geqslant \frac{1}{d}$. Then we have $\mathrm {H}(\varvec{p}^{*}) = \mathrm {H}(b) +(1-b)\log _2(d-1)$ and $H_2(\varvec{p}^{*})=~-\log _2\left( b^2 + \frac{(1-b)^2}{d-1} \right) $.

It remains to take a closer look at Eq. (7). It defines b as an implicit function of k and d. Its uniqueness is a consequence of the following claim

Claim

The function $f(b)= \mathrm {H}(b) +(1-b)\log _2(d-1)$ is strictly decreasing and concave for $b\geqslant \frac{1}{d}$.

Proof

(Proof of Appendix A ). The derivative equals $\frac{\partial f}{\partial b}=-\log _2\frac{b}{1-b}-\log _2 (d-1)$ and hence, for $\frac{1}{d}< b < 1$, is at most $-\log _2\frac{\frac{1}{d}}{1-\frac{1}{d}}-\log _2 (d-1) = 0$. The second derivative is $\frac{\partial ^2 f}{\partial b^2}= -\frac{\log _2\mathrm {e}}{b(1-b)}$. Thus, the claim follows.

The statement follows now by Appendices A and B.

B Proof of Lemma 2

Proof

Let $\varDelta = \log _2 d - k$ be the gap in the Shannon Entropy. Note that from Eq. (7) and the inequality $-2 \leqslant d( \log _2 (d-1) - \log _2 d) \leqslant -\log _2\mathrm {e}$ it follows that

$$\begin{aligned} -b\log _2 b -(1-b)\log _2 (1-b) -b\log _2 d= -\varDelta + C_1(d)\cdot d^{-1} \end{aligned}$$

where $\log _2\mathrm {e} \leqslant C_1 \leqslant 2$. Note that $f\left( \frac{1}{2}\right) = -1+\frac{1}{2}\log _2(d-1) < \log _2 d -1$. Since $\varDelta \leqslant 1$ implies $f(b) \geqslant \log _2 d-1$, by Appendix A we conclude that $b<\frac{1}{2}$. Next, observe that $1 \leqslant \frac{-(1-b)\log _2(1-b)}{b} \leqslant \log _2\mathrm {e}$, for $0 < b<\frac{1}{2}$. This means that $-(1-b)\log _2(1-b) = -b\log _2 C_2(d)$ where $\frac{1}{\mathrm {e}}\leqslant C_2(d) \leqslant \frac{1}{2}$. Now we have

$$\begin{aligned} -b\log _2 ( C_2(d)\cdot d\cdot b ) = -\varDelta + C_1(d)\cdot d^{-1}. \end{aligned}$$

Let $y = C_2(d)\cdot d\cdot b$. Our equation is equivalent to $y\ln y = C_3(d)\cdot d\cdot \varDelta -C_1(d)C_3(d)$. where $C_3=C_2/\log _2\mathrm {e}$. Using the Lambert-W function, which is defined as $W(x)\cdot \mathrm {e}^{W(x)}=x$, we can solve this equations as

$$\begin{aligned} b = \frac{\mathrm {e}^{W\left( C_3(d) d \varDelta - C_3(d)C_1(d) \right) }}{C_2(d) d}. \end{aligned}$$

(22)

For $x\geqslant \mathrm {e}$ we have the well-known approximation for the Lambert W function [HH08]

$$\begin{aligned} \ln x- \ln \ln x < W(x)\leqslant \ln x- \ln \ln x + \ln (1+\mathrm {e}^{-1}). \end{aligned}$$

(23)

Provided that $C_3(d) d \varDelta - C_3(d)C_1(d) \geqslant 1$, which is satisfied if $d \varDelta \geqslant 6$, we obtain

$$\begin{aligned} b=\frac{C_3(d) d \varDelta - C_3(d)C_1(d) }{C_3(d) d \cdot \log _2\left( C_3(d) d \varDelta - C_3(d)C_1(d)\right) } \cdot C_4(d) \end{aligned}$$

(24)

where $1 \leqslant C_4(d) \leqslant 1+\mathrm {e}^{-1}$. Since the function $x\rightarrow \frac{x}{\log _2 x}$ is increasing for $x\geqslant \mathrm {e}$ and since for $d \varDelta \geqslant 13$ we have $C_3(d) d \varDelta - C_3(d)C_1(d) \geqslant \mathrm {e}$, from Eq. (24) we get

$$\begin{aligned} b&\leqslant \frac{C_3(d) d \varDelta }{C_3(d) d \cdot \log _2\left( C_3(d) d \varDelta \right) } \cdot {C_4(d)} = \frac{C_4(d)\varDelta }{\log _2\left( C_3(d) d \varDelta \right) } \end{aligned}$$

(25)

from which the right part of Eq. (8) follows by the inequalities on $C_3$ and $C_4$. For the lower bound, note that for $d \varDelta \geqslant 13$ we have $C_3(d) d \varDelta - C_3(d)C_1(d) \geqslant C_3(d) d \varDelta \cdot \frac{11}{13} $ because it reduces to $C_1(d)\leqslant 2$, and that $C_3(d) d \varDelta \cdot \frac{11}{13} \geqslant 13\cdot \frac{1}{\mathrm {e}\log _2\mathrm {e}}\cdot \frac{11}{13} >\mathrm {e}$. Therefore, by Eq. (24) and the mononicity of $\frac{x}{\log _2 x}$ we get

$$\begin{aligned} b\geqslant \frac{\frac{11}{13}C_3(d) d \varDelta }{C_3(d) d \cdot \log _2\left( \frac{11}{13}C_3(d) d \varDelta \right) } \cdot {C_4(d)} = \frac{\frac{11}{13}C_4(d) \varDelta }{ \log _2\left( \frac{11}{13}C_3(d) d \varDelta \right) }, \end{aligned}$$

(26)

from which the left part of Eq. (8) follows by the inequalities on $C_3$ and $C_4$.

References

[AIS11]
A proposal for: Functionality classes for random number generators1, Technical report AIS 30, Bonn, Germany, September 2011. http://tinyurl.com/bkwt2wf
[AOST14]
Acharya, J., Orlitsky, A., Suresh, A.T., Tyagi, H.: The complexity of estimating renyi entropy, CoRR abs/1408.1000 (2014)Google Scholar
[BDK+11]
Barak, B., Dodis, Y., Krawczyk, H., Pereira, O., Pietrzak, K., Standaert, F.-X., Yu, Y.: Leftover hash lemma, revisited. In: Rogaway, P. (ed.) CRYPTO 2011. LNCS, vol. 6841, pp. 1–20. Springer, Heidelberg (2011) CrossRefGoogle Scholar
[BK12]
Barker, E.B., Kelsey, J.M.: Sp 800–90a recommendation for random number generation using deterministic random bit generators, Technical report, Gaithersburg, MD, United States (2012)Google Scholar
[BKMS09]
Bouda, J., Krhovjak, J., Matyas, V., Svenda, P.: Towards true random number generation in mobile environments. In: Jøsang, A., Maseng, T., Knapskog, S.J. (eds.) NordSec 2009. LNCS, vol. 5838, pp. 179–189. Springer, Heidelberg (2009) CrossRefGoogle Scholar
[BL05]
Bucci, M., Luzzi, R.: Design of testable random bit generators. In: Rao, J.R., Sunar, B. (eds.) CHES 2005. LNCS, vol. 3659, pp. 147–156. Springer, Heidelberg (2005) CrossRefGoogle Scholar
[BST03]
Barak, B., Shaltiel, R., Tromer, E.: True random number generators secure in a changing environment. In: Walter, C.D., Koç, Ç.K., Paar, C. (eds.) CHES 2003. LNCS, vol. 2779, pp. 166–180. Springer, Heidelberg (2003) CrossRefGoogle Scholar
[Cac97]
Cachin, C.: Smooth entropy and rényi entropy. In: Fumy, W. (ed.) EUROCRYPT 1997. LNCS, vol. 1233, pp. 193–208. Springer, Heidelberg (1997) CrossRefGoogle Scholar
[Cor99]
Coron, J.-S.: On the security of random sources. In: Imai, H., Zheng, Y. (eds.) PKC 1999. LNCS, vol. 1560, p. 29. Springer, Heidelberg (1999) CrossRefGoogle Scholar
[CW79]
Carter, J.L., Wegman, M.N.: Universal classes of hash functions. J. Comput. Syst. Sci. 18(2), 143–154 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
[DPR+13]
Dodis, Y., Pointcheval, D., Ruhault, S., Vergniaud, D., Wichs, D.: Security analysis of pseudo-random number generators with input: /dev/random is not robust. In: Proceedings of the 2013 ACM SIGSAC Conference on Computer and Communications Security, CCS 2013, pp. 647–658. ACM, New York (2013)Google Scholar
[DY13]
Dodis, Y., Yu, Y.: Overcoming weak expectations. In: Sahai, A. (ed.) TCC 2013. LNCS, vol. 7785, pp. 1–22. Springer, Heidelberg (2013) CrossRefGoogle Scholar
[HH08]
Hoorfar, A., Hassani, M.: Inequalities on the lambert w function and hyperpower function. J. Inequal. Pure Appl. Math. 9(2), 07–15 (2008)MathSciNetzbMATHGoogle Scholar
[HILL88]
Hstad, J., Impagliazzo, R., Levin, L.A., Luby, M.: Pseudo-random generation from one-way functions. In: Proceedings of the 20TH STOC, pp. 12–24 (1988)Google Scholar
[HILL99]
Hastad, J., Impagliazzo, R., Levin, L.A., Luby, M.: A pseudorandom generator from any one-way function. SIAM J. Comput. 28(4), 1364–1396 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
[Hol06]
Holenstein, T.: Pseudorandom generators from one-way functions: a simple construction for any hardness. In: Halevi, S., Rabin, T. (eds.) TCC 2006. LNCS, vol. 3876, pp. 443–461. Springer, Heidelberg (2006) CrossRefGoogle Scholar
[Hol11]
Holenstein, T.: On the randomness of repeated experimentGoogle Scholar
[LPR11]
Lauradoux, C., Ponge, J., Röck, A.: Online Entropy Estimation for Non-Binary Sources and Applications on iPhone. Rapport de recherche, Inria (2011) Google Scholar
[Mau92]
Maurer, U.: A universal statistical test for random bit generators. J. Cryptology 5, 89–105 (1992)MathSciNetzbMATHGoogle Scholar
[NZ96]
Nisan, N., Zuckerman, D.: Randomness is linear in space. J. Comput. Syst. Sci. 52(1), 43–52 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
[RTS00]
Radhakrishnan, J., Ta-Shma, A.: Bounds for dispersers, extractors, and depth-two superconcentrators. SIAM J. Discrete Math. 13, 2000 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
[RW04]
Renner, R., Wolf, S.: Smooth renyi entropy and applications. In: Proceedings of the International Symposium on Information Theory, ISIT 2004, p. 232. IEEE (2004)Google Scholar
[RW05]
Renner, R.S., Wolf, S.: Simple and tight bounds for information reconciliation and privacy amplification. In: Roy, B. (ed.) ASIACRYPT 2005. LNCS, vol. 3788, pp. 199–216. Springer, Heidelberg (2005) CrossRefGoogle Scholar
[Sha48]
Shannon, C.E.: A mathematical theory of communication. Bell Syst. Tech. J. 27(3), 379–423 (1948)MathSciNetCrossRefzbMATHGoogle Scholar
[Sha11]
Shaltiel, R.: An introduction to randomness extractors. In: Aceto, L., Henzinger, M., Sgall, J. (eds.) ICALP 2011, Part II. LNCS, vol. 6756, pp. 21–41. Springer, Heidelberg (2011) CrossRefGoogle Scholar
[Shi15]
Shikata, J.: Design and analysis of information-theoretically secure authentication codes with non-uniformly random keys. IACR Cryptology ePrint Arch. 2015, 250 (2015)Google Scholar
[VSH11]
Voris, J., Saxena, N., Halevi, T.: Accelerometers and randomness: perfect together. In: Proceedings of the Fourth ACM Conference on Wireless Network Security, WiSec 2011, pp. 115–126. ACM, New York (2011)Google Scholar

Copyright information

Authors and Affiliations

Maciej Skórski
- 1
Email author

1.Cryptology and Data Security GroupUniversity of WarsawWarsawPoland

Shannon Entropy Versus Renyi Entropy from a Cryptographic Viewpoint

Abstract

Keywords

Notes

Acknowledgment

References

Copyright information

Authors and Affiliations

Personalised recommendations

Cite paper