Skip to main content
SpringerLink
Log in

Entropy and Concentration

  • Chapter
  • First Online:
Harmonic and Applied Analysis

Part of the book series: Applied and Numerical Harmonic Analysis ((ANHA))

  • 970 Accesses

Abstract

The chapter presents the entropy method to prove concentration inequalities for functions of independent random variables. After the introduction of a general framework the famous bounded difference inequality, versions of Bernstein’s inequality, and the Gaussian concentration inequality are derived. Applications include vector-valued concentration, random matrices, and the suprema of empirical processes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 44.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 59.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 59.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adamczak, R., et al.: Moment inequalities for u-statistics. Ann. Probab. 34(6), 2288–2314 (2006)

    Article  MathSciNet  Google Scholar 

  2. Arcones, M.A.: A Bernstein-type inequality for u-statistics and u-processes. Stat. Probab. Lett. 22(3), 239–247 (1995)

    Article  MathSciNet  Google Scholar 

  3. Bartlett, P., Mendelson, S.: Rademacher and gaussian complexities: risk bounds and structural results. J. Mach. Learn. Res. 3, 463–482 (2002)

    MathSciNet  MATH  Google Scholar 

  4. Bernstein, S.: Theory of Probability. Moscow (1927)

    Google Scholar 

  5. Boltzmann, L.: Über die Beziehung zwischen dem zweiten Hauptsatze des mechanischen Wärmetheorie und der Wahrscheinlichkeitsrechnung, respective den Sätzen über das Wärmegleichgewicht. Kk Hof-und Staatsdruckerei (1877)

    Google Scholar 

  6. Boucheron, S., Lugosi, G., Massart, P.: Concentration Inequalities. Oxford University Press, Oxford (2013)

    Google Scholar 

  7. Boucheron, S., Lugosi, G., Massart, P., et al.: Concentration inequalities using the entropy method. Ann. Probab. 31(3), 1583–1614 (2003)

    Article  MathSciNet  Google Scholar 

  8. Boucheron, S., Lugosi, G., Massart, P., et al.: On concentration of self-bounding functions. Electron. J. Probab. 14, 1884–1899 (2009)

    Article  MathSciNet  Google Scholar 

  9. Bousquet, O.: A Bennett concentration inequality and its application to suprema of empirical processes. C.R. Math. 334(6), 495–500 (2002)

    Article  MathSciNet  Google Scholar 

  10. Chatterjee, S.: Concentration inequalities with exchangeable pairs. Ph.D. thesis, Citeseer (2005)

    Google Scholar 

  11. Chebyshev, P.L.: Sur les valeurs limites des intégrales. Imprimerie de Gauthier-Villars (1874)

    Google Scholar 

  12. Gibbs, J.W.: Elementary Principles in Statistical Mechanics: Developed with Especial Reference to the Rational Foundations of Thermodynamics. C. Scribner’s Sons, New York (1902)

    Google Scholar 

  13. Giné, E., Latała, R., Zinn, J.: Exponential and moment inequalities for u-statistics. High Dimensional Probability II, pp. 13–38. Springer, Berlin (2000)

    Google Scholar 

  14. Gross, L.: Logarithmic sobolev inequalities. Am. J. Math. 97(4), 1061–1083 (1975)

    Article  MathSciNet  Google Scholar 

  15. Hoeffding, W.: A class of statistics with asymptotically normal distribution. Ann. Math. Stat., pp. 293–325 (1948)

    Google Scholar 

  16. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58, 301 (1963)

    Article  MathSciNet  Google Scholar 

  17. Houdré, C.: The iterated jackknife estimate of variance. Statist. Probab. Lett. 35(2), 197–201 (1997)

    Article  MathSciNet  Google Scholar 

  18. Ledoux, M.: The Concentration of Measure Phenomenon, vol. 89. American Mathematical Society, Providence (2001)

    Google Scholar 

  19. Lieb, E.H.: Some convexity and subadditivity properties of entropy. Inequalities, pp. 67–79. Springer, Berlin (2002)

    Google Scholar 

  20. Massart, P., et al.: About the constants in Talagrand’s concentration inequalities for empirical processes. Ann. Probab. 28(2), 863–884 (2000)

    Article  MathSciNet  Google Scholar 

  21. Maurer, A., et al.: A Bernstein-type inequality for functions of bounded interaction. Bernoulli 25(2), 1451–1471 (2019)

    Article  MathSciNet  Google Scholar 

  22. McAllester, D., Ortiz, L.: Concentration inequalities for the missing mass and for histogram rule error. J. Mach. Learn. Res. 4(Oct), 895–911 (2003)

    Google Scholar 

  23. McDiarmid, C.: Concentration. Probabilistic Methods of Algorithmic Discrete Mathematics, pp. 195–248. Springer, Berlin (1998)

    Google Scholar 

  24. de la Peña, V.H.: Decoupling and khintchine’s inequalities for u-statistics. Ann. Probab., pp. 1877–1892 (1992)

    Google Scholar 

  25. Popper, K.R.: Logik der forschung (1934). The Logic of Scientific Discovery. [Google Scholar] (1968)

    Google Scholar 

  26. Steele, J.M.: An Efron-Stein inequality for nonsymmetric statistics. Ann. Probab., pp. 753–758 (1986)

    Google Scholar 

  27. Talagrand, M.: Concentration of measure and isoperimetric inequalities in product spaces. Publications Mathématiques de l’Institut des Hautes Etudes Scientifiques 81(1), 73–205 (1995)

    Google Scholar 

  28. Talagrand, M.: A new look at independence. Ann. Probab., pp. 1–34 (1996)

    Google Scholar 

  29. Vershynin, R.: High-dimensional Probability: An Introduction with Applications in Data Science, vol. 47. Cambridge University Press, Cambridge (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Istituto Italiano di Tecnologia, Via Morego 30, 16163, Genova, Italy

    Andreas Maurer

Authors
  1. Andreas Maurer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andreas Maurer .

Editor information

Editors and Affiliations

  1. Dipartimento di Matematica, Università di Genova, Genova, Italy

    Filippo De Mari

  2. Dipartimento di Matematica, Università di Genova, Genova, Italy

    Ernesto De Vito

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Maurer, A. (2021). Entropy and Concentration. In: De Mari, F., De Vito, E. (eds) Harmonic and Applied Analysis. Applied and Numerical Harmonic Analysis. Birkhäuser, Cham. https://doi.org/10.1007/978-3-030-86664-8_2

Download citation

Publish with us

Policies and ethics

Search

Navigation