Foundations of Computational Mathematics

, Volume 12, Issue 4, pp 389–434 | Cite as

User-Friendly Tail Bounds for Sums of Random Matrices

Open Access
Article

Abstract

This paper presents new probability inequalities for sums of independent, random, self-adjoint matrices. These results place simple and easily verifiable hypotheses on the summands, and they deliver strong conclusions about the large-deviation behavior of the maximum eigenvalue of the sum. Tail bounds for the norm of a sum of random rectangular matrices follow as an immediate corollary. The proof techniques also yield some information about matrix-valued martingales.

In other words, this paper provides noncommutative generalizations of the classical bounds associated with the names Azuma, Bennett, Bernstein, Chernoff, Hoeffding, and McDiarmid. The matrix inequalities promise the same diversity of application, ease of use, and strength of conclusion that have made the scalar inequalities so valuable.

Keywords

Discrete-time martingale Large deviation Probability inequality Random matrix Sum of independent random variables 

Mathematics Subject Classification (2000)

60B20 60F10 60G50 60G42 

References

  1. 1.
    N. Ailon, B. Chazelle, The fast Johnson–Lindenstrauss transform and approximate nearest neighbors, SIAM J. Comput. 39(1), 302–322 (2009). MathSciNetMATHCrossRefGoogle Scholar
  2. 2.
    D. Achlioptas, F. McSherry, Fast computation of low-rank matrix approximations, J. Assoc. Comput. Mach. 54(2), Article 10 (2007) (electronic). MathSciNetGoogle Scholar
  3. 3.
    R. Ahlswede, A. Winter, Strong converse for identification via quantum channels, IEEE Trans. Inf. Theory 48(3), 569–579 (2002). MathSciNetMATHCrossRefGoogle Scholar
  4. 4.
    R. Bhatia, Matrix Analysis. Graduate Texts in Mathematics, vol. 169 (Springer, Berlin, 1997), p. 10. CrossRefGoogle Scholar
  5. 5.
    R. Bhatia, Positive Definite Matrices (Princeton Univ. Press, Princeton, 2007). Google Scholar
  6. 6.
    V. Bogdanov, Gaussian Measures (American Mathematical Society, Providence, 1998). Google Scholar
  7. 7.
    A. Buchholz, Operator Khintchine inequality in non-commutative probability, Math. Ann. 319, 1–16 (2001). MathSciNetMATHCrossRefGoogle Scholar
  8. 8.
    A. Buchholz, Optimal constants in Khintchine-type inequalities for Fermions, Rademachers and q-Gaussian operators, Bull. Pol. Acad. Sci., Math. 53(3), 315–321 (2005). MathSciNetMATHCrossRefGoogle Scholar
  9. 9.
    H. Chernoff, A measure of the asymptotic efficiency for tests of a hypothesis based on the sum of observations, Ann. Math. Stat. 23(4), 493–507 (1952). MathSciNetMATHCrossRefGoogle Scholar
  10. 10.
    D. Cristofides, K. Markström, Expansion properties of random Cayley graphs and vertex transitive graphs via matrix martingales, Random Struct. Algorithms 32(8), 88–100 (2008). CrossRefGoogle Scholar
  11. 11.
    E. Candès, J.K. Romberg, Sparsity and incoherence in compressive sampling, Inverse Probl. 23(3), 969–985 (2007). MATHCrossRefGoogle Scholar
  12. 12.
    V.H. de la Peña, E. Giné, Decoupling: From Dependence to Independence, Probability and Its Applications (Springer, Berlin, 2002). Google Scholar
  13. 13.
    K.R. Davidson, S. J. Szarek. Local operator theory, random matrices, and Banach spaces, in Handbook of Banach Space Geometry, ed. by W.B. Johnson, J. Lindenstrauss (Elsevier, Amsterdam, 2002), pp. 317–366. Google Scholar
  14. 14.
    A. Dembo, O. Zeitouni, Large Deviations: Techniques and Applications, 2nd edn. (Springer, Berlin, 1998). MATHGoogle Scholar
  15. 15.
    E.G. Effros, A matrix convexity approach to some celebrated quantum inequalities, Proc. Natl. Acad. Sci. USA 106(4), 1006–1008 (2009). MathSciNetMATHCrossRefGoogle Scholar
  16. 16.
    H. Epstein, Remarks on two theorems of E. Lieb, Commun. Math. Phys. 31, 317–325 (1973). MATHCrossRefGoogle Scholar
  17. 17.
    D.A. Freedman, On tail probabilities for martingales, Ann. Probab. 3(1), 100–118 (1975). MATHCrossRefGoogle Scholar
  18. 18.
    Y. Gordon, Some inequalities for Gaussian processes and applications, Isr. J. Math. 50(4), 265–289 (1985). MATHCrossRefGoogle Scholar
  19. 19.
    Y. Gordon, Majorization of Gaussian processes and geometric applications, Probab. Theory Relat. Fields 91(2), 251–267 (1992). MATHCrossRefGoogle Scholar
  20. 20.
    D. Gross, Recovering low-rank matrices from few coefficients in any basis, IEEE Trans. Inf. Theory 57(3), 1548–1566 (2011). CrossRefGoogle Scholar
  21. 21.
    N.J. Higham, Functions of Matrices: Theory and Computation (Society for Industrial and Applied Mathematics, Philadelphia, 2008). MATHCrossRefGoogle Scholar
  22. 22.
    R.A. Horn, C.R. Johnson, Matrix Analysis (Cambridge Univ. Press, Cambridge, 1985). MATHGoogle Scholar
  23. 23.
    R.A. Horn, C.R. Johnson, Topics in Matrix Analysis (Cambridge Univ. Press, Cambridge, 1994). MATHGoogle Scholar
  24. 24.
    N. Halko, P.-G. Martinsson, J.A. Tropp, Finding structure with randomness: stochastic algorithms for constructing approximate matrix decompositions, SIAM Rev. 53(2), 217–288 (2011). MathSciNetMATHCrossRefGoogle Scholar
  25. 25.
    F. Hansen, G.K. Pedersen, Jensen’s operator inequality, Bull. Lond. Math. Soc. 35, 553–564 (2003). MathSciNetMATHCrossRefGoogle Scholar
  26. 26.
    M. Junge, Q. Xu, On the best constants in some non-commutative martingale inequalities, Bull. Lond. Math. Soc. 37, 243–253 (2005). MathSciNetMATHCrossRefGoogle Scholar
  27. 27.
    M. Junge, Q. Xu, Noncommutative Burkholder/Rosenthal inequalities II: Applications, Isr. J. Math. 167, 227–282 (2008). MathSciNetMATHCrossRefGoogle Scholar
  28. 28.
    R. Latała, Some estimates of norms of random matrices, Proc. Am. Math. Soc. 133(5), 1273–1282 (2005). MATHCrossRefGoogle Scholar
  29. 29.
    E.H. Lieb, Convex trace functions and the Wigner–Yanase–Dyson conjecture, Adv. Math. 11, 267–288 (1973). MathSciNetMATHCrossRefGoogle Scholar
  30. 30.
    G. Lindblad, Expectations and entropy inequalities for finite quantum systems, Commun. Math. Phys. 39, 111–119 (1974). MathSciNetMATHCrossRefGoogle Scholar
  31. 31.
    F. Lust-Piquard, Inégalités de Khintchine dans C p (1<p<∞), C. R. Math. Acad. Sci. Paris 303(7), 289–292 (1986). MathSciNetMATHGoogle Scholar
  32. 32.
    F. Lust-Piquard, G. Pisier, Noncommutative Khintchine and Paley inequalities, Ark. Mat. 29(2), 241–260 (1991). MathSciNetMATHCrossRefGoogle Scholar
  33. 33.
    M. Ledoux, M. Talagrand, Probability in Banach Spaces: Isoperimetry and Processes (Springer, Berlin, 1991). MATHGoogle Scholar
  34. 34.
    G. Lugosi, Concentration-of-measure inequalities (2009), Available at http://www.econ.upf.edu/~lugosi/anu.pdf.
  35. 35.
    P. Massart, Concentration Inequalities and Model Selection: Ecole d’Eté de Probabilités de Saint-Flour XXXIII—2003. Lecture Notes in Mathematics, vol. 1896 (Springer, Berlin, 2007). MATHGoogle Scholar
  36. 36.
    C. McDiarmid, Concentration, in Probabilistic Methods for Algorithmic Discrete Mathematics. Algorithms and Combinatorics, vol. 16 (Springer, Berlin, 1998), pp. 195–248. Google Scholar
  37. 37.
    R. Motwani, P. Raghavan, Randomized Algorithms (Cambridge Univ. Press, Cambridge, 1995). MATHGoogle Scholar
  38. 38.
    A. Nemirovski, Sums of random symmetric matrices and quadratic optimization under orthogonality constraints, Math. Program., Ser. B 109, 283–317 (2007). MathSciNetMATHCrossRefGoogle Scholar
  39. 39.
    R.I. Oliveira, Concentration of the adjacency matrix and of the Laplacian in random graphs with independent edges (2010), arXiv:0911.0600.
  40. 40.
    R.I. Oliveira, Sums of random Hermitian matrices and an inequality by Rudelson, Electron. Commun. Probab. 15, 203–212 (2010). MathSciNetMATHCrossRefGoogle Scholar
  41. 41.
    B.N. Parlett, The Symmetric Eigenvalue Problem. Classics in Applied Mathematics, vol. 20 (Society for Industrial and Applied Mathematics, Philadelphia, 1987). Google Scholar
  42. 42.
    V.I. Paulsen, Completely Bounded Maps and Operator Algebras. Cambridge Studies in Advanced Mathematics, vol. 78 (Cambridge Univ. Press, Cambridge, 2002). MATHGoogle Scholar
  43. 43.
    D. Petz, A survey of certain trace inequalities, in Functional Analysis and Operator Theory. Banach Center Publications, vol. 30 (Polish Acad. Sci., Warsaw, 1994), pp. 287–298. Google Scholar
  44. 44.
    G. Pisier, Introduction to Operator Spaces (Cambridge Univ. Press, Cambridge, 2003). Google Scholar
  45. 45.
    B. Recht, Simpler approach to matrix completion, J. Mach. Learn. Res. (2009, to appear). Available at http://pages.cs.wisc.edu/brecht/papers/09.Recht.ImprovedMC.pdf.
  46. 46.
    M. Rudelson, Random vectors in the isotropic position, J. Funct. Anal. 164, 60–72 (1999). MathSciNetMATHCrossRefGoogle Scholar
  47. 47.
    M.B. Ruskai, Inequalities for quantum entropy: a review with conditions for equality, J. Math. Phys. 43(9), 4358–4375 (2002). Erratum: J. Math. Phys. 46(1), 0199101 (2005). MathSciNetMATHCrossRefGoogle Scholar
  48. 48.
    M. Rudelson, R. Vershynin, Sampling from large matrices: an approach through geometric functional analysis, J. Assoc. Comput. Mach. 54(4), Article 21 (2007) (electronic) 19 pp. MathSciNetCrossRefGoogle Scholar
  49. 49.
    Y. Seginer, The expected norm of random matrices, Comb. Probab. Comput. 9, 149–166 (2000). MathSciNetMATHCrossRefGoogle Scholar
  50. 50.
    A.M.-C. So, Moment inequalities for sums of random matrices and their applications in optimization, Math. Prog. Ser. A (2009) (electronic). Google Scholar
  51. 51.
    A. Sankar, D.A. Spielman, S.-H. Teng, Smoothed analysis of the condition numbers and growth factors of matrices, SIAM J. Matrix Anal. Appl. 28(2), 446–476 (2006). MathSciNetMATHCrossRefGoogle Scholar
  52. 52.
    N. Tomczak-Jaegermann, The moduli of smoothness and convexity and the Rademacher averages of trace classes S p (1≤p<∞), Stud. Math. 50, 163–182 (1974). MathSciNetMATHGoogle Scholar
  53. 53.
    J.A. Tropp, On the conditioning of random subdictionaries, Appl. Comput. Harmon. Anal. 25, 1–24 (2008). MathSciNetMATHCrossRefGoogle Scholar
  54. 54.
    J.A. Tropp, Improved analysis of the subsampled randomized Hadamard transform. Adv. Adapt. Data Anal. (2010, to appear). Available at arXiv:1011.1595.
  55. 55.
    J.A. Tropp, Freedman’s inequality for matrix martingales, Electron. Commun. Probab. 16, 262–270 (2011). MathSciNetMATHCrossRefGoogle Scholar
  56. 56.
    J.A. Tropp, From the joint convexity of quantum relative entropy to a concavity theorem of Lieb. Proc. Amer. Math. Soc. (2011, to appear). Available at arXiv:1101.1070.
  57. 57.
    J.A. Tropp, User-friendly tail bounds for matrix martingales, ACM Report 2011-01, California Inst. Tech., Pasadena, CA (2011). Google Scholar
  58. 58.
    R. Vershynin, A note on sums of independent random matrices after Ahlswede–Winter (2009), Available at http://www-personal.umich.edu/~romanv/teaching/reading-group/ahlswede-winter.pdf.

Copyright information

© The Author(s) 2011

Authors and Affiliations

  1. 1.Computing & Mathematical Sciences, MC 305-16California Institute of TechnologyPasadenaUSA

Personalised recommendations