Abstract
Covariance matrix estimation is central to many applications in statistics and allied fields. A useful estimator in this context was proposed by Stein which regularizes the sample covariance matrix by shrinking its eigenvalues together. This estimator can sometimes yield estimates of the eigenvalues that are negative or differ in order from the observed eigenvalues. In order to rectify this problem, Stein also proposed an ad hoc “isotonizing” procedure which pools together eigenvalue estimates in such a way that the original ordering and positivity of the estimates are enforced. From numerical studies, Stein’s “isotonized” estimator is known to have good risk properties in comparison with the maximum likelihood estimator. However, it remains unclear what role is played by the isotonizing procedure in the remarkable risk reductions achieved by Stein’s estimator. Through two distinct lines of investigations, it is established that Stein’s estimator without the isotonizing algorithm gives only modest risk reductions. In cases where the isotonizing algorithm is frequently used, however, Stein’s estimator can lead to significant risk reductions for certain domains of the parameter. In other cases, Stein’s estimator can even yield risk increases, such as when (1) the theoretical eigenvalues are well separated, and/or (2) when the sample size is moderate to large, leading to over-shrinkage.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Our R package that implements Stein’s isotonized estimator is available on CRAN at https://cran.r-project.org/web/packages/stcov/.
References
Daniels M, Kass R (2001) Shrinkage estimators for covariance matrices. Biometrics 57(4):1173–1184
El Karoui N (2008) Spectrum estimation for large dimensional covariance matrices using random matrix theory. Ann Stat 36(6):2757–2790
Fisher TJ, Sun XQ (2011) Improved stein-type shrinkage estimators for the high-dimensional multivariate normal covariance matrix. Comp Stat Data Anal 55(5):1909–1918
Hamimeche S, Lewis A (2009) Properties and use of CMB power spectrum likelihoods. Phys Rev D. 79(8):083012
Johnstone IM (2001) On the distribution of the largest eigenvalue in principal component analysis. Ann Stat 29(2):295–327
Khare K, Rajaratnam B (2011) Wishart distributions for decomposable covariance graph models. Ann Stat 39(1):514–555
Ledoit O, Wolf M (2004) A well-conditioned estimator for large-dimensional covariance matrices. J Multivariate Anal 88:365–411
Ledoit O, Wolf M (2014) Optimal estimation of a large-dimensional covariance matrix under Stein’s loss. Working Paper N. 122, Department of Economics, University of Zurich
Li W, Chen J, Qin Y, Bai Z, Yao J (2013) Estimation of the population spectral distribution from a large dimensional sample covariance matrix. J Stat Plan Inference 143(11):1887–1897
Lin S, Perlman M (1985) A Monte Carlo comparison of four estimators of a covariance matrix. In: Krishnaiah P (ed) Multivariate Analysis, vol 6. North Holland, Amsterdam, pp 411–429
Loh WL (1991) Estimating covariance matrices. Ann Stat 19(1):283–296
Pope A, Szapudi I (2008) Shrinkage estimation of the power spectrum covariance matrix. Mon Not R Astron Soc 389(2):766–774
Pourahmadi M (2011) Covariance estimation: The GLM and regularization perspectives. Stat Sci 26(3):369–387
Rajaratnam B, Vincenzi D (2015) A theoretical study of Stein’s covariance estimator. Biometrika (in press) https://hal.archives-ouvertes.fr/hal-01243388
Rajaratnam B, Massam H, Carvalho C (2008) Flexible covariance estimation in graphical Gaussian models. Ann Stat 36(6):2818–2849
Schäfer J, Strimmer K (2005) A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Stat Appl Genet Mol Biol 4(1):32
Stein C (1975) Estimation of a covariance matrix. Rietz Lecture
Stein C (1977) Lectures on the theory of estimation of many parameters (in Russian). In: Ibragimov IA, Nikulin MS (eds) Studies in the Statistical Theory of Estimation, Part I, Proceedings of Scientific Seminars of the Steklov Institute, Leningrad Division, vol 74, pp 4–65
Stein C (1986) Lectures on the theory of estimation of many parameters. J Math Sci 34(1):1373–1403
Touloumis A (2015) Nonparametric Stein-type shrinkage covariance matrix estimators in high-dimensional settings. Comput Stat Data Anal 83:251–261
Wang C, Pan G, Tong T, Zhu L (2015) Shrinkage estimation of large dimensional precision matrix using random matrix theory. Stat Sin 25:993–1008
Won J, Lim J, Kim S, Rajaratnam B (2013) Condition-number-regularized covariance estimation. J R Stat Soc Ser B Stat Methodol 75(3):427–450
Yang R, Berger JO (1994) Estimation of a covariance matrix using the reference prior. Ann Stat 22(3):1195–1211
Acknowledgments
The authors would like to gratefully acknowledge funding from the France–Stanford Center for Interdisciplinary Studies which facilitated their collaborations. We thank Prof. Charles Stein for his encouragement and enthusiasm when this work was initiated. BR was supported in part by the National Science Foundation under Grant Nos. DMS 0906392, DMS-CMG 1025465, AGS-1003823, DMS-1106642 and Grants NSA H98230-11-1-0194, DARPA-YFA N66001-11-1-4131, and SUWIEVP10-SUFSC10-SMSCVISG0906. BN was supported in part by grant DARPA-YFA N66001-11-1-4131.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Naul, B., Rajaratnam, B. & Vincenzi, D. The role of the isotonizing algorithm in Stein’s covariance matrix estimator. Comput Stat 31, 1453–1476 (2016). https://doi.org/10.1007/s00180-016-0672-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-016-0672-4