Statistics and Computing

, Volume 20, Issue 3, pp 267–282 | Cite as

Optimal linear projections for enhancing desired data statistics

  • Evgenia RubinshteinEmail author
  • Anuj Srivastava


Problems involving high-dimensional data, such as pattern recognition, image analysis, and gene clustering, often require a preliminary step of dimension reduction before or during statistical analysis. If one restricts to a linear technique for dimension reduction, the remaining issue is the choice of the projection. This choice can be dictated by desire to maximize certain statistical criteria, including variance, kurtosis, sparseness, and entropy, of the projected data. Motivations for such criteria comes from past empirical studies of statistics of natural and urban images. We present a geometric framework for finding projections that are optimal for obtaining certain desired statistical properties. Our approach is to define an objective function on spaces of orthogonal linear projections—Stiefel and Grassmann manifolds, and to use gradient techniques to optimize that function. This construction uses the geometries of these manifolds to perform the optimization. Experimental results are presented to demonstrate these ideas for natural and facial images.


Dimension reduction Linear projection Numerical optimization on Grassmann and Stiefel manifolds Stochastic optimization Optimization algorithm 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Bell, A.J., Sejnowski, T.J.: An information maximization approach to blind separation and blind deconvolution. Neural Comput. 7, 1129–1159 (1995) CrossRefGoogle Scholar
  2. Comon, P.: Independent component analysis, a new concept? Signal Process. Special issue on higher-order statistics 36(3), 287–314 (1994) zbMATHCrossRefGoogle Scholar
  3. Cook, D.: Testing predictor contributions in sufficient dimension reduction. Ann. Stat. 32(3), 1062–1092 (2004) zbMATHCrossRefGoogle Scholar
  4. Cook, D., Li, B.: Dimension reduction for conditional mean in regression. Ann. Stat. 30(2), 455–474 (2002) zbMATHCrossRefMathSciNetGoogle Scholar
  5. Donoho, D.L., Flesia, A.G.: Can recent innovations in harmonic analysis “explain” key findings in natural image statistics? Netw. Comput. Neural Syst. 12(3), 371–393 (2001) zbMATHGoogle Scholar
  6. Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20(2), 303–353 (1998) zbMATHCrossRefMathSciNetGoogle Scholar
  7. Field, D.J.: What is the goal of sensory coding? Neural Comput. 6(4), 559–601 (1994) CrossRefMathSciNetGoogle Scholar
  8. Fiori, S.: A minor subspace algorithm based on neural Stiefel dynamics. Int. J. Neural Syst. 19(5), 339–350 (2002a) CrossRefGoogle Scholar
  9. Fiori, S.: A theory for learning based on rigid bodies dynamics. IEEE Trans. Neural Netw. 13(3), 521–531 (2002b) CrossRefGoogle Scholar
  10. Geman, S., Hwang, C.-R.: Diffusions for global optimization. SIAM J. Control Optim. 24(5), 1031–1043 (1987) CrossRefMathSciNetGoogle Scholar
  11. Golub, G.H., Van Loan, C.: Matrix Computations. The John Hopkins University Press, Baltimore (1989) zbMATHGoogle Scholar
  12. Hyvärinen, A.: Fast and robust fixed-point algorithm for independent component analysis. IEEE Trans. Neural Netw. 10(3), 626–634 (1999) CrossRefGoogle Scholar
  13. Hyvärinen, A., Karhunen, J., Oja, E.: Independent Component Analysis. Wiley, New York (2001) CrossRefGoogle Scholar
  14. Johnson, R.A., Wichern, D.W.: Applied Multivariate Statistical Analysis. Prentice Hall, New York (2001) Google Scholar
  15. Liu, X., Srivastava, A., Gallivan, K.A.: Optimal linear representations of images for object recognition. In: Proceedings of 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 229–234 (2003) Google Scholar
  16. Mallat, S.G.: Theory for multiresolution signal decomposition: The wavelet representation. IEEE Trans. Pattern Anal. Mach. Intell. 11(7), 674–693 (1989) zbMATHCrossRefGoogle Scholar
  17. Olshausen, B.A., Field, D.J.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381, 607–609 (1996a) CrossRefGoogle Scholar
  18. Olshausen, B.A., Field, D.J.: Natural image statistics and efficient coding. Netw. Comput. Neural Syst. 7, 333–339 (1996b) CrossRefGoogle Scholar
  19. Robert, C.P., Casella, G.: Monte Carlo Statistical Methods. Springer Texts in Statistics. Springer, New York (1999) zbMATHGoogle Scholar
  20. Srivastava, A.: A Bayesian approach to geometric subspace estimation. IEEE Trans. Signal Process. 48(5), 1390–1400 (2000) CrossRefMathSciNetGoogle Scholar
  21. Srivastava, A., Lee, A.B., Simoncelli, E.P., Zhu, S.-C.: On advances in statistical modeling of natural images. J. Math. Imaging Vis. 18, 17–33 (2003) zbMATHCrossRefMathSciNetGoogle Scholar
  22. Srivastava, A., Liu, X.: Tools for application-driven linear dimension reduction. J. Neurocomput. 67, 136–160 (2005) CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  1. 1.Vladivostok State University of Economics and ServiceVladivostokRussia
  2. 2.Department of StatisticsFlorida State UniversityTallahasseeUSA

Personalised recommendations