Overview and Recent Advances in Partial Least Squares

  • Roman Rosipal
  • Nicole Krämer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3940)


Partial Least Squares (PLS) is a wide class of methods for modeling relations between sets of observed variables by means of latent variables. It comprises of regression and classification tasks as well as dimension reduction techniques and modeling tools. The underlying assumption of all PLS methods is that the observed data is generated by a system or process which is driven by a small number of latent (not directly observed or measured) variables. Projections of the observed data to its latent structure by means of PLS was developed by Herman Wold and coworkers [48,49,52].


Mean Square Error Partial Little Square Partial Little Square Regression Canonical Correlation Analysis Principal Component Regression 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Almøy, T.: A simulation study on comparison of prediction models when only a few components are relevant. Computational Statistics and Data Analysis 21, 87–107 (1996)CrossRefzbMATHGoogle Scholar
  2. 2.
    Aronszajn, N.: Theory of reproducing kernels. Transactions of the American Mathematical Society 68, 337–404 (1950)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Baffi, G., Martin, E.B., Morris, A.J.: Non-linear projection to latent structures revisited (the neural network PLS algorithm). Computers Chemical Engineering 23, 1293–1307 (1999)CrossRefGoogle Scholar
  4. 4.
    Barker, M., Rayens, W.S.: Partial least squares for discrimination. Journal of Chemometrics 17, 166–173 (2003)CrossRefGoogle Scholar
  5. 5.
    Bartlett, M.S.: Further aspects of the theory of multiple regression. In: Proceedings of the Cambridge Philosophical Society, vol. 34, pp. 33–40 (1938)Google Scholar
  6. 6.
    De Bie, T., Cristianini, N., Rosipal, R.: Eigenproblems in Pattern Recognition. In: Bayro-Corrochano, E. (ed.) Handbook of Geometric Computing: Applications in Pattern Recognition, Computer Vision, Neuralcomputing, and Robotics, pp. 129–170. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  7. 7.
    Butler, N.A., Denham, M.C.: The peculiar shrinkage properties of partial least squares regression. Journal of the Royal Statistical Society: B 62, 585–593 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    de Jong, S.: SIMPLS: an alternative approach to partial least squares regression. Chemometrics and Intelligent Laboratory Systems 18, 251–263 (1993)CrossRefGoogle Scholar
  9. 9.
    de Jong, S., Wise, B.M., Ricker, N.L.: Canonical partial least squares and continuum power regression. Journal of Chemometrics 15, 85–100 (2001)CrossRefGoogle Scholar
  10. 10.
    Frank, I.E.: A nonlinear PLS model. Chemolab 8, 109–119 (1990)Google Scholar
  11. 11.
    Frank, I.E., Friedman, J.H.: A Statistical View of Some Chemometrics Regression Tools. Technometrics 35, 109–147 (1993)CrossRefzbMATHGoogle Scholar
  12. 12.
    Goutis, C.: Partial least squares yields shrinkage estimators. The Annals of Statistics 24, 816–824 (1996)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, Heidelberg (2001)CrossRefzbMATHGoogle Scholar
  14. 14.
    Helland, I.S.: Some theoretical aspects of partial least squares regression. Chemometrics and Intelligent Laboratory Systems 58, 97–107 (1999)CrossRefGoogle Scholar
  15. 15.
    Hestenes, M., Stiefel, E.: Methods for conjugate gradients for solving linear systems. Journal of Research of the National Bureau of Standards 49, 409–436 (1952)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Hoerl, A.E., Kennard, R.W.: Ridge regression: bias estimation for nonorthogonal problems. Technometrics 12, 55–67 (1970)CrossRefzbMATHGoogle Scholar
  17. 17.
    Höskuldsson, A.: PLS Regression Methods. Journal of Chemometrics 2, 211–228 (1988)CrossRefGoogle Scholar
  18. 18.
    Hulland, J.S.: Use of partial least squares (PLS) in strategic management research: A review of four recent studies. Strategic Management Journal 20, 195–204 (1999)CrossRefGoogle Scholar
  19. 19.
    Lanczos, C.: An iteration method for the solution of the eigenvalue problem of linear differential and integral operators. Journal of Research of the National Bureau of Standards 45, 225–280 (1950)MathSciNetCrossRefGoogle Scholar
  20. 20.
    Lewi, P.J.: Pattern recognition, reflection from a chemometric point of view. Chemometrics and Intelligent Laboratory Systems 28, 23–33 (1995)CrossRefGoogle Scholar
  21. 21.
    Lingjærde, O.C., Christophersen, N.: Shrinkage Structure of Partial Least Squares. Scandinavian Journal of Statistics 27, 459–473 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Lobaugh, N.J., West, R., McIntosh, A.R.: Spatiotemporal analysis of experimental differences in event-related potential data with partial least squares. Psychophysiology 38, 517–530 (2001)CrossRefGoogle Scholar
  23. 23.
    Manne, R.: Analysis of Two Partial-Least-Squares Algorithms for Multivariate Calibration. Chemometrics and Intelligent Laboratory Systems 2, 187–197 (1987)CrossRefGoogle Scholar
  24. 24.
    Mardia, K.V., Kent, J.T., Bibby, J.M.: Multivariate Analysis. Academic Press, London (1997)zbMATHGoogle Scholar
  25. 25.
    Martens, M., Martens, H.: Partial Least Squares Regression. In: Piggott, J.R. (ed.) Statistical Procedures in Food Research, pp. 293–359. Elsevier Applied Science, London (1986)Google Scholar
  26. 26.
    Massy, W.F.: Principal components regression in exploratory statistical research. Journal of the American Statistical Association 60, 234–256 (1965)CrossRefGoogle Scholar
  27. 27.
    Momma, M.: Efficient Computations via Scalable Sparse Kernel Partial Least Squares and Boosted Latent Features. In: Proceedings of SIGKDD International Conference on Knowledge and Data Mining, Chicago, IL, pp. 654–659 (2005)Google Scholar
  28. 28.
    Nguyen, D.V., Rocke, D.M.: Tumor classification by partial least squares using microarray gene expression data. Bioinformatics 18, 39–50 (2002)CrossRefGoogle Scholar
  29. 29.
    Nilsson, J., de Jong, S., Smilde, A.K.: Multiway Calibration in 3D QSAR. Journal of Chemometrics 11, 511–524 (1997)CrossRefGoogle Scholar
  30. 30.
    Parlett, B.: The symmetric eigenvalue problem. SIAM, Philadelphia (1998)CrossRefzbMATHGoogle Scholar
  31. 31.
    Phatak, A., de Hoog, F.: Exploiting the connection between PLS, Lanczos, and conjugate gradients: Alternative proofs of some properties of PLS. Journal of Chemometrics 16, 361–367 (2003)CrossRefGoogle Scholar
  32. 32.
    Phatak, A., Rilley, P.M., Penlidis, A.: The asymptotic variance of the univariate PLS estimator. Linear Algebra and its Applications 354, 245–253 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  33. 33.
    Rännar, S., Lindgren, F., Geladi, P., Wold, S.: A PLS kernel algorithm for data sets with many variables and fewer objects. Part 1: Theory and algorithm. Chemometrics and Intelligent Laboratory Systems 8, 111–125 (1994)Google Scholar
  34. 34.
    Rosipal, R., Trejo, L.J.: Kernel Partial Least Squares Regression in Reproducing Kernel Hilbert Space. Journal of Machine Learning Research 2, 97–123 (2001)zbMATHGoogle Scholar
  35. 35.
    Rosipal, R., Trejo, L.J., Matthews, B.: Kernel PLS-SVC for Linear and Nonlinear Classification. In: Proceedings of the Twentieth International Conference on Machine Learning, Washington, DC, pp. 640–647 (2003)Google Scholar
  36. 36.
    Sampson, P.D., Streissguth, A.P., Barr, H.M., Bookstein, F.L.: Neurobehavioral effects of prenatal alcohol: Part II. Partial Least Squares analysis. Neurotoxicology and tetralogy 11, 477–491 (1989)CrossRefGoogle Scholar
  37. 37.
    Saunders, C., Gammerman, A., Vovk, V.: Ridge Regression Learning Algorithm in Dual Variables. In: Proceedings of the 15th International Conference on Machine Learning, Madison, WI, pp. 515–521 (1998)Google Scholar
  38. 38.
    Schölkopf, B., Smola, A.J.: Learning with Kernels – Support Vector Machines, Regularization, Optimization and Beyond. The MIT Press, Cambridge (2002)Google Scholar
  39. 39.
    Smilde, A., Bro, R., Geladi, P.: Multi-way Analysis: Applications in the Chemical Sciences. Wiley, Chichester (2004)CrossRefGoogle Scholar
  40. 40.
    Stone, M., Brooks, R.J.: Continuum Regression: Cross-validated Sequentially Constructed Prediction Embracing Ordinary Least Squares, Partial Least Squares and Principal Components Regression. Journal of the Royal Statistical Society: B 52, 237–269 (1990)MathSciNetzbMATHGoogle Scholar
  41. 41.
    Vinod, H.D.: Canonical ridge and econometrics of joint production. Journal of Econometrics 4, 147–166 (1976)CrossRefzbMATHGoogle Scholar
  42. 42.
    Wahba, G.: Splines Models of Observational Data. Series in Applied Mathematics, vol. 59. SIAM, Philadelphia (1990)CrossRefzbMATHGoogle Scholar
  43. 43.
    Wallerius, J., Trejo, L.J., Matthew, R., Rosipal, R., Caldwell, J.A.: Robust feature extraction and classification of EEG spectra for real-time classification of cognitive state. In: Proceedings of 11th International Conference on Human Computer Interaction, Las Vegas, NV (2005)Google Scholar
  44. 44.
    Wangen, L.E., Kowalsky, B.R.: A multiblock partial least squares algorithm for investigating complex chemical systems. Journal of Chemometrics 3, 3–20 (1989)CrossRefGoogle Scholar
  45. 45.
    Wegelin, J.A.: A survey of Partial Least Squares (PLS) methods, with emphasis on the two-block case. Technical report, Department of Statistics, University of Washington, Seattle (2000)Google Scholar
  46. 46.
    Westerhuis, J., Kourti, T., MacGregor, J.: Analysis of multiblock and hierarchical PCA and PLS models. Journal of Chemometrics 12, 301–321 (1998)CrossRefGoogle Scholar
  47. 47.
    Wold, H.: Path models with latent variables: The NIPALS approach. In: Blalock, H.M., et al. (eds.) Quantitative Sociology: International perspectives on mathematical and statistical model building, pp. 307–357. Academic Press, London (1975)CrossRefGoogle Scholar
  48. 48.
    Wold, H.: Soft modeling: the basic design and some extensions. In: Jöreskog, J.-K., Wold, H. (eds.) Systems Under Indirect Observation, vol. 2, pp. 1–53. North Holland, Amsterdam (1982)Google Scholar
  49. 49.
    Wold, H.: Partial least squares. In: Kotz, S., Johnson, N.L. (eds.) Encyclopedia of the Statistical Sciences, vol. 6, pp. 581–591. John Wiley & Sons, Chichester (1985)Google Scholar
  50. 50.
    Wold, S.: Nonlinear partial least squares modeling II, Spline inner relation. Chemolab 14, 71–84 (1992)Google Scholar
  51. 51.
    Wold, S., Kettaneh-Wold, N., Skagerberg, B.: Nonlinear PLS modelling. Chemometrics and Intelligent Laboratory Systems 7, 53–65 (1989)CrossRefGoogle Scholar
  52. 52.
    Wold, S., Ruhe, H., Wold, H., Dunn III., W.J.: The collinearity problem in linear regression. The partial least squares (PLS) approach to generalized inverse. SIAM Journal of Scientific and Statistical Computations 5, 735–743 (1984)CrossRefzbMATHGoogle Scholar
  53. 53.
    Worsley, K.J.: An overview and some new developments in the statistical analysis of PET and fMRI data. Human Brain Mapping 5, 254–258 (1997)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Roman Rosipal
    • 1
  • Nicole Krämer
    • 2
  1. 1.Austrian Research Institute for Artificial IntelligenceViennaAustria
  2. 2.Department of Computer Science and Electrical EngineeringTU BerlinBerlinGermany

Personalised recommendations