Neural Processing Letters

, 29:109

Local Dimensionality Reduction for Non-Parametric Regression


DOI: 10.1007/s11063-009-9098-0

Cite this article as:
Hoffmann, H., Schaal, S. & Vijayakumar, S. Neural Process Lett (2009) 29: 109. doi:10.1007/s11063-009-9098-0


Locally-weighted regression is a computationally-efficient technique for non-linear regression. However, for high-dimensional data, this technique becomes numerically brittle and computationally too expensive if many local models need to be maintained simultaneously. Thus, local linear dimensionality reduction combined with locally-weighted regression seems to be a promising solution. In this context, we review linear dimensionality-reduction methods, compare their performance on non-parametric locally-linear regression, and discuss their ability to extend to incremental learning. The considered methods belong to the following three groups: (1) reducing dimensionality only on the input data, (2) modeling the joint input-output data distribution, and (3) optimizing the correlation between projection directions and output data. Group 1 contains principal component regression (PCR); group 2 contains principal component analysis (PCA) in joint input and output space, factor analysis, and probabilistic PCA; and group 3 contains reduced rank regression (RRR) and partial least squares (PLS) regression. Among the tested methods, only group 3 managed to achieve robust performance even for a non-optimal number of components (factors or projection directions). In contrast, group 1 and 2 failed for fewer components since these methods rely on the correct estimate of the true intrinsic dimensionality. In group 3, PLS is the only method for which a computationally-efficient incremental implementation exists. Thus, PLS appears to be ideally suited as a building block for a locally-weighted regressor in which projection directions are incrementally added on the fly.


Correlation Dimensionality reduction Factor analysis Incremental learning Kernel function Locally-weighted regression Partial least squares Principal component analysis Principal component regression Reduced-rank regression 

Copyright information

© Springer Science+Business Media, LLC. 2009

Authors and Affiliations

  • Heiko Hoffmann
    • 1
    • 2
  • Stefan Schaal
    • 3
  • Sethu Vijayakumar
    • 1
  1. 1.IPAB, School of InformaticsUniversity of EdinburghEdinburghUK
  2. 2.Biomedical EngineeringUniversity of Southern CaliforniaLos AngelesUSA
  3. 3.Computer Science and NeuroscienceUniversity of Southern CaliforniaLos AngelesUSA

Personalised recommendations