On Evolutionary Approaches to Unsupervised Nearest Neighbor Regression

  • Oliver Kramer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7248)


The detection of structures in high-dimensional data has an important part to play in machine learning. Recently, we proposed a fast iterative strategy for non-linear dimensionality reduction based on the unsupervised formulation of K-nearest neighbor regression. As the unsupervised nearest neighbor (UNN) optimization problem does not allow the computation of derivatives, the employment of direct search methods is reasonable. In this paper we introduce evolutionary optimization approaches for learning UNN embeddings. Two continuous variants are based on the CMA-ES employing regularization with domain restriction, and penalizing extension in latent space. A combinatorial variant is based on embedding the latent variables on a grid, and performing stochastic swaps. We compare the results on artificial dimensionality reduction problems.


Dimensionality Reduction Latent Space Locally Linear Embedding Dimensionality Reduction Method Evolution Strategy 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beyer, H.G., Schwefel, H.P.: Evolution strategies - A comprehensive introduction. Natural Computing 1, 3–52 (2002)MathSciNetzbMATHCrossRefGoogle Scholar
  2. 2.
    Carreira-Perpiñán, M.Á., Lu, Z.: Parametric dimensionality reduction by unsupervised regression. In: Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1895–1902 (2010)Google Scholar
  3. 3.
    Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, Berlin (2009)zbMATHCrossRefGoogle Scholar
  4. 4.
    Jolliffe, I.: Principal component analysis. Springer Series in Statistics. Springer, New York (1986)Google Scholar
  5. 5.
    Klanke, S., Ritter, H.: Variants of unsupervised kernel regression: General cost functions. Neurocomputing 70(7-9), 1289–1303 (2007)CrossRefGoogle Scholar
  6. 6.
    Kramer, O.: Dimensionality reduction by unsupervised k-nearest neighbor regression. In: Proceedings of the International Conference on Machine Learning and Applications (ICMLA), pp. 275–278. IEEE Computer Society Press (2011)Google Scholar
  7. 7.
    Kramer, O., Gieseke, F.: A stochastic optimization approach for unsupervised kernel regression. In: Genetic and Evolutionary Methods (GEM), pp. 156–161 (2011)Google Scholar
  8. 8.
    Lawrence, N.D.: Probabilistic non-linear principal component analysis with gaussian process latent variable models. Journal of Machine Learning Research 6, 1783–1816 (2005)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Meinicke, P.: Unsupervised Learning in a Generalized Regression Framework. Ph.D. thesis, University of Bielefeld (2000)Google Scholar
  10. 10.
    Meinicke, P., Klanke, S., Memisevic, R., Ritter, H.: Principal surfaces from unsupervised kernel regression. IEEE Trans. Pattern Anal. Mach. Intell. 27(9), 1379–1391 (2005)CrossRefGoogle Scholar
  11. 11.
    Ostermeier, A., Gawelczyk, A., Hansen, N.: A derandomized approach to self adaptation of evolution strategies. Evolutionary Computation 2(4), 369–380 (1994)CrossRefGoogle Scholar
  12. 12.
    Pearson, K.: On lines and planes of closest fit to systems of points in space. Philosophical Magazine 2(6), 559–572 (1901)Google Scholar
  13. 13.
    Roweis, S.T., Saul, L.K.: Nonlinear dimensionality reduction by locally linear embedding. Science 290, 2323–2326 (2000)CrossRefGoogle Scholar
  14. 14.
    Smola, A.J., Mika, S., Schölkopf, B., Williamson, R.C.: Regularized principal manifolds. J. Mach. Learn. Res. 1, 179–209 (2001)MathSciNetzbMATHGoogle Scholar
  15. 15.
    Tenenbaum, J.B., Silva, V.D., Langford, J.C.: A global geometric framework for nonlinear dimensionality reduction. Science 290, 2319–2323 (2000)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Oliver Kramer
    • 1
  1. 1.Fakultät II, Department for Computer ScienceCarl von Ossietzky Universität OldenburgOldenburgGermany

Personalised recommendations