The Peaking Phenomenon in Semi-supervised Learning

  • Jesse H. KrijtheEmail author
  • Marco Loog
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10029)


For the supervised least squares classifier, when the number of training objects is smaller than the dimensionality of the data, adding more data to the training set may first increase the error rate before decreasing it. This, possibly counterintuitive, phenomenon is known as peaking. In this work, we observe that a similar but more pronounced version of this phenomenon also occurs in the semi-supervised setting, where instead of labeled objects, unlabeled objects are added to the training set. We explain why the learning curve has a more steep incline and a more gradual decline in this setting through simulation studies and by applying an approximation of the learning curve based on the work by Raudys and Duin.


Semi-supervised learning Peaking Least squares classifier Pseudo-inverse 



This work was funded by project P23 of the Dutch public/private research network COMMIT.


  1. 1.
    Chapelle, O., Schölkopf, B., Zien, A.: Semi-Supervised Learning. MIT press, Cambridge (2006)CrossRefGoogle Scholar
  2. 2.
    Duin, R.P.W.: Small sample size generalization. In: Proceedings of the Scandinavian Conference on Image Analysis, pp. 957–964 (1995)Google Scholar
  3. 3.
    Duin, R.P.: Classifiers in almost empty spaces. In: Proceedings of the 15th International Conference on Pattern Recognition, pp. 1–7 (2000)Google Scholar
  4. 4.
    Fan, B., Lei, Z., Li, S.Z.: Normalized LDA for semi-supervised learning. In: International Conference on Automatic Face & Gesture Recognition, pp. 1–6 (2008)Google Scholar
  5. 5.
    Hughes, G.F.: On the mean accuracy of statistical pattern recognizers. IEEE Trans. Inf. Theor. 14(1), 55–63 (1968)CrossRefGoogle Scholar
  6. 6.
    Jain, A.K., Chandrasekaran, B.: Dimensionality and sample size considerations in pattern recognition practice. In: Krishnaiah, P.R., Kanal, L. (eds.) Handbook of Statistics, vol. 2, pp. 835–855. North-Holland Publishing Company, Amsterdam (1982)Google Scholar
  7. 7.
    Krijthe, J.H., Loog, M.: Implicitly constrained semi-supervised least squares classification. In: Fromont, E., Bie, T., Leeuwen, M. (eds.) IDA 2015. LNCS, vol. 9385, pp. 158–169. Springer, Heidelberg (2015). doi: 10.1007/978-3-319-24465-5_14 CrossRefGoogle Scholar
  8. 8.
    Lichman, M.: UCI Machine Learning Repository (2013).
  9. 9.
    Loog, M., Duin, R.P.W.: The dipping phenomenon. In: Gimefarb, G., Hancock, E., Imiya, A., Kuijper, A., Kudo, M., Omachi, S., Windeatt, T., Windeatt, T., Yamada, K. (eds.) SSPR & SPR 2012. LNCS, vol. 7626, pp. 310–317. Springer, Heidelberg (2012)Google Scholar
  10. 10.
    Opper, M.: Learning to generalize. In: Baltimore, D. (ed.) Frontiers of Life: Intelligent Systems, pp. 763–775. Academic Press, Cambridge (2001)Google Scholar
  11. 11.
    Opper, M., Kinzel, W.: Statistical mechanics of generalization. In: Domany, E., Hemmen, J.L., Schulten, K. (eds.) Physics of Neural Networks III, pp. 151–209. Springer, New York (1995)Google Scholar
  12. 12.
    Raudys, S., Duin, R.P.W.: Expected classification error of the Fisher linear classifier with pseudo-inverse covariance matrix. Pattern Recogn. Lett. 19(5–6), 385–392 (1998)CrossRefzbMATHGoogle Scholar
  13. 13.
    Shaffer, J.P.: The Gauss-Markov theorem and random regressors. Am. Stat. 45(4), 269–273 (1991)MathSciNetGoogle Scholar
  14. 14.
    Skurichina, M., Duin, R.P.W.: Stabilizing classifiers for very small samplesizes. In: Proceedings of the 13th International Conference on Pattern Recognition, pp. 891–896 (1996)Google Scholar
  15. 15.
    Skurichina, M., Duin, R.P.W.: Regularisation of linear classifiers by adding redundant features. Pattern Anal. Appl. 2(1), 44–52 (1999)CrossRefzbMATHGoogle Scholar
  16. 16.
    Wyman, F.J., Young, D.M., Turner, D.W.: A comparison of asymptotic error rate expansions for the sample linear discriminant function. Pattern Recogn. 23(7), 775–783 (1990)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.Pattern Recognition LaboratoryDelft University of TechnologyDelftNetherlands
  2. 2.Department of Molecular EpidemiologyLeiden University Medical CenterLeidenNetherlands
  3. 3.The Image SectionUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations