Online Manifold Regularization: A New Learning Setting and Empirical Study

  • Andrew B. Goldberg
  • Ming Li
  • Xiaojin Zhu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5211)


We consider a novel “online semi-supervised learning” setting where (mostly unlabeled) data arrives sequentially in large volume, and it is impractical to store it all before learning. We propose an online manifold regularization algorithm. It differs from standard online learning in that it learns even when the input point is unlabeled. Our algorithm is based on convex programming in kernel space with stochastic gradient descent, and inherits the theoretical guarantees of standard online algorithms. However, naïve implementation of our algorithm does not scale well. This paper focuses on efficient, practical approximations; we discuss two sparse approximations using buffering and online random projection trees. Experiments show our algorithm achieves risk and generalization accuracy comparable to standard batch manifold regularization, while each step runs quickly. Our online semi-supervised learning setting is an interesting direction for further theoretical development, paving the way for semi-supervised learning to work on real-world life-long learning tasks.


Online Learning Online Algorithm Reproduce Kernel Hilbert Space Generalization Error Concept Drift 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Brefeld, U., Büscher, C., Scheffer, T.: Multiview discriminative sequential learning. In: Gama, J., Camacho, R., Brazdil, P.B., Jorge, A.M., Torgo, L. (eds.) ECML 2005. LNCS (LNAI), vol. 3720, pp. 60–71. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  2. 2.
    Zinkevich, M.: Online convex programming and generalized infinitesimal gradient ascent. In: ICML 2003 (2003)Google Scholar
  3. 3.
    Chapelle, O., Zien, A., Schölkopf, B. (eds.): Semi-supervised learning. MIT Press, Cambridge (2006)Google Scholar
  4. 4.
    Zhu, X.: Semi-supervised learning literature survey. Technical Report 1530, Department of Computer Sciences, University of Wisconsin, Madison (2005)Google Scholar
  5. 5.
    Kivinen, J., Smola, A.J., Williamson, R.C.: Online learning with kernels. IEEE Transactions on Signal Processing 52(8), 2165–2176 (2004)CrossRefMathSciNetGoogle Scholar
  6. 6.
    Schölkopf, B., Smola, A.J.: Learning with Kernels. MIT Press, Cambridge (2002)Google Scholar
  7. 7.
    Belkin, M., Niyogi, P., Sindhwani, V.: Manifold regularization: A geometric framework for learning from labeled and unlabeled examples. Journal of Machine Learning Research 7, 2399–2434 (2006)MathSciNetGoogle Scholar
  8. 8.
    Sindhwani, V., Niyogi, P., Belkin, M.: Beyond the point cloud: from transductive to semi-supervised learning. In: ICML 2005 (2005)Google Scholar
  9. 9.
    Zhu, X., Ghahramani, Z., Lafferty, J.: Semi-supervised learning using Gaussian fields and harmonic functions. In: ICML 2003 (2003)Google Scholar
  10. 10.
    Blum, A., Mitchell, T.: Combining labeled and unlabeled data with co-training. In: COLT 1998 (1998)Google Scholar
  11. 11.
    Sindhwani, V., Niyogi, P., Belkin, M.: A co-regularized approach to semi-supervised learning with multiple views. In: ICML 2005 (2005)Google Scholar
  12. 12.
    Brefeld, U., Gaertner, T., Scheffer, T., Wrobel, S.: Efficient co-regularized least squares regression. In: ICML 2006 (2006)Google Scholar
  13. 13.
    Joachims, T.: Transductive inference for text classification using support vector machines. In: ICML 1999 (1999)Google Scholar
  14. 14.
    Chapelle, O., Sindhwani, V., Keerthi, S.S.: Branch and bound for semi-supervised support vector machines. In: NIPS 2006 (2006)Google Scholar
  15. 15.
    Collobert, R., Sinz, F., Weston, J., Bottou, L.: Large scale transductive SVMs. The Journal of Machine Learning Research 7, 1687–1712 (2006)MathSciNetGoogle Scholar
  16. 16.
    Kimeldorf, G., Wahba, G.: Some results on Tchebychean spline functions. Journal of Mathematics Analysis and Applications 33, 82–95 (1971)zbMATHCrossRefMathSciNetGoogle Scholar
  17. 17.
    Cesa-Bianchi, N., Conconi, A., Gentile, C.: On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory 50(9), 2050–2057 (2004)CrossRefMathSciNetGoogle Scholar
  18. 18.
    Vincent, P., Bengio, Y.: Kernel matching pursuit. Machine Learning 48(1-3), 165–187 (2002)zbMATHCrossRefGoogle Scholar
  19. 19.
    Dekel, O., Shalev-Shwartz, S., Singer, Y.: The forgetron: A kernel-based perceptron on a fixed budget. In: NIPS 2005 (2005)Google Scholar
  20. 20.
    Hegde, C., Wakin, M., Baraniuk, R.: Random projections for manifold learning. In: NIPS 2007 (2007)Google Scholar
  21. 21.
    Freund, Y., Dasgupta, S., Kabra, M., Verma, N.: Learning the structure of manifolds using random projections. In: NIPS 2007 (2007)Google Scholar
  22. 22.
    Dasgupta, S., Freund, Y.: Random projection trees and low dimensional manifolds. Technical Report CS2007-0890, University of California, San Diego (2007)Google Scholar
  23. 23.
    Jebara, T., Kondor, R., Howard, A.: Probability product kernels. Journal of Machine Learning Research, Special Topic on Learning Theory 5, 819–844 (2004)MathSciNetGoogle Scholar
  24. 24.
    Tsang, I., Kwok, J.: Large-scale sparsified manifold regularization. In: NIPS 2006 (2006)Google Scholar
  25. 25.
    LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. In: Proceedings of the IEEE, vol. 86(11), pp. 2278–2324 (1998)Google Scholar
  26. 26.
    Tsang, I.W., Kwok, J.T., Cheung, P.M.: Core vector machines: Fast svm training on very large data sets. Journal of Machine Learning Research 6, 363–392 (2005)MathSciNetGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Andrew B. Goldberg
    • 1
  • Ming Li
    • 2
  • Xiaojin Zhu
    • 1
  1. 1.Department of Computer SciencesUniversity of Wisconsin-MadisonMadisonUSA
  2. 2.National Key Laboratory for Novel Software TechnologyNanjing UniversityNanjingChina

Personalised recommendations