Neural Processing Letters

, Volume 42, Issue 1, pp 187–197 | Cite as

Scalable Semi-Supervised Classification via Neumann Series

  • Chen Gong
  • Keren Fu
  • Lei Zhou
  • Jie Yang
  • Xiangjian He
Article

Abstract

Traditional graph-based semi-supervised learning (GBSSL) algorithms usually scale badly due to the expensive computational burden. The main bottleneck is that they need to compute the inversion of a huge matrix. In order to alleviate this problem, this paper proposes Neumann series approximation (NSA) to explicitly approximate the inversion process required by conventional GBSSL methodologies, which makes them computationally tractable for relatively large datasets. It is proved that the deviation between the approximation and direct inversion is bounded. Using real-world datasets related to handwritten digit recognition, speech recognition and text classification, the experimental results reveal that NSA accelerates the speed significantly without decreasing too much precision. We also empirically show that NSA outperforms other scalable approaches such as Nyström method, Takahashi equation, Lanczos process based SVD and AnchorGraph regularization, in terms of both efficiency and accuracy.

Keywords

Semi-supervised learning Scalability Neumann series Error bound 

Notes

Acknowledgments

This research is supported by NSFC, China (No: 6127325861375048), and Ph.D. Programs Foundation of Ministry of Education of China (No.20120073110018).

References

  1. 1.
    Belkin M et al (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7:2399–2434MathSciNetMATHGoogle Scholar
  2. 2.
    Campbell Y, Davis T (1995) Computing the sparse inverse subset: an inverse multifrontal approach. Technical report TR-95-021, University of Florida, Gainesville, FLGoogle Scholar
  3. 3.
    Delalleau O, Bengio Y et al (2005) Efficient non-parametric function induction in semi-supervised learning. In: Proceedings of the 10th international workshop on artificial intelligence and statistics, p 12–19Google Scholar
  4. 4.
    Fergus R, Kandola J et al (2009) Semi-supervised learning in gigantic image collections. In: Proceedings of the advances in neural information processing systems, VancouverGoogle Scholar
  5. 5.
    Fowlkes C et al (2004) Spectral grouping using the nyström method. Pattern Anal Mach Intell IEEE Trans 26(2):214–225CrossRefMATHGoogle Scholar
  6. 6.
    Garcke J, Griebel M (2005) Semi-supervised learning with sparse grids. In: Proceedings of the international conference on machine learning, BonnGoogle Scholar
  7. 7.
    Golub G, Loan V (1996) Matrix computations. Johns Hopkins University Press, BaltimoreGoogle Scholar
  8. 8.
    Kawano S et al (2012) Semi-supervised logistic discrimination via graph-based regularization. Neural Process Lett 36(3):203–216CrossRefGoogle Scholar
  9. 9.
    Larsen R (1998) Lanczos bidiagonalization with partial reorthogonalization. DAIMI Report Series 27(537):1–101Google Scholar
  10. 10.
    Liu W, He J et al (2010) Large graph construction for scalable semi-supervised learning. In: Proceedings of the international conference on machine learning, Haifa, pp 679–686Google Scholar
  11. 11.
    Salton G, McGill M (1986) Introduction to modern information retrieval, McGraw-Hill, New YorkGoogle Scholar
  12. 12.
    Shang F et al (2012) Integrating spectral kernel learning and constraints in semi-supervised classification. Neural Process Lett 36(2):101–115CrossRefGoogle Scholar
  13. 13.
    Sinha K, Belkin M (2009) Semi-supervised learning using sparse eigenfunction bases. Adv Neural Info Process Sys 22:1687–1695Google Scholar
  14. 14.
    Stewart G (1998) Matrix algorithms: basic decompositions. SIAM, PhiladelphiaCrossRefMATHGoogle Scholar
  15. 15.
    Subramanya A, Bilmes J (2011) Semi-supervised learning with measure propagation. J Mach Learn Res 12:3311–3370MathSciNetGoogle Scholar
  16. 16.
    Talwalkar A, Kumar S et al (2008) Large-scale manifold learning. In: Proceedings of the IEEE conference on the computer vision and pattern recognition (CVPR)Google Scholar
  17. 17.
    Tsang I, Kwok J (2007) Large-scale sparsified manifold regularization. In: Proceedings of the advances in neural information processing systems, VancouverGoogle Scholar
  18. 18.
    Valls G et al (2007) Semi-supervised graph-based hyperspectral image classification. Geosci Remote Sens IEEE Trans 45(10):3044–3054CrossRefGoogle Scholar
  19. 19.
    Wang J et al (2009) Linear neighborhood propagation and its applications. Pattern Anal Mach Intell IEEE Trans 31(9):1600–1615CrossRefGoogle Scholar
  20. 20.
    Zhang K, Kwok J, et al (2009) Prototype vector machine for large scale semi-supervised learning. In: Proceedings of the international conference on machine learning, pp 1233–1240Google Scholar
  21. 21.
    Zhou D, Bousquet O (2003) Learning with local and global consistency. In: Proceedings of the advances in neural information processing systems, Vancouver, pp 321–328Google Scholar
  22. 22.
    Zhu X, Ghahramani Z et al (2003) Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the international conference on machine learning, Washington, DC, pp 912–919Google Scholar
  23. 23.
    Zhu X, Goldberg B (2009) Introduction to semi-supervised learning. Morgan & Claypool Publishers, San RafaelMATHGoogle Scholar
  24. 24.
    Zhu X (2005) Lafferty: Harmonic mixtures: combining mixture models and graph-based methods for inductive and scalable semi-supervised learning. In: Proceedings of the international conference on machine learning, Bonn, pp 1052–1059Google Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Chen Gong
    • 1
  • Keren Fu
    • 1
  • Lei Zhou
    • 1
  • Jie Yang
    • 1
  • Xiangjian He
    • 2
  1. 1.Institute of Image Processing and Pattern RecognitionShanghai Jiao Tong UniversityShanghaiChina
  2. 2.Department of Computer ScienceUniversity of Technology SydneyUltimoAustralia

Personalised recommendations