Abstract
Traditional graph-based semi-supervised learning (GBSSL) algorithms usually scale badly due to the expensive computational burden. The main bottleneck is that they need to compute the inversion of a huge matrix. In order to alleviate this problem, this paper proposes Neumann series approximation (NSA) to explicitly approximate the inversion process required by conventional GBSSL methodologies, which makes them computationally tractable for relatively large datasets. It is proved that the deviation between the approximation and direct inversion is bounded. Using real-world datasets related to handwritten digit recognition, speech recognition and text classification, the experimental results reveal that NSA accelerates the speed significantly without decreasing too much precision. We also empirically show that NSA outperforms other scalable approaches such as Nyström method, Takahashi equation, Lanczos process based SVD and AnchorGraph regularization, in terms of both efficiency and accuracy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Lanczos process based SVD is implemented by using PROPACK, which is available at http://soi.stanford.edu/~rmunk/PROPACK/.
Available at: http://archive.ics.uci.edu/ml/datasets/ISOLET.
Avialable at http://qwone.com/~jason/20Newsgroups/.
References
Belkin M et al (2006) Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. J Mach Learn Res 7:2399–2434
Campbell Y, Davis T (1995) Computing the sparse inverse subset: an inverse multifrontal approach. Technical report TR-95-021, University of Florida, Gainesville, FL
Delalleau O, Bengio Y et al (2005) Efficient non-parametric function induction in semi-supervised learning. In: Proceedings of the 10th international workshop on artificial intelligence and statistics, p 12–19
Fergus R, Kandola J et al (2009) Semi-supervised learning in gigantic image collections. In: Proceedings of the advances in neural information processing systems, Vancouver
Fowlkes C et al (2004) Spectral grouping using the nyström method. Pattern Anal Mach Intell IEEE Trans 26(2):214–225
Garcke J, Griebel M (2005) Semi-supervised learning with sparse grids. In: Proceedings of the international conference on machine learning, Bonn
Golub G, Loan V (1996) Matrix computations. Johns Hopkins University Press, Baltimore
Kawano S et al (2012) Semi-supervised logistic discrimination via graph-based regularization. Neural Process Lett 36(3):203–216
Larsen R (1998) Lanczos bidiagonalization with partial reorthogonalization. DAIMI Report Series 27(537):1–101
Liu W, He J et al (2010) Large graph construction for scalable semi-supervised learning. In: Proceedings of the international conference on machine learning, Haifa, pp 679–686
Salton G, McGill M (1986) Introduction to modern information retrieval, McGraw-Hill, New York
Shang F et al (2012) Integrating spectral kernel learning and constraints in semi-supervised classification. Neural Process Lett 36(2):101–115
Sinha K, Belkin M (2009) Semi-supervised learning using sparse eigenfunction bases. Adv Neural Info Process Sys 22:1687–1695
Stewart G (1998) Matrix algorithms: basic decompositions. SIAM, Philadelphia
Subramanya A, Bilmes J (2011) Semi-supervised learning with measure propagation. J Mach Learn Res 12:3311–3370
Talwalkar A, Kumar S et al (2008) Large-scale manifold learning. In: Proceedings of the IEEE conference on the computer vision and pattern recognition (CVPR)
Tsang I, Kwok J (2007) Large-scale sparsified manifold regularization. In: Proceedings of the advances in neural information processing systems, Vancouver
Valls G et al (2007) Semi-supervised graph-based hyperspectral image classification. Geosci Remote Sens IEEE Trans 45(10):3044–3054
Wang J et al (2009) Linear neighborhood propagation and its applications. Pattern Anal Mach Intell IEEE Trans 31(9):1600–1615
Zhang K, Kwok J, et al (2009) Prototype vector machine for large scale semi-supervised learning. In: Proceedings of the international conference on machine learning, pp 1233–1240
Zhou D, Bousquet O (2003) Learning with local and global consistency. In: Proceedings of the advances in neural information processing systems, Vancouver, pp 321–328
Zhu X, Ghahramani Z et al (2003) Semi-supervised learning using Gaussian fields and harmonic functions. In: Proceedings of the international conference on machine learning, Washington, DC, pp 912–919
Zhu X, Goldberg B (2009) Introduction to semi-supervised learning. Morgan & Claypool Publishers, San Rafael
Zhu X (2005) Lafferty: Harmonic mixtures: combining mixture models and graph-based methods for inductive and scalable semi-supervised learning. In: Proceedings of the international conference on machine learning, Bonn, pp 1052–1059
Acknowledgments
This research is supported by NSFC, China (No: 6127325861375048), and Ph.D. Programs Foundation of Ministry of Education of China (No.20120073110018).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Gong, C., Fu, K., Zhou, L. et al. Scalable Semi-Supervised Classification via Neumann Series. Neural Process Lett 42, 187–197 (2015). https://doi.org/10.1007/s11063-014-9351-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-014-9351-z