Advertisement

Spectral Clustering with a Convex Regularizer on Millions of Images

  • Maxwell D. Collins
  • Ji Liu
  • Jia Xu
  • Lopamudra Mukherjee
  • Vikas Singh
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8691)

Abstract

This paper focuses on efficient algorithms for single and multi-view spectral clustering with a convex regularization term for very large scale image datasets. In computer vision applications, multiple views denote distinct image-derived feature representations that inform the clustering. Separately, the regularization encodes high level advice such as tags or user interaction in identifying similar objects across examples. Depending on the specific task, schemes to exploit such information may lead to a smooth or non-smooth regularization function. We present stochastic gradient descent methods for optimizing spectral clustering objectives with such convex regularizers for datasets with up to a hundred million examples. We prove that under mild conditions the local convergence rate is \(O(1/\sqrt{T})\) where T is the number of iterations; further, our analysis shows that the convergence improves linearly by increasing the number of threads. We give extensive experimental results on a range of vision datasets demonstrating the algorithm’s empirical behavior.

Keywords

Spectral Cluster Latent Semantic Analysis Normalize Mutual Information Nonnegative Matrix Factorization Stochastic Gradient 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Balzano, L., Nowak, R., Recht, B.: Online identification and tracking of subspaces from highly incomplete information. In: Proceedings of the Allerton Conference on Communication, Control and Computing (2010)Google Scholar
  2. 2.
    Batra, D., Agrawal, H., Banik, P., Chavali, N., Alfadda, A.: CloudCV: Large-scale distributed computer vision as a cloud service (2013), http://www.cloudcv.org
  3. 3.
    Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006, Part I. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  4. 4.
    Bickel, S., Scheffer, T.: Multi-view clustering. In: Proceedings of the IEEE International Conference on Data Mining (2004)Google Scholar
  5. 5.
    Blaschko, M.B., Lampert, C.H.: Correlational spectral clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2008)Google Scholar
  6. 6.
    Chaudhuri, K., Kakade, S.M., Livescu, K., Sridharan, K.: Multi-view clustering via canonical correlation analysis. In: Proceedings of the International Conference on Machine Learning (2009)Google Scholar
  7. 7.
    Chen, W., Song, Y., Bai, H., Lin, C., Chang, E.Y.: Parallel spectral clustering in distributed systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(3), 568–586 (2011)CrossRefGoogle Scholar
  8. 8.
    Chen, X., Cai, D.: Large scale spectral clustering with landmark-based representation. In: Proceedings of the AAAI Conference on Artificial Intelligence (2011)Google Scholar
  9. 9.
    Darken, C., Moody, J.: Towards faster stochastic gradient search. In: Advances in Neural Information Processing Systems (1993)Google Scholar
  10. 10.
    Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-Scale Hierarchical Image Database. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2009)Google Scholar
  11. 11.
    Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: Decaf: A deep convolutional activation feature for generic visual recognition. ArXiv preprint ArXiv:1310.1531 (2013)Google Scholar
  12. 12.
    Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications 20(2), 303–353 (1998)CrossRefzbMATHMathSciNetGoogle Scholar
  13. 13.
    Fowlkes, C., Belongie, S., Chung, F., Malik, J.: Spectral grouping using the Nyström method. IEEE Transactions on Pattern Analysis and Machine Intelligence 26(2), 214–225 (2004)CrossRefGoogle Scholar
  14. 14.
    Gehler, P., Nowozin, S.: On feature combination for multiclass object classification. In: Proceedings of the IEEE International Conference on Computer Vision (2009)Google Scholar
  15. 15.
    Hofmann, T.: Probabilistic latent semantic analysis. In: Proceedings of the Conference on Uncertainty in Artificial Intelligence (1999)Google Scholar
  16. 16.
    Khoa, N.L.D., Chawla, S.: Large scale spectral clustering using resistance distance and Spielman-Teng solvers. In: Ganascia, J.-G., Lenca, P., Petit, J.-M. (eds.) DS 2012. LNCS, vol. 7569, pp. 7–21. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  17. 17.
    Krishnamurthy, A., Balakrishnan, S., Xu, M., Singh, A.: Efficient active algorithms for hierarchical clustering. In: Proceedings of the International Conference on Machine Learning (2012)Google Scholar
  18. 18.
    Kulis, B., Grauman, K.: Kernelized locality-sensitive hashing for scalable image search. In: Proceedings of the IEEE International Conference on Computer Vision (2009)Google Scholar
  19. 19.
    Kumar, A., Daumé III, H.: A co-training approach for multi-view spectral clustering. In: Proceedings of the International Conference on Machine Learning (2011)Google Scholar
  20. 20.
    Kumar, A., Rai, P., Daumé III, H.: Co-regularized multi-view spectral clustering. In: Advances in Neural Information Processing Systems (2011)Google Scholar
  21. 21.
    Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2006)Google Scholar
  22. 22.
    Lehoucq, R.B., Sorensen, D.C., Yang, C.: ARPACK users’ guide: Solution of large-scale eigenvalue problems with implicitly restarted Arnoldi methods, vol. 6 (1998)Google Scholar
  23. 23.
    Li, L., Su, H., Xing, E.P., Fei-Fei, L.: Object bank: A high-level image representation for scene classification & semantic feature sparsification. In: Advances in Neural Information Processing Systems (2010)Google Scholar
  24. 24.
    Li, M., Lian, X.C., Kwok, J., Lu, B.L.: Time and space efficient spectral clustering via column sampling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2011)Google Scholar
  25. 25.
    Liu, J., Wang, C., Gao, J., Han, J.: Multi-view clustering via joint nonnegative matrix factorization. In: SIAM International Conference on Data Mining (2013)Google Scholar
  26. 26.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (2004)CrossRefGoogle Scholar
  27. 27.
    Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision 42(3), 145–175 (2001)CrossRefzbMATHGoogle Scholar
  28. 28.
    Sakai, T., Imiya, A.: Fast spectral clustering with random projection and sampling. In: Machine Learning and Data Mining in Pattern Recognition (2009)Google Scholar
  29. 29.
    Serre, T., Wolf, L., Poggio, T.: Object recognition with features inspired by visual cortex. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2005)Google Scholar
  30. 30.
    Tang, W., Lu, Z., Dhillon, I.S.: Clustering with multiple graphs. In: Proceedings of the IEEE International Conference on Data Mining (2009)Google Scholar
  31. 31.
    Torralba, A., Fergus, R., Freeman, W.T.: 80 million tiny images: a large dataset for non-parametric object and scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(11), 1958–1970 (2008)CrossRefGoogle Scholar
  32. 32.
    Tuzel, O., Porikli, F., Meer, P.: Region covariance: A fast descriptor for detection and classification. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 589–600. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  33. 33.
    Wen, Z., Yin, W.: A feasible method for optimization with orthogonality constraints. Mathematical Programming, 1–38 (2012)Google Scholar
  34. 34.
    Xu, J., Ithapu, V.K., Mukherjee, L., Rehg, J.M., Singh, V.: GOSUS: Grassmannian Online Subspace Updates with Structured-sparsity. In: Proceedings of the IEEE International Conference on Computer Vision (2013)Google Scholar
  35. 35.
    Zelnik-Manor, L., Perona, P.: Self-tuning spectral clustering. In: Advances in Neural Information Processing Systems (2004)Google Scholar
  36. 36.
    Zhou, D., Burges, C.J.C.: Spectral clustering and transductive learning with multiple views. In: Proceedings of the International Conference on Machine Learning (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Maxwell D. Collins
    • 1
  • Ji Liu
    • 2
  • Jia Xu
    • 1
  • Lopamudra Mukherjee
    • 3
  • Vikas Singh
    • 1
  1. 1.University of WisconsinMadisonUSA
  2. 2.University of RochesterUSA
  3. 3.University of WisconsinWhitewaterUSA

Personalised recommendations