The Journal of Supercomputing

, Volume 73, Issue 5, pp 2157–2186 | Cite as

DLRankSVM: an efficient distributed algorithm for linear RankSVM

Article
  • 138 Downloads

Abstract

Linear RankSVM is one of the widely used methods for learning to rank. The existing methods, such as Trust-Region Newton (TRON) method along with Order-Statistic Tree (OST), can be applied to train the linear RankSVM effectively. However, extremely lengthy training time is unacceptable when using any existing method to handle the large-scale linear RankSVM. To solve this problem, we thus focus on designing an efficient distributed method (named DLRankSVM) to train the huge-scale linear RankSVM on distributed systems. First, to efficiently reduce the communication overheads, we divide the training problem into subproblems in terms of different queries. Second, we propose an efficient heuristic algorithm to address the load balancing issue (which is a NP-complete problem). Third, using OST, we propose an efficient parallel algorithm (named PAV) to compute auxiliary variables at each computational node of the distributed system. Finally, based on PAV and the proposed heuristic algorithm, we develop DLRankSVM under the framework of TRON. The extensive empirical evaluations show that DLRankSVM not only can obtain impressive speedups on both multi-core and distributed systems, but also can perform well in prediction compared with the other state-of-the-art methods. To the best of our knowledge, this is the first research work proposed to train the huge-scale linear RankSVM in a distributed fashion.

Keywords

Distributed computing Linear RankSVM Trust-Region Newton method Order-statistic tree Learning to rank 

Notes

Acknowledgments

This research is supported in part by the National Natural Science Foundation of China under Grants No. 61309028 and No. 61472454, the National Social Science Foundation of China under Grants No. 12&ZD222, and the Project of Department of Education of Guangdong Province under Grants No. 2013KJCX0128. The authors thank the anonymous reviewers for their constructive comments and suggestions.

References

  1. 1.
    Airola A, Pahikkala T, Salakoski T (2011) Training linear ranking svms in linearithmic time using red-black trees. Pattern Recognit Lett 32(9):1328–1336CrossRefGoogle Scholar
  2. 2.
    Arabnia HR (1990) A parallel algorithm for the arbitrary rotation of digitized images using process-and-data-decomposition approach. J Parallel Distrib Comput 10(2):188–192CrossRefGoogle Scholar
  3. 3.
    Arabnia HR (1995) A distributed stereocorrelation algorithm. In: Fourth International Conference on Computer Communications and Networks. Proceedings. IEEE, pp 479–482Google Scholar
  4. 4.
    Arabnia HR, Bhandarkar SM (1996) Parallel stereocorrelation on a reconfigurable multi-ring network. J Supercomput 10(3):243–269CrossRefMATHGoogle Scholar
  5. 5.
    Arabnia HR, Fang WC, Lee C, Zhang Y (2010) Context-aware middleware and intelligent agents for smart environments. IEEE Intell Syst 25(2):10–11CrossRefGoogle Scholar
  6. 6.
    Arabnia HR, Oliver MA (1986) Fast operations on raster images with simd machine architectures. In: Computer Graphics Forum, vol 5. Wiley Online Library, pp 179–188Google Scholar
  7. 7.
    Arabnia HR, Oliver MA (1987) Arbitrary rotation of raster images with simd machine architectures. In: Computer Graphics Forum, vol 6. Wiley Online Library, pp 3–11Google Scholar
  8. 8.
    Bhandarkar SM, Arabnia HR (1995) The hough transform on a reconfigurable multi-ring network. J Parallel Distribut Comput 24(1):107–114CrossRefGoogle Scholar
  9. 9.
    Bhandarkar SM, Arabnia HR (1995) The refine multiprocessortheoretical properties and algorithms. Parallel Comput 21(11):1783–1805CrossRefGoogle Scholar
  10. 10.
    Bhandarkar SM, Arabnia HR, Smith JW (1995) A reconfigurable architecture for image processing and computer vision. Int J Pattern Recognit Artific Intell 9(02):201–229CrossRefGoogle Scholar
  11. 11.
    Bottou L, Lin CJ (2007) Support vector machine solvers. Large scale kernel machines, pp 301–320Google Scholar
  12. 12.
    Burges C, Shaked T, Renshaw E, Lazier A, Deeds M, Hamilton N, Hullender G (2005) Learning to rank using gradient descent. In: Proceedings of the 22nd International Conference on Machine learning. ACM, pp 89–96Google Scholar
  13. 13.
    Burges CJ, Svore KM, Bennett PN, Pastusiak A, Wu Q (2011) Learning to rank using an ensemble of lambda-gradient models. In: Yahoo! Learning to Rank Challenge, pp 25–35 (2011)Google Scholar
  14. 14.
    Cao Z, Qin T, Liu TY, Tsai MF, Li H (2007) Learning to rank: from pairwise approach to listwise approach. In: Proceedings of the 24th International Conference on Machine Learning. ACM, pp 129–136Google Scholar
  15. 15.
    Chapelle O (2007) Training a support vector machine in the primal. Neural Comput 19(5):1155–1178MathSciNetCrossRefMATHGoogle Scholar
  16. 16.
    Chapelle O, Chang Y (2011) Yahoo! learning to rank challenge overview. In: Yahoo! Learning to Rank Challenge, pp 1–24Google Scholar
  17. 17.
    Chapelle O, Keerthi SS (2010) Efficient algorithms for ranking with svms. Inf Retrieval 13(3):201–215CrossRefGoogle Scholar
  18. 18.
    Cortes C, Vapnik V (1995) Support-vector networks. Mach Learn 20(3):273–297MATHGoogle Scholar
  19. 19.
    Cossock D, Zhang T (2008) Statistical analysis of bayes optimal subset ranking. IEEE Trans Inf Theory 54(11):5140–5154MathSciNetCrossRefMATHGoogle Scholar
  20. 20.
    Dean J, Ghemawat S (2008) Mapreduce: simplified data processing on large clusters. Commun ACM 51(1):107–113CrossRefGoogle Scholar
  21. 21.
    Dembo RS, Steihaug T (1983) Truncated-newtono algorithms for large-scale unconstrained optimization. Math Program 26(2):190–212CrossRefMATHGoogle Scholar
  22. 22.
    Fürnkranz J, Hüllermeier E (2003) Pairwise preference learning and ranking. In: Machine Learning: ECML. Springer, pp 145–156Google Scholar
  23. 23.
    Herbrich R, Graepel T, Obermayer K (1999) Large margin rank boundaries for ordinal regression. Advances in neural information processing systems, pp 115–132Google Scholar
  24. 24.
    Ho CH, Lin CJ (2012) Large-scale linear support vector regression. J Mach Learn Res 13(1):3323–3348MathSciNetMATHGoogle Scholar
  25. 25.
    Jafri R, Ali SA, Arabnia HR (2013) Computer vision-based object recognition for the visually impaired using visual tags. In: Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp), p 1Google Scholar
  26. 26.
    Jafri R, Ali SA, Arabnia HR, Fatima S (2014) Computer vision-based object recognition for the visually impaired in an indoors environment: a survey. Vis Comput 30(11):1197–1222CrossRefGoogle Scholar
  27. 27.
    Jafri R, Arabnia HR (2008) Fusion of face and gait for automatic human recognition. In: Fifth International Conference on Information Technology: New Generations, 2008. ITNG 2008. IEEE, pp 167–173Google Scholar
  28. 28.
    Järvelin K, Kekäläinen J (2002) Cumulated gain-based evaluation of ir techniques. ACM Trans Inf Syst (TOIS) 20(4):422–446CrossRefGoogle Scholar
  29. 29.
    Jin J, Cai X, Lai G, Lin X (2015) Gpu-accelerated parallel algorithms for linear ranksvm. J Supercomput 71(11):4141–4171CrossRefGoogle Scholar
  30. 30.
    Jin J, Lin X (2014) Efficient parallel algorithms for linear ranksvm on gpu. In: Network and Parallel Computing. Springer, pp 181–194Google Scholar
  31. 31.
    Joachims T (2002) Optimizing search engines using clickthrough data. In: Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 133–142Google Scholar
  32. 32.
    Joachims T (2006) Training linear svms in linear time. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 217–226Google Scholar
  33. 33.
    Kuo TM, Lee CP, Lin CJ (2014) Large-scale Kernel Rank SVM. In: Proceedings of the 2014 SIAM International Conference on Data Mining, pp 812–820Google Scholar
  34. 34.
    Lee CP, Lin CJ (2014) Large-scale linear ranksvm. Neural Comput 26(4):781–817MathSciNetCrossRefGoogle Scholar
  35. 35.
    Li P, Wu Q, Burges CJ (2007) Mcrank: Learning to rank using multiple classification and gradient boosting. In: Advances in Neural Information Processing Systems, pp 897–904Google Scholar
  36. 36.
    Lin CJ, Moré JJ (1999) Newton’s method for large bound-constrained optimization problems. SIAM J Optim 9(4):1100–1127MathSciNetCrossRefMATHGoogle Scholar
  37. 37.
    Lin CJ, Weng RC, Keerthi SS (2008) Trust region newton method for logistic regression. J Mach Learn Res 9:627–650MathSciNetMATHGoogle Scholar
  38. 38.
    Lin CY, Tsai CH, Lee CP, Lin CJ (2014) Large-scale logistic regression and linear support vector machines using spark. In: 2014 IEEE International Conference on Big Data (Big Data). IEEE, pp 519–528Google Scholar
  39. 39.
    Liu B, Chen J, Wang X (2015) Application of learning to rank to protein remote homology detection. Bioinformatics, p btv413Google Scholar
  40. 40.
    Luper D, Cameron D, Miller J, Arabnia HR (2007) Spatial and temporal target association through semantic analysis and gps data mining. In: IKE, vol 7. Citeseer, pp 25–28Google Scholar
  41. 41.
    Paisitkriangkrai S, Shen C, van den Hengel A (2015) Learning to rank in person re-identification with metric ensembles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1846–1855Google Scholar
  42. 42.
    Rabenseifner R, Hager G, Jost G (2009) Hybrid mpi/openmp parallel programming on clusters of multi-core smp nodes. In: 17th Euromicro International Conference on Parallel, Distributed and Network-based Processing. IEEE, pp 427–436Google Scholar
  43. 43.
    Rahbarinia B, Pedram MM, Arabnia HR, Alavi Z (2010) A multi-objective scheme to hide sequential patterns. In: The 2nd International Conference on Computer and Automation Engineering (ICCAE), vol 1. IEEE, pp 153–158Google Scholar
  44. 44.
    Severyn A, Moschitti A (2015) Learning to rank short text pairs with convolutional deep neural networks. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp 373–382Google Scholar
  45. 45.
    Snir M (1998) MPI—the Complete Reference: the MPI core, vol 1. MIT PressGoogle Scholar
  46. 46.
    Taylor M, Guiver J, Robertson S, Minka T (2008) Softrank: optimizing non-smooth rank metrics. In: Proceedings of the 2008 International Conference on Web Search and Data Mining. ACM, pp 77–86Google Scholar
  47. 47.
    Teo CH, Vishwanthan S, Smola AJ, Le QV (2010) Bundle methods for regularized risk minimization. J Mach Learn Res 11:311–365MathSciNetMATHGoogle Scholar
  48. 48.
    Ter Mors A, Valk J, Witteveen C, Arabnia H, Mun Y (2004) Coordinating autonomous planners. In: IC-AI, p 795Google Scholar
  49. 49.
    Trotman A (2005) Learning to rank. Inf Retrieval 8(3):359–381CrossRefGoogle Scholar
  50. 50.
    Wani MA, Arabnia HR (2003) Parallel edge-region-based segmentation algorithm targeted at reconfigurable multiring network. J Supercomput 25(1):43–62CrossRefMATHGoogle Scholar
  51. 51.
    Xia F, Liu TY, Wang J, Zhang W, Li H (2008) Listwise approach to learning to rank: theory and algorithm. In: Proceedings of the 25th International Conference on Machine Learning. ACM, pp 1192–1199Google Scholar
  52. 52.
    Xu J, Li H (2007) Adarank: a boosting algorithm for information retrieval. In: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, pp 391–398Google Scholar
  53. 53.
    Yu H, Kim Y, Hwang S (2009) Rv-svm: An efficient method for learning ranking svm. In: Advances in Knowledge Discovery and Data Mining. Springer, pp 426–438Google Scholar
  54. 54.
    Yu J, Tao D, Wang M, Rui Y (2015) Learning to rank using user clicks and visual features for image retrieval. IEEE Trans Cybern 45(4):767–779CrossRefGoogle Scholar
  55. 55.
    Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I (2010) Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, p 10Google Scholar
  56. 56.
    Zheng Z, Zha H, Zhang T, Chapelle O, Chen K, Sun G (2008) A general boosting method and its application to learning ranking functions for web search. In: Advances in Neural Information Processing Systems, pp 1697–1704Google Scholar
  57. 57.
    Zhuang Y, Chin WS, Juan YC, Lin CJ (2014) Distributed newton method for regularized logistic regression. Department of Computer Science and Information Engineering, National Taiwan University, Tech. RepGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  • Jing Jin
    • 1
  • Guoming Lai
    • 2
  • Xiaola Lin
    • 1
  • Xianggao Cai
    • 3
  1. 1.School of Data and Computer ScienceSun Yat-sen UniversityGuangzhouChina
  2. 2.Computer Engineering Technical CollegeGuangdong Institute of Science and TechnologyZhuhaiChina
  3. 3.School of Information ManagementSun Yat-sen UniversityGuangzhouChina

Personalised recommendations