Bringing High Performance Computing to Big Data Algorithms

  • H. Anzt
  • J. Dongarra
  • M. Gates
  • J. KurzakEmail author
  • P. Luszczek
  • S. Tomov
  • I. Yamazaki


Many ideas of High Performance Computing are applicable to Big Data problems. The more so now, that hybrid, GPU computing gains traction in mainstream computing applications. This work discusses the differences between the High Performance Computing software stack and the Big Data software stack and then focuses on two popular computing workloads, the Alternating Least Squares algorithm and the Singular Value Decomposition, and shows how their performance can be maximized using hybrid computing techniques.


Singular Value Decomposition Thread Block Latent Semantic Indexing Alternate Little Square Implicit Feedback 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Apache, Mahout version 0.9 (2015a).
  2. 2.
    Apache, Spark version 1.5 (2015b).
  3. 3.
    J. Baglama, L. Reichel, Augmented implicitly restarted Lanczos bidiagonalization methods. SIAM J. Sci. Comput. 27, 19–42 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    J. Bennett, S. Lanning, The netflix prize, in Proceedings of the KDD Cup Workshop 2007 (ACM, New York, 2007), pp 3–6.
  5. 5.
    M.W. Berry, Large scale sparse singular value computations. Int. J. Supercomput. Appl. 6, 13–49 (1992)Google Scholar
  6. 6.
    T. Bertin-Mahieux, D.P. Ellis, B. Whitman, P. Lamere, The million song dataset, in Proceedings of the 12th International Conference on Music Information Retrieval (ISMIR) (2011)Google Scholar
  7. 7.
    C. Bishop, Pattern Recognition and Machine Learning (Springer, New York, 2006)zbMATHGoogle Scholar
  8. 8.
    P. Biswas, T.C. Lian, T.C. Wang, Y. Ye, Semidefinite programming based algorithms for sensor network localization. ACM Trans. Sensor Networks (TOSN) 2(2), 188–220 (2006)CrossRefGoogle Scholar
  9. 9.
    E.J. Candès, B. Recht, Exact matrix completion via convex optimization. Found. Comput. Math. 9(6), 717–772 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    P. Chen, D. Suter, Recovering the missing components in a large noisy low-rank matrix: application to SFM. IEEE Trans. Pattern Anal. Mach. Intell. 26(8), 1051–1063 (2004)CrossRefGoogle Scholar
  11. 11.
    Committee on the Analysis of Massive Data, Committee on Applied and Theoretical Statistics, Board on Mathematical Sciences and Their Applications, Division on Engineering and Physical Sciences, National Research Council (2013). Frontiers in Massive Data Analysis. The National Academies PressGoogle Scholar
  12. 12.
    Dato, GraphLab version 1.3 (2015).
  13. 13.
    S. Deerwester, S. Dumais, G. Furnas, T. Landauer, R. Harshman, Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci. 41, 391–407 (1990)CrossRefGoogle Scholar
  14. 14.
    DOE Office of Science, Synergistic challenges in data-intensive science and exascale computing. DOE Advanced Scientific Computing Advisory Committee (ASCAC) (2013). Data Subcommittee ReportGoogle Scholar
  15. 15.
    S.H. Fuller, L.I. Millett, The Future of Computing Performance: Game Over Or Next Level? (National Academy Press, Washington, DC, 2011)Google Scholar
  16. 16.
    M. Gates, H. Anzt, J. Kurzak, J. Dongarra, Accelerating collaborative filtering using concepts from high performance computing, in 2015 IEEE International Conference on Big Data (Big Data) (IEEE, 2015), pp. 667–676Google Scholar
  17. 17.
    D. Goldberg, D. Nichols, B.M. Oki, D. Terry, Using collaborative filtering to weave an information tapestry. Commun. ACM 35(12), 61–70 (1992)CrossRefGoogle Scholar
  18. 18.
    G. Golub, C. van Loan, Matrix Computations, 4th edn. (The Johns Hopkins University Press, Baltimore, 2012)zbMATHGoogle Scholar
  19. 19.
    G. Golub, F. Luk, M. Overton, A block Lanczos method for computing the singular values and corresponding singular vectors of a matrix. ACM Trans. Math. Softw. 7, 149–169 (1981)CrossRefzbMATHGoogle Scholar
  20. 20.
    S. Graham, M. Snir, C. Patterson, Getting Up to Speed: The Future of Supercomputing (The National Academies Press, Washington, DC, 2004)Google Scholar
  21. 21.
    N. Halko, P. Martinsson, J. Tropp, Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53(2), 217–288 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    M. Hoemmen, Communication-avoiding Krylov subspace methods. Ph.D. thesis, University of California, Berkeley (2010)Google Scholar
  23. 23.
    Y. Hu, Y. Koren, C. Volinsky, Collaborative filtering for implicit feedback datasets, in IEEE International Conference on Data Mining (ICDM) (2008), pp. 263–272Google Scholar
  24. 24.
    Innovative Computing Lab, BEAST (2015).
  25. 25.
    Intel Corp, Developer Reference for Intel Math Kernel Library (2015).
  26. 26.
    Intel Corp, Intel Data Analytics Acceleration Library 2016, Developer Guide (2016)Google Scholar
  27. 27.
    P. Jain, P. Netrapalli, S. Sanghavi, Low-rank matrix completion using alternating minimization, in Proceedings of the Forty-Fifth annual ACM Symposium on Theory of Computing (ACM, 2013), pp 665–674Google Scholar
  28. 28.
    I. Karasalo, Estimating the covariance matrix by signal subspace averaging. IEEE Trans. Acoust. Speech Signal Process. 34(1), 8–12 (1986)MathSciNetCrossRefGoogle Scholar
  29. 29.
    T. Kolda, D. O’Leary, A semidiscrete matrix decomposition for latent semantic indexing information retrieval. ACM Trans. Inf. Syst. 16(4), 322–346 (1998)CrossRefGoogle Scholar
  30. 30.
    Y. Koren, Factorization meets the neighborhood: a multifaceted collaborative filtering model, in Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’08 (ACM, New York, 2008), pp. 426–434Google Scholar
  31. 31.
    R. Krovetz, W.B. Croft, Lexical ambiguity and information retrieval. ACM Trans. Inf. Syst. 10(2), 115–141 (1992)CrossRefGoogle Scholar
  32. 32.
    J. Kurzak, S. Tomov, J. Dongarra, Autotuning gemm kernels for the Fermi GPU. IEEE Trans. Parallel Distrib. Syst. 23(11), 2045–2057 (2012)CrossRefGoogle Scholar
  33. 33.
    J. Kurzak, H. Anzt, M. Gates, J. Dongarra, Implementation and tuning of batched Cholesky factorization and solve for NVIDIA GPUs. Trans. Parallel Distrib. Syst. (2015). doi: 10.1109/TPDS.2015.2481890
  34. 34.
    C. Lam, Hadoop in Action (Manning Publications Co., Stamford, 2010)Google Scholar
  35. 35.
    D. Laney, 3D data management: controlling data volume, velocity, and variety. Application Delivery Strategies by META Group Inc., File: 949 (2001)Google Scholar
  36. 36.
    E. Liberty, F. Woolfe, P.G. Martinsson, V. Rokhlin, M. Tygert, Randomized algorithms for the low-rank approximation of matrices. Proc. National Acad. Sci. 104(51), 20167–20172 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  37. 37.
    Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, j.M. Hellerstein, GraphLab: a new framework for parallel machine learning. CoRR abs/1006.4990 (2010).
  38. 38.
    Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, J.M. Hellerstein, Distributed GraphLab: a framework for machine learning and data mining in the cloud. Proc. VLDB Endow. 5(8), 716–727 (2012)CrossRefGoogle Scholar
  39. 39.
    P. Luszczek, M. Gates, J. Kurzak, A. Danalis, J. Dongarra, Search space generation and pruning system for autotuners, in International Workshop on Automatic Performance Tuning (iWAPT 2016) (2016, submitted)Google Scholar
  40. 40.
    D. Lyubimov, Command line interface, stochastic SVD. Technical report, The Apache Software Foundation (2014).
  41. 41.
    M.W. Mahoney, Randomized algorithms for matrices and data. Found. Trends\(\textregistered \) Mach. Learn. 3(2), 123–224 (2011)Google Scholar
  42. 42.
    P.G. Martinsson, V. Rockhlin, M. Tygert, A randomized algorithm for the approximation of matrices. Technical report, DTIC Document (2006)Google Scholar
  43. 43.
    P. McJones, Eachmovie collaborative filtering data set. DEC Systems Research Center 249 (1997)Google Scholar
  44. 44.
    X. Meng, J. Bradley, B. Yavuz, E. Sparks, S. Venkataraman, D. Liu, J. Freeman, D. Tsai, M. Amde, S. Owen et al., MLlib: Machine learning in Apache Spark (2015). arXiv preprint arXiv:150506807
  45. 45.
    P. Menozzi, A. Piazza, L. C-Sforza, Synthetic maps of human gene frequencies in Europeans. Science 201, 786–792 (1978)CrossRefGoogle Scholar
  46. 46.
    NVIDIA Corp, cuBLAS Library User Guide, v7.0 (2015a)Google Scholar
  47. 47.
    NVIDIA Corp, CUDA C Programming Guide, v7.0 (2015b)Google Scholar
  48. 48.
    S. Owen, R. Anil, T. Dunning, E. Friedman, Mahout in Action (Manning Publications Co., Greenwich, 2011)Google Scholar
  49. 49.
    P. Paschou, E. Ziv, E. Burchard, S. Choudhry, W. R-Cintron, M. Mahoney, P. Drineas, PCA-correlated SNPs for structure identification in worldwide human populations. PLoS Genet. 3, 1672–1686 (2007)CrossRefGoogle Scholar
  50. 50.
    A. Paterek, Improving regularized singular value decomposition for collaborative filtering, in Proceedings of KDD Cup and Workshop (2007), pp. 39–42Google Scholar
  51. 51.
    N. Patterson, A. Price, D. Reich, Population structure and eigenanalysis. PLoS Genet. 2(12), 2074–2093 (2006)CrossRefGoogle Scholar
  52. 52.
    A. Price, N. Patterson, R. Plenge, M. Weinblatt, N. Shadick, D. Reich, Principal components analysis corrects for stratification in genome-wide association studies. Nature Genet. 38(8), 904–909 (2006)CrossRefGoogle Scholar
  53. 53.
    R.A. Rossi, N.K. Ahmed, The network data repository with interactive graph analytics and visualization, in Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence (2015).
  54. 54.
    G. Salton, M. McGill, Introduction to Modern Information Retrieval (McGraw-Hill, New York, 1983)zbMATHGoogle Scholar
  55. 55.
    B. Sarwar, G. Karypis, J. Konstan, J. Riedl, Analysis of recommendation algorithms for e-commerce, in Proceedings of the 2nd ACM Conference on Electronic Commerce (2000), pp 158–167Google Scholar
  56. 56.
    A. Stathopoulos, K. Wu, A block orthogonalization procedure with constant synchronization requirements. SIAM J. Sci. Comput. 23(6), 2165–2182 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  57. 57.
    W. Tan, L. Cao, L.L. Fong, Faster and cheaper: Parallelizing large-scale matrix factorization on gpus. CoRR abs/1603.03820 (2016).
  58. 58.
    J. Tougas, R. Spiteri, Updating the partial singular value decomposition in latent semantic indexing. Comput. Statist. Data Anal. 52, 174–183 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  59. 59.
    E. Vecharynski, Y. Saad, Fast updating algorithms for latent semantic indexing. SIAM J. Matrix Anal. Appl. 35(3), 1105–1131 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  60. 60.
    T. White, Hadoop: The Definitive Guide (O’Reilly Media, Inc., Sebastopol, 2012)Google Scholar
  61. 61.
    K. Wu, H. Simon, Thick-restart Lanczos method for large symmetric eigenvalue problems. SIAM J. Matrix Anal. Appl. 22(2), 602–616 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  62. 62.
    I. Yamazaki, K. Wu, A communication-avoiding thick-restart lanczos method on a distributed-memory system, in Proceedings of the 2011 International Conference on Parallel Processing, Euro-Par’11 (Springer, Berlin, 2012), pp. 345–354Google Scholar
  63. 63.
    I. Yamazaki, H. Anzt, S. Tomov, M. Hoemmen, J. Dongarra Improving the performance of CA-GMRES on multicores with multiple GPUs, in Proceedings of the IEEE International Parallel and Distributed Symposium (IPDPS) (2014a), pp. 382–391Google Scholar
  64. 64.
    I. Yamazaki, T. Mary, J. Kurzak, S. Tomov, Access-averse framework for computing low-rank matrix approximations, in Proceedings of the International Workshop on High Performance Big Graph Data Management, Analysis, and Minig (2014b), pp. 70–77Google Scholar
  65. 65.
    I. Yamazaki, S. Rajamanickam, E. Boman, M. Hoemmen, M. Heroux, S. Tomov, Domain decomposition preconditioners for communication-avoiding Krylov methods on a hybrid CPU/GPU cluster, in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (2014c), pp. 933–944Google Scholar
  66. 66.
    I. Yamazaki, J. Kurzak, P. Luszczek, J. Dongarra, Randomized algorithms to update partial singular value decomposition on a hybrid CPU/GPU cluster, in Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC) (2015), pp. 345–354Google Scholar
  67. 67.
    M. Zaharia, M. Chowdhury, M.J. Franklin, S. Shenker, I. Stoica, Spark: cluster computing with working sets, in Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, vol. 10 (2010), p.10Google Scholar
  68. 68.
    H. Zha, H. Simon, On updating problems in latent semantic indexing. SIAM J. Sci. Comput. 21(2), 782–791 (1999)MathSciNetCrossRefzbMATHGoogle Scholar
  69. 69.
    H. Zha, O. Marques, H. Simon, Large-scale SVD and subspace-based methods for information retrieval, in Solving Irregularly Structured Problems in Parallel, vol. 1457, Lecture Notes in Computer Science, ed. by A. Ferreira, J. Rolim, H. Simon, S.-H. Teng (Springer, Heidelberg, 1998), pp. 29–42CrossRefGoogle Scholar
  70. 70.
    Y. Zhou, D. Wilkinson, R. Schreiber, R. Pan, Large-scale parallel collaborative filtering for the netflix prize in Proceedings of the 4th International Conference on Algorithmic Aspects in Information and Management, AAIM’08 (Springer, Berlin, 2008), pp. 337–348Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • H. Anzt
    • 1
  • J. Dongarra
    • 1
  • M. Gates
    • 1
  • J. Kurzak
    • 1
    Email author
  • P. Luszczek
    • 1
  • S. Tomov
    • 1
  • I. Yamazaki
    • 1
  1. 1.Innovative Computing LaboratoryUniversity of TennesseeKnoxvilleUSA

Personalised recommendations