General Parallel Execution Model for Large Matrix Workloads
Large scale statistical computing is crucial for extracting useful information from huge amount of data for both large companies and research scientists. The Solutions developed by high-performance communities have been more limited to clusters or high-end machines for decades. The cost of maintaining such dedicated clusters are prohibiting, people start to look at cloud computing where we can rent a cluster by time and pay-as-we-go. In a cloud setting, system features including fault tolerance and scalability become important. In this paper, we proposed a simple and universal parallel execution model for large matrix workloads. We implement the model in Hadoop MapReduce framework using map-only jobs. Because of the superiority of the model, experiments show that our Hadoop-based execution engine can reduce the execution time of matrix multiplication by half comparing with previous works.
KeywordsGeneral parallel execution Model Large matrix workloads Hodoop
Our research was supported by the Natural Science Foundation of China under grant No: 61462037 and the Natural Science Foundation of Jiangxi under grant No: 20142BAB217014.
- 2.Toledo, S., Gustavson, F.G.: The design and implementation of SOLAR, a portable library for scalable out-of-core linear algebra computations. In: Proceedings of the Fourth Workshop on I/O in Parallel and Distributed Systems, pp. 28–40 (1996)Google Scholar
- 3.Wu, P., Chen, Z.Z.: FT-ScaLAPACK: correcting soft errors on-line for ScaLAPACK cholesky, QR and LU factorization routines. In: Proceedings of the 23rd International Symposium on High-performance Parallel and Distributed Computing (HPDC 2014), Vancouver, BC, Canada (2014)Google Scholar
- 4.Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: 6th Conference on Symposium on Operating Systems Design & Implementation (OSDI 2004), Berkeley, CA, USA, vol. 6, pp. 107–113 (2004)Google Scholar
- 6.Das, S., Sismanis, Y., Beyer, K.S., Gemulla, R., Haas, P.J., McPherson, J.: Ricardo: integrating R and Hadoop. In: Proceedings of the 2010 International Conference on Management of Data, Indianapolis, Indiana, USA, pp. 987–998 (2010)Google Scholar
- 8.Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Proceedings of the NIPS, Denver, CO, USA (2000)Google Scholar