Abstract
LU decomposition is one of the most efficient algorithms that can be applied to various operations such as the solving of linear equations, finding the determinant of a given matrix and matrix inversion. In any real-time application involving one of the above application scenarios, the matrix size will be gigantic and will not be able to be efficiently decomposed on a single node. In this paper, we propose a scalable algorithm for LU decomposition which is frugal in terms of time. This is accomplished by pipelining block LU decomposition on a multi-node Apache Spark system that is integrated with Apache Ignite. With the introduction of Ignite RDD, the entire dataset is available with all the nodes due to the availability of a shared Ignite cache layer, and only references to memory locations need to be passed across nodes. This is especially significant with respect to exa-scale computing where network latency is a major issue. The proposed algorithm is future-oriented and ready to deal with an efficient decomposition of large matrices with time complexity of O(N2).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anderson E, Bai Z et al (1990) Lapack: a portable linear algebra library for high-performance computers. In ACM/IEEE Conference on Supercomputing, pp 2–11
Dongarra J, Luszczek P (1999) Linpack benchmark. In: Author F, Author S, Author T (eds) Encyclopedia of parallel computing, 2011, 2nd edn. Publisher, Springer, pp 1033–1036
Bientinesi P, Gunter B, Geijn RA (2008) Families of algorithms related to the inversion of a symmetric positive definite matrix. ACM Trans Math Softw (TOMS) 35(1)
Agullo E, Bouwmeester H et al (2011) Towards an efficient tile matrix inversion of symmetric positive definite matrices on multicore architectures. High performance computing for computational science–VECPAR, Springer, pp 129–138
Dongarra J, Faverge M et al (2011) High performance matrix inversion based on lu factorization for multicore architectures. In: ACM workshop on many task computing on grids and supercomputers, pp 33–42
Chen C, Fang J, Tang T, Yang C (2017) LU factorization on heterogeneous systems: an energy-efficient approach towards high performance. Comput Arch Sci Comput 99(8):791–811
Tomov S, Dongarra J, Baboulin M (2010a) Towards dense linear algebra for hybrid GPU accelerated manycore systems. Parallel Comput 36(5–6):232–240
Volkov V, Demmel JW, LU (2008) QR and Cholesky factorizations using vector capabilities of GPUs. Technical Report UCB/EECS-2008–49, EECS Department, University of California, Berkeley, Calif, USA
Van de Velde EF (1990) Experiments with multicomputer LU-decomposition. Concurrency: Pract Experience 2(1):1–26
Kurzak J, Luszczek P, Faverge M, Dongarra J (2013) LU factorization with partial pivoting for a multicore system with accelerators. IEEE Trans Parallel Distrib Syst 24(8):1613–1621
Deisher M, Smelyanskiy M, Nickerson B, Lee VW, Chuvelev M, Dubey P (2011) Designing and dynamically load balancing hybrid LU for multi/many-core. Comput Sci Res Dev 26(3–4):211–220
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Harini, S., Ravikumar, A., Thakkar, K. (2021). Parallel LU Decomposition Algorithm for Exa-Scale Computing Using Spark Ignite. In: Goyal, D., Gupta, A.K., Piuri, V., Ganzha, M., Paprzycki, M. (eds) Proceedings of the Second International Conference on Information Management and Machine Intelligence. Lecture Notes in Networks and Systems, vol 166. Springer, Singapore. https://doi.org/10.1007/978-981-15-9689-6_26
Download citation
DOI: https://doi.org/10.1007/978-981-15-9689-6_26
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-9688-9
Online ISBN: 978-981-15-9689-6
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)