A Memory Efficient Parallel All-Pairs Computation Framework: Computation – Communication Overlap

  • Venkata Kasi Viswanath YeleswarapuEmail author
  • Arun K. Somani
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10777)


All-Pairs problems require each data element in a set of N data elements to be paired with every other data element for specific computation using the two data elements. Our framework aims to address recurring problems of scalability, distributing equal work load to all nodes and by reducing memory footprint. We reduce memory footprint of All-Pairs problems, by reducing memory requirement from \(N/\sqrt{P}\) to 3N/P. A bio-informatics application is implemented to demonstrate the scalability ranging up to 512 cores for the data set we experimented, redundancy management, and speed up performance of the framework.


Communication - computation overlap High performance computing All-Pairs problems Parallel computing MPI 



The research reported in this paper is partially supported by the Philip and Virginia Sproul Professor Endowment and HPC@ISU equipment at Iowa State University, some of which has been purchased through funding provided by NSF under MRI grant number NSF CNS grant number 1229081 and NSF CRI grant number 1205413. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies.


  1. 1.
    Hedegaard, R.: Handshake problem, January 2016.
  2. 2.
    Watson-Haigh, N.S., Kadarmideen, H.N., Reverter, A.: PCIT: an R package for weighted gene co-expression networks based on partial correlation and information theory approaches. Bioinformatics 26(3), 411–413 (2010)CrossRefGoogle Scholar
  3. 3.
    Chapman, T., Kalyanaraman, A.: An OpenMP algorithm and implementation for clustering biological graphs. In: Proceedings of the First Workshop on Irregular Applications: Architectures and Algorithm, p. 310. ACM (2011)Google Scholar
  4. 4.
    Phillips, P., et al.: Overview of the face recognition grand challenge. In: IEEE Computer Vision and Pattern Recognition (2005)Google Scholar
  5. 5.
    Moretti, C., Bui, H., Hollingsworth, K., Rich, B., Flynn, P., Thain, D.: All-Pairs: an abstraction for data-intensive computing on campus grids. IEEE Trans. Parallel Distrib. Syst. 21(1), 33–46 (2010)CrossRefGoogle Scholar
  6. 6.
    Chae, H., Jung, I., Lee, H., Marru, S., Lee, S.-W., Kim, S.: Bio and health informatics meets cloud: BioVLab as an example. Health Inf. Sci. Syst. 1(1), 6 (2013)CrossRefGoogle Scholar
  7. 7.
    Kleinheksel, C.J., Somani, A.K.: Scaling distributed all-pairs algorithms. Information Science and Applications (ICISA) 2016. LNEE, vol. 376, pp. 247–257. Springer, Singapore (2016). CrossRefGoogle Scholar
  8. 8.
    Plimpton, S.: Fast parallel algorithms for short-range molecular dynamics. J. Comput. Phys. 117(1), 119 (1995)CrossRefzbMATHGoogle Scholar
  9. 9.
    Driscoll, M., Georganas, E., Koanantakool, P., Solomonik, E., Yelick, K.: A communication-optimal n-body algorithm for direct interactions. In: Proceedings of the IEEE 27th International Symposium on in Parallel and Distributed Processing (IPDPS), pp. 1075–1084. IEEE (2013)Google Scholar
  10. 10.
    Doerfler, D., Brightwell, R.: Measuring MPI send and receive overhead and application availability in high performance network interfaces. In: Mohr, B., Träff, J.L., Worringen, J., Dongarra, J. (eds.) EuroPVM/MPI 2006. LNCS, vol. 4192, pp. 331–338. Springer, Heidelberg (2006). CrossRefGoogle Scholar
  11. 11.
    Mishra, P., Somani, A.K.: Host managed contention avoidance storage solutions for Big Data. J. Big Data 4(1), 18 (2017)CrossRefGoogle Scholar
  12. 12.
    Ozkural, E., Aykanat, C: 1-D and 2-D parallel algorithms for all-pairs similarity problem. CoRR abs/1402.3010 (2014)Google Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  • Venkata Kasi Viswanath Yeleswarapu
    • 1
    Email author
  • Arun K. Somani
    • 1
  1. 1.Department of Electrical and Computer EngineeringIowa State UniversityAmesUSA

Personalised recommendations