Multilevel Data Processing Using Parallel Algorithms for Analyzing Big Data in High-Performance Computing

  • Awais Ahmad
  • Anand Paul
  • Sadia Din
  • M. Mazhar Rathore
  • Gyu Sang Choi
  • Gwanggil Jeon
Part of the following topical collections:
  1. Special Issue on Programming Models and Algorithms for Data Analysis in HPC Systems


The growing gap between users and the Big Data analytics requires innovative tools that address the challenges faced by big data volume, variety, and velocity. Therefore, it becomes computationally inefficient to analyze such massive volume of data. Moreover, advancements in the field of Big Data application and data science poses additional challenges, where High-Performance Computing solution has become a key issue and has attracted attention in recent years. However, these systems are either memoryless or computational inefficient. Therefore, keeping in view the aforementioned needs, there is a requirement for a system that can efficiently analyze a stream of Big Data within their requirements. Hence, this paper presents a system architecture that enhances the working of traditional MapReduce by incorporating parallel processing algorithm. Moreover, complete four-tier architecture is also proposed that efficiently aggregate the data, eliminate unnecessary data, and analyze the data by the proposed parallel processing algorithm. The proposed system architecture both read and writes operations that enhance the efficiency of the Input/Output operation. To check the efficiency of the proposed algorithms exploited in the proposed system architecture, we have implemented our proposed system using Hadoop and MapReduce. MapReduce is supported by a parallel algorithm that efficiently processes a huge volume of data sets. The system is implemented using MapReduce tool at the top of the Hadoop parallel nodes to generate and process graphs with near real-time. Moreover, the system is evaluated in terms of efficiency by considering the system throughput and processing time. The results show that the proposed system is more scalable and efficient.


Big Data HPC Parallel Processing algorithm Four-tier system architecture 



This work is supported by BK21 Plus project (SW Human Resource Development Program for Supporting Smart Life) funded by the Ministry of Education, School of Computer Science and Engineering, Kyungpook National University, Korea (21A20131600005) and NRF Grant funded by the Korean Government (NRF-2015R1D1A1A01058171).


  1. 1.
    Ahmad, A., Paul, A., Rathore, M.M.: An efficient divide-and-conquer approach for big data analytics in machine-to-machine communication. Neurocomputing 174, 439–453 (2016)CrossRefGoogle Scholar
  2. 2.
    NOAA. Overview of Current Atmospheric Reanalysis. (2016)
  3. 3.
    Ahmad, A., Paul, A., Rathore, M., Chang, H.: An efficient multidimensional big data fusion approach in machine-to-machine communication. ACM Trans. Embed. Comput. Syst. (TECS) 15(2), 39 (2016)Google Scholar
  4. 4.
    Rathore, M.M., Ullah, A.P., Ahmad, A., Chen, B.-W., Huang, B., Ji, W.: Real-time big data analytical architecture for remote sensing application. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 8(10), 4610–4621 (2015)CrossRefGoogle Scholar
  5. 5.
    Haderer, N., Romain, R., Seinturier, L.: Dynamic deployment of sensing experiments in the wild using smartphones. In: IFIP International Conference on Distributed Applications and Interoperable Systems, pp. 43–56. Springer, Berlin, Heidelberg (2013)Google Scholar
  6. 6.
    Mosser, S., Fleurey, F., Morin, B., Chauvel, F., Solberg, A., Goutier, I.: Sensapp as a reference platform to support cloud experiments: from the internet of things to the internet of services. In: 2012 14th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp. 400–406. IEEE (2012)Google Scholar
  7. 7.
    Mosser, S., Logre, I., Ferry, N., Collet, P.: From sensors to visualization dashboards: need for language composition. In: Globalization of Modeling Languages workshop (GeMOC’13) (2013)Google Scholar
  8. 8.
    Awais, A., Paul, A., Rathore, M.M., Chang, H.: Smart cyber society: integration of capillary devices with high usability based on cyber–physical system. Future Gen. Comput. Syst. 56, 493–503 (2016)CrossRefGoogle Scholar
  9. 9.
    Labrinidis, A., Jagadish, H.V.: Challenges and opportunities with big data. Proc. VLDB Endow. 5(12), 2032–2033 (2012)CrossRefGoogle Scholar
  10. 10.
    Chen, C., Lang, M., Chen, Y.: Multilevel active storage for big data applications in high performance computing. In: 2013 IEEE International Conference on Big Data, pp. 169–174. IEEE (2013)Google Scholar
  11. 11.
    Felix, E.J., Fox, K., Regimbal, K., Nieplocha, J.: Active storage processing in a parallel file system. In: Proceedings of the 6th LCI International Conference on Linux Clusters: The HPC Revolution, p. 85 (2006)Google Scholar
  12. 12.
    Thakur, R., Gropp, W., Lusk, E.: Data sieving and collective I/O in ROMIO. In: The Seventh Symposium on the Frontiers of Massively Parallel Computation, 1999. Frontiers’ 99, pp. 182–189. IEEE (1999)Google Scholar
  13. 13.
    Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  14. 14.
    Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google file system. ACM SIGOPS Oper. Syst. Rev. 37(5), 29–43 (2003)CrossRefGoogle Scholar
  15. 15.
    Yoo, R.M., Romano, A., Kozyrakis, C.: Phoenix rebirth: Scalable MapReduce on a large-scale shared-memory system. In: IEEE International Symposium on Workload Characterization, 2009. IISWC 2009, pp. 198–207. IEEE (2009)Google Scholar
  16. 16.
    Ranger, C., Raghuraman, R., Penmetsa, A., Bradski, G., Kozyrakis, C.: Evaluating mapreduce for multi-core and multiprocessor systems. In: 2007 IEEE 13th International Symposium on High Performance Computer Architecture, pp. 13–24. IEEE (2007)Google Scholar
  17. 17.
    Rafique, M.M., Rose, B., Butt, A.R., Nikolopoulos, D.S.: Supporting MapReduce on large-scale asymmetric multi-core clusters. ACM SIGOPS Oper. Syst. Rev. 43(2), 25–34 (2009)CrossRefGoogle Scholar
  18. 18.
    Lee, K.H., Lee, Y.J., Choi, H., Chung, Y.D., Moon, B.: Parallel data processing with MapReduce: a survey. AcM sIGMoD Rec. 40(4), 11–20 (2012)CrossRefGoogle Scholar
  19. 19.
    Shim, K.: MapReduce algorithms for big data analysis. Proc. VLDB Endow. 5(12), 2016–2017 (2012)CrossRefGoogle Scholar
  20. 20.
    Panda, B., Herbach, J.S., Basu, S., Bayardo, R.J.: Planet: massively parallel learning of tree ensembles with mapreduce. Proc. VLDB Endow. 2(2), 1426–1437 (2009)CrossRefGoogle Scholar
  21. 21.
    Dean, J., Ghemawat, S.: MapReduce: a flexible data processing tool. Commun. ACM 53(1), 72–77 (2010)CrossRefGoogle Scholar
  22. 22.
    Ekanayake, J., Pallickara, S., Fox, G.: Mapreduce for data intensive scientific analyses. In: IEEE Fourth International Conference on eScience, 2008. eScience’08, pp. 277–284. IEEE (2008)Google Scholar
  23. 23.
    Rathore, M.M., Ahmad, A., Paul, A., Rho, S.: Exploiting encrypted and tunneled multimedia calls in high-speed big data environment. Multimed. Tools Appl. 1–26 (2017)Google Scholar
  24. 24.
    Paul, A., Ahmad, A., Rathore, M.M., Jabbar, S.: Smartbuddy: defining human behaviors using big data analytics in social internet of things. IEEE Wirel. Commun. 23(5), 68–74 (2016)CrossRefGoogle Scholar
  25. 25.
    Rathore, M.M., Paul, A., Ahmad, A., Jeon, G.: IoT-based big data: from smart city towards next generation super city planning. Int. J. Semant. Web Inf. Syst. 13(1), 28–47 (2017)CrossRefGoogle Scholar
  26. 26.
    Shvachko, K., Kuang, H., Radia, S., Chansler, R.: The hadoop distributed file system. In: 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–10. IEEE (2010)Google Scholar
  27. 27.
  28. 28.
    Gordon at San Diego Supercomputer Center.
  29. 29.
    Gropp, W., Lusk, E., Sterling, T.: Enabling Technologies in Beowulf Cluster Computing with Linux, 2nd edn, vol. 3. The MIT Press, Cambridge, MA, London, England, p. 14 (2003)Google Scholar
  30. 30.
    Sterling, T.L., Salmon, J., Becker, D.J., Savarese, D.F.: How to Build a Beowulf: A Guide to the Implementation and Application of PC Clusters. MIT Press, Cambridge, MA (1999)Google Scholar
  31. 31.
    Engelmann, C., Ong, H., Scott, S.L.: Middleware in modern high performance computing system architectures. In: International Conference on Computational Science, pp. 784–791. Springer, Berlin, Heidelberg (2007)Google Scholar
  32. 32.
    Castain, R.H., Kulkarni, O.: MapReduce and Lustre: Running Hadoop in a High Performance Computing Environment.
  33. 33.
    Wasi-ur Rahman, Md., Lu, X., Islam, N.S., Rajachandrasekar, R., Panda, D.K.: MapReduce over Lustre: Can RDMA-Based Approach Benefit? In: tEuropean Conference on Parallel Processing, pp. 644–655. Springer, Berlin (2014)Google Scholar
  34. 34.
    Wasi-ur-Rahman, Md., Islam, N.S., Lu, X., Jose, J., Subramoni, H., Wang, H., Panda, D.K.: High-performance RDMA-based design of Hadoop MapReduce over InfiniBand. In: 2013 IEEE 27th International Parallel and Distributed Processing Symposium Workshops and PhD Forum (IPDPSW), pp. 1908–1917. IEEE (2013)Google Scholar
  35. 35.
    Wasi-ur Rahman, Md., Lu, X., Islam, N.S., Panda, D.K.: HOMR: a hybrid approach to exploit maximum overlapping in MapReduce over high performance interconnects. In: Proceedings of the 28th ACM international conference on Supercomputing, pp. 33–42. ACM (2014)Google Scholar
  36. 36.
    Lu, X., Islam, N.S., Wasi-Ur-Rahman, Md., Jose, J., Subramoni, H., Wang, H., Panda, D.K.: High-performance design of Hadoop RPC with RDMA over InfiniBand. In: 2013 42nd International Conference on Parallel Processing, pp. 641–650. IEEE (2013). doi: 10.1109/ICPP.2013.78
  37. 37.
    Available online: 14/10/2014, 2312.
  38. 38.
    ESA: ENVISAT Altimetry Level 2 User Manual V1.4 2011. [Available online: 15/10/2014, 0333]

Copyright information

© Springer Science+Business Media New York 2017

Authors and Affiliations

  1. 1.Department of Information and Communication EngineeringYeungnam UniversityGyeongbukRepublic of Korea
  2. 2.School of Computer Science and EngineeringKyungpook National UniversityDaeguRepublic of Korea
  3. 3.Department of Embedded Systems EngineeringIncheon National UniversityIncheonKorea

Personalised recommendations