Skip to main content
SpringerLink
Log in

Hadoop Scalability and Performance Testing in Homogeneous Clusters

  • Conference paper
  • First Online:
Proceedings of ICETIT 2019

Part of the book series: Lecture Notes in Electrical Engineering ((LNEE,volume 605))

Abstract

Big data is a term used to refer to the datasets that are too large (Ex. GBs, TBs, PBs, ZBs, etc.) or complex for traditional data processing application software. Distributed and parallel processing becomes increasingly important for big data. There are two most popular parallel and distributed processing frameworks available, namely Hadoop and Spark. Hadoop and Spark are open-source software frameworks for reliable, scalable, and distributed computing. Hadoop is created by Apache Software Foundation. This framework allows the processing of extremely large datasets on clusters of computers using a simple programming model called MapReduce. It works on a distributed file system called HDFS (Hadoop Distributed File System) to run on commodity hardware. It is designed to scale up horizontally from a single machine to thousands of machines, each offering local computation and storage. Performance of Hadoop cluster depends on the application and several parameters. In this paper we aim to study the performance of Hadoop homogeneous cluster by tuning a few parameters like cluster size, dataset size, and HDFS block size, etc.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 259.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 329.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 329.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. https://hadoop.apache.org/

  2. http://www.cs.rpi.edu/zaki/Workshops/FIMI/data/

  3. Babu, S.: Towards automatic optimization of MapReduce programs. In: Proceedings of the 1st ACM Symposium on Cloud Computing, pp. 137–142. ACM (2010)

    Google Scholar 

  4. Bansal, G., Gupta, A., Pyne, U., Singhal, M., Banerjee, S.: A framework for performance analysis and tuning in hadoop based clusters. In: Smarter Planet and Big Data Analytics Workshop (SPBDA 2014), held in conjunction with International Conference on Distributed Computing and Networking (ICDCN 2014), Coimbatore, India (2014)

    Google Scholar 

  5. Barry, D., Tinetti, F.G., Real, I., Jaramillo, R.: Hadoop scalability and performance testing in heterogeneous clusters, July 2015

    Google Scholar 

  6. Chen, X., Liang, Y., Li, G.R., Chen, C., Liu, S.Y.: Optimizing performance of hadoop with parameter tuning. In: ITM Web of Conferences, vol. 12, p. 03040. EDP Sciences (2017)

    Google Scholar 

  7. Cheng, D., Rao, J., Guo, Y., Jiang, C., Zhou, X.: Improving performance of heterogeneous MapReduce clusters with adaptive task tuning. IEEE Trans. Parallel Distrib. Syst. 28(3), 774–786 (2017)

    Article  Google Scholar 

  8. Dadheech, P., Goyal, D., Srivastava, S., Kumar, A.: Performance improvement of heterogeneous hadoop clusters using query optimization (2018)

    Google Scholar 

  9. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)

    Article  Google Scholar 

  10. Elmagarmid, A.K., Rusinkiewicz, M., Sheth, A., Sheth, A.: Management of Heterogeneous and Autonomous Database Systems. Morgan Kaufmann, Burlington (1999)

    Google Scholar 

  11. Feller, E., Ramakrishnan, L., Morin, C.: Performance and energy efficiency of big data applications in cloud environments: a hadoop case study. J. Parallel Distrib. Comput. 79, 80–89 (2015)

    Article  Google Scholar 

  12. Ghazi, M.R., Gangodkar, D.: Hadoop, MapReduce and HDFS: a developers perspective. Procedia Comput. Sci. 48, 45–50 (2015)

    Article  Google Scholar 

  13. Ghemawat, S., Gobioff, H., Leung, S.T.: The Google file system (2003)

    Google Scholar 

  14. Gopalani, S., Arora, R.: Comparing apache spark and map reduce with performance analysis using k-means. Int. J. Comput. Appl. 113(1), 8–11 (2015)

    Google Scholar 

  15. Guo, S.: Hadoop Operations and Cluster Management Cookbook. Packt Publishing Ltd., Birmingham (2013)

    Google Scholar 

  16. Holmes, A.: Hadoop in Practice. Manning Publications Co., New York (2012)

    Google Scholar 

  17. Ibrahim, S., Phan, T.D., Carpen-Amarie, A., Chihoub, H.E., Moise, D., Antoniu, G.: Governing energy consumption in hadoop through cpu frequency scaling: an analysis. Future Gener. Comput. Syst. 54, 219–232 (2016)

    Article  Google Scholar 

  18. Liu, F.H., Liou, Y.R., Lo, H.F., Chang, K.C., Lee, W.T.: The comprehensive performance rating for hadoop clusters on cloud computing platform. Int. J. Inf. Electron. Eng. 4(6), 480 (2014)

    Google Scholar 

  19. Maurya, M., Mahajan, S.: Performance analysis of MapReduce programs on hadoop cluster. In: 2012 World Congress on Information and Communication Technologies, pp. 505–510. IEEE (2012)

    Google Scholar 

  20. Mavridis, I., Karatza, H.: Performance evaluation of cloud-based log file analysis with apache hadoop and apache spark. J. Syst. Softw. 125, 133–151 (2017)

    Article  Google Scholar 

  21. Ousterhout, K., Rasti, R., Ratnasamy, S., Shenker, S., Chun, B.G.: Making sense of performance in data analytics frameworks. In: 12th fUSENIXg Symposium on Networked Systems Design and Implementation (fNSDIg 2015), pp. 293–307 (2015)

    Google Scholar 

  22. Ozsu, M.T., Valduriez, P.: Principles of Distributed Database Systems. Springer, New York (2011)

    Google Scholar 

  23. Pal, A., Jain, K., Agrawal, P., Agrawal, S.: A performance analysis of MapReduce task with large number of files dataset in big data using hadoop. In: 2014 Fourth International Conference on Communication Systems and Network Technologies, pp. 587–591. IEEE (2014)

    Google Scholar 

  24. Parsian, M.: Data Algorithms: Recipes for Scaling Up with Hadoop and Spark. O’Reilly Media Inc., Boston (2015)

    Google Scholar 

  25. Ren, Z., Wan, J., Shi, W., Xu, X., Zhou, M.: Workload analysis, implications, and optimization on a production hadoop cluster: a case study on taobao. IEEE Trans. Serv. Comput. 7(2), 307–321 (2014)

    Article  Google Scholar 

  26. Rizki, R., Rakhmatsyah, A., Nugroho, M.A.: Performance analysis of container based hadoop cluster: Openvz and LXC. In: 2016 4th International Conference on Information and Communication Technology (ICoICT), pp. 1–4. IEEE (2016)

    Google Scholar 

  27. Shafer, J., Rixner, S., Cox, A.L.: The hadoop distributed file system: balancing portability and performance. In: 2010 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pp. 122–133. IEEE (2010)

    Google Scholar 

  28. Singh, M., Ali, A.: Big Data Analytics with Microsoft HDInsight in 24 Hours, Sams Teach Yourself. Sams Publishing, Indianapolis (2015)

    Google Scholar 

  29. Song, G., Meng, Z., Huet, F., Magoules, F., Yu, L., Lin, X.: A hadoop MapReduce performance prediction method. In: 2013 IEEE 10th International Conference on High Performance Computing and Communications and 2013 IEEE International Conference on Embedded and Ubiquitous Computing, pp. 820–825. IEEE (2013)

    Google Scholar 

  30. Wang, G., Butt, A.R., Pandey, P., Gupta, K.: Using realistic simulation for performance analysis of MapReduce setups. In: Proceedings of the 1st ACM Workshop on Large-Scale System and Application Performance, pp. 19–26. ACM (2009)

    Google Scholar 

  31. White, T.: Hadoop: The Definitive Guide. O’Reilly Media Inc., Sebastopol (2012)

    Google Scholar 

  32. Wu, D., Gokhale, A.: A self-tuning system based on application profiling and performance analysis for optimizing hadoop MapReduce cluster configuration. In: 20th Annual International Conference on High Performance Computing, pp. 89–98. IEEE (2013)

    Google Scholar 

  33. Xu, Z., Shi, Y.: Exploring big data analysis: fundamental scientific problems. Ann. Data Sci. 2(4), 363–372 (2015)

    Article  Google Scholar 

  34. Zhang, C., De Sterck, H., Aboulnaga, A., Djambazian, H., Sladek, R.: Case study of scientific data processing on a cloud using hadoop. In: International Symposium on High Performance Computing Systems and Applications, pp. 400–415. Springer (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. B V Raju Institute of Technology, Narsapur, 502313, Telangana, India

    Chiranjeevi Manike, Ashok Kumar Nanda & Tejashwini Gajulagudem

Authors
  1. Chiranjeevi Manike
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ashok Kumar Nanda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tejashwini Gajulagudem
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chiranjeevi Manike .

Editor information

Editors and Affiliations

  1. Department of Computer Science and Engineering, Jaypee University of Information Technology, Kandaghat, India

    Pradeep Kumar Singh

  2. Department of Electrical Engineering, IIT Delhi, Delhi, Delhi, India

    Bijaya Ketan Panigrahi

  3. School of Computer and Information Sciences, University of Hyderabad, Hyderabad, Telangana, India

    Nagender Kumar Suryadevara

  4. Institute of Information Technology and Management, Delhi, Delhi, India

    Sudhir Kumar Sharma

  5. USICT, GGSIPU, Delhi, Delhi, India

    Amit Prakash Singh

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Manike, C., Nanda, A.K., Gajulagudem, T. (2020). Hadoop Scalability and Performance Testing in Homogeneous Clusters. In: Singh, P., Panigrahi, B., Suryadevara, N., Sharma, S., Singh, A. (eds) Proceedings of ICETIT 2019. Lecture Notes in Electrical Engineering, vol 605. Springer, Cham. https://doi.org/10.1007/978-3-030-30577-2_81

Download citation

Publish with us

Policies and ethics

Search

Navigation