Abstract
Recently, businesses have started using MapReduce as a popular computation framework for processing large amount of data, such as spam detection, and different data mining tasks, in both public and private clouds. Two of the challenging questions in such environments are (1) choosing suitable values for MapReduce configuration parameters – e.g., number of mappers, number of reducers, and DFS block size–, and (2) predicting the amount of resources that a user should lease from the service provider. Currently, the tasks of both choosing configuration parameters and estimating required resources are solely the users’ responsibilities. In this paper, we present an approach to provision the total CPU usage in clock cycles of jobs in MapReduce environment. For a MapReduce job, a profile of total CPU usage in clock cycles is built from the job past executions with different values of two configuration parameters e.g., number of mappers, and number of reducers. Then, a polynomial regression is used to model the relation between these configuration parameters and total CPU usage in clock cycles of the job. We also briefly study the influence of input data scaling on measured total CPU usage in clock cycles. This derived model along with the scaling result can then be used to provision the total CPU usage in clock cycles of the same jobs with different input data size. We validate the accuracy of our models using three realistic applications (WordCount, Exim MainLog parsing, and TeraSort). Results show that the predicted total CPU usage in clock cycles of generated resource provisioning options are less than 8% of the measured total CPU usage in clock cycles in our 20-node virtual Hadoop cluster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Hadoop example for Exim logs with Python, http://blog.gnucom.cc/2010/hadoop-example-for-exim-logs-with-python/
Rizvandi, N.B., Boloori, A.J., Kamyabpour, N., Zomaya, A.: MapReduce Implementation of Prestack Kirchhoff Time Migration (PKTM) on Seismic Data. Presented at the The 12th International Conference on Parallel and Distributed Computing, Applications and Technologies (PDCAT), Gwangju, Korea (2011)
Arumugam, K., Tan, Y.S., Lee, B.S., Kanagasabai, R.: “Cloud-enabling Sequence Alignment with Hadoop MapReduce: A Performance Analysis. Presented at the 2012 4th International Conference on Bioinformatics and Biomedical Technology (2012)
Kavulya, S., Tan, J., Gandhi, R., Narasimhan, P.: An Analysis of Traces from a Production MapReduce Cluster. Presented at the Proceedings of the 2010 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing (2010)
Kambatla, K., Pathak, A., Pucha, H.: Towards Optimizing Hadoop Provisioning in the Cloud. Presented at the the 2009 Conference on Hot Topics in Cloud Computing, San Diego, California (2009)
Zaharia, M., Konwinski, A., Joseph, A.D., Katz, R., Stoica, I.: Improving MapReduce Performance in Heterogeneous Environments. In: 8th USENIX Symposium on Operating Systems Design and Implementation (OSDI 2008), December 18, pp. 29–42 (2008)
Verma, A., Cherkasova, L., Campbell, R.H.: Resource Provisioning Framework for MapReduce Jobs with Performance Goals. In: Kon, F., Kermarrec, A.-M. (eds.) Middleware 2011. LNCS, vol. 7049, pp. 165–186. Springer, Heidelberg (2011)
Chen, Y., Ganapathi, A.S., Fox, A., Katz, R.H., Patterson, D.A.: Statistical Workloads for Energy Efficient MapReduce. University of California at Berkeley, Technical Report No. UCB/EECS-2010-6 (2010)
Karloff, H., Suri, S., Vassilvitskii, S.: A model of computation for MapReduce. Presented at the Proceedings of the Twenty-First Annual ACM-SIAM Symposium on Discrete Algorithms, Austin, Texas (2010)
Kambatla, K., Pathak, A., Pucha, H.: Towards optimizing hadoop provisioning in the cloud. Presented at the Proceedings of the 2009 Conference on Hot Topics in Cloud Computing, San Diego, California (2009)
Rizvandi, N.B., Taheri, J., Zomaya, A.Y., Moraveji, R.: A Study on Using Uncertain Time Series Matching Algorithms in Map-Reduce Applications. In: Concurrency and Computation: Practice and Experience (2012)
Wieder, A., Bhatotia, P., Post, A., Rodrigues, R.: Brief Announcement: Modelling MapReduce for Optimal Execution in the Cloud. Presented at the Proceeding of the 29th ACM SIGACT-SIGOPS Symposium on Principles of Distributed Computing, Zurich, Switzerland (2010)
Wieder, A., Bhatotia, P., Post, A., Rodrigues, R.: Conductor: orchestrating the clouds. Presented at the 4th International Workshop on Large Scale Distributed Systems and Middleware, Zurich, Switzerland (2010)
Oprescu, A.-M., Kielmann, T.: Bag-of-Tasks Scheduling under Budget Constraints. Presented at the IEEE Second International Conference on Cloud Computing Technology and Science (CloudCom), Indianapolis, IN, U.S.A (2010)
Wood, T., Cherkasova, L., Ozonat, K., Shenoy, P.D.: Profiling and Modeling Resource Usage of Virtualized Applications. In: Issarny, V., Schantz, R. (eds.) Middleware 2008. LNCS, vol. 5346, pp. 366–387. Springer, Heidelberg (2008)
Rizvandi, N.B., Nabavi, A., Hessabi, S.: An Accurate Fir Approximation of Ideal Fractional Delay Filter with Complex Coefficients in Hilbert Space. Journal of Circuits, Systems, and Computers 14, 497–506 (2005)
Chisnall, D.: The definitive guide to the xen hypervisor, 1st edn. Prentice Hall Press (2007)
Hadoop-0.20.2, http://www.apache.org/dyn/closer.cgi/hadoop/core
Sysstat-9.1.6, http://perso.orange.fr/sebastien.godard/
Optimizing Hadoop Deployments, Intel Corporation (2009)
Mao, A., Morris, R., Kaashoek, M.F.: Optimizing MapReduce for Multicore Architectures. Massachusetts Institute of Technology (2010)
Babu, S.: Towards automatic optimization of MapReduce programs. Presented at the 1st ACM Symposium on Cloud Computing, Indianapolis, Indiana, USA (2010)
Sort Benchmark Home Page, http://sortbenchmark.org/
Wang, G., Butt, A.R., Pandey, P., Gupta, K.: A Simulation Approach to Evaluating Design Decisions in MapReduce Setups. Presented at the MASCOTS (2009)
Wang, G., Butt, A.R., Pandey, P., Gupta, K.: Using realistic simulation for performance analysis of mapreduce setups. Presented at the Proceedings of the 1st ACM Workshop on Large-Scale System and Application Performance, Garching, Germany (2009)
Moise, D., Trieu, T.-T.-L., Boug, L., #233, Antoniu, G.: Optimizing intermediate data management in MapReduce computations. Presented at the Proceedings of the First International Workshop on Cloud Computing Platforms, Salzburg, Austria (2011)
Gillick, D., Faria, A., DeNero, J.: MapReduce: Distributed Computing for Machine Learning, www.icsi.berkeley.edu/~arlo/publications/gillick_cs262a_proj.pdf2008
Rizvandi, N.B., Taheri, J., Zomaya, A.Y.: On using Pattern Matching Algorithms in MapReduce Applications. Presented at the The 9th IEEE International Symposium on Parallel and Distributed Processing with Applications (ISPA), Busan, South Korea (2011)
Islam, S., Keung, J., Lee, K., Liu, A.: Empirical prediction models for adaptive resource provisioning in the cloud. Future Generation Comp. Syst. 28, 155–162 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Rizvandi, N.B., Taheri, J., Moraveji, R., Zomaya, A.Y. (2012). On Modelling and Prediction of Total CPU Usage for Applications in MapReduce Environments. In: Xiang, Y., Stojmenovic, I., Apduhan, B.O., Wang, G., Nakano, K., Zomaya, A. (eds) Algorithms and Architectures for Parallel Processing. ICA3PP 2012. Lecture Notes in Computer Science, vol 7439. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-33078-0_30
Download citation
DOI: https://doi.org/10.1007/978-3-642-33078-0_30
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-33077-3
Online ISBN: 978-3-642-33078-0
eBook Packages: Computer ScienceComputer Science (R0)