Skip to main content
SpringerLink
Log in

Analyzing the impact of various parameters on job scheduling in the Google cluster dataset

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Cloud architecture and its operations interest both general consumers and researchers. Google, as a technology giant, offers cloud services globally. This paper analyzes the Google cluster usage trace, focusing on three key aspects: task execution times, rescheduling frequency, and the relationship between task priority and rescheduling. Firstly, we examine how memory and processor performance impact task execution times across different machines. Next, we investigate how the number of task constraints influences rescheduling frequency and overall environmental efficiency. Furthermore, we analyze how task priority affects rescheduling and explore its correlation with task constraints. The results reveal that doubling the memory size can accelerate tasks by a factor of nine and that 90% of rescheduling is associated with tasks having less than seven constraints. We aim to enhance data center performance by identifying bottlenecks in the Google Cluster Dataset and providing recommendations for all cloud service providers. Our key findings indicate that memory plays a more significant role than the processor, and tasks with higher constraints have a less pronounced impact on rescheduling than anticipated.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Khaledian, N., Khamforoosh, K., Azizi, S., Maihami, V.: IKH-EFT: an improved method of workflow scheduling using the krill herd algorithm in the fog-cloud environment. Sustain. Comput.: Inform. Syst. 37, 100834 (2023)

    Google Scholar 

  2. Rosà, A., Chen, L.Y., Birke, R., Binder, W.: Demystifying casualties of evictions in big data priority scheduling. ACM SIGMETRICS Perform. Eval. Rev. 42(4), 12–21 (2015)

    Article  Google Scholar 

  3. Chen, X., Lu, C. D., Pattabiraman, K.: Failure analysis of jobs in compute clouds: a Google cluster case study. In 2014 IEEE 25th International Symposium on Software Reliability Engineering (pp. 167–177). IEEE. (2014)

  4. Rzadca, K., Findeisen, P., Swiderski, J., Zych, P., Broniek, P., Kusmierek, J., Wilkes, J.: Autopilot: workload autoscaling at Google. In proceedings of the fifteenth european conference on computer systems (pp. 1–16), (2020)

  5. Anil, R., Capan, G., Drost-Fromm, I., Dunning, T., Friedman, E., Grant, T., Yılmazel, Ö.: Apache mahout: machine learning on distributed dataflow systems. J. Mach. Learn. Res. 21(127), 1–6 (2020)

    Google Scholar 

  6. Gévay, G.E., Soto, J., Markl, V.: Handling iterations in distributed dataflow systems. ACM Comput. Surv. (CSUR) 54(9), 1–38 (2021)

    Article  Google Scholar 

  7. Tirmazi, M., Barker, A., Deng, N., Haque, M.E., Qin, Z.G., Hand, S., Wilkes, J.: Borg: the next generation. In proceedings of the fifteenth european conference on computer systems (pp. 1–14), (2020)

  8. Fernández-Cerero, D., Varela-Vaca, Á.J., Fernández-Montes, A., Gómez-López, M.T., Alvárez-Bermejo, J.A.: Measuring data-centre workflows complexity through process mining: the Google cluster case. J. Supercomput. 76, 2449–2478 (2020)

    Article  Google Scholar 

  9. Gog, I., Schwarzkopf, M., Gleave, A., Watson, R.N., Hand, S.: Firmament: Fast, centralized cluster scheduling at scale. In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI 16) (pp. 99–115), (2016)

  10. Fernández Cerero, D., Fernández Montes González, A., Jakóbik, A., Kolodziej, J.: Stackelberg game-based models in energy-aware cloud scheduling. In ECMS 2018: 32nd European Conference on Modelling and Simulation (2018). European Council for Modelling and Simulation, (2018)

  11. Khaledian, N., Khamforoosh, K., Akraminejad, R., Abualigah, L., Javaheri, D.: An energy-efficient and deadline-aware workflow scheduling algorithm in the fog and cloud environment. Computing 106(1), 109–137 (2024)

    Article  Google Scholar 

  12. Fernández-Cerero, D., Jakóbik, A., Grzonka, D., Kołodziej, J., Fernández-Montes, A.: Security supportive energy-aware scheduling and energy policies for cloud environments. J. Parallel Distrib. Comput. 119, 191–202 (2018)

    Article  Google Scholar 

  13. Maala, H.H., Yousif, S.A.: Cluster trace analysis for performance enhancement in cloud computing environments. J. Theor. Appl. Inf. Technol. 97(7), 2019 (2019)

    Google Scholar 

  14. R. Koch, "The 20/80Principle: the secret of achieving more with less.," Doubleday, (1999)

  15. Van Loo, T., Jindal, A., Benedict, S., Chadha, M., Gerndt, M.: Scalable infrastructure for workload characterization of cluster traces. (2022), arXiv preprint

  16. Adil, I.H., Wahid, A., Mantell, E.H.: Split sample skewness. Commun. Stat. Theory Methods 50(22), 5171–5188 (2021)

    Article  MathSciNet  Google Scholar 

  17. Olabisi, D., Abubakar, S.K., Abdullahi, A.T.: demystifying dew computing: concept, architecture and research opportunities. Int. J. Comput. Trends Technol. 70, 39–43 (2022)

    Article  Google Scholar 

  18. Reiss, C., Tumanov, A., Ganger, G.R., Katz, R.H., Kozuch, M.A.: Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In Proceedings of the third ACM symposium on cloud computing (pp. 1–13), (2012)

  19. Umer, A., Mian, A.N., Rana, O.: Predicting machine behavior from Google cluster workload traces. Concurr. Comput.: Pract. Exp. 35(5), e7559 (2023)

    Article  Google Scholar 

  20. Jassas, M. S., Mahmoud, Q. H.: Failure characterization and prediction of scheduling jobs in Google cluster traces. In 2019 IEEE 10th GCC Conference & Exhibition (GCC) (pp. 1–7). IEEE. (2019)

  21. Wang, H., Jiang, C., Xie, B.: Missing data analysis and prediction: a Google cluster case study. (2022)

  22. Ngang'a, D.N., Cheruiyot, W.K., Njagi, D. A Machine Learning Framework for Predicting Failures in Cloud Data Centers-A Case of Google Cluster-Azure Clouds and Alibaba Clouds. Available at SSRN 4404569

  23. Soualhia, M., Khomh, F., Tahar, S.: Predicting scheduling failures in the cloud: A case study with Google clusters and Hadoop on Amazon EMR. In 2015 IEEE 17th International Conference on High Performance Computing and Communications, 2015 IEEE 7th International Symposium on Cyberspace Safety and Security, and 2015 IEEE 12th International Conference on Embedded Software and Systems (pp. 58–65). IEEE. (2015)

  24. Chen, S., Yang, C., Huang, W., Liang, W., Ke, N., Souri, A., Li, K.C.: Fairness constraint efficiency optimization for multiresource allocation in a cluster system serving internet of things. Int. J. Commun. Syst. 36(3), e5395 (2023)

    Article  Google Scholar 

  25. Wilkes, J.: More Google cluster data. Google research blog, Nov, (2011)

  26. Gupta, S., Dileep, A.D.: Long range dependence in cloud servers: a statistical analysis based on Google workload trace. Computing 102(4), 1031–1049 (2020)

    Article  MathSciNet  Google Scholar 

  27. Subramanian, N.V., Sriram, V.S.: Load-aware VM migration using hypergraph based CDB-LSTM. Intell. Autom. Soft Comput. 35(3), 3279–3294 (2023)

    Article  Google Scholar 

  28. Berisha, B., Mëziu, E., Shabani, I.: Big data analytics in Cloud computing: an overview. J. Cloud Comput. 11(1), 24 (2022)

    Article  Google Scholar 

  29. Osborne, J.W., Overbay, A.: The power of outliers (and why researchers should always check for them). Pract. Assess. Res. Eval. 9(1), 6 (2019)

    Google Scholar 

  30. Seo, S.: A review and comparison of methods for detecting outliers in univariate data sets (Doctoral dissertation, University of Pittsburgh), (2006)

  31. Brys, G., Hubert, M., Struyf, A.: A robust measure of skewness. J. Comput. Graph. Stat. 13(4), 996–1017 (2004)

    Article  MathSciNet  Google Scholar 

  32. Tawhid, A., Teotia, T., Elmiligi, H.: Machine Learning for Optimizing Healthcare Resources Machine Learning, Big Data, and IoT for Medical Informatics, pp. 215–239. Academic Press, Cambridge (2021)

    Book  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Graduate School of Engineering Science and Technology, National Yunlin University of Science and Technology, 123 University Road, Douliou, 64002, Yunlin, Taiwan

    Danyal Shahmirzadi

  2. Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Esch-sur-Alzette, Luxembourg

    Navid Khaledian

  3. Future Technology Research Center, National Yunlin University of Science and Technology, Douliou, Yunlin, Taiwan

    Amir Masoud Rahmani

Authors
  1. Danyal Shahmirzadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Navid Khaledian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amir Masoud Rahmani
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

"D.S: Writing original draft, Methodology, Software, N.K: preparation, visualization; A.M.R.: conceptualization, validation, supervision, and investigation. All authors reviewed the manuscript."

Corresponding author

Correspondence to Amir Masoud Rahmani.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shahmirzadi, D., Khaledian, N. & Rahmani, A.M. Analyzing the impact of various parameters on job scheduling in the Google cluster dataset. Cluster Comput (2024). https://doi.org/10.1007/s10586-024-04377-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10586-024-04377-8

Keywords

Search

Navigation