Advertisement

ABS-YARN: A Formal Framework for Modeling Hadoop YARN Clusters

  • Jia-Chun Lin
  • Ingrid Chieh Yu
  • Einar Broch Johnsen
  • Ming-Chang Lee
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9633)

Abstract

In cloud computing, software which does not flexibly adapt to deployment decisions either wastes operational resources or requires reengineering, both of which may significantly increase costs. However, this could be avoided by analyzing deployment decisions already during the design phase of the software development. Real-Time ABS is a formal language for executable modeling of deployed virtualized software. Using Real-Time ABS, this paper develops a generic framework called ABS-YARN for YARN, which is the next generation of the Hadoop cloud computing platform with a state-of-the-art resource negotiator. We show how ABS-YARN can be used for prototyping YARN and for modeling job execution, allowing users to rapidly make deployment decisions at the modeling level and reduce unnecessary costs. To validate the modeling framework, we show strong correlations between our model-based analyses and a real YARN cluster in different scenarios with benchmarks.

Keywords

Execution Time Cloud Computing Virtual Machine Slave Node Concurrent Object 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgement

The authors thank NCLab at National Chiao Tung University, Taiwan for providing computation facilities for the YARN cluster used in our experiments.

References

  1. 1.
    Agha, G.A.: ACTORS: A Model of Concurrent Computations in Distributed Systems. The MIT Press, Cambridge (1986)Google Scholar
  2. 2.
    Albert, E., Arenas, P., Flores-Montoya, A., Genaim, S., Gómez-Zamalloa, M., Martin-Martin, E., Puebla, G., Román-Díez, G.: SACO: static analyzer for concurrent objects. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014 (ETAPS). LNCS, vol. 8413, pp. 562–567. Springer, Heidelberg (2014)CrossRefGoogle Scholar
  3. 3.
    Albert, E., de Boer, F.S., Hähnle, R., Johnsen, E.B., Schlatte, R., Tapia Tarifa, S.L., Wong, P.Y.H.: Formal modeling and analysis of resource management for cloud architectures: An industrial case study using Real-Time ABS. J. Serv. Oriented Comput. Appl. 8(4), 323–339 (2014)CrossRefGoogle Scholar
  4. 4.
    Amnell, T., Fersman, E., Mokrushin, L., Pettersson, P., Yi, W.: TIMES: a tool for schedulability analysis and code generation of real-time systems. In: Larsen, K.G., Niebert, P. (eds.) FORMATS 2003. LNCS, vol. 2791, pp. 60–72. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  5. 5.
  6. 6.
    Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., Zaharia, M.: A view of cloud computing. Commun. ACM 53(4), 50–58 (2010)CrossRefGoogle Scholar
  7. 7.
    Armstrong, J.: Programming Erlang: Software for a Concurrent World. Pragmatic Bookshelf, Raleigh (2007)Google Scholar
  8. 8.
    Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.-P.: Performance evaluation and model checking join forces. Commun. ACM 53(9), 76–85 (2010)CrossRefGoogle Scholar
  9. 9.
    Barbanera, F., Bugliesi, M., Dezani-Ciancaglini, M., Sassone, V.: Space-aware ambients and processes. Theor. Comput. Sci. 373(1–2), 41–69 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Bjørk, J., de Boer, F.S., Johnsen, E.B., Schlatte, R., Tapia, S.L.: Tarifa.: User-defined schedulers for real-time concurrent objects. Innov. Syst. Softw. Eng. 9(1), 29–43 (2013)CrossRefGoogle Scholar
  11. 11.
    Caromel, D., Henrio, L.: A Theory of Distributed Objects. Springer, New York (2005)zbMATHGoogle Scholar
  12. 12.
    Clavel, M., Durán, F., Eker, S., Lincoln, P., Martí-Oliet, N., Meseguer, J., Talcott, C. (eds.): All About Maude - A High-Performance Logical Framework. LNCS, vol. 4350. Springer, Heidelberg (2007)zbMATHGoogle Scholar
  13. 13.
  14. 14.
    Hähnle, R., Johnsen, E.B.: Designing resource-aware cloud applications. IEEE Comput. 48(6), 72–75 (2015)CrossRefGoogle Scholar
  15. 15.
    Hammoud, S., Li, M., Liu, Y., Alham, N.K., Liu, Z.: MRSim: A discrete event based MapReduce simulator. In: Seventh International Conference on Fuzzy Systems and Knowledge Discovery, FSKD 2010, pp. 2993–2997. IEEE (2010)Google Scholar
  16. 16.
    Johnsen, E.B., Hähnle, R., Schäfer, J., Schlatte, R., Steffen, M.: ABS: a core language for abstract behavioral specification. In: Aichernig, B.K., de Boer, F.S., Bonsangue, M.M. (eds.) Formal Methods for Components and Objects. LNCS, vol. 6957, pp. 142–164. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  17. 17.
    Johnsen, E.B., Schlatte, R., Tapia Tarifa, S.L.: Modeling resource-aware virtualized applications for the cloud in real-time ABS. In: Aoki, T., Taguchi, K. (eds.) ICFEM 2012. LNCS, vol. 7635, pp. 71–86. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  18. 18.
    Johnsen, E.B., Schlatte, R., Tapia Tarifa, S.L.: Integrating deployment architectures and resource consumption in timed object-oriented models. J. Log. Algebraic Methods Program. 84(1), 67–91 (2015)CrossRefzbMATHGoogle Scholar
  19. 19.
    Kolberg, W., Marcos, P.D.B., Anjos, J.C., Miyazaki, A.K., Geyer, C.R., Arantes, L.B.: MRSG - a MapReduce simulator over SimGrid. Parallel Comput. 39(4), 233–244 (2013)CrossRefGoogle Scholar
  20. 20.
    Koralov, L.B., Sinai, Y.G.: Theory of Probability and Random Processes. Springer-Verlag, Berling (2007)CrossRefzbMATHGoogle Scholar
  21. 21.
    Liu, Y., Li, M., Alham, N.K., Hammoud, S.: HSim: a MapReduce simulator in enabling cloud computing. Future Gener. Comput. Syst. 29(1), 300–308 (2013)CrossRefGoogle Scholar
  22. 22.
    Luo, C., Zhan, J., Jia, Z., Wang, L., Lu, G., Zhang, L., Xu, C.-Z., Sun, N.: Cloudrank-d: benchmarking and ranking cloud computing systems for data processing applications. Front. Comput. Sci. 6(4), 347–362 (2012)MathSciNetGoogle Scholar
  23. 23.
    Murthy, A., Vavilapalli, V., Eadline, D., Niemiec, J., Markham, J.: Apache Hadoop YARN: Moving Beyond MapReduce and Batch Processing with Apache Hadoop 2. Addison-Wesley Professional, San Francisco (2014)Google Scholar
  24. 24.
    Palanisamy, B., Singh, A., Liu, L., Bryan, L.: Cura: A cost-optimized model for MapReduce in a cloud. In: IEEE 27th International Symposium on Parallel and Distributed Processing, pp. 1275–1286. IEEE (2013)Google Scholar
  25. 25.
    Reiss, C., Tumanov, A., Ganger, G.R., Katz, R.H., Kozuch, M.A.: Towards understanding heterogeneous clouds at scale: Google traceanalysis.Technical Report ISTC-CC-TR-12-101, Intel Science and TechnologyCenter for Cloud Computing, Carnegie Mellon University, April 2012. http://www.pdl.cmu.edu/PDL-FTP/CloudComputing/ISTC-CC-TR-12-101.pdf
  26. 26.
    Sgroi, M., Lavagno, L., Watanabe, Y., Sangiovanni-Vincentelli, A.: Synthesis of embedded software using free-choice petri nets. In: Proceedings of the Design Automation Conference, DAC 1999, pp. 805–810. ACM (1999)Google Scholar
  27. 27.
    Vavilapalli, V.K., Murthy, A.C., Douglas, C., Agarwal, S., Konar, M., Evans, R., Graves, T., Lowe, J., Shah, H., Seth, S., Saha, B., Curino, C., O’Malley, O., Radia, S., Reed, B., Baldeschwieler, E.: Apache Hadoop YARN: yet another resource negotiator. In: Lohman, G.M. (ed.) ACM Symposium on Cloud Computing (SOCC 2013), pp. 5:1–5:16 (2013)Google Scholar
  28. 28.
    Verhoef, M., Larsen, P.G., Hooman, J.: Modeling and validating distributed embedded real-time systems with VDM++. In: Misra, J., Nipkow, T., Sekerinski, E. (eds.) FM 2006. LNCS, vol. 4085, pp. 147–162. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  29. 29.
    Wang, G., Butt, A.R., Pandey, P., Gupta, K.: A simulation approach to evaluating design decisions in MapReduce setups. In: IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2009, pp. 1–11. IEEE (2009)Google Scholar
  30. 30.
    Wong, P.Y.H., Albert, E., Muschevici, R., Proença, J., Schäfer, J., Schlatte, R.: The ABS tool suite: modelling, executing and analysing distributed adaptable object-oriented systems. J. Softw. Tools Technol. Transf. 14(5), 567–588 (2012)CrossRefGoogle Scholar
  31. 31.

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Jia-Chun Lin
    • 1
  • Ingrid Chieh Yu
    • 1
  • Einar Broch Johnsen
    • 1
  • Ming-Chang Lee
    • 1
  1. 1.Department of InformaticsUniversity of OsloOsloNorway

Personalised recommendations