A Scalable Platform for Monitoring Data Intensive Applications

  • Ioan DrăganEmail author
  • Gabriel Iuhasz
  • Dana Petcu


Latest advances in information technology and the widespread growth in different areas are producing large amounts of data. Consequently, in the past decade a large number of distributed platforms for storing and processing large datasets have been proposed. Whether in development or in production, monitoring the applications running on these platforms is not an easy task, dedicated tools and platforms were proposed for this task yet none are specially designed for Big Data frameworks. In this paper we present a distributed, scalable, highly available platform able to collect, store, query and process monitoring data obtained from multiple Big Data frameworks. Alongside the architecture we experimentally show that the solution proposed is scalable and can handle a substantial quantity of monitoring data.


Big data Cloud computing Monitoring big data applications Scalable monitoring 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.



This work has received funding from the EC-funded project H2020 DICE (Agreement 644869), which aims at providing a toolchain that makes the task of developing Big Data applications less daunting and the H2020 ASPIDE project (Agreement 801091). This work was partially supported by grants from Romanian Ministry of Research and Innovation, grant Acronim (PNIII-P4-ID-PCE-2016-0842) and grant BID (PNIII-P1-PDI-PFE-2018-028).


  1. 1.
    Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: Cluster Computing with Working Sets. In: Proceedings of the 2Nd USENIX Conference on Hot Topics in Cloud Computing, HotCloud’10, pp. 10–10. USENIX Association, Berkeley (2010)Google Scholar
  2. 2.
    Casale, G., Ardagna, D., Artac, M., Barbier, F., Di Nitto, E., Henry, A., Iuhasz, G., Joubert, C., Merseguer, J., Munteanu, V.I., Perez, J.F., Petcu, D., Rossi, M., Sheridan, C., Spais, I., Vladuic, D.: Dice: Quality-driven development of data-intensive cloud applications. In: 7th IEEE/ACM International Workshop on Modeling in Software Engineering, miSE 2015, Florence, pp, 78–83 (2015)Google Scholar
  3. 3.
    Villalpando, L.E.B., April, A., Abran, A.: Performance analysis model for big data applications in cloud computing. J. Cloud Comput. 3(1), 1–20 (2014)Google Scholar
  4. 4.
    Gutierrez-Aguado, J., Calero, J.M., Villanueva, W.D.: Iaasmon: Alcaraz Monitoring architecture for public cloud computing data centers. J. Grid Comput. 14(2), 283–297 (2016)Google Scholar
  5. 5.
    Antypas, V., Zacheilas, N., Kalogeraki, V.: Dynamic Reduce Task Adjustment for Hadoop Workloads. In: Proceedings of the 19Th Panhellenic Conference on Informatics, PCI ’15, pp. 203–208. ACM, New York (2015)Google Scholar
  6. 6.
    Kertesz, A., Kecskemeti, G., Oriol, M., Kotcauer, P., Acs, S., Rodríguez, M., Mercè, O., Marosi, A.Cs., Marco, J., Franch, X.: Enhancing federated cloud management with an integrated service monitoring approach. J. Grid Comput. 11(4), 699–720 (2013)Google Scholar
  7. 7.
    Birje, M.N., Manvi, S.S.: Wigrimma: A wireless grid monitoring model using agents. J. Grid Comput. 9(4), 549–572 (2011)Google Scholar
  8. 8.
    Alhamazani, K., Ranjan, R., Mitra, K., Rabhi, F., Jayaraman, P.P., Khan, S.U., Guabtni, A., Bhatnagar, V.: An overview of the commercial cloud monitoring tools Research dimensions, design issues, and state-of-the-art. Computing 97(4), 357–377 (2015)MathSciNetGoogle Scholar
  9. 9.
    Iuhasz, G., Dragan, I.: An overview of monitoring tools for big data and cloud applications. In: 2015 17th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp. 363–366 (2015)Google Scholar
  10. 10.
    Venner, J., Wadkar, S., Siddalingaiah, M.: Pro Apache Hadoop. Apress (2014)Google Scholar
  11. 11.
    Jacobs, M.: Challenges and Lessons Learned Building Monitoring And diagnostics Tools for Hadoop. In: Proceedings of the 2012 Workshop on Management of Big Data Systems, MBDS ’12, pp. 33–34. ACM, New York (2012)Google Scholar
  12. 12.
    Aceto, G., Botta, A., de Donato, W.: Antonio pescapè Cloud monitoring: A survey. Comput. Netw. 57(9), 2093–2115 (2013)Google Scholar
  13. 13.
    Garduno, E., Kavulya, S.P., Tan, J., Gandhi, R., Narasimhan, P.: Theia: Visual Signatures for Problem Diagnosis in Large Hadoop Clusters. In: Proceedings of the 26Th International Conference on Large Installation System Administration: Strategies, Tools, and Techniques, Lisa’12, pp. 33–42. USENIX Association, Berkeley (2012)Google Scholar
  14. 14.
    Fowler, M.: Microservice overview (2016)Google Scholar
  15. 15.
    Newman, S.: Building Microservices: Designing fine-grained Systems. O’Reilly Media, Incorporated (2015)Google Scholar
  16. 16.
    Marz, N., Warren, J.: Big Data: Principles and best practices of scalable realtime data systems. Manning Publications (2015)Google Scholar
  17. 17.
    Gormley, C., Tong, Z.: Elasticsearch: The Definitive Guide. O’Reilly Media (2015)Google Scholar
  18. 18.
    McCandless, M., Hatcher, E., Gospodnetić, O.: Lucene in Action. Manning Pubs Co Series. Manning (2010)Google Scholar
  19. 19.
    Turnbull, J.: The logstash book: James Turnbull (2013)Google Scholar
  20. 20.
    Kreps, J., Narkhede, N., Rao, J.: Kafka: A distributed messaging system for log processing. In: Proceedings of 6th International Workshop on Networking Meets Databases (NetDB), Athens, GreeceGoogle Scholar
  21. 21.
    Grinberg, M.: Flask Web Development: Developing Web Applications with Python, 1st edn. O’Reilly Media, Inc., Philadelphia (2014)Google Scholar
  22. 22.
    Artac, M., Borovsak, T., Di Nitto, E., Guerriero, M., Tamburri, D.A.: Model-driven continuous deployment for quality devops. In: Proceedings of the 2nd International Workshop on Quality-Aware DevOps, QUDOS@ISSTA 2016, pp. 40–41. Saarbru̇cken, Germany (2016)Google Scholar
  23. 23.
    Kennedy, S., Jiu, L.: Facilitating Collaboration and Interaction across the Enterprise with Oslc. In: Proceedings of the 2013 Conference of the Center for Advanced Studies on Collaborative Research, CASCON ’13, pp. 374–375. IBM Corp, Riverton (2013)Google Scholar
  24. 24.
    Chodorow, K., Dirolf, M.: MongoDB: The Definitive Guide: Powerful and Scalable Data Storage. O’Reilly Media (2010)Google Scholar
  25. 25.
    Santomaggio, G., Boschi, S.: RabbitMQ Cookbook. Packt Publ., Birmingham (2013)Google Scholar
  26. 26.
    Wu, X., Liu, Y., Gorton, I.: Exploring performance models of hadoop applications on cloud architecture. In: 2015 11th International ACM SIGSOFT Conference on Quality of Software Architectures (QoSA), pp. 93–101 (2015)Google Scholar
  27. 27.
    Chandola, V., Banerjee, A., Kumar, V.: Anomaly detection: A survey. ACM Comput. Surv. 41(3), 15:1–15:58 (2009)Google Scholar
  28. 28.
    Gander, M., Felderer, M., Katt, B., Tolbaru, A., Breu, R., Moschitti, A.: Anomaly detection in the cloud: Detecting security incidents via machine learning. In: Moschitti, A., Plank, B. (eds.) Trustworthy Eternal Systems via Evolving Software, Data and Knowledge, volume 379 of Communications in Computer and Information Science, pp 103–116. Springer, Berlin (2013)Google Scholar
  29. 29.
    Iuhasz, G., Pop, D.: Monitoring and data warehousing tools – initial version. DICE EU H2020 Project Deliverable (2016)Google Scholar
  30. 30.
    Sagha, H., Bayati, H., Millán, J.D.R., Chavarriaga, R.: On-line anomaly detection and resilience in classifier ensembles. Pattern Recogn. Lett. 34(15), 1916–1927 (2013)Google Scholar
  31. 31.
    Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software: An update. SIGKDD Explor. Newsl. 11(1), 10–18 (2009)Google Scholar
  32. 32.
    Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G.S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I., Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y., Zheng, X.: Tensorflow: Large-scale machine learning on heterogeneous systems. Software available from (2015)Google Scholar
  34. 34.
    Casale, G., et al.: D1.2 dice requirement specification Technical report (2016)Google Scholar
  35. 35.
    Grossman, R.L., Bailey, S., Ramu, A., Malhi, B., Hallstrom, P., Pulleyn, I., Qin, X.: The management and mining of multiple predictive models using the predictive modeling markup language. Inf. Softw. Technol. 41(9), 589–595 (1999)Google Scholar
  36. 36.
    Bersani, M.M., Marconi, F., Rossi, M.: Trace checking of streaming applications through dice-tract. In: Companion of the 2018 ACM/SPEC International Conference on Performance Engineering, ICPE ’18, pp. 159–160. ACM, New York (2018)Google Scholar
  37. 37.
    Bersani, M.M., Bianculli, D., Ghezzi, C., Krstić, S., San Pietro, P.: Smt-based checking of soloist over sparse traces. In: Gnesi, S., Rensink, A. (eds.) Fundamental Approaches to Software Engineering, pp 276–290. Springer, Berlin (2014)Google Scholar
  38. 38.
    Doan, D.N., Iuhasz, G.: Tuning logstash garbage collection for high throughput in a monitoring platform. In: 2016 18th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC), pp. 359–365 (2016)Google Scholar
  39. 39.
    Bersani, M.M., Marconi, F., Rossi, M., Erascu, M.: A Tool for Verification of Big-Data Applications. In: Proceedings of the 2Nd International Workshop on Quality-Aware DevOps, QUDOS 2016, pp. 44–45. ACM, New York (2016)Google Scholar
  40. 40.
    Torres, I., Joubert, C., Di Nitto, E., Ridene, Y., Iuhasz, G., Kinouani, J.Y., Artac, M., Borovsak, T., Casale, G., Jamshidi, P., Zhai, Y., Li, C., Zhu, L., Bersani, M.M., Marconi, F., Guerriero, M., Tamburri, D.A., Sheridan, C., Requeno, J.I., Merseguer, J., Ardagna, D., Parant, L.-A.: Final assessment report and impact analysis (d6.4). Technical report, H2020 DICE (2018)Google Scholar
  41. 41.
    Bernardi, S., Requeno, J.I., Joubert, C., Romeu, A.: A Systematic Approach for Performance Evaluation Using Process Mining: The Posidonia Operations Case Study. In: Proceedings of the 2Nd International Workshop on Quality-Aware DevOps, QUDOS 2016, pp. 24–29. ACM, New York (2016)Google Scholar
  42. 42.
    Henry, A., Ridene, Y.: Legacy Data Migration and Fraud Detection Using Blu Age and Big Data. Technical report, BluAge (2015)Google Scholar
  43. 43.
    Perarnau, S., Thakur, R., Iskra, K., Raffenetti, K., Cappello, F., Gupta, R., Beckman, P., Snir, M., Hoffmann, H., Schulz, M., Rountree, B.: Distributed Monitoring and Management of Exascale Systems in the Argo Project, pp 173–178. Springer International Publishing, Cham (2015)Google Scholar

Copyright information

© Springer Nature B.V. 2019

Authors and Affiliations

  1. 1.Institute e-AustriaTimişoaraRomania
  2. 2.“Victor Babeş” University of Medicine and PharmacyTimişoaraRomania
  3. 3.West University of TimişoaraTimişoaraRomania

Personalised recommendations