Empirical Software Engineering

, Volume 23, Issue 6, pp 3394–3441 | Cite as

A systematic approach for performance assessment using process mining

An industrial experience report
  • Simona Bernardi
  • Juan L. Domínguez
  • Abel Gómez
  • Christophe Joubert
  • José Merseguer
  • Diego Perez-Palacin
  • José I. Requeno
  • Alberto Romeu
Experience Report


Software performance engineering is a mature field that offers methods to assess system performance. Process mining is a promising research field applied to gain insight on system processes. The interplay of these two fields opens promising applications in the industry. In this work, we report our experience applying a methodology, based on process mining techniques, for the performance assessment of a commercial data-intensive software application. The methodology has successfully assessed the scalability of future versions of this system. Moreover, it has identified bottlenecks components and replication needs for fulfilling business rules. The system, an integrated port operations management system, has been developed by Prodevelop, a medium-sized software enterprise with high expertise in geospatial technologies. The performance assessment has been carried out by a team composed by practitioners and researchers. Finally, the paper offers a deep discussion on the lessons learned during the experience, that will be useful for practitioners to adopt the methodology and for researcher to find new routes.


Software performance Process mining Unified modeling language Complex event processing Stochastic petri net 



Special thanks to Ismael Torres and Sergi Soler, from Prodevelop, for their help in collecting the logs of the case study. This work has been supported by the European Commission under the H2020 Research and Innovation program [DICE, Grant Agreement No. 644869], the Spanish Ministry of Economy and Competitiveness [ref. CyCriSec-TIN2014-58457-R], and the Aragonese Government [ref. T94, DIStributed COmputation (DISCO)].


  1. Adriansyah A, Buijs J (2012) Mining process performance from event logs: the BPI Challenge 2012 case study. BPM reports, no. 1215.
  2. AISencoding (2012) Automatic Identification System - Encoding Guide., accessed 04/29/2016
  3. AISreceiver (2016) Installation and Quick Reference Guide - SRL-200/G AIS Receiver.
  4. AIVM/AIVDO (2015) AIVDM/AIVDO protocol decoding., accessed 04/29/2016
  5. Ajmone-Marsan M, Balbo G, Conte G, Donatelli S, Franceschinis G (1994) Modelling with Generalized Stochastic Petri Nets, 1st edn. Wiley, New YorkzbMATHGoogle Scholar
  6. Averill ML (2015) Simulation Modeling and Analysis. McGraw-Hill, New YorkGoogle Scholar
  7. Balsamo S, Di Marco A, Inverardi P, Simeoni M (2004) Model-based performance prediction in software development: A survey. IEEE Trans Softw Eng 30(5):295–310CrossRefGoogle Scholar
  8. Bass L, Weber I, Zhu L (2015) DevOps: A software architect’s perspective. Addison-Wesley Professional, BostonGoogle Scholar
  9. Becker S, Koziolek H, Reussner R (2009) The Palladio Component Model for Model-driven Performance Prediction. J Syst Softw 82(1):3–22CrossRefGoogle Scholar
  10. Bernardi S, Campos J, Merseguer J (2011) Timing-Failure Risk Assessment of UML Design Using Time Petri Net Bound Techniques. IEEE Trans Ind Inf 7(1):90–104CrossRefGoogle Scholar
  11. Bernardi S, Merseguer J, Petriu DC (2012) Dependability modeling and analysis of software systems specified with UML. ACM Comput Surv 45(1):1–48CrossRefGoogle Scholar
  12. Brünink M, Rosenblum DS (2016) Mining performance specifications. In: Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM, pp 39–49Google Scholar
  13. Brunnert A, Krcmar H (2017) Continuous performance evaluation and capacity planning using resource profiles for enterprise applications. J Syst Softw 123:239–262CrossRefGoogle Scholar
  14. Casale G, Ardagna D, Artac M, Barbier F, Nitto ED, Henry A, Iuhasz G, Joubert C, Merseguer J, Munteanu V I, Perez JF, Petcu D, Rossi M, Sheridan C, Spais I, Vladuic D (2015) DICE: Quality-Driven Development of Data-Intensive Cloud Applications. In: 7th IEEE/ACM International Workshop on Modeling in Software Engineering, MiSE 2015, Florence, Italy, May, vol 16-17, pp 78–83Google Scholar
  15. Celonis (2011) Celonis PI.
  16. Ceravolo P, Damiani E, Torabi M, Barbon S (2017) Toward a new generation of log pre-processing methods for process mining. In: International Conference on Business Process Management, Springer, pp 55–70Google Scholar
  17. Cortellessa V, Marco AD, Inverardi P (2011) Model-based software performance analysis, 1st edn. Springer Publishing Company Incorporated, BerlinCrossRefGoogle Scholar
  18. Dipartimento di informatica Università di Torino (2016) GRaphical Editor and Analyzer for Timed and Stochastic Petri Nets. - accessed 01/05/2016
  19. Diwan A, Hauswirth M, Mytkowicz T, Sweeney PF (2011) TraceAnalyzer: A system for processing performance traces. Soft Practice and Experience 41(3):267–282CrossRefGoogle Scholar
  20. Ferme V, Pautasso C (2017) Towards holistic continuous software performance assessment. In: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion, ACM, New York, NY, USA, ICPE ’17 Companion, pp 159–164Google Scholar
  21. Foundation TE (2017) Eclipse - The Eclipse Foundation open source community website., last accessed Jan. 2017
  22. Gómez-Martínez E, Gonzalez-Cabero R, Merseguer J (2014) Performance assessment of an architecture with adaptative interfaces for people with special needs. Empir Softw Eng 19(6):1967–2018. CrossRefGoogle Scholar
  23. Gómez A, Joubert C, Merseguer J (2016) A tool for assessing performance requirements of data-intensive applications. In: Proceedings of the XXIV National Conference of Concurrency and Distributed Systems (JCDS 2016), pp 159–169., accessed 01/23/2017
  24. Günther CW, Rozinat A (2012) Disco: Discover Your Processes. BPM (Demos) 940:40–44Google Scholar
  25. Hernȧndez S, van Zelst SJ, Ezpeleta J, van der Aalst W (2015) Handling big(ger) logs: Connecting ProM 6 to apache hadoop. In: Proceedings of the BPM Demo Session 2015,, vol 1418 , pp 80–84Google Scholar
  26. Huber N, Walter J, Bähr M, Kounev S (2015) Model-based autonomic and performance-aware system adaptation in heterogeneous resource environments: A case study. In: International Conference on Cloud and Autonomic Computing, pp 181–191Google Scholar
  27. Huber N, Brosig F, Spinner S, Kounev S, Bähr M (2017) Model-based self-aware performance and resource management using the descartes modeling language. IEEE Trans Softw Eng 43(5):432–452CrossRefGoogle Scholar
  28. ISO (2008) Systems and software engineering – High-level Petri nets – Part 2: Transfer format. ISO/IEC 15909-2:2011, International Organization for Standardization, Geneva, SwitzerlandGoogle Scholar
  29. Joishi J, Sureka A (2015) Vishleshan: performance comparison and programming process mining algorithms in graph-oriented and relational database query languages. In: Proceedings of the 19th International Database Engineering & Applications Symposium. ACM, pp 192–197Google Scholar
  30. Kounev S (2006) Performance modeling and evaluation of distributed component-based systems using queueing petri nets. IEEE Trans Softw Eng 32 (7):486–502CrossRefGoogle Scholar
  31. Kounev S, Huber N, Brosig F, Zhu X (2016) A model-based approach to designing self-aware it systems and infrastructures. IEEE Comput 49(7):53–61. CrossRefGoogle Scholar
  32. Lazowska ED, Zahorjan J, Graham GS, Sevcik KC (1984) Quantitative system performance: Computer system analysis using queueing network models. Prentice-Hall, Inc, Upper Saddle RiverGoogle Scholar
  33. Lȯpez-Grao J, Merseguer J, Campos J (2004) From UML activity diagrams to Stochastic Petri nets: application to software performance engineering. In: Proceedings of the Fourth International Workshop on Software and Performance, WOSP 2004, Redwood Shores, California, USA, January 14-16, 2004, pp 25–36Google Scholar
  34. Maggi F M, Mooij A J, Van der Aalst W (2013) Analyzing vessel behavior using process mining. In: van de Laar P, Tretmans J, Borth M (eds) Situation Awareness with Systems of Systems. Springer, pp 133–148. Google Scholar
  35. Menasce DA, Almeida VA, Dowdy LW, Dowdy L (2004) Performance by design: computer capacity planning by example. Prentice Hall Professional, Upper Saddle RiverGoogle Scholar
  36. NIST/SEMATECH (2013) e-Handbook of Statistical Methods. Accessed 10/15/2016
  37. OMG (2011) UML Profile for MARTE: Modeling and Analysis of Real-time Embedded Systems, Version 1.1. Tech. rep., Object Management Group.
  38. PNMLframework (2017) PNML Framework.
  39. Prodevelop (1993) Prodevelop- Integrating Tech.
  40. ProM Tools (2017) ProM Tools.
  41. QPRprocessAnalyzer (2011) QPR process analyzer.
  42. RapidMiner (2018) Rapid miner.
  43. RapidProM (2018) RapidProM: Bringing process mining to analytic workflows.
  44. Rubin VA, Mitsyuk AA, Lomazova IA, van der Aalst WM (2014) Process mining can be applied to software too!. In: Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ACMGoogle Scholar
  45. Runeson P, Höst M (2008) Guidelines for conducting and reporting case study research in software engineering. Empir Softw Eng 14(2):131. CrossRefGoogle Scholar
  46. Shavor S, D’Anjou J, Fairbrother S, Kehn D, Kellerman J, McCarthy P (2003) The java developer’s guide to eclipse. Addison-Wesley Longman Publishing Co., Inc., BostonGoogle Scholar
  47. Smith CU (1990) Performance Engineering of Software Systems, 1st edn. Addison-Wesley Longman Publishing Co., Inc., BostonGoogle Scholar
  48. Smith CU (2002) Williams LG. Addison Wesley Longman Publishing Co., Inc., Redwood CityGoogle Scholar
  49. Tarvo A, Reiss SP (2018) Automatic performance prediction of multithreaded programs: a simulation approach. Autom Softw Eng 25(1):101–155CrossRefGoogle Scholar
  50. The Eclipse Foundation (2016) Papyrus.
  51. UML2 (2015) Unified Modeling Language: Infrastructure. Version 2.5, OMG document: formal/2015- 03-01Google Scholar
  52. Van Dongen BF et al (2005) The ProM framework: A new era in process mining tool support. In: Applications and Theory of Petri Nets 2005, Springer, pp 444–454Google Scholar
  53. Walter J, van Hoorn A, Koziolek H, Okanovic D, Kounev S (2016) Asking “What”?, Automating the “How”?: The Vision of Declarative Performance Engineering. In: Proceedings of the 7th ACM/SPEC International Conference on Performance Engineering, ICPE 2016, Delft, The Netherlands, March 12-16, 2016, pp 91–94Google Scholar
  54. Walter J, Stier C, Koziolek H, Kounev S (2017) An expandable extraction framework for architectural performance models. In: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering Companion, ACM, New York, NY, USA, ICPE ’17 Companion, pp 165–170Google Scholar
  55. XES (2016) Extensible Event Stream. IEEE Task FOrce on Process Mining, [Online; accessed 18-April-2016]Google Scholar
  56. Yu X, Han S, Zhang D, Xie T (2014) Comprehending performance from real-world execution traces: A device-driver case. ACM SIGPLAN Notices 49(4):193–206Google Scholar
  57. Van der Aalst W (2011) Process mining - discovery, conformance and enhancement of business processes. Springer, Berlin. zbMATHGoogle Scholar
  58. Van der Aalst W (2014) Business intelligence: Third european summer school, eBISS 2013, Dagstuhl Castle, Germany, July 7-12, 2013, Tutorial Lectures. In: Springer International Publishing, chap Process Mining in the Large: A Tutorial, pp 33–76Google Scholar
  59. Van der Aalst W, Adriansyah A, van Dongen B (2012) Replaying history on process models for conformance checking and performance analysis. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 2(2):182–192Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Departamento de Informática e Ingeniería de SistemasUniversidad de ZaragozaZaragozaSpain
  2. 2.Prodevelop SLValenciaSpain
  3. 3.Internet Interdisciplinary InstituteUniversitat Oberta de CatalunyaBarcelonaSpain
  4. 4.Department of Computer ScienceLinnaeus UniversityVäxjöSweden

Personalised recommendations