Software & Systems Modeling

, Volume 14, Issue 4, pp 1481–1504 | Cite as

On the comprehension of workflows modeled with a precise style: results from a family of controlled experiments

  • Gianna Reggio
  • Filippo Ricca
  • Giuseppe Scanniello
  • Francesco Di Cerbo
  • Gabriella Dodero
Regular Paper


In this paper, we present the results from a family of experiments conducted to assess whether the level of formality/precision in workflow modeling, based on UML activity diagrams, influences two aspects of construct comprehensibility: correctness of understanding and task completion time. In particular, we have considered two styles for workflow modeling with different levels of formality: a precise style (with specific rules and imposed constraints) and an ultra-light style (no rules, no imposed constraints). Experiments were conducted with 111 participants (Bachelor and Master students). In each experiment, participants accomplished comprehension tasks on two workflows, modeled either with the precise style or with a lighter variant. The main results from our data analysis can be summarized as follows: (i) all participants achieved a significantly better comprehension of workflows written in the precise style, (ii) the style had no significant impact on task completion time, (iii) more experienced participants benefited more, with respect to less experienced ones, from the precise style, as for their correctness of understanding, and (iv) all participants found the precise style useful in comprehending workflows.


Family of experiments Precise and Ultra-light styles  UML activity diagrams Workflow modeling 



We would like to thank all the participants in the experiments.


  1. 1.
    Abrahão, S.M.A., Gravino, C., Pelozo, E.I., Scanniello, G., Tortora, G.: Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: results from a family of five experiments. IEEE Trans. Softw. Eng. 39(3), 327–342 (2013)CrossRefGoogle Scholar
  2. 2.
    Agarwal, R., De, P., Sinha, A.P.: Comprehending object and process models: an empirical study. IEEE Trans. Softw. Eng. 25(4), 541–556 (1999)CrossRefGoogle Scholar
  3. 3.
    Ali, S., Yue, T., Briand, L.C.: Does aspect-oriented modeling help improve the readability of UML state machines? Softw. Syst. Model. pp. 1–33 (2012)Google Scholar
  4. 4.
    Ambler, S.W.: The Elements of UML 2.0 Style. Cambridge University Press, Cambridge (2005)CrossRefGoogle Scholar
  5. 5.
    Aranda, J., Ernst, N., Horkoff, J., Easterbrook, S.: A framework for empirical evaluation of model comprehensibility. In: Proceedings of the International Workshop on Modeling in Software Engineering, MISE ’07, p. 7-, Washington, DC, USA, 2007. IEEE, Computer SocietyGoogle Scholar
  6. 6.
    Astesiano, E., Reggio, G., Ricca, F.: Modeling business within a UML-based rigorous software development approach. In: Degano, P., DeNicola, R., Meseguer, J. (eds.) Concurrency, Graphs and Models, number 5065 in LNCS, pp. 261–277. Springer, Berlin (2008)Google Scholar
  7. 7.
    Baker, R.: Modern permutation test software. In: Edgington, E. (ed.) Randomization Tests, Marcel Decker (1995)Google Scholar
  8. 8.
    Basili, V.R., Caldiera, G., Rombach, D.H.: The Goal Question Metric Paradigm, Encyclopedia of Software Engineering. Wiley, London (1994)Google Scholar
  9. 9.
    Basili, V.R., Shull, F., Lanubile, F.: Building knowledge through families of experiments. IEEE Trans. Softw. Eng. 25(4), 456–473 (1999)CrossRefGoogle Scholar
  10. 10.
    Bauer, M.I., Johnson-Laird, P.N.: How diagrams can improve reasoning. Psychol. Sci. 4, 372–378 (1993)CrossRefGoogle Scholar
  11. 11.
    Birkmeier, D., Overhage, S.: Is BPMN really first choice in joint architecture development? an empirical study on the usability of BPMN and UML activity diagrams for business users. In: Research into Practice: Reality and Gaps, number 6093 in LNCS, pp. 119–134. Springer, Berlin (2010)Google Scholar
  12. 12.
    Briand, L.C., Labiche, Y., Di Penta, M., Yan-Bondoc, H.D.: An experimental investigation of formality in UML-based development. IEEE Trans. Softw. Eng. 31(10), 833–849 (2005)CrossRefGoogle Scholar
  13. 13.
    Broy, M., Cengarle, M.V.: UML formal semantics: lessons learned. Softw. Syst. Model. 10(4), 441–446 (2011)CrossRefGoogle Scholar
  14. 14.
    Carver, J., Jaccheri, L., Morasca, S., Shull, F.: Issues in using students in empirical studies in software engineering education. In: 9th International Symposium on Software Metrics, p. 239, Washington, DC, USA, 2003. IEEE CSGoogle Scholar
  15. 15.
    De Lucia, A., Francese, R., Tortora, G.: Deriving workflow enactment rules from UML activity diagrams: a case study. In: IEEE Symposium on Human Centric Computing Languages and. Environments, pp. 211–218 (2003)Google Scholar
  16. 16.
    Di Cerbo, F., Dodero, G., Reggio, G., Ricca, F., Scanniello, G.: Precise vs. ultra-light activity diagrams—an experimental assessment in the context of business process modelling. In: International Conference on Product Focused Software Development and Process Improvement, number 6759 in LNCS, pp. 291–305. Springer (2011)Google Scholar
  17. 17.
    Di Nitto, E., Lavazza, L., Schiavoni, M., Tracanella, E., Trombetta, M.: Deriving executable process descriptions from UML. In: 22rd International Conference on Software Engineering, pp. 155–165 (2002)Google Scholar
  18. 18.
    Fowler, M.: UML Distilled: A Brief Guide to the Standard Object Modeling Language, 3rd edn. Addison-Wesley Longman Publishing Co. Inc., Boston (2003)Google Scholar
  19. 19.
    Gogolla, M., Richters, M.: On constraints and queries in UML. In: UML Workshop, pp. 109–121 (1997)Google Scholar
  20. 20.
    Gross, A., Doerr, J.: EPC vs. UML activity diagram—two experiments examining their usefulness for requirements engineering. In: Proceedings of Requirements Engineering Conference, pp. 47–56, Washington, DC, USA, 2009. IEEE CSGoogle Scholar
  21. 21.
    Havey, M.: Essential Business Process Modeling. O’Reilly Media Inc (2005)Google Scholar
  22. 22.
    Hedges, L.V., Olkin, I.: Statistical Methods for Meta-Analysis. Academic Press, New York (1985)zbMATHGoogle Scholar
  23. 23.
    Jurack, S., Lambers, L., Mehner, K., Taentzer, G., Wierse, G.: Object flow definition for refined activity diagrams. In: 12th International Conference on Fundamental Approaches to Software Engineering, pp. 49–63, Springer, Berlin (2009)Google Scholar
  24. 24.
    Kampenes, V.B., Dybå, T., Hannay, J.E., Sjøberg, D.I.K.: Systematic review: a systematic review of effect size in software engineering experiments. Inf. Softw. Technol. 49, 1073–1086 (2007)CrossRefGoogle Scholar
  25. 25.
    Kim, J., Hahn, J., Hahn, H.: How do we understand a system with (so) many diagrams? cognitive integration processes in diagrammatic reasoning. Inf. Syst. Res. 11(3), 284–303 (2000)CrossRefGoogle Scholar
  26. 26.
    Kitchenham, B., Pfleeger, S., Pickard, L., Jones, P., Hoaglin, D., El Emam, K., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8), 721–734 (2002)CrossRefGoogle Scholar
  27. 27.
    Kitchenham, B., Al-Khilidar, H., Babar, M., Berry, M., Cox, K., Keung, J., Kurniawati, F., Staples, M., Zhang, H., Zhu, L.: Evaluating guidelines for reporting empirical software engineering studies. Empir. Softw. Eng. 13, 97–121 (2008)CrossRefGoogle Scholar
  28. 28.
    Marchetto, A., Ricca, F.: From objects to services: toward a stepwise migration approach for Java applications. Int. J. Softw. Tools Technol. Transf. 11, 427–440 (2009)CrossRefGoogle Scholar
  29. 29.
    Mendling, J., Reijers, H., van der Aalst, W.: Seven process modeling guidelines (7pmg). Inf. Softw. Technol. 52(2), 127–136 (2010)CrossRefGoogle Scholar
  30. 30.
    Mendona, M.G., Maldonado, J.C., de Oliveira, M.C.F., Carver, J., Fabbri, S.C.P.F.F., Shull, F., Travassos, G.H., Hohn, E.N., Basili, V.R.: A framework for software engineering experimental replications. In: International Conference on Engineering of Complex Computer Systems, pp. 203–212 IEEE, 2008Google Scholar
  31. 31.
    Motulsky, H.: Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking. Oxford University Press, Oxford (2010)Google Scholar
  32. 32.
    Nugroho, A., Flaton, B., Chaudron, M.R.V.: Empirical analysis of the relation between level of detail in UML models and defect density. In: Proceedings of International Conference on Model Driven Engineering Languages and Systems, pp. 600–614 (2008)Google Scholar
  33. 33.
    Nugroho, A.: Level of detail in UML models and its impact on model comprehension: a controlled experiment. Inf. Softw. Technol. 51(12), 1670–1685 (2009)CrossRefGoogle Scholar
  34. 34.
    OMG. Business process model and notation (BPMN) Version 2.0. OMG Final Adopted Specification, Object Management Group (2006)Google Scholar
  35. 35.
    Oppenheim, A.N.: Questionnaire Design, Interviewing and Attitude Measurement. Pinter, London (1992)Google Scholar
  36. 36.
    Organization for the Advancement of Structured Information Standards (OASIS). Web Services Business Process Execution Language—Version 2.0. OASIS Standard (2007)Google Scholar
  37. 37.
    Peixoto, D., Batista, V., Atayde, A., Borges, E., Resende, R. ,Pádua, C. : A comparison of BPMN and UML 2.0 activity diagrams. In: VII Simposio Brasileiro de Qualidade de Software (2008)Google Scholar
  38. 38.
    Ramsey, H.R., Atwood, M.E., Van Doren, J.R.: Flowcharts versus program design languages: an experimental comparison. Commun. ACM 26(6), 445–449 (1983)CrossRefGoogle Scholar
  39. 39.
    Reggio, G., Leotta, M.,Ricca, F. : Precise is better than light—a document analysis study about quality of business process models. In: Proceedings of EmpiRE 2011, pp. 61–68. IEEE Digital Library (2011)Google Scholar
  40. 40.
    Reggio, G., Ricca, F., Astesiano, E., Leotta, M.: On business process modelling with the UML: a discipline and four styles. Technical Report DISI-TR-11-03, DISI—University of Genova, Italy, April 2011. Available at
  41. 41.
    Reggio, G., Ricca, F., Scanniello, G., Di Cerbo, F., Dodero, G.: A precise style for business process modelling: results from two controlled experiments. In: Model Driven Engineering Languages and Systems, 14th International Conference, MODELS 2011, Wellington, New Zealand, October 16–21, 2011. Proceedings, volume 6981 of LNCS, pp. 138–152. Springer (2011)Google Scholar
  42. 42.
    Ricca, F., Di Penta, M., Torchiano, M., Tonella, P., Ceccato, M.: How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: a series of four experiments. IEEE Trans. Softw. Eng. 36(1), 96–118 (2010)CrossRefGoogle Scholar
  43. 43.
    Rychly, M., Weiss, P.: Modeling of service oriented architecture: from business process to service realisation. In: Proceedings of International Working Conference on Evaluation of Novel Approaches to Software Engineering, pp. 140–146. Institute for Systems and Technologies of Information, Control and Communication (2008)Google Scholar
  44. 44.
    Scaife, M., Rogers, Y.: External cognition: how do graphical representations work? Int. J. Hum.-Comput. Stud. 45(2), 185–213 (1996)CrossRefGoogle Scholar
  45. 45.
    Scanlan, D.A.: Structured flowcharts outperform pseudocode: an experimental comparison. IEEE Softw. 6(5), 28–36 (1989)CrossRefGoogle Scholar
  46. 46.
    Scanniello, G., Gravino, C., Genero, M., Cruz-Lemus, J.A., Tortora, G.: On the impact of UML analysis models on source code comprehensibility and modifiability. ACM Trans. Soft. Eng. Meth. (to appear)Google Scholar
  47. 47.
    Scheer, A.: ARIS-Business Process Modeling. Springer, Berlin (2000)CrossRefGoogle Scholar
  48. 48.
    Shull, F., Mendonça, M., Basili, V., Carver, J., Maldonado, J.C., Fabbri, S., Travassos, G., Ferreira, M.: Knowledge-sharing issues in experimental software engineering. Empir. Softw. Eng. 9(1–2), 111–137 (2004)CrossRefGoogle Scholar
  49. 49.
    Shull, F.J., Carver, J.C., Vegas, S., Juristo, N.: The role of replications in empirical software engineering. Empir. Softw. Eng. 13(2), 211–218 (2008)CrossRefGoogle Scholar
  50. 50.
    Sjoberg, D.I.K., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanovic, A., Liborg, N., Rekdal, A.C.: A survey of controlled experiments in software engineering. IEEE Trans. Softw. Eng. 31(9), 733–753 (2005)CrossRefGoogle Scholar
  51. 51.
    Staron, M., Kuzniarz, L., Wohlin, C.: Empirical assessment of using stereotypes to improve comprehension of UML models: a set of experiments. J. Syst. Softw. 79(5), 727–742 (2006)CrossRefGoogle Scholar
  52. 52.
    Svahnberg, M., Aurum, A., Wohlin, C.: Using students as subjects—an empirical evaluation. In: Symposium on Empirical Software Engineering and Measurement, pp. 288–290, Kaiserslautern, Germany, 2008. IEEE Computer SocietyGoogle Scholar
  53. 53.
    UML Revision Task Force. OMG Unified Modeling Language (OMG UML), Superstructure, V 2.4.1 (2011)Google Scholar
  54. 54.
    Vegas, S., Juzgado, N.J., Moreno, A.M., Solari, M., Letelier, P.: Analysis of the influence of communication between researchers on experiment replication. In: International Symposium on Empirical Software Engineering, pp. 28–37, Rio de Janeiro, Brazil, 2006. IEEE Computer SocietyGoogle Scholar
  55. 55.
    Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., Kluwer, A .: Wesslén. Experimentation in Software Engineering—An Introduction. Kluwer (2000) Google Scholar
  56. 56.
    Zimmerman, M.K., Lundqvist, K., Leveson, N.G.: Investigating the readability of state-based formal requirements specification languages. In: Proceedings of International Conference on Software Engineering, pp. 33–43 (2002)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Gianna Reggio
    • 1
  • Filippo Ricca
    • 1
  • Giuseppe Scanniello
    • 2
  • Francesco Di Cerbo
    • 3
  • Gabriella Dodero
    • 4
  1. 1.DIBRISUniversità di GenovaGenoaItaly
  2. 2.DiMIEUniversità della BasilicataPotenzaItaly
  3. 3.SAP ResearchSophia AntipolisValbonneFrance
  4. 4.IDSEFree University of Bozen-BolzanoBolzanoItaly

Personalised recommendations