Measuring the Understandability of Business Process Models - Are We Asking the Right Questions?
In this paper, we show how experiments on the understandability of business process models can depend on the exact wording used in the experiments’ questionnaires. For this purpose, we partially replicated a published experiment. We asked a group of students a number of questions on relations between tasks in a business process model. Alternatively, we used a set of modified questions which were aimed to ask for exactly the same relations. The result was that there was a significant difference in the number of correct answers between the two systems to construct a question. We argue that a non-negligible part of the wrong answers given in the experiment did not result from problems to understand the model, but rather from problems to understand the question. It follows that it is dangerous to draw conclusions from such an experiment until enough effort has been taken to select appropriate questions.
KeywordsBusiness Process Correct Answer Activity Period Wrong Answer Business Process Model
Unable to display preview. Download preview PDF.
- 3.Rittgen, P.: Quality and perceived usefulness of process models. In: SAC 2010: Proceedings of the 2010 ACM Symposium on Applied Computing, pp. 65–72. ACM, New York (2010)Google Scholar
- 4.Schuman, H., Presser, S.: Questions and Answers in Attitude Surveys. Academic Press, San Diego (1981)Google Scholar
- 6.Cardoso, J.: Process control-flow complexity metric: An empirical validation. In: IEEE International Conference on Services Computing, pp. 167–173 (2006)Google Scholar
- 8.Aguilar, E.R., Sanchez, L., Carballeira, F.G., Ruiz, F., Piattini, M., Caivano, D., Visaggio, G.: Prediction models for BPMN usability and maintainability. In: 2009 IEEE Conference on Commerce and Enterprise Computing, pp. 383–390 (2009)Google Scholar
- 11.Lara Proano, M.D.: Visual layout for drawing understandable process models. Master’s thesis, Technische Universiteit Eindhoven (2008)Google Scholar
- 14.Recker, J., Dreiling, A.: Does it matter which process modelling language we teach or use? In: 18th Australasian Conference on Information Systems (2007)Google Scholar
- 15.Mendling, J., Strembeck, M.: Influence factors of understanding business process models. In: 11th International Conference, Business Information Systems, BIS 2008, Innsbruck, Austria, pp. 142–153. Springer, Heidelberg (2008)Google Scholar
- 18.Melcher, J., Seese, D.: Process measurement: Insights from software measurement on measuring process complexity, quality and performance. Technical report, Universität Karlsruhe, TH (2008)Google Scholar
- 19.Melcher, J., Seese, D.: Towards validating prediction systems for process understandability: Measuring process understandability. In: 10th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, pp. 564–571. IEEE Computer Society, Los Alamitos (2008)Google Scholar
- 20.Melcher, J., Mendling, J., Reijers, H.A., Seese, D.: On measuring the understandability of process models. In: Revised Papers of the BPM 2009 International Workshops. LNBIP, vol. 43, pp. 465–476. Springer, Ulm (2010)Google Scholar
- 21.Dwyer, M.B., Avrunin, G.S., Corbett, J.C.: Patterns in property specifications for finite-state verification. In: Proc. of the 21st International Conference on Software Engineering, pp. 411–420. IEEE Computer Society Press, Los Alamitos (1999)Google Scholar
- 22.Sudman, S., Bradburn, N.M.: Asking Questions: A Practical Guide to Questionnaire Design. Jossey-Bass, San Francisco (1982)Google Scholar
- 24.Aranda, J., Ernst, N., Horkoff, J., Easterbrook, S.M.: A framework for empirical evaluation of model comprehensibility. In: International Workshop on Modeling in Software Engineering, MiSE 2007 (2007)Google Scholar
- 25.Patig, S.: A practical guide to testing the understandability of notations. In: Fifth Asia-Pacific Conference on Conceptual Modelling (APCCM 2008). CRPIT, vol. 79, pp. 49–58. Australian Computer Society (2008)Google Scholar