Data-Driven Usability Test Scenario Creation

  • Maikel L. van EckEmail author
  • Else Markslag
  • Natalia Sidorova
  • Angelique Brosens-Kessels
  • Wil M. P. van der Aalst
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11262)


In this paper, we present a data-driven approach to enable the creation of evidence-based usability test scenarios. By utilising product usage data to create usability test scenarios, we aim to improve the reliability of the test results and to provide better insights into product usability. The approach consists of four elements: the collection of product usage data, the transformation of these data into logs of user activities, the creation of models of user behaviour, and the guided creation of usability test scenarios based on the models. We discuss the challenges that can be encountered when applying this approach based on our experiences with two case studies in product development. We have created a prototype scenario planning tool and performed a preliminary evaluation of the tool with usability engineers working at Philips Healthcare. The evaluation shows that tool-supported evidence-based usability test creation would be valuable in their daily work.


User-centered design Usability testing Data-driven design Process mining 



This work has been conducted in the context of the IMPULS collaboration project of Eindhoven University of Technology and Philips: “Mine your own body”, and within the ENABLE-S3 project that has received funding from the ECSEL Joint Undertaking under Grant Agreement No 692455. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and Austria, Denmark, Germany, Finland, Czech Republic, Italy, Spain, Portugal, Poland, Ireland, Belgium, France, Netherlands, United Kingdom, Slovakia, Norway.


  1. 1.
    van der Aalst, W.M.P.: Process Mining - Data Science in Action, 2nd edn. Springer, Berlin (2016). Scholar
  2. 2.
    Bernhaupt, R., Palanque, P., Manciet, F., Martinie, C.: User-test results injection into task-based design process for the assessment and improvement of both usability and user experience. In: Bogdan, C., et al. (eds.) HCSE/HESSD -2016. LNCS, vol. 9856, pp. 56–72. Springer, Cham (2016). Scholar
  3. 3.
    Billestrup, J., Bruun, A., Stage, J.: Usability problems experienced by different groups of skilled internet users: gender, age, and background. In: Bogdan, C., et al. (eds.) HCSE/HESSD -2016. LNCS, vol. 9856, pp. 45–55. Springer, Cham (2016). Scholar
  4. 4.
    Billman, D., Fayollas, C., Feary, M., Martinie, C., Palanque, P.: Complementary tools and techniques for supporting fitness-for-purpose of interactive critical systems. In: Bogdan, C., et al. (eds.) HCSE/HESSD -2016. LNCS, vol. 9856, pp. 181–202. Springer, Cham (2016). Scholar
  5. 5.
    Bogers, S., Frens, J.W., van Kollenburg, J., Deckers, E., Hummels, C.: Connected baby bottle: a design case study towards a framework for data-enabled design. In: Proceedings of the 2016 ACM Conference on Designing Interactive Systems, DIS 2016, Brisbane, QLD, Australia, 4–8 June, pp. 301–311 (2016)Google Scholar
  6. 6.
    Campos, J.C., et al.: A more intelligent test case generation approach through task models manipulation. In: PACMHCI 1(EICS), pp. 9:1–9:20 (2017)Google Scholar
  7. 7.
    Cugola, G., Margara, A.: Processing flows of information: from data stream to complex event processing. ACM Comput. Surv. 44(3), 15:1–15:62 (2012)CrossRefGoogle Scholar
  8. 8.
    van Eck, M.L., Sidorova, N., van der Aalst, W.M.P.: Discovering and exploring state-based models for multi-perspective processes. In: La Rosa, M., Loos, P., Pastor, O. (eds.) BPM 2016. LNCS, vol. 9850, pp. 142–157. Springer, Cham (2016). Scholar
  9. 9.
    van Eck, M.L., Sidorova, N., van der Aalst, W.M.P.: Enabling process mining on sensor data from smart products. In: Tenth IEEE International Conference on Research Challenges in Information Science, RCIS 2016, Grenoble, France, 1–3 June, pp. 1–12 (2016)Google Scholar
  10. 10.
    Harrison, M.D., Masci, P., Campos, J.C., Curzon, P.: Verification of user interface software: the example of use-related safety requirements and programmable medical devices. IEEE Trans. Hum Mach. Syst. 47(6), 834–846 (2017)CrossRefGoogle Scholar
  11. 11.
    Hoornaar, T.J.: Extracting real-life workflow models from relational data and using these to generate field-based usability testing scenarios at Philips Healthcare. Master’s thesis, Technische Universiteit Eindhoven (2017)Google Scholar
  12. 12.
    Liu, Y., Nie, L., Liu, L., Rosenblum, D.S.: From action to activity: sensor-based activity recognition. Neurocomputing 181, 108–115 (2016)CrossRefGoogle Scholar
  13. 13.
    Mannhardt, F., de Leoni, M., Reijers, H.A., van der Aalst, W.M.P., Toussaint, P.J.: Guided process discovery-a pattern-based approach. Inf. Syst. (2018, submitted)Google Scholar
  14. 14.
    Mori, G., Paternò, F., Santoro, C.: CTTE: support for developing and analyzing task models for interactive system design. IEEE Trans. Software Eng. 28(8), 797–813 (2002)CrossRefGoogle Scholar
  15. 15.
    Norman, D.A., Draper, S.W.: User Centered System Design: New Perspectives on Human-Computer Interaction. Lawrence Erlbaum Associates Inc., New Jersey (1986)CrossRefGoogle Scholar
  16. 16.
    Ogata, S., Goto, Y., Okano, K.: Framework for relative web usability evaluation on usability features in MDD. In: Bogdan, C., et al. (eds.) HCSE/HESSD -2016. LNCS, vol. 9856, pp. 73–85. Springer, Cham (2016). Scholar
  17. 17.
    Holmström Olsson, H., Bosch, J.: Towards data-driven product development: a multiple case study on post-deployment data usage in software-intensive embedded systems. In: Fitzgerald, B., Conboy, K., Power, K., Valerdi, R., Morgan, L., Stol, K.-J. (eds.) LESS 2013. LNBIP, vol. 167, pp. 152–164. Springer, Heidelberg (2013). Scholar
  18. 18.
    Paetsch, F., Eberlein, A., Maurer, F.: Requirements engineering and agile software development. In: 12th IEEE International Workshops on Enabling Technologies (WETICE 2003), Infrastructure for Collaborative Enterprises, 9–11 June 2003, Linz, Austria, pp. 308–313 (2003)Google Scholar
  19. 19.
    Rubin, J., Chisnell, D.: Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, 2nd edn. Wiley, New York (2008)Google Scholar
  20. 20.
    Silva, T.R., Hak, J.-L., Winckler, M.: Testing prototypes and final user interfaces through an ontological perspective for behavior-driven development. In: Bogdan, C., et al. (eds.) HCSE/HESSD -2016. LNCS, vol. 9856, pp. 86–107. Springer, Cham (2016). Scholar
  21. 21.
    Verbeek, H.M.W., Buijs, J.C.A.M., van Dongen, B.F., van der Aalst, W.M.P.: XES, XESame, and ProM 6. In: Soffer, P., Proper, E. (eds.) CAiSE Forum 2010. LNBIP, vol. 72, pp. 60–75. Springer, Heidelberg (2011). Scholar

Copyright information

© IFIP International Federation for Information Processing 2019

Authors and Affiliations

  • Maikel L. van Eck
    • 1
    Email author
  • Else Markslag
    • 2
  • Natalia Sidorova
    • 1
  • Angelique Brosens-Kessels
    • 2
  • Wil M. P. van der Aalst
    • 1
    • 3
  1. 1.Eindhoven University of TechnologyEindhovenThe Netherlands
  2. 2.PhilipsAmsterdamThe Netherlands
  3. 3.RWTH Aachen UniversityAachenGermany

Personalised recommendations