Comparison of Questionnaire Based and User Model Based Usability Evaluation Methods

  • Meng LiEmail author
  • Armagan Albayrak
  • Yu Zhang
  • Daan van Eijk
  • Zengyao Yang
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 824)


The usability now serves as a fundamental quality of a computational device, e.g. smartphone. Moreover, the smartphone has firmly embedded into our daily life as an indispensable part, so the context and style that user may interact with them are largely different from a decade ago. Nowadays, testing usability with end user has become a common sense. Thus, how valid a usability evaluation method could assess the ‘extent to which a product can be used by specified users’ (ISO 9241-11) to facilitate software design becomes an interesting question to explore.

In this research, three usability evaluation methods are compared. Among these methods, IsoMetrics is a standard questionnaire aiming at offer usability data for summative and formative evaluation; SUMI aims to assess quality of software product from end users perspective; User Model Checklist is a method based on user’s cognition-motor chain in specific tasks. The coverage and amount of usability issues, user’s effort of evaluation and software developer’s feedback on evaluation result are compared under a simulated usability test on SMS function with a smartphone. The result indicate that User Model Checklist could cover 90.4% of the usability issues found by IsoMetrics and SUMI, while 26.3% usability issues found by User Model Checklist could not be covered by IsoMetrics and SUMI. Users put highest effort on accomplish IsoMetrics and lowest effort on User Model Checklist. Moreover, the feedbacks from the developers show that the User Model Checklist requires lower usability knowledge, offers clearer improvement points and supports detailed design better.


Usability evaluation comparison IsoMetrics SUMI User Model Checklist 


  1. 1.
    Macleod M, Bowden R, Bevan N, Curson I (1997) The MUSiC performance measurement method. Behav Inf Technol 16(4–5):279–293CrossRefGoogle Scholar
  2. 2.
    Nokia Corporation (2004) Series 60 Developer Platform 2.0: Usability Guidelines For Enterprise Applications.
  3. 3.
    Weiss S (2003) Handheld usability. WileySons, ChichesterGoogle Scholar
  4. 4.
    Nielsen J (2012) Usability 101: Introduction to Usability.
  5. 5.
    Nielsen J (1994) Usability engineering. Elsevier, LondonzbMATHGoogle Scholar
  6. 6.
    Lathan CE, Newman DJ, Sebrechts MM, Doarn CR (1997) Evaluating a web-based interface for internet telemedicine. NASA, WashingtonGoogle Scholar
  7. 7.
  8. 8.
    Hornbæk K (2006) Current practice in measuring usability: Challenges to usability studies and research. Int J Hum Comput Stud 64(2):79–102CrossRefGoogle Scholar
  9. 9.
    Hartson HR, Andre TS, Williges RC (2003) Criteria for evaluating usability evaluation methods. Int J Hum Comput Interact 15(1):145–181CrossRefGoogle Scholar
  10. 10.
    Li LS (2007) Design investigation. China Architecture & Building Press, BeijingGoogle Scholar
  11. 11.
    International Organization for Standardization (2002) ISO/TR 16982:2002(E) Ergonomics of human-system interaction—Usability method supporting human-centred design. ISO, SwitzerlandGoogle Scholar
  12. 12.
    Gediga G, Hamborg KC, Düntsch I (1999) The IsoMetrics usability inventory: an operationalization of ISO 9241-10 supporting summative and formative evaluation of software systems. Behav Inf Technol 18(3):151–164CrossRefGoogle Scholar
  13. 13.
    Kirakowski J, Corbett M (1993) SUMI: the software usability measurement inventory. Br J Educ Technol 24(3):210–212CrossRefGoogle Scholar
  14. 14.
    Follmer S, Leithinger D, Olwal A, Hogge A, Ishii H (2013) inFORM: dynamic physical affordances and constraints through shape and object actuation. In: UIST, vol 13Google Scholar
  15. 15.
    Solso RL, Johnson HH (1989) An introduction to experimental design in psychology: a case approach, 3rd edn. Harper & Row Publishers Inc., New YorkGoogle Scholar
  16. 16.
    Rasmussen J (1983) Skills, rules, and knowledge; signals, signs, and symbols, and other distinctions in human performance models. IEEE Trans Syst Man Cybern 3:257–266CrossRefGoogle Scholar
  17. 17.
    International Organization for Standardization (2006) ISO 9241-110:2006 Ergonomics of human-system interaction – Part 110: Dialogue principles. ISO, SwitzerlandGoogle Scholar
  18. 18.
    Li LS (2004) Human computer interface design. Science Press, BeijingGoogle Scholar
  19. 19.
    Berenguer A, Goncalves J, Hosio S, Ferreira D, Anagnostopoulos T, Kostakos V (2017) Are Smartphones Ubiquitous?: an in-depth survey of smartphone adoption by seniors. IEEE Consum Electron Mag 6(1):104–110CrossRefGoogle Scholar
  20. 20.
    Lee H, Ahn H, Nguyen TG, Choi SW, Kim DJ (2017) Comparing the self-report and measured smartphone usage of college students: a pilot study. Psychiatry Invest 14(2):198–204CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.Delft University of TechnologyDelftNetherlands
  2. 2.Xi’an Jiaotong UniversityXi’anChina

Personalised recommendations