Skip to main content
SpringerLink
Log in

A systematic review of software usability studies

  • Original Research
  • Published:
International Journal of Information Technology Aims and scope Submit manuscript

Abstract

The aim of this review is to summarize, analyze various research studies and identify different research gaps regarding usability standards and models, usability evaluation methods, usability metric, usability at different phases of software development life cycle and application domains of usability. This systematic review of usability studies between 1990 and 2016 has been conducted and 150 studies are identified. We conclude that researchers have not reached at consensus w.r.t. software usability models. We identify that Efficiency, Effectiveness, Satisfaction, and Learnability are commonly addressed attributes in various existing software usability models and standards. Further, developers do not have sufficient knowledge to decide the appropriate usability evaluation method to use in given domain. On the contrary, Usability Testing, Heuristic Evaluation and Questionnaire are identified frequently used methods for usability evaluation. Our findings investigate different metrics and measurement approaches used for usability estimation. But, current methods for usability measurement in practice do not include all ISO and ANSI defined aspects of usability into a single metric. Although, we identify studies concerning the integration of usability and software engineering into a single framework with generalizable results, their practical implementation is still missing and significantly needed. Conversely, this study highlights the fact that around 71% of studies address usability related issues during Design-Phase of software development life cycle. At present, usability issues have been identified in various domains but around 33.82% of studies identify that usability evaluation approach is widely used in Web-Domain.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. These online databases are mentioned in Sect. 2.2.

  2. Associate professors are from University School of Information and Communication Technology, Guru Gobind Singh Indraprastha University, Delhi, India.

  3. List of publication sources from international journals and conferences is provided in Table 4.

  4. 2-researchers are Kalpna Sagar, a PhD-Student and Dr. Anju Saha, an Associate Professor in University School of Information and Communication Technology, GGSIPU, Delhi, India.

  5. This list is provided in Table 6.

  6. Final scores details are presented in Table 5.

  7. These questions are mentioned in Table 3.

  8. We have included 89 research publications from SCI-Indexed journals and 29 from Non-SCI journals.

  9. Final score is computed for each selected primary study after adding the values assigned to each quality assessment question7.

  10. 5-Research questions are presented in Table 1.

  11. A complete list of usability metric or measurement approach is provided in Table 13.

References

  1. Institute of Electrical and Electronics Engineers (1990) IEEE standard glossary of software engineering terminology, IEEE std. 610.12-1990. Author, Los Alamitos

  2. ISO 9126 (1991) Information technology-software product evaluation-quality characteristics and guidelines for their use. Geneva

  3. Nielsen J (1993) Usability engineering. Academic press, London, ISBN: 978-0-12-518406-9

  4. International Organization for Standardization (1998) ISO 9241-11:1998, Ergonomic requirements for office work with visual display terminals (vdts), Part 11: guidance on usability. Author, Geneva

    Google Scholar 

  5. Maguire M (2001) Context of use within usability activities. Int J Hum Comput Stud 55:453–483

    Article  MATH  Google Scholar 

  6. Seffah A, Donyaee M, Kline RB et al (2006) Usability measurement and metrics: a consolidated model. Softw Qual J 14:159–178 [SPRINGER, (S107)]

    Article  Google Scholar 

  7. Seffah A, Metzker E (2004) The obstacles and myths of usability and software engineering. Commun ACM 47(12):71–76 (S24)

    Article  Google Scholar 

  8. Bertoa MF, Troya JM, Vallecillo A (2005) Measuring the usability of software components. J Syst Softw 79:427–439 (S48)

    Article  Google Scholar 

  9. Borgholm T, Madsen KH (1999) Cooperative usability practices. Commun ACM 42(5):91–97 (S103)

    Article  Google Scholar 

  10. Graham MJ, Kubose TK, Jordan D et al (2004) Heuristic evaluation of infusion pumps: implications for patient safety in intensive care units. Int J Med Inf 73(11–12):771–779 (S36)

    Article  Google Scholar 

  11. Schusteritsch R, Wei CY, Larosa M (2007) Towards the perfect infrastructure for usability testing on mobile devices. In: Proceeding of CHI EA ‘07 CHI ‘07 Extended Abstracts on Human Factors in Computing Systems, ACM, pp 1839–1844. (S25)

  12. Bodker S, Buur J (2002) The design collaboratorium—a place for usability design. ACM Trans Comput Hum Interact 9(2):152–169 (S15)

    Article  Google Scholar 

  13. Clamann M, Kaber BD (2004) Applicability of usability evaluation techniques to aviation systems. Int J Aviat Psychol 14(4):385–420 (S123)

    Article  Google Scholar 

  14. Peevers G, Douglas G, Jack AM (2008) A usability comparison of three alternative message formats for an sms banking service. Int J Hum Comput Stud 66:113–123 [ELSEVIER, (S128)]

    Article  Google Scholar 

  15. Weir SC, Anderson NJ, Jack AM (2006) On the role of metaphor and language in design of third party payments in eBanking: usability and quality. Int J Hum Comput Stud 64:770–784 [ELSEVIER, (S129)]

    Article  Google Scholar 

  16. Chan AJ, Islam MK, Rosewall T et al (2012) Applying usability heuristics to radiotherapy systems. Radiother Oncol 102(1):142–147 [ELSEVIER, (S37)]

    Article  Google Scholar 

  17. Kushniruka AW, Triola MM, Borycki EM et al (2005) Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application. Int J Med Inf 74:519–526 [ELSEVIER, (S26)]

    Article  Google Scholar 

  18. Bastien JMC Usability testing (2010) a review of some methodological and technical aspects of the method. Int J Med Inf 79(4):18–23 [ELSEVIER, (S92)]

    Article  Google Scholar 

  19. Mansar LS, Jariwala S, Shahzad M et al (2012) A usability testing experiment for a localized weight loss mobile application. Proc Technol 5:839–848 [ELSEVIER, (S118)]

    Article  Google Scholar 

  20. Hochhauser M, Gal E (2015) Weiss LP Negotiation strategy video modeling training for adolescents with autism spectrum disorder: a usability study. Int J Hum Comput Interact 31:472–480 (S146)

    Article  Google Scholar 

  21. Kjeldskov J, Skov MB, Stage J (2004) Instant data analysis: conducting usability evaluations in a day. In: Proceeding NordiCHI ‘04 Proceedings of the third Nordic conference on Human-computer interaction, ACM, pp 233–240 (S57)

  22. Hua L, Gong Y (2013) Usability evaluation of a voluntary patient safety reporting system: understanding the difference between predicted and observed time values by retrospective think-aloud protocols. Human-computer interaction. Springer, New York, pp 94–100 (S75)

    Google Scholar 

  23. Zapata CB, Fernandez-Aleman LJ, Idri A et al (2015) Empirical studies on usability of mhealth apps: a systematic literature review. J Med Syst 39:1 [Springer, (S134)]

    Article  Google Scholar 

  24. Lewis RJ (2015) Introduction to the special issue on usability and user experience: psychometrics. Int J Hum Comput Interact 31:481–483

    Article  Google Scholar 

  25. Lewis RJ, Utesch SB (2015) Maher ED Measuring perceived usability: the sus, umux-lite and altusability. Int J Hum Comput Interact 31:496–505 (S147)

    Article  Google Scholar 

  26. Kortum P, Sorber M (2015) Measuring the usability of mobile applications for phones and tablets. Int J Hum Comput Interact 31:518–529 (S29)

    Article  Google Scholar 

  27. Hoehle H, Aljafari R, Ventakatesh V (2016) Leveraging microsoft’s mobile usability guidelines: conceptualizing and developing scales for mobile application usability. Int J Hum Comput Stud 89:35–53 [ELSEVIER, (S113)]

    Article  Google Scholar 

  28. Moumane K, Idri A, Abran A (2016) Usability evaluation of mobile applications using iso 9241 and iso 25062 standards. SpringerPlus, 5:1–15 (S135)

  29. McClellan AM, Karumur PR, Vogel IR et al (2016) Designing an educational website to improve quality of supportive oncology care for women with ovarian cancer: an expert usability review and analysis. Int J Hum Comput Stud 32:297–307 (S139)

    Article  Google Scholar 

  30. Rodriguez DF, Acuna TS, Juristo N (2015) Design and programming patterns for implementing usability functionalities in web applications. J Syst Softw 105:107–124 [ELSEVIER, (S106)]

    Article  Google Scholar 

  31. Bringula PR (2016) Factors affecting web portal information services usability: a canonical correlation analysis. Int J Hum Comput Interact 32(10):814–826 (S148)

    Google Scholar 

  32. Battleson B, Booth A, Weintrop J (2001) Usability testing of an academic library web site: a case study. J Acad Librariansh 27(3):188–198

    Article  Google Scholar 

  33. Ardito C, Costabile FM, Marsico DM (2006) An approach to usability evaluation of e-learning applications. Univers Access Inf Soc 4:270–283

    Article  Google Scholar 

  34. Rupp AM, Oppold P, McConnell SD (2015) Evaluating input device usability as a function of task difficulty in a tracking task. Ergonomics 58(5):722–735 (S140)

    Article  Google Scholar 

  35. Jankowski J (2015) Grabowski A Usability evaluation of vr interface for mobile robot teleoperation. Int J Hum Comput Interact 31:882–889 (S20)

    Article  Google Scholar 

  36. Kolade WYA (2015) Integrating usability work into a large inter-organisational agile development project: tactics developed by usability designers. J Syst Softw 100:54–66 [ELSEVIER, (S110)]

    Article  Google Scholar 

  37. Gonzalez MP, Lores J (2008) Granollers A Enhanching usability testing through datamining techniques: a novel approach to detecting usability problem patterns for a context of use. Inf Softw Technol 50:547–568 [ELSEVIER, (S86)]

    Article  Google Scholar 

  38. Kalpna S, Anju S (2016) Enhancing usability inspection through data-mining techniques: an automated approach for detecting usability problem patterns of academic websites. International conference on intelligent human computer interaction. Springer, Cham

  39. Kalpna S, Anju S (2017) Qualitative usability feature selection with ranking: a novel approach for ranking the identified usability problematic attributes for academic websites using data-mining techniques. Human-centric computing and information sciences, Springer, New York [Accepted]

  40. Panach IJ, Juristo N, Valverde F et al (2015) A framework to identify primitives that represent usability within model-driven development methods. Inf Softw Technol 58:338–354 [ELSEVIER, (S115)]

    Article  Google Scholar 

  41. Stapić Z (2012) Procedures for performing systematic literature review in software engineering. CECIIS 2012-23rd International Conference

  42. Fernandez A, Insfran E, Abrahão S (2011) Usability evaluation methods for the web: a systematic mapping study. Inf Softw Technol 53:789–817 [ELSEVIER, (S102)]

    Article  Google Scholar 

  43. Bevan N, Azuma M (1997) Quality in use: incorporating human factors into the software engineering lifecycle. In: Proceedings of the third IEEE international symposium and forum on software engineering standards, pp 169–179 (S1)

  44. Clemmensen T (2011) Templates for cross-cultural and culturally specific usability testing: results from field studies and ethnographic interviewing in three countries. Int J Hum Comput Int 27(7):634–669 (S31)

    Article  Google Scholar 

  45. Heiskari HJ, Kauppinen M, Runonen M et al. (2009) Bridging the gap between usability and requirements engineering. In: 17th IEEE international in requirements engineering conference, pp 303–308 (S2)

  46. Winter J, Rönkkö K (2009) Satisfying stakeholders’ needs—balancing agile and formal usability test results. e-Inf Softw Eng J 3:1 (S32)

  47. Lallemand C (2011) Toward a closer integration of usability in software development: a study of usability inputs in a model driven engineering process. In: in proceedings of the 3rd ACM SIGCHI symposium on Engineering interactive computing systems, pp 299–302 (S3)

  48. Polgár BP (2015) Using the cognitive walkthrough method in software process improvement. e-Inf Softw Eng J 9(1):79–85 (S33)

    Google Scholar 

  49. Burr J (1999) Bagger K Replacing usability testing with user dialogue. Commun ACM 42(5):63–66 (S4)

    Article  Google Scholar 

  50. Lee Y, Kozar AK (2012) Understanding of website usability: specifying and measuring constructs and their relationships. Decis Support Syst 52:450–463 [ELSEVIER, (S34)]

    Article  Google Scholar 

  51. Bass L (2003) John BE Linking usability to software architecture patterns through general scenarios. J Syst Softw 66:187–197 (S5)

    Article  Google Scholar 

  52. Vilbergsdottira GS, Hvannbergb TE, Law ELC (2014) Assessing the reliability, validity and acceptance of a classificationscheme of usability problems (cup). J Syst Softw 87:18–37 [ELSEVIER, (S35)]

    Article  Google Scholar 

  53. Folmer E, Bosch J (2004) Architecting for usability: a survey. J Syst Softw 70:61–78 (S6)

    Article  Google Scholar 

  54. Nebe K, Paelke V (2009) Usability-engineering-requirements as a basis for the integration with software engineering. In: Proceedings of 13th International Conference Human-Computer Interaction, Part I, pp 652–659. (S7)

  55. Folmer E, Gurp JV, Bosch J (2005) Software architecture analysis of usability. In: Proceedings of international conference eng. human-computer interaction and interactive systems, pp 38–58 (S8)

  56. Makri S, Blandford A, Cox LA et al (2011) Evaluating the information behaviour methods:formative evaluations of two methods for assessing the functionality and usability of electronic information resources. Int J Hum Comput Stud 69:455–482 [ELSEVIER, (S38)]

    Article  Google Scholar 

  57. John BE, Bass L, Segura MIS et al. (2005) Bringing usability concerns to the design of software architecture. In: Proceedings of international conference on engineering human-computer interaction and interactive systems, pp 1–19 (S9)

  58. Hiniker A, Sobel K, Hong S, Suh H et al (2016) Hidden symbols: how informal symbolism in digital interfaces disrupts usability for preschoolers. Int J Hum Comput Stud 90:53–67 [ELSEVIER, (S39)]

    Article  Google Scholar 

  59. Juristo N, Moreno AM, Segura MIS (2007) Analysing the impact of usability on software design. J Syst Softw 80(9):1506–1516 (S10)

    Article  Google Scholar 

  60. Khawaja AM, Chen F (2014) Marcus N Measuring cognitive load using linguistic features: implications for usability evaluation and adaptive interaction design. Int J Hum Comput Interact 30:343–368 (S40)

    Article  Google Scholar 

  61. Seffah A, Mohamed T, Mammar HH et al (2008) Reconciling usability and interactive system architecture using patterns. J Syst Softw 81:1845–1852 (S11)

    Article  Google Scholar 

  62. Aryana B, Clemmensen T (2013) Mobile usability: experiences from iran and turkey. Int J Hum Comput Interact 29:220–242 (S41)

    Article  Google Scholar 

  63. Juristo N, Moreno AM, Segura MS (2007) Guidelines for eliciting usability functionalities. IEEE Trans Softw Eng 33(11):744–758 (S12)

    Article  Google Scholar 

  64. Tamir DE, Mueller CJ (2010) Pinpointing usability issues using an effort based framework. In: IEEE, international conference on systems man and cybernetics, pp 931–938 (S42)

  65. Seffah A, Djouab R, Antunes H (2001) Comparing and reconciling usability-centered and use case-driven requirements engineering processes. Aust Comput Sci Commun IEEE Comput Soc 23(5):132–139 (S13)

    Google Scholar 

  66. Moorthy JTS, Ibrahim SB, Mahrin MN (2013) Formulation of usability risk assessment model, IEEE, Conference on open systems (S43)

  67. Beu A, Honold P (2000) Yuan X How to build up an infrastructure for intercultural usability engineering. Int J Hum Comput Interact 12(3 and 4):347–358 (S14)

    Article  Google Scholar 

  68. Nielsen J, Landauer KT (1993) A mathematical model of the finding of usability problems, ACM, Conference on Human Factors in Computing Systems pp 206–213 (S44)

  69. Dubey KS, Rana A (2011) Usability estimation of software system by using object-oriented metrics. ACM SIGSOFT Softw Eng Notes 36(2):371–382 (S45)

    Article  Google Scholar 

  70. Roberts JM, Newton JE, Lagattolla DF et al (2013) Objective versus subjective measures of paris metro map usability: investigating traditional octolinear versus all-curves schematics. Int J Hum Comput Stud 71:363–386 [ELSEVIER, (S16)]

    Article  Google Scholar 

  71. Sauro J, Kindlund E (2005) A method to standardize usability metrics into a single score, ACM, CHI. 1-58113-998-5 (S46)

  72. Hertzum M (2006) Problem prioritization in usability evaluation: from severity assessments toward impact on design. Int J Hum Comput Interact 21(2):125–146 (S17)

    Article  Google Scholar 

  73. Hornbaek K (2006) Current practice in measuring usability: challenges to usability studies and research. Int J Hum Comput Stud 64:79–102 (S47)

    Article  Google Scholar 

  74. Metzker M, Offergeld M (2001) An interdisciplinary approach for successfully integrating human-centered design methods into development processes practiced by industrial software development organizations. In: Proceedings of International Conference on Engineering for Human-Computer Interaction. Springer, New York (S18)

  75. Hertzum MA, Clemmensen T (2012) How do usability professionals construe usability? Int J Hum Comput Stud 70:26–42 [ELSEVIER, (S19)]

    Article  Google Scholar 

  76. Erickson W, Trerise S, Lee C et al (2013) The accessibility and usability of college websites: is your website presenting barriers to potential students? Commun Coll J Res Pract 37:864–876 (S49)

    Article  Google Scholar 

  77. Dubey KS, Rana A (2010) Assessment of usability metrics for object-oriented software system. ACM SIGSOFT Softw Eng Notes 35 (S50)

  78. Lavie T, Gilad OT (2011) Meyer J Aesthetics and usability of in-vehicle navigation displays. Int J Hum Comput Stud 69:80–99 [ELSEVIER, (S21)]

    Article  Google Scholar 

  79. Frojkaer E, Hertzum M, Hornbaek K (2000) Measuring usability: are effectiveness, efficiency and satisfaction really correlated, ACM Press, CHI, pp 345–352 (S51)

  80. Carvajal L, Moreno AM, Segura MIS et al (2013) Usability through software design. IEEE Trans Softw Eng 39(11):1582–1596 (S22)

    Article  Google Scholar 

  81. Babiker EM, FH Boyle et al. (1991) A metric for hypertext usability, ACM Press. In: Proceedings of 11th annual international conference on system documentation, pp 95–104 (S52)

  82. Carvajal L (, ) Usability-enabling guidelines: a design pattern and software plug-in solution. In: ESEC/FSE Doctoral symposium, ACM (S23)

  83. Beckles B, Welch V (2005) Basney J Mechanisms for increasing the usability of grid security. Int J Hum Comput Stud 63:74–101 [ELSEVIER, (S53)]

    Article  Google Scholar 

  84. McGee M (2004) Master usability scaling: magnitude estimation and master scaling applied to usability measurement, ACM. In: Proceedings of CHI, pp 335–342 (S54)

  85. Nielsen J, Levy J (1994) Measuring usability: preference vs. performance. Commun ACM 37:66–76 (S55)

    Article  Google Scholar 

  86. Sauro J, Kindlund E (2005) Making sense of usability metrics: usability and six sigma, UPA conference (S56)

  87. Teruel AM, Navarro E, Jaquero LV et al (2014) A cscw requirements engineering case tool: development and usability evaluation. Inf Softw Technol 56:922–949 [ELSEVIER, (S27)]

    Article  Google Scholar 

  88. Lindgaard G (2015) Challenges to assessing usability in the wild: a case study. Int J Hum Comput Interact 31:618–631 (S28)

    Article  Google Scholar 

  89. Skov MB, Stage J (2005) Supporting problem identification in usability evaluation, CHI. In: Proceedings of OZCHI. 1-59593-222-4. [S58]

  90. Mapayi T, Olaniyan OM, Isamotu NO et al (2013) Evaluating usability factors in different authentication methods using artificial neural network. IEEE. Afr J Comput ICT 6(1):69–78 (S59)

    Google Scholar 

  91. Johannessen JHG, Hornbæk K (2014) Must evaluation methods be about usability? devising and assessing the utility inspection method. Behav Inf Technol 33(2):195–206 (S30)

    Article  Google Scholar 

  92. Ivory MY, Hearst MA (2001) The state of the art in automated usability evaluation of user interfaces. ACM Comput Surv 33:470–516 (S60)

    Article  Google Scholar 

  93. Lee S, Cho JE (2007) Usability evaluation of korean e-government portal, SPRINGER. In: universal access in human-computer interaction applications and services, pp 64–72 (S61)

  94. González M, Masip L, Granollers A et al (2009) Quantitative analysis in a heuristic evaluation experiment. Adv Eng Softw 40:1271–1278 [Elsevier, (S62)]

    Article  Google Scholar 

  95. Delice EK (2009) Z Gungor The usability analysis with heuristics evaluation and analytic hierarchy process. Int J Ind Ergon 39:934–993 [Elsevier, (S63)]

    Article  Google Scholar 

  96. Ríos AD, García VA, Rey ME et al (2010) Usability: a critical analysis and taxonomy. Int J Hum Comput Interact 26(1):53–74 (S108)

    Article  Google Scholar 

  97. Bradner E, Dawe M (2008) Parts of the sum: a case study of usability benchmarking using the sum metric. In: UPA international conference (S64)

  98. Winter S, Wagner S, Deissenboeck F (2008) A comprehensive model of usability, Springer, International Federation for Information Processing, pp 106–122 (S109)

  99. Kline RB, Seffah A (2005) Evaluation of integrated software development environments: challenges and results from three empirical studies. Int J Hum Comput Stud 63:607–627 [ELSEVIER, (S65)]

    Article  Google Scholar 

  100. Tamir D, Komogortsev OV, Mueller CJ (2008) An effort and time based measure of usability, ACM, WoSQ. 978-1-60558-023-4 (S66)

  101. Scheller T, Kuhn E (2015) Automated measurement of api usability: the api concepts framework. Inf Softw Technol 61:145–162 [ELSEVIER, (S111)]

    Article  Google Scholar 

  102. Tullis T, Fleischman S, McNulty M et al. (2002) An empirical comparison of lab and remote usability testing of web sites, In: Proceedings of Usability Professionals Conference (S67)

  103. Santana VFD, Baranauskas MCC (2015) Welfit: a remote evaluation tool for identifying web usage patterns through client-side logging. Int J Hum Comput Stud 76:40–49 [ELSEVIER, (S112)]

    Article  Google Scholar 

  104. Følstad A (2007) Work-domain experts as evaluators: usability inspection of domain-specific work support systems. Int J Hum Comput Interact 22(3):217–245 (S68)

    Article  Google Scholar 

  105. Frøkjær E, Hornbæk K (2005) Cooperative usability testing: complementing usability tests with user-supported interpretation sessions. In: CHI’ 05 extended abstracts on human factors in computing systems, ACM, pp 1383–1386 (S69)

  106. Brown M, Sharples S, Harding J (2013) Introducing pegi: a usability process for the practical evaluation of geographic information. Int J Hum Comput Interact 71:668–678 [ELSEVIER, (S114)]

    Google Scholar 

  107. Thompson KE, Rozanski EP, Haake AR (2004) Here, there, anywhere: remote usability testing that works in, SIGITE, ACM, New York, pp 132–137 (S70)

  108. Bak JO, Nguyen K, Risgaard P et al (2008) Obstacles to usability evaluation in practice: a survey of software development organizations, ACM. In: Proceedings of Nordi CHI (S71)

  109. Harrison R, Flood D, Duce D (2013) Usability of mobile applications: literature review and rationale for a new usability model. J Interact Sci 1:1

    Article  Google Scholar 

  110. Winter J, Hinely M (2011) Examining correlations in usability data to effectivize usability testing. e-Inf Softw Eng J 5(1):25–37 (S72)

    Google Scholar 

  111. Castilla D, Palacios GA, Lopez BJ et al (2013) Process of design and usability evaluation of a telepsychology web and virtual reality system for elderly: butler. Int J Hum Comput Stud 71:350–362 [ELSEVIER, (S117)]

    Article  Google Scholar 

  112. Bruun A, Stage J (2012) Training software development practitioners in usability testing: an assessment acceptance and prioritization, OZCHI, ACM (S73)

  113. Sauer J, Seibel K, Ruttinger B (2014) The influence of user expertise and prototype fidelity in usability tests. Appl Ergon 41:130–140 [ELSEVIER, (S74)]

    Article  Google Scholar 

  114. Hurtado N, Ruiz M, Orta E et al (2015) Using simulation to aid decision making in managing the usability evaluation process. Inf Softw Technol 57:509–526 (S119)

    Article  Google Scholar 

  115. Hasan L, Morris A, Probets S (2012) A comparison of usability evaluation methods for evaluating e-commerce websites. Behav Inf Technol 31(7):707–737 (S120)

    Article  Google Scholar 

  116. Hong JI, Heer J, Waterson S et al (2001) Webquilt: a proxy-based approach to remote web usability testing. ACM Trans Inf Syst 19(3):263–285 (S76)

    Article  Google Scholar 

  117. Kjeldskov J, Stage J (2004) New techniques for usability evaluation of mobile systems. Int J Hum Comput Stud 60:599–620 [ELSEVIER, (S121)]

    Article  Google Scholar 

  118. Nielsen J, Molich R (1990) Heuristic evaluation of user interfaces. In: Proceedings ACM CHI pp 249–256 (S77)

  119. Ji GY, Park HJ, Lee C et al (2006) A usability checklist for the usability evaluation of mobile phone user interface. Int J Hum Comput Interact 3(3):207–231 (S122)

    Article  Google Scholar 

  120. Nielsen J (1994) Enhancing the explanatory power of usability heuristics, ACM, CHI Human factors in computing systems, 0-89791-650-6/94/0152 (S78)

  121. Torrente SCM, Prieto MBA, Gutiérrez AD et al (2013) Sirius: a heuristic-based framework for measuring web usability adapted to the type of website. J Syst Softw 86:649–663 [ELSEVIER, (S79)]

    Article  Google Scholar 

  122. Folmer E, Welie VM, Bosch J (2006) Bridging patterns: an approach to bridge gaps between se and hci. Inf Softw Technol 48:69–89 [ELSEVIER, (S124)]

    Article  Google Scholar 

  123. Andreasen MS, Nielsen HV, Schrøder SO (2007) What happened to remote usability testing? an empirical study of three methods, CHI, ACM, pp 1405–1414. (S80)

  124. Roberts LV, Fels ID (2006) Methods for inclusion: employing think aloud protocols in software usability studies with individuals who are deaf. Int J Hum Comput Stud 64:489–501 [ELSEVIER, (S125)]

    Article  Google Scholar 

  125. Wright PC, Monk AF (1991) The use of think—aloud evaluation methods in design. ACM SIGCHI 23(1):55–57 (S81)

    Article  Google Scholar 

  126. Nivala MA, Sarjakoski TL, Sarjakoski T (2007) Usability methods familiarity among map application developers. Int J Hum Comput Stud 65:784–795 (S126)

    Article  Google Scholar 

  127. Wharton C, Rieman J, Lewis C et al (1994) (1994) The cognitive walkthrough method: a practitioner’s guide. In: Nielsen J, Mack RL (eds) Usability inspection methods. Wiley, Hoboken, pp 105–140 (S82)

    Google Scholar 

  128. Propp S, Buchholz G, Forbrig P (2009) Integration of usability evaluation and model-based software development. Adv Eng Softw 40:1223–1230 [ELSEVIER, (S127)]

    Article  MATH  Google Scholar 

  129. Leow CM, Wang KYL, Lau HS et al (2016) Usability of rpg-based learning framework. Int J Hum Comput Interact. doi:10.1080/10447318.2016.1183863 (S83)

    Google Scholar 

  130. Følstada A, Hornbæk K (2010) Work-domain knowledge in usability evaluation: experiences with cooperative usability testing. J Syst Softw 83(11):2019–2030 (S84)

    Article  Google Scholar 

  131. Skov MB, Stage J (2004) Integrating usability design and evaluation: training novice evaluators in usability testing. In: Proceedings of the workshop on Improving the interplay between Usability Evaluation and User Interface Design, ACM (S85)

  132. Isa WAM, Suhami MR, Safie NI et al (2011) Accessing the usability and accessbility of malaysia e-government website. Am J Econ Bus Adm 3(1):40–46 (S130)

    Google Scholar 

  133. Yammiyavar P, Clemmensen T (2008) Kumar J Influence of cultural background on non-verbal communication in a usability testing situation. Int J Des 2(2):31–40 (S131)

    Google Scholar 

  134. Hartson HR, Castillo JC, Kelso J (1996) Remote evaluation: the network as an extension of usability laboratory, ACM, CHI. 089791-777-4/96/04 (S87)

  135. Hertzum M, Clemmensen T, Hornbæk K et al (2011) Personal usability constructs: how people construe usability across nationalities and stakeholder groups. Int J Hum Comput Interact 27(8):729–761 (S132)

    Google Scholar 

  136. Nielsen J (1992) Finding usability problems through heuristics evaluation. In: Proceedings ACM CHI pp 373–380 (S88)

  137. Rivero L, Conte T (2013) Using an empirical study to evaluate the feasibility of a new usability inspection technique for paper based prototypes of web applications. J Softw Eng Res Dev 1:2 [Springer, (S133)]

    Article  Google Scholar 

  138. Mankoff J, Dey AK, Hsieh G et al (2003) Heuristic evaluation of ambient displays, ACM, CHI: new horizons, 5:1 (S89)

  139. Ko MS, Chang SW, Ji GY (2013) Usability principles for augmented reality applications in a smartphone environment. Int J Hum Comput Interact 29:501–515 (S90)

    Article  Google Scholar 

  140. Sawyer P, Flanders A, Wixon D (1996) Making a difference-the impact of inspections. In: Proceedings of CHI, ACM press (S91)

  141. Hamborg CK, Vehse B, Bludau BH (2004) Questionnaire based usability evaluation of hospital information systems. Electron J Inf Syst Eval 7(1):21–30 (S136)

    Google Scholar 

  142. Arnhold M, Quade M, Kirch W (2014) Mobile applications for diabetics: a systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older. J Med Internet Res 14(4):1–18 (S137)

    Google Scholar 

  143. Paradowski M, Fletcher A (2004) Using task analysis to improve usability of fatigue modelling software. Int J Hum Comput Stud 60:101–115 [ELSEVIER, (S93)]

    Article  Google Scholar 

  144. Mirkovic J, Kaufman RD (2014) Rutland MC (2014) Supporting cancer patients in illness management: usability evaluation of a mobile app. JMIR MHEALTH AND UHEALTH 2(3):1–21 (S138)

    Article  Google Scholar 

  145. Holzinger A (2005) Usability engineering methods for software developers. Commun ACM 48(1):71–74 (S94)

    Article  Google Scholar 

  146. Spencer R (2000) The streamlined cognitive walkthrough method: working around social constraints encountered in software development company. In: Proceedings of the conference on human factors in computing systems (CHI), ACM press, New York (S95)

  147. Christophersen T (2011) Konradi U Reliability, validity and sensitivity of a single-item measure of online store usability. Int J Hum Comput Stud 69:269–280 [ELSEVIER, (S96)]

    Article  Google Scholar 

  148. Datcu D, Lukosch S, Brazier F (2015) On the usability and effectiveness of different interaction types in augmented reality. Int J Hum Comput Interact 31:193–209 (S141)

    Article  Google Scholar 

  149. Leuthold S, Avila BAJ, K K (2008) Beyond web content accessibility guidelines: design of enhanced text user interfaces for blind internet users. Int J Hum Comput Stud 66:257–270 [ELSEVIER, (S97)]

    Article  Google Scholar 

  150. Bias GR, Moon MB (2015) Hoffman RR Concept mapping usability evaluation: an exploratory study of a new usability inspection method. Int J Hum Comput Interact 31(9):571–583 (S142)

    Article  Google Scholar 

  151. Kim JK, Han HS, Yun HM et al (2002) A systematic procedure for modeling usability based on product design variables: a case study in audiovisual consumer electronic products. Int J Occup Saf Ergon 8(3):387–406 (S98)

    Article  Google Scholar 

  152. Tung LL, Xu Y, Tan BF (2009) Attributes of web site usability: a study of web users with the repertory grid technique. Int J Electron Commer 13(4):97–126 (S143)

    Article  Google Scholar 

  153. Chung W, Chen H, Chaboya GL et al (2005) Evaluating event visualization: a usability study of coplink spatio-temporal visualizer. Int J Hum Comput Stud 62:127–157 [ELSEVIER, (S99)]

    Article  Google Scholar 

  154. Choi KI, Kim SW, Lee D et al (2015) A weighted qfd-based usability evaluation method for elderly in smart cars. Int J Hum Comput Interact 31(10):703–716 (S144)

    Article  Google Scholar 

  155. Roth V, Straub T, Richter K (2005) Security and usability engineering with particular attention to electronic mail. Int J Hum Comput Stud 63:51–73 [ELSEVIER, (S100)]

    Article  Google Scholar 

  156. Nawaz A (2013) Clemmensen T Website usability in asia”from within: an overview of a decade of literature. Int J Hum Comput Interact 29:256–273 (S145)

    Article  Google Scholar 

  157. Howell M, Love S, Turner M (2006) Visualisation improves the usability of voice-operated mobile phone services. Int J Hum Comput Stud 64:754–769 [ELSEVIER, (S101)]

    Article  Google Scholar 

  158. West R, Lehman KR (2006) Automated summative usability studies: an empirical evaluation, ACM, CHI 2006 Proceedings. 1-59593-178-3/06/0004 (S104)

  159. Shamim A, Balakrishnan V, Tahir M et al (2016) Age and domain specific usability analysis of opinion visualisation techniques. Behav Inf Technol 35(8):680–689 (S149)

    Article  Google Scholar 

  160. Wixon D (2003) Evaluating usability methods: why the current literature fails the practitioner, ACM, New York, 1072-5220/03/0700 (S105)

  161. Hanrath S, Kottman M (2015) Use and usability of a discovery tool in an academic library. J Web Librariansh 9:1–21 (S150)

    Article  Google Scholar 

  162. Abran A, Khelifi A, Suryn W (2003) Usability meanings and interpretations in iso standards. Softw Qual J 11:325–338

    Article  Google Scholar 

  163. Arms WY (2000) Digital libraries. MIT Pr, Cambridge

    Google Scholar 

  164. Bevan N, Kirakowsi J, Maissel J (1991) What is usability? In: Proceedings of the 4th international conference on HCI, pp 651–655

  165. Boëhm B (1978) Characteristics of software quality, Vol 1 TRW series on software technology. North-Holland, Amsterdam

    Google Scholar 

  166. Booth P (1989) An introduction to human-computer interaction. Lawrence Erlbaum Associates Publishers, Hove/East Sussex

    Google Scholar 

  167. Lewis RJ (1995) IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact 7:57–78

    Article  Google Scholar 

  168. Blandford A, George B (2002) Usability for digital libraries. In: Proceedings of the second ACM/lEEECS joint conference on digital libraries. ACM Press, New York, p 424

  169. Brinck Tom, Gergle Darren, Wood Scott D (2002) Designing web sites that work: usability for the web. Morgan Kaufmann, San Francisco

    Google Scholar 

  170. Constantine L, Lockwood LAD (1999) Software for use: a practical guide to the models and methods of usage-centered design. Addison-Wesley, New York

    Google Scholar 

  171. Dix A, Finley J, Abowd G et al (1998) Human-computer interaction, 2nd edn. Prentice-Hall, Englewood Cliffs

    Google Scholar 

  172. Donyaee M, Seffah A (2001) Quim: an integrated model for specifying and measuring quality in use. In: Eighth IFIP Conference on Human Computer Interaction. Tokyo, Japan

  173. Grady RB (1992) Practical software metrics for project management and process improvement. Prentice Hall, Englewood Cliffs

    Google Scholar 

  174. Gould JD (1988) How to design usable systems. In: Helander M (ed) Handbook of human computer interaction. Elsevier, New York, pp 57–89

    Google Scholar 

  175. Hix D, Hartson HR (1993) Developing user interfaces: ensuring usability through product & process. Wiley, New York

    MATH  Google Scholar 

  176. Löwgren J (1993) Human-computer interaction: What every system developer should know. Studentlitteratur, Lund

    Google Scholar 

  177. McCall JA, Richards PK, Walters GF (1977) Factors in software quality, vol II. Rome Aid Defence Centre, Italy

    Google Scholar 

  178. Macleod M, Bowden R, Bevan N et al (1997) The music performance method. Behav Inf Technol 16:279–293

    Article  Google Scholar 

  179. Preece J, Benyon D, Davies G et al (1993) A guide to usability: human factors in computing. Addison-Wesley, Reading

    Google Scholar 

  180. Preece J, Rogers Y, Sharp H et al (1994) Human-computer interaction. Addison-Wesley, Reading

    Google Scholar 

  181. Quesenbery W (2001) What does usability mean: looking beyond ease of use. In: Proceedings of the 18th Annual Conference Society for Technical Communications

  182. Shneiderman B, Plaisant C (2005) Designing the user interface: strategies for effective human-computer interaction. Addison Wesley, Boston

    Google Scholar 

  183. Shackel B (1991) Usability—context, framework, definition, design and evaluation. In: Shackel B, Richardson SJ (eds) Human factors for informatics usability. Cambridge University Press, New York, pp 21–37

    Google Scholar 

  184. Bevan N (1995) Human-computer interaction standards. In: Proceedings of the 6th international conference on human computer interaction, Elsevier, Amsterdam

  185. Bevan N (2009) International standards for usability should be more widely used. J Usability Stud 4(3):106–113

    Google Scholar 

  186. Bevan N, Schoeffel R (2001) A proposed standard for consumer product usability. In: Proceedings of Ist international conference on universal access in human computer interaction, pp 557—561

  187. DIN EN ISO 13407 (1999s) Human-centered design processes for interactive systems, CEN—European Committee for Standardization, Brussels

  188. ISO/PAS 18152 (2003) Ergonomics of human-system interaction-specification for the process assessment of human-system issues. ISO, Genf

  189. Lewis RJ (2014) Usability: lessons learned…and yet to be learned. Int J Hum Comput Interact 30:663–684

    Article  Google Scholar 

  190. Seffah A, Mammar HH (2009) Usability engineering laboratories: limitations and challenges toward a unifying tools/practices environment. Behav Inf Technol 28(3):281–291 (94)

    Article  Google Scholar 

  191. Kortum Philip T, Bangor Aaron (2013) Usability ratings for everyday products measured with the System Usability Scale. Int J Hum Comput Interact 29(2):67–76

    Article  Google Scholar 

  192. Kortum P, Peres CS (2014) The relationship between system effectiveness and subjective usability scores using the system usability scale. Int J Hum Comput Interact 30:575–584

    Article  Google Scholar 

Download references

Acknowledgements

Darshan Lal, Dipta Sakkarwal, Ricky Tharan, Shruti Singh, Surbhi Garg and Upasana Kundra are gratefully acknowledged for their assistance in providing their suggestions for improvement of this SLR. We also thank Associate Professors of Guru Gobind Singh Indraprastha University for their help in conceptualizing this work.

Author information

Authors and Affiliations

  1. University School of Information and Communication Technology GGSIPU, Sector 16-c, Delhi, 110078, India

    Kalpna Sagar & Anju Saha

Authors
  1. Kalpna Sagar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anju Saha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kalpna Sagar.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sagar, K., Saha, A. A systematic review of software usability studies. Int. j. inf. tecnol. (2017). https://doi.org/10.1007/s41870-017-0048-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s41870-017-0048-1

Keywords

Search

Navigation