Getting a Technology-Based Diabetes Intervention Ready for Prime Time: a Review of Usability Testing Studies
Consumer health technologies can educate patients about diabetes and support their self-management, yet usability evidence is rarely published even though it determines patient engagement, optimal benefit of any intervention, and an understanding of generalizability. Therefore, we conducted a narrative review of peer-reviewed articles published from 2009 to 2013 that tested the usability of a web- or mobile-delivered system/application designed to educate and support patients with diabetes. Overall, the 23 papers included in our review used mixed (n = 11), descriptive quantitative (n = 9), and qualitative methods (n = 3) to assess usability, such as documenting which features performed as intended and how patients rated their experiences. More sophisticated usability evaluations combined several complementary approaches to elucidate more aspects of functionality. Future work pertaining to the design and evaluation of technology-delivered diabetes education/support interventions should aim to standardize the usability testing processes and publish usability findings to inform interpretation of why an intervention succeeded or failed and for whom.
KeywordsUsability testing Technology Diabetes Review User-centered design Think aloud Cognitive walkthrough Heuristic evaluation Intervention
Dr. Lyles is supported by a Career Development Award from the Agency for Healthcare Research and Quality (K99 HS022408). Dr. Sarkar is also supported by the Agency for Healthcare Research and Quality (R21 HS021322). Dr. Osborn is supported by a Career Development Award from the National Institute of Diabetes and Digestive and Kidney Diseases (K01 DK087894). The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent these funding entities. We thank Lina Tieu, MPH, for her assistance in reviewing and finalizing the manuscript.
Compliance with Ethics Guidelines
Conflict of Interest
Courtney R. Lyles, Urmimala Sarkar, and Chandra Y. Osborn declare that they have no conflict of interest.
Human and Animal Rights and Informed Consent
This article does not contain any studies with human or animal subjects performed by any of the authors.
Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance
- 3.Or CK, Tao D. Does the use of consumer health information technology improve outcomes in the patient self-management of diabetes? A meta-analysis and narrative review of randomized controlled trials. Int J Med Inform 2014.Google Scholar
- 13.Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, Britten N, Roen K, Duffy S. Guidance on the conduct of narrative synthesis in systematic reviews. A Product from the ESRC Methods Programme 2006:1–92.Google Scholar
- 19.Wharton C, Rieman J, Lewis C, Polson P. The cognitive walkthrough method: a practitioner’s guide. In: Usability Inspection Methods. New York: Wiley; 1994. p. 105–41.Google Scholar
- 20.Lewis C, Rieman J. Chapter 4. Evaluating the design without users. In Task-centered user interface design: a practical introduction. Online via shareware; 1993, 1994.Google Scholar
- 21.Nielsen J, Molich R. Heuristic evaluation of user interfaces. In Proc ACM CHI'90 Conf; April 1–5; Seattle, WA. 1990: 249–256.Google Scholar
- 23.••Nijland N, van Gemert-Pijnen JE, Kelders SM, Brandenburg BJ, Seydel ER. Factors influencing the use of a Web-based application for supporting the self-care of patients with type 2 diabetes: a longitudinal study. J Med Internet Res. 2011;13:e71. This study used four usability approaches and collected data on reasons for discontinued use of a website. It only took 20 users to identify 166 unique usability problems, and users attributed a suboptimal user interface and limited interactivity as reasons for discontinued use.PubMedCrossRefPubMedCentralGoogle Scholar
- 24.•Froisland DH, Arsand E, Skarderud F. Improving diabetes care for young people with type 1 diabetes through visual learning on mobile phones: mixed-methods study. J Med Internet Res. 2012;14:e111. This study illustrates how to conduct a comprehensive qualitative and quantitative usability testing approach to identify technical problems, obtain users’ ideas for improvements, and, in turn, inform further development and testing.PubMedCrossRefPubMedCentralGoogle Scholar
- 26.Brooke J. SUS: a “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, editors. Usability evaluation in industry. London: Taylor and Francis; 1996.Google Scholar
- 27.••Ossebaard HC, Seydel ER, van Gemert-Pijnen L. Online usability and patients with long-term conditions: a mixed-methods approach. Int J Med Inform. 2012;81:374–87. This study employed four usability testing approaches, including software that recorded screen action during think aloud sessions. Utilization of a variety of methods uncovered negative user experiences, negative think aloud expressions, incomplete user tasks by most participants, and usability issues pertaining to navigation and non-personalized content that could be improved.PubMedCrossRefGoogle Scholar
- 29.Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. In Human Factors Group. Boca Raton, FL: IBM Corporation; 1993.Google Scholar
- 30.Monkman H, Kushniruk A. A health literacy and usability heuristic evaluation of a mobile consumer health application. Stud Health Technol Inf. 2013;192:724–8.Google Scholar
- 31.Broderick J, Devine T, Langhans E, Lemerise AJ, Lier S, Harris L. Designing health literate mobile apps. Washington: Institute of Medicine; 2014.Google Scholar
- 33.Sarkar U, Karter AJ, Liu JY, Adler NE, Nguyen R, Lopez A, et al. The literacy divide: health literacy and the use of an internet-based patient portal in an integrated health system—results from The Diabetes Study of Northern California (DISTANCE). J Health Commun. 2010;15 Suppl 2:183–96.PubMedCrossRefPubMedCentralGoogle Scholar
- 34.Smith A. Older adults and technology use. Washington: Pew Research Center; 2014.Google Scholar
- 37.Gude WT, Simon AC, Peute LW, Holleman F, Hoekstra JB, Peek N, et al. Formative usability evaluation of a web-based insulin self-titration system: preliminary results. Stud Health Technol Inf. 2012;180:1209–11.Google Scholar
- 39.Nielsen J, Mack RL, editors. Usability inspection methods. New York: Wiley; 1994.Google Scholar
- 44.Chomutare T, Tatara N, Arsand E, Hartvigsen G. Designing a diabetes mobile application with social network support. Stud Health Technol Inf. 2013;188:58–64.Google Scholar
- 49.Padman R, Jaladi S, Kim S, Kumar S, Orbeta P, Rudolph K, et al. An evaluation framework and a pilot study of a mobile platform for diabetes self-management: insights from pediatric users. Stud Health Technol Inf. 2013;192:333–7.Google Scholar
- 51.Tatara N, Arsand E, Bratteteig T, Hartvigsen G. Usage and perceptions of a mobile self-management application for people with type 2 diabetes: qualitative study of a five-month trial. Stud Health Technol Inf. 2013;192:127–31.Google Scholar