Current Diabetes Reports

, 14:534 | Cite as

Getting a Technology-Based Diabetes Intervention Ready for Prime Time: a Review of Usability Testing Studies

  • Courtney R. LylesEmail author
  • Urmimala Sarkar
  • Chandra Y. Osborn
Psychosocial Aspects (KK Hood, Section Editor)
Part of the following topical collections:
  1. Topical Collection on Psychosocial Aspects


Consumer health technologies can educate patients about diabetes and support their self-management, yet usability evidence is rarely published even though it determines patient engagement, optimal benefit of any intervention, and an understanding of generalizability. Therefore, we conducted a narrative review of peer-reviewed articles published from 2009 to 2013 that tested the usability of a web- or mobile-delivered system/application designed to educate and support patients with diabetes. Overall, the 23 papers included in our review used mixed (n = 11), descriptive quantitative (n = 9), and qualitative methods (n = 3) to assess usability, such as documenting which features performed as intended and how patients rated their experiences. More sophisticated usability evaluations combined several complementary approaches to elucidate more aspects of functionality. Future work pertaining to the design and evaluation of technology-delivered diabetes education/support interventions should aim to standardize the usability testing processes and publish usability findings to inform interpretation of why an intervention succeeded or failed and for whom.


Usability testing Technology Diabetes Review User-centered design Think aloud Cognitive walkthrough Heuristic evaluation Intervention 



Dr. Lyles is supported by a Career Development Award from the Agency for Healthcare Research and Quality (K99 HS022408). Dr. Sarkar is also supported by the Agency for Healthcare Research and Quality (R21 HS021322). Dr. Osborn is supported by a Career Development Award from the National Institute of Diabetes and Digestive and Kidney Diseases (K01 DK087894). The contents of this manuscript are solely the responsibility of the authors and do not necessarily represent these funding entities. We thank Lina Tieu, MPH, for her assistance in reviewing and finalizing the manuscript.

Compliance with Ethics Guidelines

Conflict of Interest

Courtney R. Lyles, Urmimala Sarkar, and Chandra Y. Osborn declare that they have no conflict of interest.

Human and Animal Rights and Informed Consent

This article does not contain any studies with human or animal subjects performed by any of the authors.


Papers of particular interest, published recently, have been highlighted as: • Of importance •• Of major importance

  1. 1.
    Bu D, Pan E, Walker J, Adler-Milstein J, Kendrick D, Hook JM, et al. Benefits of information technology-enabled diabetes management. Diabetes Care. 2007;30:1137–42.PubMedCrossRefGoogle Scholar
  2. 2.
    Archer E, Groessl EJ, Sui X, McClain AC, Wilcox S, Hand GA, et al. An economic analysis of traditional and technology-based approaches to weight loss. Am J Prev Med. 2012;43:176–82.PubMedCrossRefGoogle Scholar
  3. 3.
    Or CK, Tao D. Does the use of consumer health information technology improve outcomes in the patient self-management of diabetes? A meta-analysis and narrative review of randomized controlled trials. Int J Med Inform 2014.Google Scholar
  4. 4.
    LeRouge C, Wickramasinghe N. A review of user-centered design for diabetes-related consumer health informatics technologies. J Diabetes Sci Technol. 2013;7:1039–56.PubMedPubMedCentralGoogle Scholar
  5. 5.
    Corry MD, Frick TW, Hansen L. User-centered design and usability testing of a Web site: an illustrative case study. Etr & D-Educ Tech Res. 1997;45:65–76.CrossRefGoogle Scholar
  6. 6.
    Schleyer TK, Thyvalikakath TP, Hong J. What is user-centered design? J Am Dent Assoc. 2007;138:1081–2.PubMedCrossRefGoogle Scholar
  7. 7.
    Jaspers MW. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform. 2009;78:340–53.PubMedCrossRefGoogle Scholar
  8. 8.
    Napolitano MA, Hayes S, Russo G, Muresu D, Giordano A, Foster GD. Using avatars to model weight loss behaviors: participant attitudes and technology development. J Diabetes Sci Technol. 2013;7:1057–65.PubMedPubMedCentralGoogle Scholar
  9. 9.
    Bus SA, Waaijman R, Nollet F. New monitoring technology to objectively assess adherence to prescribed footwear and assistive devices during ambulatory activity. Arch Phys Med Rehabil. 2012;93:2075–9.PubMedCrossRefGoogle Scholar
  10. 10.
    Hazenberg CE, Bus SA, Kottink AI, Bouwmans CA, Schonbach-Spraul AM, van Baal SG. Telemedical home-monitoring of diabetic foot disease using photographic foot imaging—a feasibility study. J Telemed Telecare. 2012;18:32–6.PubMedCrossRefGoogle Scholar
  11. 11.
    Morak J, Schwarz M, Hayn D, Schreier G. Feasibility of mHealth and Near Field Communication technology based medication adherence monitoring. Conf Proc IEEE Eng Med Biol Soc. 2012;2012:272–5.PubMedGoogle Scholar
  12. 12.
    Keith-Hynes P, Guerlain S, Mize B, Hughes-Karvetski C, Khan M, McElwee-Malloy M, et al. DiAs user interface: a patient-centric interface for mobile artificial pancreas systems. J Diabetes Sci Technol. 2013;7:1416–26.PubMedGoogle Scholar
  13. 13.
    Popay J, Roberts H, Sowden A, Petticrew M, Arai L, Rodgers M, Britten N, Roen K, Duffy S. Guidance on the conduct of narrative synthesis in systematic reviews. A Product from the ESRC Methods Programme 2006:1–92.Google Scholar
  14. 14.
    Britto MT, Jimison HB, Munafo JK, Wissman J, Rogers ML, Hersh W. Usability testing finds problems for novice users of pediatric portals. J Am Med Inform Assoc. 2009;16:660–9.PubMedCrossRefPubMedCentralGoogle Scholar
  15. 15.
    Urowitz S, Wiljer D, Dupak K, Kuehner Z, Leonard K, Lovrics E, et al. Improving diabetes management with a patient portal: a qualitative study of diabetes self-management portal. J Med Internet Res. 2012;14:e158.PubMedCrossRefPubMedCentralGoogle Scholar
  16. 16.
    Dick JJ, Nundy S, Solomon MC, Bishop KN, Chin MH, Peek ME. Feasibility and usability of a text message-based program for diabetes self-management in an urban African-American population. J Diabetes Sci Technol. 2011;5:1246–54.PubMedCrossRefPubMedCentralGoogle Scholar
  17. 17.
    Boren MT, Ramey J. Thinking aloud: reconciling theory and practice. IEEE Trans Prof Commun. 2000;43:261–78.CrossRefGoogle Scholar
  18. 18.
    Simon AC, Holleman F, Gude WT, Hoekstra JB, Peute LW, Jaspers MW, et al. Safety and usability evaluation of a web-based insulin self-titration system for patients with type 2 diabetes mellitus. Artif Intell Med. 2013;59:23–31.PubMedCrossRefGoogle Scholar
  19. 19.
    Wharton C, Rieman J, Lewis C, Polson P. The cognitive walkthrough method: a practitioner’s guide. In: Usability Inspection Methods. New York: Wiley; 1994. p. 105–41.Google Scholar
  20. 20.
    Lewis C, Rieman J. Chapter 4. Evaluating the design without users. In Task-centered user interface design: a practical introduction. Online via shareware; 1993, 1994.Google Scholar
  21. 21.
    Nielsen J, Molich R. Heuristic evaluation of user interfaces. In Proc ACM CHI'90 Conf; April 1–5; Seattle, WA. 1990: 249–256.Google Scholar
  22. 22.
    Demidowich AP, Lu K, Tamler R, Bloomgarden Z. An evaluation of diabetes self-management applications for Android smartphones. J Telemed Telecare. 2012;18:235–8.PubMedCrossRefGoogle Scholar
  23. 23.••
    Nijland N, van Gemert-Pijnen JE, Kelders SM, Brandenburg BJ, Seydel ER. Factors influencing the use of a Web-based application for supporting the self-care of patients with type 2 diabetes: a longitudinal study. J Med Internet Res. 2011;13:e71. This study used four usability approaches and collected data on reasons for discontinued use of a website. It only took 20 users to identify 166 unique usability problems, and users attributed a suboptimal user interface and limited interactivity as reasons for discontinued use.PubMedCrossRefPubMedCentralGoogle Scholar
  24. 24.•
    Froisland DH, Arsand E, Skarderud F. Improving diabetes care for young people with type 1 diabetes through visual learning on mobile phones: mixed-methods study. J Med Internet Res. 2012;14:e111. This study illustrates how to conduct a comprehensive qualitative and quantitative usability testing approach to identify technical problems, obtain users’ ideas for improvements, and, in turn, inform further development and testing.PubMedCrossRefPubMedCentralGoogle Scholar
  25. 25.
    Bangor A, Kortum PT, Miller JT. An empirical evaluation of the System Usability Scale (SUS). Int J Hum Comput Interact. 2008;24:574–94.CrossRefGoogle Scholar
  26. 26.
    Brooke J. SUS: a “quick and dirty” usability scale. In: Jordan PW, Thomas B, Weerdmeester BA, McClelland AL, editors. Usability evaluation in industry. London: Taylor and Francis; 1996.Google Scholar
  27. 27.••
    Ossebaard HC, Seydel ER, van Gemert-Pijnen L. Online usability and patients with long-term conditions: a mixed-methods approach. Int J Med Inform. 2012;81:374–87. This study employed four usability testing approaches, including software that recorded screen action during think aloud sessions. Utilization of a variety of methods uncovered negative user experiences, negative think aloud expressions, incomplete user tasks by most participants, and usability issues pertaining to navigation and non-personalized content that could be improved.PubMedCrossRefGoogle Scholar
  28. 28.
    Yen PY, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc. 2012;19:413–22.PubMedCrossRefPubMedCentralGoogle Scholar
  29. 29.
    Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. In Human Factors Group. Boca Raton, FL: IBM Corporation; 1993.Google Scholar
  30. 30.
    Monkman H, Kushniruk A. A health literacy and usability heuristic evaluation of a mobile consumer health application. Stud Health Technol Inf. 2013;192:724–8.Google Scholar
  31. 31.
    Broderick J, Devine T, Langhans E, Lemerise AJ, Lier S, Harris L. Designing health literate mobile apps. Washington: Institute of Medicine; 2014.Google Scholar
  32. 32.
    Chakkalakal RJ, Kripalani S, Schlundt DG, Elasy TA, Osborn CY. Disparities in using technology to access health information: race versus health literacy. Diabetes Care. 2014;37:e53–4.PubMedCrossRefGoogle Scholar
  33. 33.
    Sarkar U, Karter AJ, Liu JY, Adler NE, Nguyen R, Lopez A, et al. The literacy divide: health literacy and the use of an internet-based patient portal in an integrated health system—results from The Diabetes Study of Northern California (DISTANCE). J Health Commun. 2010;15 Suppl 2:183–96.PubMedCrossRefPubMedCentralGoogle Scholar
  34. 34.
    Smith A. Older adults and technology use. Washington: Pew Research Center; 2014.Google Scholar
  35. 35.
    Lyles CR, Harris LT, Jordan L, Grothaus L, Wehnes L, Reid RJ, et al. Patient race/ethnicity and shared medical record use among diabetes patients. Med Care. 2012;50:434–40.PubMedCrossRefGoogle Scholar
  36. 36.
    Sarkar U, Karter AJ, Liu JY, Adler NE, Nguyen R, Lopez A, et al. Social disparities in internet patient portal use in diabetes: evidence that the digital divide extends beyond access. J Am Med Inform Assoc. 2011;18:318–21.PubMedCrossRefPubMedCentralGoogle Scholar
  37. 37.
    Gude WT, Simon AC, Peute LW, Holleman F, Hoekstra JB, Peek N, et al. Formative usability evaluation of a web-based insulin self-titration system: preliminary results. Stud Health Technol Inf. 2012;180:1209–11.Google Scholar
  38. 38.
    Jennings A, Powell J, Armstrong N, Sturt J, Dale J. A virtual clinic for diabetes self-management: pilot study. J Med Internet Res. 2009;11:e10.PubMedCrossRefPubMedCentralGoogle Scholar
  39. 39.
    Nielsen J, Mack RL, editors. Usability inspection methods. New York: Wiley; 1994.Google Scholar
  40. 40.
    Zayas-Caban T, Marquard JL, Radhakrishnan K, Duffey N, Evernden DL. Scenario-based user testing to guide consumer health informatics design. AMIA Annu Symp Proc. 2009;2009:719–23.PubMedPubMedCentralGoogle Scholar
  41. 41.
    Arsand E, Tatara N, Ostengen G, Hartvigsen G. Mobile phone-based self-management tools for type 2 diabetes: the few touch application. J Diabetes Sci Technol. 2010;4:328–36.PubMedCrossRefPubMedCentralGoogle Scholar
  42. 42.
    Cafazzo JA, Casselman M, Hamming N, Katzman DK, Palmert MR. Design of an mHealth app for the self-management of adolescent type 1 diabetes: a pilot study. J Med Internet Res. 2012;14:e70.PubMedCrossRefPubMedCentralGoogle Scholar
  43. 43.
    Carroll AE, DiMeglio LA, Stein S, Marrero DG. Using a cell phone-based glucose monitoring system for adolescent diabetes management. Diabetes Educ. 2011;37:59–66.PubMedCrossRefGoogle Scholar
  44. 44.
    Chomutare T, Tatara N, Arsand E, Hartvigsen G. Designing a diabetes mobile application with social network support. Stud Health Technol Inf. 2013;188:58–64.Google Scholar
  45. 45.
    DeShazo J, Harris L, Turner A, Pratt W. Designing and remotely testing mobile diabetes video games. J Telemed Telecare. 2010;16:378–82.PubMedCrossRefGoogle Scholar
  46. 46.
    Garcia-Saez G, Hernando ME, Martinez-Sarriegui I, Rigla M, Torralba V, Brugues E, et al. Architecture of a wireless personal assistant for telemedical diabetes care. Int J Med Inform. 2009;78:391–403.PubMedCrossRefGoogle Scholar
  47. 47.
    Mulvaney SA, Anders S, Smith AK, Pittel EJ, Johnson KB. A pilot test of a tailored mobile and web-based diabetes messaging system for adolescents. J Telemed Telecare. 2012;18:115–8.PubMedCrossRefPubMedCentralGoogle Scholar
  48. 48.
    Osborn CY, Mulvaney SA. Development and feasibility of a text messaging and interactive voice response intervention for low-income, diverse adults with type 2 diabetes mellitus. J Diabetes Sci Technol. 2013;7:612–22.PubMedCrossRefPubMedCentralGoogle Scholar
  49. 49.
    Padman R, Jaladi S, Kim S, Kumar S, Orbeta P, Rudolph K, et al. An evaluation framework and a pilot study of a mobile platform for diabetes self-management: insights from pediatric users. Stud Health Technol Inf. 2013;192:333–7.Google Scholar
  50. 50.
    Tani S, Marukami T, Matsuda A, Shindo A, Takemoto K, Inada H. Development of a health management support system for patients with diabetes mellitus at home. J Med Syst. 2010;34:223–8.PubMedCrossRefGoogle Scholar
  51. 51.
    Tatara N, Arsand E, Bratteteig T, Hartvigsen G. Usage and perceptions of a mobile self-management application for people with type 2 diabetes: qualitative study of a five-month trial. Stud Health Technol Inf. 2013;192:127–31.Google Scholar
  52. 52.
    Vuong AM, Huber Jr JC, Bolin JN, Ory MG, Moudouni DM, Helduser J, et al. Factors affecting acceptability and usability of technological approaches to diabetes self-management: a case study. Diabetes Technol Ther. 2012;14:1178–82.PubMedCrossRefPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Courtney R. Lyles
    • 1
    • 5
    Email author
  • Urmimala Sarkar
    • 1
    • 5
  • Chandra Y. Osborn
    • 2
    • 3
    • 4
    • 6
  1. 1.UCSF Center for Vulnerable Populations at San Francisco General HospitalSan FranciscoUSA
  2. 2.Department of MedicineVanderbilt University Medical CenterNashvilleUSA
  3. 3.Department of Biomedical InformaticsVanderbilt University Medical CenterNashvilleUSA
  4. 4.Diabetes Research & Training Center, Center for Diabetes Translational ResearchVanderbilt University Medical CenterNashvilleUSA
  5. 5.Division of General Internal Medicine at San Francisco General HospitalUniversity of California San FranciscoSan FranciscoUSA
  6. 6.Division of General Internal Medicine & Public HealthVanderbilt University Medical CenterNashvilleUSA

Personalised recommendations