Internal and Emergency Medicine

, Volume 12, Issue 4, pp 535–543 | Cite as

Evaluation of internet-based patient education materials from internal medicine subspecialty organizations: will patients understand them?

  • David R. Hansberry
  • Nitin Agarwal
  • Elizabeth S. John
  • Ann M. John
  • Prateek Agarwal
  • James C. Reynolds
  • Stephen R. Baker


The majority of Americans use the Internet daily, if not more often, and many search online for health information to better understand a diagnosis they have been given or to research treatment options. The average American reads at an eighth-grade level. The purpose of this study is to evaluate the readability of online patient education materials on the websites of 14 professional organizations representing the major internal medicine subspecialties. We used ten well-established quantitative readability scales to assess written text from patient education materials published on the websites of the major professional organizations representing the following subspecialty groups: allergy and immunology, cardiology, endocrinology, gastroenterology, geriatrics, hematology, hospice and palliative care, infectious disease, nephrology, oncology, pulmonology and critical care, rheumatology, sleep medicine, and sports medicine. Collectively the 540 articles analyzed were written at an 11th-grade level (SD 1.4 grade levels). The sleep medicine and nephrology websites had the most readable materials, written at an academic grade level of 8.5 ± 1.5 and 9.0 ± 0.2, respectively. Material at the infectious disease site was written at the most difficult level, with average readability corresponding to grades 13.9 ± 0.3. None of the patient education materials we reviewed conformed to the American Medical Association (AMA) and the National Institutes of Health (NIH) guidelines requiring that patient education articles be written at a third- to seventh-grade reading level. If these online resources were rewritten, it is likely that more patients would derive benefit from reading them.


Internal medicine Internet Literacy Patient education Readability 



Contributors: We appreciate the detailed review by Diana Winters.

Compliance with ethical standards

Conflict of interest/disclosures

Data from the American Gastroenterological Association (AGA) website is included in a separate manuscript under review at the International Journal of Colorectal Disease that discusses the readability of online gastroenterology-based patient education resources.

Statement of human and animal rights

Institutional IRB approval was not required as there were no human or animal subjects.

Informed consent

Informed consent was not required as there were no human subjects.


  1. 1.
    Internet & American Life Project (2011) Demographics of internet users. Pew Research Center, WashingtonGoogle Scholar
  2. 2.
    Fox S (2010) Mobile Health. Pew Internet & American Life Project, WashingtonGoogle Scholar
  3. 3.
    Fox S (2011) The Social Life of Health InformationGoogle Scholar
  4. 4.
    Andreassen HK, Bujnowska-Fedak MM, Chronaki CE, Dumitru RC, Pudule I, Santana S, Voss H, Wynn R (2007) European citizens’ use of E-health services: a study of seven countries. BMC Public Health 7:53. doi: 10.1186/1471-2458-7-53 CrossRefPubMedPubMedCentralGoogle Scholar
  5. 5.
    Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, Viswanath K (2005) Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Int Med 165(22):2618–2624. doi: 10.1001/archinte.165.22.2618 CrossRefGoogle Scholar
  6. 6.
    Rice RE (2006) Influences, usage, and outcomes of Internet health information searching: multivariate results from the Pew surveys. Int J Med Inform 75(1):8–28. doi: 10.1016/j.ijmedinf.2005.07.032 ([pii]: S1386-5056(05)00146-2) CrossRefPubMedGoogle Scholar
  7. 7.
    Diaz JAGR, Ng JJ, Reinert SE, Friedmann PD, Moulton AW (2002) Patients’ Use of the Internet for Medical Information. J Gen Intern Med 17:180–185CrossRefPubMedPubMedCentralGoogle Scholar
  8. 8.
    NCES (2008) America’s Health Literacy: Why We Need Accessible Health InformationGoogle Scholar
  9. 9.
    Covering Kids & Families (2005) Health Literacy Style ManualGoogle Scholar
  10. 10.
    Kutner M, Greenberg E, Jin Y, Paulsen C (2006) The Health Literacy of America’s Adults: Results From the 2003 National Assessment of Adult Literacy (NCES 2006–483). National Center for Education Statistics, Washington, DCGoogle Scholar
  11. 11.
    Weis BD (2003) Health Literacy: A Manual for Clinicians. American Medical Association, American Medical Foundation, Chicago, ILGoogle Scholar
  12. 12.
    Simply Put—A Guide for Creating Easy-to-Understand Materials (2009) US Department of Health and Human Services—Centers for Disease Control and Prevention (CDC), 3 edn. Atlanta, GeorgiaGoogle Scholar
  13. 13.
    National Institutes of Health How to Write Easy to Read Health Materials (2016) National Library of Medicine. Accessed 11 Nov 2016
  14. 14.
    Agarwal N, Hansberry DR, Sabourin V, Tomei KL, Prestigiacomo CJ (2013) A comparative analysis of the quality of patient education materials from medical specialties. JAMA Intern Med 8(173):1257–1259. doi: 10.1001/jamainternmed.2013.6060 CrossRefGoogle Scholar
  15. 15.
    Hansberry DR, Agarwal N, Shah R, Schmitt PJ, Baredes S, Setzen M, Carmel PW, Prestigiacomo CJ, Liu JK, Eloy JA (2014) Analysis of the readability of patient education materials from surgical subspecialties. Laryngoscope 124(2):405–412. doi: 10.1002/lary.24261 CrossRefPubMedGoogle Scholar
  16. 16.
    Hansberry DR, Agarwal N, Gonzales SF, Baker SR (2014) Are we effectively informing patients? A quantitative analysis of online patient education resources from the American Society of Neuroradiology. Am J Neuroradiol 35(7):1270–1275. doi: 10.3174/ajnr.A3854 CrossRefPubMedGoogle Scholar
  17. 17.
    Hansberry DR, Ramchand T, Patel S, Kraus C, Jung J, Agarwal N, Gonzales SF, Baker SR (2014) Are We Failing to Communicate? Internet-Based Patient Education Materials and Radiation Safety. Eur J Radiol 83(9):1698–1702CrossRefPubMedGoogle Scholar
  18. 18.
    Agarwal N, Feghhi DP, Gupta R, Hansberry DR, Heary RF, Goldstein IM (2014) A comparative analysis of minimally invasive and open spine surgery patient education resources. J Neurosurg Spine 21(3):468–474CrossRefPubMedGoogle Scholar
  19. 19.
    Agarwal N, Chaudhari A, Hansberry DR, Tomei KL, Prestigiacomo CJ (2013) A comparative analysis of neurosurgical online education materials to assess patient comprehension. J Clin Neurosci 20(10):1357–1361. doi: 10.1016/j.jocn.2012.10.047 CrossRefPubMedGoogle Scholar
  20. 20.
    Feghhi DP, Agarwal N, Hansberry DR, Berberian WS, Sabharwal S (2014) Critical Review of Patient Education Materials from the American Academy of Orthopedic Surgeons. Am J Orthop 43(8):168–174Google Scholar
  21. 21.
    Hansberry DR, John A, John E, Agarwal N, Gonzales SF, Baker SR (2014) A critical review of the readability of online patient education resources from Am J Roentgenol 202(3):566–575CrossRefGoogle Scholar
  22. 22.
    Hansberry DR, Agarwal N, Baker SR (2015) Health literacy and online education resources: an opportunity to educate our patients. Am J Roentgenol 204(1):111–116CrossRefGoogle Scholar
  23. 23.
    Hansberry D, Kraus C, Agarwal N, Baker S, Gonzales S (2014) Health literacy in vascular and interventional radiology: a comparative analysis of online patient education resources. Cardiovasc Interv Radiol 37(4):1034–1040. doi: 10.1007/s00270-013-0752-6 CrossRefGoogle Scholar
  24. 24.
    Hansberry DR, Suresh R, Agarwal N, Heary RF, Goldstein IM (2013) Quality assessment of online patient education resources for peripheral neuropathy. J Peripher Nerv Syst 18(1):44–47. doi: 10.1111/jns5.12006 CrossRefPubMedGoogle Scholar
  25. 25.
    Agarwal N, Sarris C, Hansberry DR, Lin MJ, Barrese JC, Prestigiacomo CJ (2013) Quality of patient education materials for rehabilitation after neurological surgery. Neuro Rehabil 32(4):817–821. doi: 10.3233/nre-130905 Google Scholar
  26. 26.
    Misra P, Agarwal N, Kasabwala K, Hansberry DR, Setzen M, Eloy JA (2013) Readability analysis of healthcare-oriented education resources from the american academy of facial plastic and reconstructive surgery (AAFPRS). Laryngoscope 123(1):90–96. doi: 10.1002/lary.23574 CrossRefPubMedGoogle Scholar
  27. 27.
    Kasabwala K, Agarwal N, Hansberry DR, Baredes S, Eloy JA (2012) Readability Assessment of Patient Education Materials from the American Academy of Otolaryngology—Head and Neck Surgery Foundation. Otolaryngol Head Neck Surg 147(3):466–471. doi: 10.1177/0194599812442783 CrossRefPubMedGoogle Scholar
  28. 28.
    Eloy JA, Li S, Kasabwala K, Agarwal N, Hansberry DR, Baredes S, Setzen M (2012) Readability Assessment of Patient Education Materials on Major Otolaryngology Association Websites. Otolaryngol Head Neck Surg 147(3):466–471. doi: 10.1177/0194599812456152 CrossRefPubMedGoogle Scholar
  29. 29.
    Kasabwala K, Misra P, Hansberry DR, Agarwal N, Baredes S, Setzen M, Anderson Eloy J (2013) Readability assessment of the American Rhinologic Society patient education materials. Int Forum Allergy Rhinol 3(4):325–333. doi: 10.1002/alr.21097 CrossRefPubMedGoogle Scholar
  30. 30.
    Coleman M, Liau TL (1975) A computer readability formula designed for machine scoring. J Appl Psychol 60:2CrossRefGoogle Scholar
  31. 31.
    Kincaid EH (1992) The Medicare program. Exploring federal health care policy. N C Med J 53(11):596–601PubMedGoogle Scholar
  32. 32.
    Flesch R (1948) A new readability yardstick. J Appl Psychol 32(3):221–233CrossRefPubMedGoogle Scholar
  33. 33.
    Caylor JS, Sticht TG, Fox LC, Ford JP (1973) Methodologies for determining reading requirements of military occupational specialties (Tech Report No. 73-5). Alexandria, VAGoogle Scholar
  34. 34.
    Fry E (1968) A readability formula that saves time. J Read 11:4Google Scholar
  35. 35.
    Gunning R (1952) The technique of clear writing. McGraw-Hill, New YorkGoogle Scholar
  36. 36.
    Chall JS (1995) Readability revisited: The new Dale-Chall readability formula. Brookline Books, Cambridge, MAGoogle Scholar
  37. 37.
    Raygor AL (1977) The Raygor readability estimate: a quick and easy way to determine difficulty. In: Pearson PD (eds) Reading: theory rap (ed) National Reading Conference. Clemson, SC, pp 259–263Google Scholar
  38. 38.
    McLaughlin GH (1969) SMOG grading: a new readability formula. J Read 12(8):8Google Scholar
  39. 39.
    Weiss BD, Blanchard JS, McGee DL, Hart G, Warren B, Burgoon M, Smith KJ (1994) Illiteracy among Medicaid recipients and its relationship to health care costs. J Health Care Poor Underserved 5(2):99–111. doi: 10.1353/hpu.2010.0272 CrossRefPubMedGoogle Scholar
  40. 40.
    US Department of Health and Human Services: Office of Disease Prevention and Health Promotion (2010) National Action Plan to Improve Health Literacy, vol 15, 2002/05/04 edn. Washington, DCGoogle Scholar

Copyright information

© SIMI 2017

Authors and Affiliations

  • David R. Hansberry
    • 1
  • Nitin Agarwal
    • 2
  • Elizabeth S. John
    • 3
  • Ann M. John
    • 4
  • Prateek Agarwal
    • 5
  • James C. Reynolds
    • 6
  • Stephen R. Baker
    • 7
  1. 1.Department of RadiologyThomas Jefferson University HospitalsPhiladelphiaUSA
  2. 2.Department of Neurological SurgeryUniversity of Pittsburgh Medical CenterPittsburghUSA
  3. 3.Department of MedicineUniversity of Central Florida College of MedicineOrlandoUSA
  4. 4.Department of Medicine, New Jersey Medical School, RutgersThe State University of New JerseyNewarkUSA
  5. 5.Perelman School of MedicineUniversity of PennsylvaniaPhiladelphiaUSA
  6. 6.Division of Gastroenterology, Department of Medicine, Perelman School of MedicineUniversity of PennsylvaniaPhiladelphiaUSA
  7. 7.Department of Radiology, New Jersey Medical School, RutgersThe State University of New JerseyNewarkUSA

Personalised recommendations