Advancing health literacy has the potential to improve patient outcomes. The American Academy of Orthopaedic Surgeons’ (AAOS) online patient education materials serve as a tool to improve health literacy for orthopaedic patients; however, it is unknown whether the materials currently meet the National Institutes of Health/American Medical Association’s recommended sixth grade readability guidelines for health information or the mean US adult reading level of eighth grade.
The purposes of this study were (1) to evaluate the mean grade level readability of online AAOS patient education materials; and (2) to determine what proportion of the online materials exceeded recommended (sixth grade) and mean US (eighth grade) reading level.
Reading grade levels for 99.6% (260 of 261) of the online patient education entries from the AAOS were analyzed using the Flesch-Kincaid formula built into Microsoft Word software.
Mean grade level readability of the AAOS patient education materials was 9.2 (SD ± 1.6). Two hundred fifty-one of the 260 articles (97%) had a readability score above the sixth grade level. The readability of the AAOS articles exceeded the sixth grade level by an average of 3.2 grade levels. Of the 260 articles, 210 (81%) had a readability score above the eighth grade level, which is the average reading level of US adults.
Most of the online patient education materials from the AAOS had readability levels that are far too advanced for many patients to comprehend. Efforts to adjust the readability of online education materials to the needs of the audience may improve the health literacy of orthopaedic patients. Patient education materials can be made more comprehensible through use of simpler terms, shorter sentences, and the addition of pictures. More broadly, all health websites, not just those of the AAOS, should aspire to be comprehensible to the typical reader.
In the United States, the mean adult reading level is an eighth grade level [21, 22, 43, 51]. Fourth grade text is difficult to comprehend for one of five US adults , and nearly 50% of adults “experience considerable difficulty in performing tasks that required them to integrate or synthesize information from complex or lengthy texts” . The readability of education materials is measured by the grade level at which the written material is comprehended and educational materials should be written at the appropriate grade level for individuals to understand the content.
The “capacity to obtain, interpret, and understand basic health information and services and the competence to use such information and services to enhance health” defines a person’s health literacy . Health-related quality of life is independently predicted by the level of one’s health literacy [49, 55, 57]. Moreover, lower health literacy correlates with higher rates of complication , hospitalization [6, 7, 48], increased costs [27, 53, 62], and overall worse health [8, 11, 29, 48, 50, 61].
Given adult reading skills in the United States and the important effects of health literacy on actual health, the National Institutes of Health (NIH) and American Medical Association (AMA) officially recommend that patient education materials not exceed the sixth grade reading level [18, 22, 36, 59, 60]. As suggested by several past studies, patient education materials may be too complex for some individuals to comprehend [2, 5, 17, 22, 38, 47, 54, 58].
The Internet has become a major source of patient education. Eight million individuals in the United States go online for health information everyday . Studies have demonstrated orthopaedic patients use the Internet extensively for health education materials [9, 10, 15, 31]. Patients likely also use online education materials without being able to discuss its content with an orthopaedic surgeon.
The American Academy of Orthopedic Surgeons (AAOS) is an important purveyor of online health information about musculoskeletal conditions for patients. The AAOS is “the world’s largest medical association of musculoskeletal specialists” and “the preeminent provider of musculoskeletal education” . The AAOS is a particularly high-traffic educational platform with more than 35,000 people visiting the website daily  and 6361 other websites linking directly to the website . Surgeons often print online patient education handouts for their patients  and commonly refer patients specifically to the AAOS for reliable, peer-reviewed, up-to-date, and accurate content .
The goal of our study was to analyze if the readability of the AAOS online patient education materials are in accordance with AMA/NIH readability recommendations [18, 22, 36, 59, 60]. Specifically, we sought (1) to evaluate the mean grade level readability of online AAOS patient education materials; and (2) to determine what proportion of the online materials exceed recommended (sixth grade) and mean US (eighth grade) reading levels.
Materials and Methods
Our study analyzed the AAOS online patient educational materials found at www.orthoinfo.org/. The website was accessed between January 1 and February 2, 2014, and again on July 31, 2014. All 261 articles under “Parts of the Body” were evaluated, except the one article “Treatment Options for Low Back Pain” (video), which was excluded from analysis because it consisted predominantly of a video and contained minimal text.
For each of the 260 articles, text was copied into separate Microsoft Word 2010 (Microsoft Corporation, Redmond, WA, USA) documents in plain text format. Following the recommendation of Flesch  and others , all numbers, abbreviations, decimals, bullets, semicolons, paragraph breaks, colons, and dashes within a sentence were removed to avoid underestimating readability level. Any text not directly related to patient education was deleted, including author information, hyperlinks, copyright notes, date stamps, tables, and citations.
For each document, the well-validated Flesch-Kincaid Grade Level (FKGL) formula was used to analyze written material readability in terms of academic grade levels [2, 5, 13, 17, 44, 47, 54, 58, 63]. Although there are various readability assessment tools used in the healthcare setting, the FK formula is the most widely validated method of analyzing readability [2, 5, 13, 17, 44, 47, 54, 58, 63]. The FKGL was obtained using the Microsoft Word built-in readability calculator. Sequentially, selection of document tools “Review,” “Spelling & Grammar,” “Options,” and “Show Readability Statistics” enabled the FKGL calculator to automatically display a given document’s FKGL after the “Spelling & Grammar” function was used. The FKGL formula is: 0.39 × [average number of words per sentence] + 11.8 × [average number of syllables per word] − 15.59.
The FKGLs of the AAOS articles were compared with the AMA/NIH-recommended readability level (sixth grade) [18, 22, 36, 59, 60] as well as the mean US adult reading level (eighth grade) [21, 43, 51] with data throughout the article being presented as mean ± SD except where otherwise specified and comparisons being performed using unpaired t-tests. The cutoff for statistical significance was p < 0.05.
For all AAOS patient education articles, mean grade level readability was 9.2 ± 1.6. The articles under “Shoulder & Elbow” patient education materials had a mean FKGL of 9.9 ± 1.4; “Hand & Wrist” articles had a mean FKGL of 8.7 ± 0.9; “Hip & Thigh” articles had a mean FKGL of 9.6 ± 1.5; “Knee & Lower Leg” articles had a mean FKGL of 9.2 ± 1.8; “Foot & Ankle” articles had a mean FKGL of 8.4 ± 1.1; and “Neck & Back” articles had a mean FKGL of 9.2 ± 1.7. Grade level readability for all sections was greater (p < 0.05) than the recommended sixth grade reading level (Table 1).
Of the total 260 articles we evaluated, 251 (97%) had a readability score above the sixth grade level. Readability of the articles exceeded the sixth grade level by an average of 3.2 grade levels (95% confidence interval [CI], 3.0–3.4; p < 0.001). The readability score for 210 (81%) of the 260 articles was above the eighth grade level. Readability of the articles exceeded the eighth grade level by an average of 1.2 grade levels (95% CI, 1.0–1.4; p < 0.001; Fig. 1).
Advancing health literacy has the potential to improve patient outcomes. It is therefore important that trusted and much-used sources of health information like the AAOS be comprehensible to most users, but it is unknown whether they are. Readability analysis of the AAOS online patient education materials has broad applicability and relevance. Because the AAOS is an expert website that is visited daily by more than 35,000 individuals , linked to 6361 other websites , and a commonly recommended and used patient education resource of orthopaedic surgeons [26, 28, 31, 42, 47], it serves as an important example bringing attention to the effects of readability of patient education materials on health literacy and health outcomes. This analysis highlights the need for orthopaedic surgeons and societies to appreciate the concept of readability when creating any patient materials, including educational materials and consent forms.
Despite the heavy use of web-based health information [9, 10, 15, 24, 31], only 12% of adults have the health literacy skills to navigate their health issues. According to the National Assessment of Adult Literacy , the mean reading level for adults in the United States is of an eighth grade level [21, 22, 43, 51]. Moreover, even patients with stronger literacy skills prefer lower grade level online content in simpler formats [19, 56]. Taken together, the AMA/NIH recommend that patient education materials not exceed a sixth grade reading level [18, 22, 36, 59, 60].
In this study, we found that the average readability of AAOS online patient education materials exceeds the AMA/NIH-recommended health information readability level and the average US adult reading level. A substantial portion of the intended patient audience may not effectively comprehend the content, which can negatively affect health literacy, contributing to poorer health outcomes, increased hospitalizations, and higher healthcare costs .
Our study had several limitations. Although the FKGL scoring system is well validated [2, 5, 13, 17, 44, 47, 54, 58, 63], it only factors in written text—not illustrated, video, or audio content, which may improve understanding of content. An important additional potential study limitation is the assumption that the reading skills of those reading the AAOS patient education materials are equivalent to the general US adult population. However, improved Internet access makes the online population increasingly similar to the general population . Online readers may have reading skills comparable to the general population; one-third of those seeking health information online have a high school diploma or less education . Of the adults with “below basic health literacy,” 20% get their health information online . Consequently, most studies assess online material readability compared with mean adult reading skill level [12, 14, 19, 25, 26, 34, 56]. Because www.orthoinfo.org is publically accessible, the reading level of these materials should match the reading skills of any potential orthopaedic patient or the general population.
Orthopaedic surgeons in practice may not depend on written AAOS patient teaching information alone to prevent postsurgical disasters related to uninformed patient behavior. Any written patient education material should take an immediate backseat to the critical process and content of the dialogue between the treating surgeon and the patient. Ostensibly, physicians should be able to judge the general reading comprehension level of each patient. Subsequently, the informing surgeon should adjust his or her teaching points based on an eye-to-eye assessment of the patient’s capacity to absorb and fully comprehend the specifics of the informed consent dialogue.
Sabharwal et al.  analyzed AAOS patient education materials produced between September 1999 and July 2006 and found the average FKGL was 10.4. Our findings suggest that the AAOS has improved the readability of its patient education materials by 1.2 grade levels over the last 8 years; however, further improvement is still needed. Such minimal improvement over time may reflect a lack of awareness in the orthopaedic community regarding the concept of readability.
Patient education material readability can be analyzed simply using widely available software as demonstrated by the methods of this study. Patient education materials can then be made more comprehensible through use of simpler terms, shorter sentences, and the addition of pictures [2, 18]. Additional media formats such as brief YouTube videos gauged to a sixth grade level of science comprehension may improve communication. Multiple other health information websites (eg, [1, 32, 35, 37, 39–41, 46]) have proven it is actually possible to achieve appropriate level patient education materials. Making the material more readable must and can be done without losing important content. There are multiple publically accessible tools (eg, [16, 20, 52]) that can be used to substitute complex medical jargon with simpler lay terms and improve the readability of patient educations materials. If leading expert websites, like the AAOS, improve readability of their patient education materials, other health information websites could follow suit, starting a trend toward optimizing all patient education materials.
With increased use of online health information, the study of health education is a particularly relevant subject. The full value of online patient education materials cannot be realized unless readers can comprehend the content. To improve health literacy and, in turn, health outcomes, modifying readability of patient education materials to levels comprehendible by the average reader should be emphasized.
Abrams MA, Dreyer BP, eds. Plain Language Pediatrics: Health Literacy Strategies and Communication Resources for Common Pediatric Topics. Elk Grove Village, IL, USA: American Academy of Pediatrics; 2008.
Albright J, de Guzman C, Acebo P, Paiva D, Faulkner M, Swanson J. Readability of patient education materials: implications for clinical practice. Appl Nurs Res. 1996;9:139–143.
Alexa Internet, Inc. How popular is aaos.org. Available at: www.alexa.com/siteinfo/orthoinfo.org. Accessed October 31, 2014.
American Academy of Orthopaedic Surgeons. About the AAOS. Available at: http://www.aaos.org/about. Accessed November 2, 2014.
Badarudeen S, Sabharwal S. Readability of patient education materials from the American Academy of Orthopaedic Surgeons and Pediatric Orthopaedic Society of North America web sites. J Bone Joint Surg Am. 2008;90:199–204.
Baker DW, Gazmararian JA, Williams MV, Scott T, Parker RM, Green D, Ren J, Peel J. Functional health literacy and the risk of hospital admission among Medicare managed care enrollees. Am J Public Health. 2002;92:1278–1283.
Baker DW, Parker RM, Williams MV, Clark WS. Health literacy and the risk of hospital admission. J Gen Intern Med. 1998;13:791–798.
Baker DW, Parker RM, Williams MV, Clark WS, Nurss J. The relationship of patient reading ability to self-reported health and use of health services. Am J Public Health. 1997;87:1027–1030.
Beall MS 3rd, Beall MS Jr, Greenfield ML, Biermann JS. Patient Internet use in a community outpatient orthopaedic practice. Iowa Orthop J. 2002;22:103–107.
Beall MS 3rd, Golladay GJ, Greenfield ML, Hensinger RN, Biermann JS. Use of the Internet by pediatric orthopaedic outpatients. J Pediatr Orthop. 2002;22:261–264.
Bennett CL, Ferreira MR, Davis TC, Kaplan J, Weinberger M, Kuzel T, Seday MA, Sartor O. Relation between literacy, race, and stage of presentation among low-income patients with prostate cancer. J Clin Oncol. 1998;16:3101–3104.
Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, Kanouse DE, Muñoz JA, Puyol JA, Lara M, Watkins KE, Yang H, McGlynn EA. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA. 2001;285:2612–2621.
Bluman EM, Foley RP, Chiodo CP. Readability of the Patient Education Section of the AOFAS website. Foot Ankle Int. 2009;30:287–291.
Boulos MN. British internet-derived patient information on diabetes mellitus: is it readable? Diabetes Technol Ther. 2005;7:528–535.
Brooks BA. Using the Internet for patient education. Orthop Nurs. 2001;20:69–77.
Centers for Disease Control and Prevention, National Center for Health Marketing. Plain Language Thesaurus for Health Communications. Atlanta, GA, USA: Centers for Disease Control and Prevention; 2009.
Cooley ME, Moriarty H, Berger MS, Selm-Orr D, Coyle B, Short T. Patient literacy and the readability of written cancer educational materials. Oncol Nurs Forum. 1995;22:1345–1351.
Cotugna N, Vickery CE, Carpenter-Haefele KM. Evaluation of literacy level of patient education pages in health-related journals. J Community Health. 2005;30:213–219.
D’Alessandro DM KP, Johnson-West J. The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med. 2001;155:807–812.
Dale E, O’Rourke J. The Living Word Vocabulary: The Words We Know, a National Vocabulary Inventory. Chicago, IL, USA: Field Enterprises Educational Corp; 1976.
Doak CC, Doak LG, Friedell GH, Meade CD. Improving comprehension for cancer patients with low literacy skills: strategies for clinicians. CA Cancer J Clin. 1998;48:151–162.
Doak CC, Doak LG, Root JH. Teaching Patients with Low Literacy Skills. 2nd ed. Philadelphia, PA, USA: JB Lippincott; 1996.
Flesch RF. How to Write Plain English: A Book for Lawyers, Consumers. 1st ed. New York, NY, USA: Barnes and Noble; 1979.
Fox SC. Controlling emesis: what is it worth? J Clin Oncol. 2006;24:5616–5617.
Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006;33:352–373.
Graber MA, Roller CM, Kaeble B. Readability levels of patient education material on the World Wide Web. J Fam Pract. 1999;48:58–61.
Howard DH, Gazmararian J, Parker RM. The impact of low health literacy on the medical costs of Medicare managed care enrollees. Am J Med. 2005;118:371–377.
Jackson RH, Davis TC, Bairnsfather LE, George RB, Crouch MA, Gault H. Patient reading ability: an overlooked problem in health care. South Med J. 1991;84:1172–1175.
Kalichman SC, Rompa D. Functional health literacy is associated with health status and health-related knowledge in people living with HIV-AIDS. J Acquir Immune Defic Syndr. 2000;25:337–344.
Kirsch IS, Jungeblut A, Jenkins L, Kolstad A. Adult Literacy in America: a First Look at the Results of the National Adult Literacy Survey. Washington, DC, USA: National Center for Educational Statistics; 1993.
Krempec J, Hall J, Biermann JS. Internet use by patients in orthopaedic surgery. Iowa Orthop J. 2003;23:80–82.
Kurtz-Rossi S, Lane MA, McKinney J, Frost J, Smith G. Staying Healthy: an English Learner’s Guide to Health Care and Healthy Living. Orlando, FL, USA: Florida Literacy Coalition, Inc; 2008.
Kutner M, Greenburg E, Jin Y, Paulsen C. The Health Literacy of America’s Adults: Results from the 2003 National Assessment of Adult Literacy. Washington, DC, USA: National Center for Education Statistics; 2006.
Lewiecki EM, Rudolph LA, Kiebzak GM, Chavez JR, Thorpe BM. Assessment of osteoporosis-website quality. Osteoporosis Int. 2006;17:741–752.
March of Dimes Foundation. Thinking About Your Family Health History. White Plains, NY, USA: March of Dimes Foundation; 2012.
MedlinePlus. How to Write Easy-to-read Health Materials. Bethesda, MD, USA: US National Library of Medicine, National Institutes of Health; 2013.
Michigan Cancer Consortium. Making the Choice. Deciding What to Do About Early Stage Prostate Cancer. Okemos, MI, USA: Michigan Cancer Consortium; 2014.
Murero M, D’Ancona G, Karamanoukian H. Use of the Internet by patients before and after cardiac surgery: telephone survey. J Med Internet Res. 2001;3:E27.
National Cancer Institutes, National Institutes of Health. When Your Brother or Sister Has Cancer: a Guide for Teens. Bethesda, MD, USA: National Institutes of Health; 2013. NIH Publication No. 13-5733.
National Institute on Aging. Safe Use of Medicines: Take Your Medicines the Right Way—Each Day! Bethesda, MD, USA: National Institutes of Health; 2010. NIH Publication No. 10-7348.
National Institute on Aging. Caring for a Person With Alzheimer’s Disease: Your Easy-to-use Guide From the National Institute on Aging. Bethesda, MD, USA: National Institutes of Health; 2013. NIH Publication No. 12-6173.
Office of Communication, Centers for Disease Control and Prevention. Scientific and Technical Information Simply Put. 2nd ed. Atlanta, GA, USA: Centers for Disease Control and Prevention & Agency for Toxic Substances and Disease Registry; 1999.
Organisation for Economic Co-operation and Development. Time for the US to Reskill? What the Survey of Adult Skills Says: OECD Skills Studies. East Jerusalem, Israel: OECD Publishing; 2013.
Polishchuk DL, Hashem J, Sabharwal S. Readability of online patient education materials on adult reconstruction web sites. J Arthroplasty. 2012;27:716–719.
Purvis JM, Alexander AH, Einhorn TA, Griffin LY. American Orthopaedic Association symposium: evaluating the flood of orthopaedic media and marketing information. J Bone Joint Surg Am. 2005;87:1392–1398.
River Valley Healthy Communities Coalition, National Network of Libraries of Medicine—New England Region. Who Can You Trust? Health Information and the Internet. Rumford, ME, USA: River Valley Communities Coalition; 2009.
Sabharwal S, Badarudeen S, Unes Kunju S. Readability of online patient education materials from the AAOS web site. Clin Orthop Relat Res. 2008;466:1245–1250.
Schillinger D, Grumbach K, Piette J, Wang F, Osmond D, Daher C, Palacios J, Sullivan GD, Bindman AB. Association of health literacy with diabetes outcomes. JAMA. 2002;288:475–482.
Song L, Mishel M, Bensen JT, Chen RC, Knafl GJ, Blackard B, Farnan L, Fontham E, Su LJ, Brennan CS, Mohler JL, Godley PA. How does health literacy affect quality of life among men with newly diagnosed clinically localized prostate cancer? Findings from the North Carolina-Louisiana Prostate Cancer Project (PCaP). Cancer. 2012;118:3842–3851.
Sudore RL, Yaffe K, Satterfield S, Harris TB, Mehta KM, Simonsick EM, Newman AB, Rosano C, Rooks R, Rubin SM, Ayonayon HN, Schillinger D. Limited literacy and mortality in the elderly: the health, aging, and body composition study. J Gen Intern Med. 2006;21:806–812.
US Department of Education, National Center for Education Statistics. The NAEP Reading Achievement Levels by Grade. Washington, DC, USA: US Department of Education; 2011.
US Institute of Medicine. Health Literacy: a Prescription to End Confusion. Washington, DC, USA: National Academies Press; 2004.
Vernon J. Low Health Literacy: an Economic Drain on the US Healthcare System. Washington, DC, USA: National Press Club; October 10, 2007.
Vives M, Young L, Sabharwal S. Readability of spine-related patient education materials from subspecialty organization and spine practitioner websites. Spine (Phila Pa 1976). 2009;34:2826–2831.
Wallace LS, Rogers ES, Weiss BD. Relationship between health literacy and health-related quality of life among Tennesseans. Tenn Med. 2008;101:35–39.
Wallace LS, Turner LW, Ballard JE, Keenum AJ, Weiss BD. Evaluation of web-based osteoporosis educational materials. J Womens Health. 2005;14:936–945.
Wang C, Li H, Li L, Xu D, Kane RL, Meng Q. Health literacy and ethnic disparities in health-related quality of life among rural women: results from a Chinese poor minority area. Health Qual Life Outcomes. 2013;11:153.
Wang SW, Capo JT, Orillaza N. Readability and comprehensibility of patient education material in hand-related web sites. J Hand Surg Am. 2009;34:1308–1315.
Weiss BD. Health Literacy: a Manual for Clinicians. Chicago, IL, USA: American Medical Association, American Medical Foundation; 2003.
Weiss BD, Coyne C. Communicating with patients who cannot read. N Engl J Med. 1997;337:272–274.
Weiss BD, Hart G, McGee DL, D’Estelle S. Health status of illiterate adults: relation between literacy and health status among persons with low literacy skills. J Am Board Fam Pract. 1992;5:257–264.
Weiss BD, Palmer R. Relationship between health care costs and very low literacy skills in a medically needy and indigent Medicaid population. J Am Board Fam Pract. 2004;17:44–47.
Yi PH, Ganta A, Hussein KI, Frank RM, Jawa A. Readability of arthroscopy-related patient education materials from the American Academy of Orthopaedic Surgeons and Arthroscopy Association of North America web sites. Arthroscopy. 2013;29:1108–1112.
Each author certifies that he or she, or a member of his or her immediate family, has no funding or commercial associations (eg, consultancies, stock ownership, equity interest, patent/licensing arrangements, etc) that might pose a conflict of interest in connection with the submitted article.
All ICMJE Conflict of Interest Forms for authors and Clinical Orthopaedics and Related Research ® editors and board members are on file with the publication and can be viewed on request.
Clinical Orthopaedics and Related Research ® neither advocates nor endorses the use of any treatment, drug, or device. Readers are encouraged to always seek additional information, including FDA-approval status, of any drug or device prior to clinical use.
About this article
Cite this article
Eltorai, A.E.M., Sharma, P., Wang, J. et al. Most American Academy of Orthopaedic Surgeons’ Online Patient Education Material Exceeds Average Patient Reading Level. Clin Orthop Relat Res 473, 1181–1186 (2015). https://doi.org/10.1007/s11999-014-4071-2
- Health Literacy
- Grade Level
- Sixth Grade
- Reading Level
- Patient Education Material