Assessing the Readability of Covid-19 Testing Messages on the Internet

Abstract

The COVID-19 pandemic first became evident at the end of 2019, and because of the many unknown aspects of this emerging infectious disease, the internet quickly became a source of information for consumers. It is important for any vital information to be written unambiguously, and at a level that can be understood by all people regardless of education levels. The purpose of this study was to assess the readability of 50 sources of COVID19 testing information online. Only 6 websites out of 50 received an appropriate readability score on more than one assessment. One-sample, one-tailed t-tests (α = 0.05, df = 49) were used to see if the websites with information on COVID-19 testing are being written at appropriate reading levels. The resulting p-values indicate that each p-value recorded is substantially below 0.05, it is very unlikely that websites on this topic are being written at the recommended levels. Even the optimal messages on COVID-19 reflect a confusing and rapidly changing public health crisis, however if messages are kept simple and clear, individuals will have the best possible chance of optimizing behavioral mitigation strategies. These are compelling reasons for informational hosts to take necessary steps to ensure that messages are written in as simple terms as possible. To this end, it is suggested that internet sites dispersing COVID-19 testing information build in text analysis methods for all published messages, particularly those meant to inform best health practices in the time of a pandemic.

Introduction

The first announcements of an illness causing pneumonia-like symptoms had emerged in Wuhan China were evident at the end of 2019 [1, 2] and the illness was later identified as COVID-19, caused by the SARS-CoV-2 virus. Now a pandemic, [3] with 90,343,519 confirmed cases of COVID-19 and 1,936,133 deaths worldwide as of January 11, 2021 [4]. In the United States, specifically there have been 22,409,480 confirmed cases of COVID-19 and 374,341 deaths [4]. The need for adequate testing measures quickly became clear, and on March 21, 2020, the World Health Organization (WHO) highlighted that there were shortfalls in testing capacity [1]. Their laboratory testing strategies were expanded upon and reiterated as the pandemic persisted [1].

According to the Food and Drug administration, at the current time in the United states, “there are two types of diagnostic tests – molecular tests, such as RT-PCR tests, that detect the virus’s genetic material, and antigen tests that detect specific proteins from the virus [5].” Additionally, although not diagnostic, antibody tests determine if a person has developed antibodies indicating a person has had the virus. Molecular tests such as the reverse transcription molecular polymerase chain reaction (RT-PCR) and antigen tests indicate if a person is currently infected and antibody testing discerns if a person has had an infection in the past [5]. Most molecular PCR and antigen tests utilize a nasal swab to gather respiratory fluid and cell samples, whereas antibody tests use a blood draw or finger stick [5].

Despite the development of tests for COVID-19, the testing process has not been so straightforward. Challenges include, but are not limited to timing of the sample [6, 7] cost, availability, and placement of testing location centers [8], personnel issues [9, 10], variation in sensitivity and specificity [11, 12], and lag time for results [13]. Given these challenges and changes to guidelines for testing [14], the general public was left seeking answers to questions related to testing. An increasing number of people turn to the internet for information. In fact, 35 % of U.S. adults indicate that they have used the internet as a source of medical information, often without verifying their findings with a physician [15] despite the fact that this information guides medical decisions [16].

Often, medical information is difficult to understand, and even though it is highly accessible to a lay audience, few U.S. adults have proficient levels of health literacy [17]. Often health information on the internet is written at levels that are difficult to read for the average adult. The importance of COVID-19 testing in reducing community transmission and saving lives cannot be understated. [18] Therefore, it is important that online information on this topic be as accessible as possible. Therefore, the purpose of this study was to determine the level of difficulty of written information about COVID testing that is readily available on the internet.

Methods

The methods for this study follow our prior work on general COVID-19 information [19]. Fifty English language websites were gathered by the researcher to establish the sample size for the study. Google.com was utilized as the preferred search engine, primarily due to its popularity and accessibility across all populations. The search keywords used to generate the sample size was “COVID 19 testing.” URLs of the first eligible 50 websites were recorded in a Microsoft Excel spreadsheet. Sponsored or advertised weblinks were excluded from the sample. Additionally, those webpages functioning as home pages for other COVID 19 testing related resources were also excluded. For a website to qualify and be eligible in the study sample, the page at minimum, must have any direct information regarding COVID 19 testing. Overall, a total of 8 search result pages were reviewed, with 15 skipped websites that met the exclusionary criteria of a home page. Of note, these 15 websites do not include any sponsored or advertised URLs as they were not tracked for this study.

The preferred browser utilized in this study was Google Chrome, attributed by its reliability, user friendliness, speed, and security. Search results have the potential to be uniquely tailored to the user according to any preferences, saved data, and search histories. To prevent this, prior to conducting the search, Google Chrome was first cleared of all browsing and search history, cache, cookies, and other data.

Readable.io [20] was used as the readability scoring tool. The Readable service generates five common and recommended readability tests: Flesch-Kincaid Grade Level (FKGL), Gunning Fog Index (GFI), Coleman-Liau Index (CLI), the Simple Measure of Gobbledygook (SMOG) Grade level, and Flesch-Kincaid Reading Ease (FRE). FKGL and FRE are identical in that they are intended to calculate and analyze sentence word length, but differ in their scoring schema. FKGL develops a score similar to a United States grade level, while FRE deploys a ranged scoring system (0-100) where higher results indicate a generally easier read, fit for the average consumer, whereas lower scores indicate content best suited for university graduates or professionals. GFI similarly deploys a grade level scoring pattern with a focus on measuring the average words per sentence in addition to the number of polysyllabic words. CLI calculates the number of letters in a word and number of words in a sentence, which is then scored through the use of a grade level pattern similar to the United States Education system [20]. Lastly, the SMOG grade level uses a syllable counting system with a subsection of text to establish the lowest education level to understand the material [21]. These five readability scores were calculated for each of the 50 websites.

Microsoft Excel was utilized for all data capture tasks. The researcher created a code book with each row capturing different categories. These categories include: website characteristics like type (.gov, .com, .edu, etc.), creation/written dates, as well as the results of the readability tests (FKGL, GFI, CLI, SMOG, and FRE). As per the protocol at William Paterson University, the Institutional Review Board does not review studies that do not involve human subjects.

Results

In total, there were 50 websites on COVID-19 testing included in this readability study. For each site, a readability score was determined using five different readability assessments: FRE, FKGL, GFI, CLI, and SMOG. The FRE test is scored from 0 to 100, with scores from 80 to 100 considered to be easy-to-read, 60–79 average-to-read, and 0–59 difficult-to-read. The remaining four measures assign a score based on reading grade level, i.e. 7 indicates a grade 7 reading level. Based on recommendations from public health professionals, the analysis here considers a 6th grade reading level or an FRE score of easy to be appropriate [22]. All data entry and analysis was completing using Microsoft Excel.

Overall, the analysis indicates that online materials for COVID-19 testing are not being written at an appropriate reading level. None of the websites reviewed tested at the recommended level on all five readability assessments. In fact, only 6 websites received an appropriate readability score on more than one assessment. Of these 6, two were deemed appropriate by 3 tests (FKGL, GFI, and CLI) and four were deemed appropriate by 2 tests (FHGL and GFI). The remaining 9 received an appropriate score by only one of the five assessments.

Table 1 displays the distribution of websites by reading assessment and subsequently by difficulty level. The FRE, CLI, and SMOG characterized a majority of the sites as difficult to read. The FKGL, on the other hand, classified a majority of the sites (54 %) as average and falling within the 7th to 10th grade reading level range.

Table 1 The distribution of websites by readability assessment and then by difficulty level

One-sample, one-tailed t-tests (α = 0.05, df = 49) were used to see if the websites with information on COVID-19 testing are being written at appropriate reading levels. The resulting p-values are included in Table 2 along with mean readability score and standard deviation information. Given that each p-value recorded is substantially below 0.05, it is very unlikely that websites on this topic are being written at the recommended levels.

Table 2 Mean and Standard Deviation Statistics

Table 3 shows the distribution of the 50 websites by URL extension. Roughly half of sites (n = 24) had a .com extension and roughly half (n = 26) had an extension other than .com. These included .org. .gov, .net, .edu, and other. To determine if there was any difference in readability score between .com sites and non-.com sites, one-sided two-sample t-tests (α = 0.05) were performed for each readability test. The resulting p-values and additional descriptive statistics are given in Table 4. Note that a significant difference was only found for one readability test: the GFI (p = .0296). Thus, it appears that .com sites on COVID-19 do have significantly higher GFI scores than non-.com sites on the same material. Still, both categories had average GFI scores above what is considered appropriate.

Table 3 The distribution of the sites sampled by URL extension
Table 4 Readability for .com vs. non-.com sites

Discussion

Results of this analysis showed that the COVID-19 testing information reviewed on 50 prominent internet sites was written at reading levels considerably higher than recommended. This represents a miscalculation of either the text composition or readership of messages on the part of the hosts; either of which poses an obstacle to the mitigation of illness through behavioral strategies such as testing. It is known that simple, well composed messages can facilitate the management of public health crises. However, because information on COVID-19 is inherently protean and politicized, conflicting messages and contingencies have left many readers paralyzed in terms of best health practices [23]. Dissemination of complicated messages about testing thus add to the already stressful and confusing informational landscape of COVID-19. These problems with internet-based messaging must be corrected as soon as possible, particularly due to the forthcoming need to control vaccine distribution across the U.S. through similar communications.

In addition to being written at a more difficult reading level than advised, it is likely that other obstacles to comprehensible coronavirus testing messages are in effect. For instance, it is known that emotional states such as anxiety motivate individuals to seek out information on novel or ambiguous sources of threat [24,25,26]. At the same time, however, anxiety and stress can interfere with the visual processing and interpretation of messages related to threat [27, 28]. This can create a powerful feed-forward loop of COVID-19 information seeking, detection of complicated information on internet sites, even more of a rise in anxiety levels, and the cycle repeats, ultimately resulting in a failure of informed health behaviors related to the ongoing pandemic [19]. Objective evidence of the difficulty of understanding COVID-19 messages is found in the development of special tools meant to assist readers in deciding whether or not to obtain a test [29].

Even the optimal messages on COVID-19 reflect a confusing and rapidly changing public health crisis [30], however if messages are kept simple and clear, individuals will have the best possible chance of optimizing behavioral mitigation strategies. These are compelling reasons for informational hosts to take necessary steps to ensure that messages are written in as simple terms as possible [31]. To this end, it is suggest that internet sites dispersing COVID-19 testing information build in text analysis methods for all published messages, particularly those meant to inform best health practices in the time of a pandemic.

References

  1. 1.

    World Health Organization. (2020). Timeline of WHO’s response to COVID-19. https://www.who.int/news-room/detail/29-06-2020-covidtimeline.

  2. 2.

    Guan, W. J., Ni, Z. Y., Hu, Y., Liang, W. H., Ou, C. Q., & He, J. X. (2020). Clinical Characteristics of Coronavirus Disease 2019 in China. New England Journal of Medicine, 382, 1708–1720.

    CAS  Article  Google Scholar 

  3. 3.

    World Health Organization. (2020). WHO Director-General’s opening remarks at the media briefing on COVID-19–11 March 2020. https://www.who.int/dg/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020.

  4. 4.

    Johns Hopkins University & Medicine. (2021). COVID-19 Map – John Hopkins Coronavirus Resource Center. https://coronavirus.jhu.edu/map.html.

  5. 5.

    United States Food and Drug Administration. (2020). Coronavirus Disease 2019 Testing Basics. U.S Food & Drug Administration. https://www.fda.gov/consumers/consumer-updates/coronavirus-disease-2019-testing-basics#:~:text=Currently%20there%20are%20two %20types,proteins%20from%20the%20virus.

  6. 6.

    He, X., Lau, E. H. Y., Wu, P., Deng, X., Wang, J., & Hao, et al.. (2020). Temporal dynamics in viral shedding and transmissibility of COVID-19. Nature Medicine, 26, 672–675.

    CAS  Article  Google Scholar 

  7. 7.

    Walsh, K. A., Jordan, K., Clyne, B., Rohde, D., Drummond, L., Byrne, P., et al. (2020). SARS-CoV-2 detection, viral load and infectivity over the course of an infection. Journal of Infection, 81(3), 357–371.

    CAS  Article  Google Scholar 

  8. 8.

    Mervosh, S., & Fernandez, M. (2020). Months Into Virus Crisis, U.S. Cities Still Lack Testing Capacity. New York Times. https://www.nytimes.com/2020/07/06/us/coronavirus-test-shortage.html.

  9. 9.

    Bertuzzi, S., & Patel, R. (2020). We Need a New Kind of National Guard. New York Times. https://www.nytimes.com/2020/04/27/opinion/biomedical-national-guard-covid.html.

  10. 10.

    Brammer, C., & Elwell, T. (2020). The U.S. needs a national COVID-19 testing st rategy, not a state-by-state patchwork. STAT. https://www.statnews.com/2020/10/01/national-covid-19-testing-strategy-not-state-by-state-patchwork/.

  11. 11.

    La Marca, A., Capuzzo, M., Paglia, T., Roli, L., Trenti, T., & Nelson, S. M. (2020). Testing for SARS-CoV-2 (COVID-19): a systematic review and clinical guide to molecular and serological in-vitro diagnostic assays. Reproductive BioMedicine Online, 41(3), 483–499.

    Article  Google Scholar 

  12. 12.

    Deeks, J. J., Dinnes, J., Takwoingi, Y., Davenport, C., Spijker, R., Taylor-Phillips, S., et al. (2020). Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane Database of Systematic Reviews, 6(6), CD013652.

    Google Scholar 

  13. 13.

    Tirrell, M., Wells, N., & Miller, L. (2020). 40 % of U.S. Covid-19 tests come back too late to be clinically meaningful, data show. CNBC. https://www.cnbc.com/2020/08/15/forty-percent-of-us-covid-19-tests-come-back-too-late-to-be-clinically-meaningful-data-show.html.

  14. 14.

    Wu, K. (2020). CDC Now Says People Without COVID-19 Symptoms Do Not Need Testing. New York Times. https://www.nytimes.com/2020/08/25/health/covid-19-testing-cdc.html.

  15. 15.

    Fox, S., & Duggan, M. One in three American Adults have gone online to figure out a medical condition. Pew Research Center: Internet, Science & Tech. https://www.pewresearch.org/internet/2013/01/15/health-online-2013/.

  16. 16.

    Chen, Y.-Y., Li, C.-M., Liang, J.-C., & Tsai, C.-C. (2018). Health Information Obtained From the Internet and Changes in Medical Decision Making: Questionnaire Development and Cross-Sectional Survey. Journal of Medical Internet Research., 20(2), e47.

  17. 17.

    The Health Literacy of America’s Adults: Results From the 2003 National Assessment of Adult Literacy. Published online 2003:76.

  18. 18.

    National Institute on Aging. (2020). Why COVID-19 testing is the key to getting back to normal. U.S. Department of Health & Human Services. National Institute on Aging. https://www.nia.nih.gov/news/why-covid-19-testing-key-getting-back-normal.

  19. 19.

    Basch, C. H., Mohlman, J., Hillyer, G. C., & Garcia, P. (2020). Public health communication in time of crisis: Readability of on-line COVID-19 information. Disaster Medicine and Public Health Preparedness, 14(5), 635–637.

    Article  Google Scholar 

  20. 20.

    Readable.io. (n. d.). Measure text readability – readability and keyword density analysis and tools – unique readability tools to improve your writing! Readable.io. Retrieved January 2, 2020, from https://readable.io/.

  21. 21.

    McLaughlin, G. H. (1969). SMOG grading: A new readability formula. Journal of Reading, 12(8), 639–646.

    Google Scholar 

  22. 22.

    McKenzie, J. F., Neiger, B. L., & Thackeray, R. (2017). Planning, implementing, and evaluating health promotion programs: A primer (7th ed.). New York: Pearson.

    Google Scholar 

  23. 23.

    Luthra, S. (2020). Conflicting COVD Messages Create Cloud of Confusion Around Public Health and Prevention. Kaiser Health News. https://www.usnews.com/news/healthiest-communities/articles/2020-06-19/conflicting-coronavirus-messages-create-confusion-misinformation

  24. 24.

    Rosen, N. O., & Knauper, B. (2009). A little uncertainty goes a long way: state and trait differences in uncertainty interact to increase information seeking but also increase worry. Health Communication, 24(3), 228–238.

    Article  Google Scholar 

  25. 25.

    Minnaert, A. E. (1999). Individual differences in text comprehension as a function of test anxiety and prior knowledge. Psychological Reports, 84(1), 167–177.

    CAS  Article  Google Scholar 

  26. 26.

    Ladouceur, R., Gosselin, P., & Dugas, M. J. (2000). Experimental manipulation of intolerance of uncertainty: a study of a theoretical model of worry. Behavior Research and Therapy, 38(9), 933–941.

    CAS  Article  Google Scholar 

  27. 27.

    Basso, M. R., Schefft, B. K., Douglas Ris, M., & Dember, W. N. (1996). Mood and global-local visual processing. Journal of the International Neuropsychological Society, 2(3), 249–255.

    CAS  Article  Google Scholar 

  28. 28.

    MacLeod, C., & Cohen, I. L. (1993). Anxiety and the interpretation of ambiguity: A text comprehension study. Journal of Abnormal Psychology, 102, 238–247.

    CAS  Article  Google Scholar 

  29. 29.

    Centers for Disease Control and Prevention. (2020). Coronavirus Self-Checker. https://www.cdc.gov/coronavirus/2019-ncov/symptoms-testing/coronavirus-self-checker.html.

  30. 30.

    Fielding, J. (2020). Good Communication will help beat COVID-19. The Hill. https://thehill.com/opinion/healthcare/490410-good-communications-will-help-beat-covid-19.

  31. 31.

    Rogers, T. (2020). Why no one is reading your coronavirus emails. CNN. https://www.cnn.com/2020/03/13/opinions/coronavirus-emails-effective-messaging-rogers/index.html

Download references

Funding

Not Applicable.

Author information

Affiliations

Authors

Contributions

PG, CHB, and JM conceptualized the study. PG conducted the data collection. JF conducted the data analysis. All authors contributed to the writing of the manuscript.

Corresponding author

Correspondence to Philip Garcia.

Ethics declarations

Conflict of Interest

Authors declare that they have no conflict of interest to disclose.

Ethical Approval

As per the protocol at William Paterson University, the Institutional Review Board does not review studies that do not involve human subjects.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Garcia, P., Fera, J., Mohlman, J. et al. Assessing the Readability of Covid-19 Testing Messages on the Internet. J Community Health (2021). https://doi.org/10.1007/s10900-021-00973-6

Download citation

Keywords

  • COVID-19 testing
  • Readability
  • Internet
  • Online