Evaluating Completeness of a Radiology Glossary Using Iterative Refinement


A lay-language glossary of radiology, built to help patients better understand the content of their radiology reports, has been analyzed for its coverage and readability, but not for its completeness. We present an iterative method to sample radiology reports, identify “missing” terms, and measure the glossary’s completeness. We hypothesized that the refinement process would reduce the number of missing terms to fewer than 1 per report. A random sample of 1000 radiology reports from a large US academic health system was divided into 10 cohorts of 100 reports each. Each cohort was reviewed in sequence by two investigators to identify terms (single words and multi-word phrases) absent from the glossary. Terms marked as new were added to the glossary and hence was shown as matched in subsequent cohorts. This HIPAA-compliant study was IRB-approved; informed consent was waived. The refinement process added a mean of 288.0 new terms per 100 reports in the first 5 cohorts vs. a mean of 66.0 new terms per 100 reports in the last 5 cohorts; the difference was statistically significant (p < .01). After reviewing 500 reports, the review process found fewer than 1 new term per report in each of 500 subsequent reports. The findings suggest that 500 to 1000 reports is adequate to test the completeness of a glossary, and that the glossary after iterative refinement achieved a high level of completeness to cover the vocabulary of radiology reports.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2


  1. 1.

    Kemp JL, Mahoney MC, Mathews VP, Wintermark M, Yee J, Brown SD: Patient-centered radiology: Where are we, where do we want to be, and how do we get there? Radiology 285:601–608, 2017

    Article  PubMed  Google Scholar 

  2. 2.

    Itri JN: Patient-centered radiology. RadioGraphics 35:1835–1846, 2015

    Article  PubMed  Google Scholar 

  3. 3.

    Bruno MA, Petscavage-Thomas JM, Mohr MJ, Bell SK, Brown SD: The "open letter": radiologists' reports in the era of patient web portals. J Am Coll Radiol 11:863–867, 2014

    Article  PubMed  Google Scholar 

  4. 4.

    Gefen R, Bruno MA, Abujudeh HH: Online portals: Gateway to patient-centered radiology. AJR Am J Roentgenol:1–5, 2017

  5. 5.

    Henshaw D, Okawa G, Ching K, Garrido T, Qian H, Tsai J: Access to radiology reports via an online patient portal: Experiences of referring physicians and patients. J Am Coll Radiol 12 e581:582–586, 2015

    Article  Google Scholar 

  6. 6.

    Miles RC, Hippe DS, Elmore JG, Wang CL, Payne TH, Lee CI: Patient access to online radiology reports. frequency and sociodemographic characteristics associated with use. Acad Radiol 23:1162–1169, 2016

    PubMed  Google Scholar 

  7. 7.

    How to write easy-to-read health materials. Available at www.nlm.nih.gov/medlineplus/etr.html. Accessed 30 January 2017

  8. 8.

    Weiss BD: Health literacy and patient safety: Help patients understand: Manual for clinicians. Chicago, IL: American Medical Association Foundation. American Medical Association, 2007

    Google Scholar 

  9. 9.

    Sierra AE, Bisesi MA, Rosenbaum TL, Potchen EJ: Readability of the radiologic report. Invest Radiol 27:236–239, 1992

    Article  CAS  PubMed  Google Scholar 

  10. 10.

    Trofimova A, Vey BL, Safdar NM, Duszak R Jr, Kadom N: Radiology report readability: An opportunity to improve patient communication. J Am Coll Radiol: Published online: https://doi.org/10.1016/j.jacr.2018.1003.1032, 2018

  11. 11.

    Oh SC, Cook TS, Kahn CE Jr: PORTER: A prototype system for patient-oriented radiology reporting. J Digit Imaging 29:450–454, 2016

    Article  PubMed  PubMed Central  Google Scholar 

  12. 12.

    Martin-Carreras T, Kahn CE Jr: Coverage and readability of information resources to help patients understand radiology reports. J Am Coll Radiol. https://doi.org/10.1016/j.jacr.2017.11.019, 2017

  13. 13.

    Mervak BM, Davenport MS, Flynt KA, Kazerooni EA, Weadock WJ: What the patient wants: An analysis of radiology-related inquiries from a web-based patient portal. J Am Coll Radiol 13:1311–1318, 2016

    Article  PubMed  Google Scholar 

  14. 14.

    Okawa G, Ching K, Qian H, Feng Y: Automatic release of radiology reports via an online patient portal. J Am Coll Radiol 14:1219–1221, 2017

    Article  PubMed  Google Scholar 

  15. 15.

    Otte-Trojel T, de Bont A, Rundall TG, van de Klundert J: What do we know about developing patient portals? A systematic literature review. J Am Med Informatics Assoc 23:e162–e168, 2016

    Article  Google Scholar 

  16. 16.

    Lee CI, Langlotz CP, Elmore JG: Implications of direct patient online access to radiology reports through patient web portals. J Am Coll Radiol 13:1608–1614, 2016

    Article  PubMed  Google Scholar 

  17. 17.

    Cook TS, Oh SC, Kahn CE Jr: Patients' use and evaluation of an online system to annotate radiology reports with lay-language definitions. Acad Radiol 24:1169–1174, 2017

    Article  PubMed  Google Scholar 

  18. 18.

    Chen W, Durkin C, Huang Y, Adler B, Rust S, Lin S: Simplified readability metric drives improvement of radiology reports: An experiment on ultrasound reports at a pediatric hospital. J Digit Imaging 30:710–717, 2017

    Article  PubMed  PubMed Central  Google Scholar 

  19. 19.

    Hansberry DR, Agarwal N, Baker SR: Health literacy and online educational resources: An opportunity to educate patients. AJR Am J Roentgenol 204:111–116, 2015

    Article  PubMed  Google Scholar 

  20. 20.

    Gunn AJ, Mangano MD, Choy G, Sahani DV: Rethinking the role of the radiologist: Enhancing visibility through both traditional and nontraditional reporting practices. RadioGraphics 35:416–423, 2015

    Article  PubMed  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Charles E. Kahn Jr.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Chan, P.Y.W., Kahn, C.E. Evaluating Completeness of a Radiology Glossary Using Iterative Refinement. J Digit Imaging 32, 417–419 (2019). https://doi.org/10.1007/s10278-018-0137-0

Download citation


  • Radiology reports
  • Patient-centered care
  • Vocabulary
  • Patient portals
  • Glossary