Journal of Medical Systems

, Volume 21, Issue 1, pp 21–32 | Cite as

In Search of Controlled Evidence for Health Care Quality Improvement

  • E. Andrew Balas
  • Marcia G. Stockham
  • Joyce A. Mitchell
  • Mary Ellen Sievert
  • Bernard G. Ewigman
  • Suzanne Austin Boren


The purpose of this study was to measure the efficiency of simple searches in retrieving controlled evidence about specific primary health care quality improvement interventions and their effects. Searches were conducted to retrieve evidence on seven interventions and seven effect variables. Specific words and the closest Medical Subject Headings (MeSH) recommended by professional librarians were used to search the MEDLINE database. Searches were restricted to the MeSH publication type “randomized controlled trial.” Two reviewers independently judged retrieved citations for relevancy to the selected interventions and effects. In selecting MeSH terms, the average agreement among librarians was 64.3% (±26.1) for interventions and 57.1% (±19.9) for effects. Analysis of the 755 retrieved reports showed that MeSH term searches had an overall recall rate of 58% while the same rate for textword searches was significantly lower (11%, p < .001). The difference in overall precision rates was nonsignificant (26% versus 33%, p = .15). In the group of MeSH searches, overall precision and recall was significantly lower for effects than for interventions (12% versus 52%, p < .001 and 41% versus 69%, p < .001). Two textwords appeared in more than 25% of the benchmark collection: reminder (25.7%) and cost (25.0%). The results of this study indicate that information needs for health care quality improvement cannot be met by simple literature searches. Certain MeSH terms and combinations of textwords yield moderately efficient recall and precision in literature searches for health care quality improvement. Clinicians and physician executives gaining direct access to bibliographic database could probably be better served by structured indexing of critical aspects of randomized controlled clinical trials: design, sample, interventions, and effects.

Health Services Research MEDLINE quality assurance, health care randomized controlled trials 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Berwick, D.M., Continuous improvement as an ideal in health care. N. Eng. J. Med. 320(1):53, 1989.Google Scholar
  2. 2.
    Horowitz, R.I., The experimental paradigm and observational studies of cause-effect relationships in clinical medicine. J. Chron. Dis. 40(1):91, 1987.Google Scholar
  3. 3.
    Friedman, L.M., Furberg, C.D., & DeMets, D.L., Fundamentals of Clinical Trials (second edition), PSG, Littleton, MA, 1985.Google Scholar
  4. 4.
    North American Symptomatic Carotid Endarterectomy Trial Collaborators. Beneficial effect of carotid endarterectomy in symptomatic patients with high-grade carotid stenosis. N. Engl. J. Med. 325:445, 1991.Google Scholar
  5. 5.
    McPhee, S.J., Bird, J.A., Jenkins, C.N., & Fordham, D., Promoting cancer screening. A randomized, controlled trial of three interventions. Arch. Intern. Med. 149:1866, 1989.Google Scholar
  6. 6.
    Tape, T.G., and Campbell, J.R., Computerized medical records and preventive health care: success depends on many factors. Am. J. Med. 94:619, 1993.Google Scholar
  7. 7.
    Melin, A.L., & Bygren, L.O., Efficacy of the rehabilitation of elderly primary health are patients after short-stay hospital treatment. Med. Care 30:1004, 1992.Google Scholar
  8. 8.
    Dickersin, K., Min, Y., & Meinert, C. L., Factors influencing publication of research results. JAMA 267(3):374, 1992.Google Scholar
  9. 9.
    Funk, M.E., Reid, C.A., & McGoogan, L.S., Indexing consistency in MEDLINE. Bull. Med. Libr. Assoc. 71(2):176, 1983.Google Scholar
  10. 10.
    Silagy, C., Developing a register of randomized controlled trials in primary care. BMJ 306:897, 1993.Google Scholar
  11. 11.
    Dickersin, K., Hewitt, P., Mutch, L., Chalmers, I., & Chalmers, T.C., Perusing the literature: comparison of MEDLINE searching with a perinatal trials database. Contr. Clin. Trials 6:306, 1985.Google Scholar
  12. 12.
    Gøtzsche, P.C., & Lange, B., Comparison of search strategies for recalling double-blind trials from MEDLINE. Dan. Med. Bull. 38(6):476, 1991.Google Scholar
  13. 13.
    Kirpalani, H., Schmidt, B., McKibbon, K.A., Haynes, R.B., & Sinclair, J.C., Searching MEDLINE for randomized clinical trials involving care of the newborn. Pediatrics 83(4):543, 1989.Google Scholar
  14. 14.
    Dickersin, K., Scherer, R., & Lefebvre, C., Identifying relevant studies for systematic reviews. BMJ 309(6964):1286, 1994.Google Scholar
  15. 15.
    Hersh, W., and Hickam, D., Use of a multi-application computer worksatation in a clinical setting. Bull. Med. Libr. Assoc. 82(4):382, 1994.Google Scholar
  16. 16.
    McKinin, E.J., Sievert, M.E., Johnson, E.D., & Mitchell, J.A., The Medline/full-text research project. J. Am. Soc. Inf. Sci. 42(4):297, 1991.Google Scholar
  17. 17.
    Balas, E.A., Stockham, M.G., Mitchell, J.A., Austin, S.M., West, D.A., Ewigman, B.G., The Columbia registry of information and utilization management trials. JAMIA 2:307, 1995.Google Scholar
  18. 18.
    Balas, E.A., In Health Services in the United States. (D. Wedding, ed.), Behavior and Medicine, Mosby, St. Louis, 1995, p. 461.Google Scholar
  19. 19.
    Cohen, J., A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20(1):37, 1960.Google Scholar
  20. 20.
    National Library of Medicine. Medical subject headings, annotated alphabetic list, 1994. The Library, Bethesda, 1993.Google Scholar
  21. 21.
    Bernstein, F., The retrieval of randomized clinical trials in liver diseases from the medial literature: manual versus MEDLARS searches. Contr. Clin. Trials 9:23, 1988.Google Scholar
  22. 22.
    Jadad, A.R., McQuay, H.J., A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J. Curr. Clin. Trials 1993 (Doc No. 33), 1993.Google Scholar
  23. 23.
    Poynard, T., Conn, H.O., The retrieval of randomized clinical trials in liver disease from the medical literature: a comparison of MEDLARS and manual methods. Contr. Clin. Trials 6:271, 1985.Google Scholar
  24. 24.
    Haynes, R.B., Wilczynski, N., McKibbon, K.A., Walker, C.J., & Sinclair, J.C., Developing optimal search strategies for detecting clinically sound studies in MEDLINE. JAMIA 1:447, 1994.Google Scholar
  25. 25.
    L'Abbé, K.A., Detsky, A.S., & O'Rourke, K., Meta-analysis in clinical research. Ann. Int. Med. 107:224, 1987.Google Scholar
  26. 27.
    Sacks, H.S., Berrer, J., Reitman, D., Ancona-Berk, V.A., & Chalmers, T.C., Meta-analyses of randomized controlled trials. N. Engl. J. Med. 316:450, 1987.Google Scholar

Copyright information

© Plenum Publishing Corporation 1997

Authors and Affiliations

  • E. Andrew Balas
    • 1
    • 2
  • Marcia G. Stockham
    • 1
  • Joyce A. Mitchell
    • 2
  • Mary Ellen Sievert
    • 3
  • Bernard G. Ewigman
    • 4
  • Suzanne Austin Boren
    • 1
  1. 1.Program in Health Services ManagementUniversity of Missouri-ColumbiaColumbia
  2. 2.Medical Informatics GroupUniversity of Missouri-ColumbiaColumbia
  3. 3.School of Library and Informational SciencesUniversity of Missouri-ColumbiaColumbia
  4. 4.Family and Community MedicineUniversity of Missouri-ColumbiaColumbia

Personalised recommendations