Journal of General Internal Medicine

, Volume 27, Issue 8, pp 974–984 | Cite as

Changing Clinical Practice Through Patient Specific Reminders Available at the Time of the Clinical Encounter: Systematic Review and Meta-Analysis

  • Tim A. HoltEmail author
  • Margaret Thorogood
  • Frances Griffiths



To synthesise current evidence for the influence on clinical behaviour of patient-specific electronically generated reminders available at the time of the clinical encounter.


PubMed, Cochrane library of systematic reviews; Science Citation Index Expanded; Social Sciences Citation Index; ASSIA; EMBASE; CINAHL; DARE; HMIC were searched for relevant articles.


We included controlled trials of reminder interventions if the intervention was: directed at clinician behaviour; available during the clinical encounter; computer generated (including computer generated paper-based reminders); and generated by patient-specific (rather than condition specific or drug specific) data.


Systematic review and meta-analysis of controlled trials published since 1970. A random effects model was used to derive a pooled odds ratio for adherence to recommended care or achievement of target outcome. Subgroups were examined based on area of care and study design. Odds ratios were derived for each sub-group. We examined the designs, settings and other features of reminders looking for factors associated with a consistent effect.


Altogether, 42 papers met the inclusion criteria. The studies were of variable quality and some were affected by unit of analysis errors due to a failure to account for clustering. An overall odds ratio of 1.79 [95% confidence interval 1.56, 2.05] in favour of reminders was derived. Heterogeneity was high and factors predicting effect size were difficult to identify.


Methodological diversity added to statistical heterogeneity as an obstacle to meta-analysis. The quality of included studies was variable and in some reports procedural details were lacking.


The analysis suggests a moderate effect of electronically generated, individually tailored reminders on clinician behaviour during the clinical encounter. Future research should concentrate on identifying the features of reminder interventions most likely to result in the target behaviour.


reminder systems electronic health records computer systems decision support systems, clinical 



We thank Dr Simon Gates for analytical advice and Samantha Johnson for assistance with database searching and retrieval of papers. We also thank the authors of studies for which clarification was required for providing useful information.

Conflict of Interest

The authors declare that they do not have a conflict of interest.

Funding source

Funded through internal sources. No external funding.


  1. 1.
    Balas EA, Austin SM, Mitchell JA, Ewigman BG, Bopp KD, Brown GD. The clinical value of computerized information services - A review of 98 randomized clinical trials. Arch Fam Med. 1996;5(5):271–8.PubMedCrossRefGoogle Scholar
  2. 2.
    Bennett JW, Glasziou PP, Sim I. Review: Computerised reminders and feedback can improve provider medication management. Evid Base Med. 2003;8(6):190.CrossRefGoogle Scholar
  3. 3.
    Garg AX, Adhikari NKJ, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: A systematic review. JAMA. 2005;293(10):1223–38.PubMedCrossRefGoogle Scholar
  4. 4.
    Kawamoto K, Houlihan C, Balas E, Lobach D. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005;330(7494):765.PubMedCrossRefGoogle Scholar
  5. 5.
    Mitchell E, Sullivan F. A descriptive feast but an evaluative famine: systematic review of published articles on primary care computing during 1980-97. BMJ. 2001;322(7281):279–82E.PubMedCrossRefGoogle Scholar
  6. 6.
    Montgomery AA, Fahey T. A systematic review of the use of computers in the management of hypertension. Journal of Epidemiology and Community Health. 1998;52(8):520–5.PubMedCrossRefGoogle Scholar
  7. 7.
    Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc. 1996;3(6):399–409.PubMedCrossRefGoogle Scholar
  8. 8.
    Shiffman RN, Liaw Y, Brandt CA, Corb GJ. Computer-based guideline implementation systems: a systematic review of functionality and effectiveness. J Am Med Inform Assoc. 1999;6(2):104–14.PubMedCrossRefGoogle Scholar
  9. 9.
    Dexheimer JW, Talbot TR, Sanders DL, Rosenbloom ST, Aronsky D. Prompting clinicians about preventive care measures: a systematic review of randomized controlled trials. J Am Med Inform Assoc. 2008;15(3):311–20.PubMedCrossRefGoogle Scholar
  10. 10.
    Shojania KG, Jennings A, Mayhew A, et al. The effects of on-screen, point of care computer reminders on processes and outcomes of care. Cochrane Database Syst Rev. 2009;3:CD001096.PubMedGoogle Scholar
  11. 11.
    Shojania KG, Jennings A, Mayhew A, et al. Effect of point-of-care computer reminders on physician behaviour: a systematic review. CMAJ. 2010;182:E216–25.PubMedCrossRefGoogle Scholar
  12. 12.
    Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from (last accessed 24.1.12).
  13. 13.
    Adams G, Gulliford MC, Ukoumunne OC, et al. Patterns of intra-cluster correlation from primary care research to inform study design and analysis. J Clin Epidemiol. 2004;57(8):785–794.PubMedCrossRefGoogle Scholar
  14. 14.
    Matheny ME, Sequist TD, Seger AC, Fiskio JM, Sperling M, Bugbee D, et al. A randomized trial of electronic clinical reminders to improve medication laboratory monitoring. J Am Med Inform Assoc. 2008;15(4):424–9.PubMedCrossRefGoogle Scholar
  15. 15.
    Bates DW, Kuperman GJ, Rittenberg E, Teich JM, Fiskio J, Ma'luf N, et al. A randomized trial of a computer-based intervention to reduce utilization of redundant laboratory tests. Am J Med. 1999;106(2):144–50.PubMedCrossRefGoogle Scholar
  16. 16.
    Burack RC, Gimotty PA, George J, Simon MS, Dews P, Moncrease A. The effect of patient and physician reminders on use of screening mammography in a health maintenance organization. Results of a randomized controlled trial. Cancer. 1996;78(8):1708–21.PubMedCrossRefGoogle Scholar
  17. 17.
    Burack RC, Gimotty PA, George J, McBride S, Moncrease A, Simon MS, et al. How reminders given to patients and physicians affected pap smear use in a health maintenance organization: results of a randomized controlled trial. Cancer. 1998;82(12):2391–400.PubMedCrossRefGoogle Scholar
  18. 18.
    Chambers CV, Balaban DJ, Carlson BL, Grasberger DM. The effect of microcomputer-generated reminders on influenza vaccination rates in a university-based family practice center. J Am Board Fam Pract. 1991;4(1):19–26.PubMedGoogle Scholar
  19. 19.
    Dexter PR, Wolinsky FD, Gramelspacher GP, Zhou XH, Eckert GJ, Waisburd M, et al. Effectiveness of computer-generated reminders for increasing discussions about advance directives and completion of advance directive forms. A randomized, controlled trial. Ann Intern Med. 1998;128(2):102–10.PubMedGoogle Scholar
  20. 20.
    Dexter PR, Perkins S, Overhage JM, Maharry K, Kohler RB, McDonald CJ. A computerized reminder system to increase the use of preventive care for hospitalized patients. N Engl J Med. 2001;345(13):965–70.PubMedCrossRefGoogle Scholar
  21. 21.
    Eccles M, McColl E, Steen N, Rousseau N, Grimshaw J, Parkin D, et al. Effect of computerised evidence based guidelines on management of asthma and angina in adults in primary care: cluster randomised controlled trial. BMJ. 2002;325(7370):941.PubMedCrossRefGoogle Scholar
  22. 22.
    Filippi A, Sabatini A, Badioli L, Samani F, Mazzaglia G, Catapano A, et al. Effects of an automated electronic reminder in changing the antiplatelet drug-prescribing behavior among Italian general practitioners in diabetic patients: an intervention trial. Diabetes Care. 2003;26(5):1497–500.PubMedCrossRefGoogle Scholar
  23. 23.
    Frank O, Litt J, Beilby J. Opportunistic electronic reminders: improving performance of preventive care in general practice. Aust Fam Physician. 2004;33(1/2):87–90.PubMedGoogle Scholar
  24. 24.
    Hicks LS, Sequist TD, Ayanian JZ, Shaykevich S, Fairchild DG, Orav EJ, et al. Impact of computerized decision support on blood pressure management and control: A randomized controlled trial. J Gen Intern Med. 2007;23(4):429–41.CrossRefGoogle Scholar
  25. 25.
    Judge J, Field TS, DeFlorio M, Laprino J, Auger J, Rochon P, et al. Prescribers' responses to alerts during medication ordering in the long term care setting. J Am Med Inform Assoc. 2006;13(4):385–90.PubMedCrossRefGoogle Scholar
  26. 26.
    Kenealy T, Arroll B, Petrie KJ. Patients and computers as reminders to screen for diabetes in family practice. Randomized-controlled trial. J Gen Intern Med. 2005;20(10):916–21.PubMedCrossRefGoogle Scholar
  27. 27.
    Kralj B, Iverson D, Hotz K, Ashbury FD. The impact of computerized clinical reminders on physician prescribing behavior: Evidence from community oncology practice. Am J Med Qual. 2003;18(5):197–203.PubMedCrossRefGoogle Scholar
  28. 28.
    Krall MA, Traunweiser K, Towery W. Effectiveness of an electronic medical record clinical quality alert prepared by off-line data analysis. Medinfo. 2004;11(Pt 1):135–9.Google Scholar
  29. 29.
    Kucher N, Koo S, Quiroz R, Cooper JM, Paterno MD, Soukonnikov B, et al. Electronic alerts to prevent venous thromboembolism among hospitalized patients. N Engl J Med. 2005;352(10):969–77.PubMedCrossRefGoogle Scholar
  30. 30.
    Litzelman DK, Dittus RS, Miller ME, Tierney WM. Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols. J Gen Intern Med. 1993;8(6):311–7.PubMedCrossRefGoogle Scholar
  31. 31.
    Lo HG, Matheny ME, Seger DL, Bates DW, Gandhi TK. Impact of non-interruptive medication laboratory monitoring alerts in ambulatory care. J Am Med Inform Assoc. 2009;16(1):66–71.PubMedCrossRefGoogle Scholar
  32. 32.
    McCowan C, Neville RG, Ricketts IW, Warner FC, Hoskins G, Thomas GE. Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma. Med Inform Internet Med. 2001;26(3):191–201.PubMedCrossRefGoogle Scholar
  33. 33.
    McDonald CJ. Use of a computer to detect and respond to clinical events: Its effect on clinical behavior. Ann Intern Med. 1976;84:162–7.Google Scholar
  34. 34.
    McDonald CJ, Wilson GA, McCabe GP Jr. Physician response to computer reminders. JAMA. 1980;244(14):1579–81.PubMedCrossRefGoogle Scholar
  35. 35.
    McDowell I, Newell C, Rosser W. A randomized trial of computerized reminders for blood pressure screening in primary care. Med Care. 1989;27(3):297–305.PubMedCrossRefGoogle Scholar
  36. 36.
    McDowell I, Newell C, Rosser W. Computerized reminders to encourage cervical screening in family practice. J Fam Pract. 1989;28(4):420–4.PubMedGoogle Scholar
  37. 37.
    Murray MD, Harris LE, Overhage JM, Zhou XH, Eckert GJ, Smith FE, et al. Failure of Computerized Treatment Suggestions to Improve Health Outcomes of Outpatients with Uncomplicated Hypertension: Results of a Randomized Controlled Trial. Pharmacotherapy. 2004;24(3):324–37.PubMedCrossRefGoogle Scholar
  38. 38.
    Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med. 1996;156(14):1551–6.PubMedCrossRefGoogle Scholar
  39. 39.
    Overhage JM, Tierney WM, Zhou XH, McDonald CJ. A randomized trial of "corollary orders" to prevent errors of omission. J Am Med Inform Assoc. 1997;4(5):364–75.PubMedCrossRefGoogle Scholar
  40. 40.
    Rosser WW, McDowell I, Newell C. Use of Reminders for Preventive Procedures in Family Medicine. CMAJ. 1991;145(7):807.PubMedGoogle Scholar
  41. 41.
    Rosser WW, Hutchison BG, McDowell I, Newell C. Use of Reminders to Increase Compliance with Tetanus Booster Vaccination. CMAJ. 1992;146(6):911–7.PubMedGoogle Scholar
  42. 42.
    Rossi RA, Every NR. A computerized intervention to decrease the use of calcium channel blockers in hypertension. J Gen Intern Med. 1997;12(11):672–8.PubMedCrossRefGoogle Scholar
  43. 43.
    Rothschild J, McGurk S, Honour M, Lu L, McClendon A, Srivastava P, et al. Assessment of education and computerized decision support interventions for improving transfusion practice. Transfusion. 2007;47(2):228–39.PubMedCrossRefGoogle Scholar
  44. 44.
    Safran C, Rind DM, Davis RB, Ives D, Sands DZ, Currier J, et al. Guidelines for management of HIV infection with computer-based patient's record. Lancet. 1995;346(8971):341–6.PubMedCrossRefGoogle Scholar
  45. 45.
    Sequist TD, Gandhi TK, Karson AS, Fiskio JM, Bugbee D, Sperling M, et al. A randomized trial of electronic clinical reminders to improve quality of care for diabetes and coronary artery disease. Journal of the American Medical Informatics Association. 2005;12(4):431–7.PubMedCrossRefGoogle Scholar
  46. 46.
    Shea S, Sideli RV, DuMouchel W, Pulver G, Arons RR, Clayton PD. Computer-generated informational messages directed to physicians: effect on length of hospital stay. J Am Med Inform Assoc. 1995;2(1):58–64.PubMedCrossRefGoogle Scholar
  47. 47.
    Tamblyn R, Huang A, Perreault R, Jacques A, Roy D, Hanley J, et al. The medical office of the 21st century (MOXXI): effectiveness of computerized decision-making support in reducing inappropriate prescribing in primary care. CMAJ. 2003;169(6):549–56.PubMedGoogle Scholar
  48. 48.
    Tamblyn R, Huang A, Taylor L, Kawasumi Y, Bartlett G, Grad R, et al. A randomized trial of the effectiveness of on-demand versus computer-triggered drug decision support in primary care. J Am Med Inform Assoc. 2008;15(4):430–8.PubMedCrossRefGoogle Scholar
  49. 49.
    Tape TG, Campbell JR. Computerized medical records and preventive health care: success depends on many factors. Am J Med. 1993;94(6):619–25.PubMedCrossRefGoogle Scholar
  50. 50.
    Tierney WM, McDonald CJ, Martin DK, Rogers MP. Computerized display of past test results. Effect on outpatient testing. Ann Intern Med. 1987;107(4):569–74.PubMedGoogle Scholar
  51. 51.
    Tierney WM, Overhage JM, Murray MD, Harris LE, Zhou XH, Eckert GJ, et al. Effects of computerized guidelines for managing heart disease in primary care. J Gen Intern Med. 2003;18(12):967–76.PubMedCrossRefGoogle Scholar
  52. 52.
    Tierney W, Overhage J, Murray M, Harris L, Zhou X, Eckert G, et al. Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial. Health Serv Res. 2005;40(2):477–97.PubMedCrossRefGoogle Scholar
  53. 53.
    van Wyk MA, van der Lei J, Mosseveld M, Bohnen AM, van Bemmel JH. Assessment of decision support for blood test ordering in primary care. A randomized trial. Ann Intern Med. 2001;134(4):274–81.Google Scholar
  54. 54.
    White KS, Lindsay A, Pryor TA, Brown WF, Walsh K. Application of a computerized medical decision-making process to the problem of digoxin intoxication. J Am Coll Cardiol. 1984;4(3):571–6.PubMedCrossRefGoogle Scholar
  55. 55.
    Holt TA, Thorogood M, Griffiths F, Munday S, Friede T, Stables D. Automated electronic reminders to facilitate cardiovascular disease prevention: randomised controlled trial. Brit J Gen Pract. 2010;60(573):e137–43.CrossRefGoogle Scholar
  56. 56.
    Gandhi TJ, Sequist TD, Poon EG, et al. Primary care clinicians’ attitudes towards electronic clinical reminders and clinical practice guidelines. Proceedings of the AMIA Symposium 2003:848.Google Scholar
  57. 57.
    Krall MA, Sittig DF. Subjective assessment of usefulness and appropriate presentation mode of alerts and reminders in the outpatient setting. Proceedings of the AMIA Symposium 2001:334–338.Google Scholar
  58. 58.
    Saleem JJ, Patterson ES, Militello L, Render ML, Orshansky G, Asch SM. Exploring barriers and facilitators to the use of computerized clinical reminders. JAMIA. 2005;12:438–447.PubMedGoogle Scholar
  59. 59.
    Patterson ES, Doebbeling BN, Fung CH, Militello L, Anders S, Asch SM. Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods. J Biomed Inform. 2005;38:189–199.PubMedCrossRefGoogle Scholar
  60. 60.
    Sittig DF, Krall MA, Dykstra RH, Russell A, Chin HL. A survey of factors affecting clinician acceptance of clinical decision support. BMC Med Informat Decis Making. 2006;6:6.CrossRefGoogle Scholar
  61. 61.
    Krall MA, Sittig DF. Clinicians’ assessments of outpatient electronic medical record alert and reminder usability and usefulness requirements. Proceedings of the AMIA Symposium 2002:400–404.Google Scholar
  62. 62.
    Agrawal A, Mayo-Smith MF. Adherence to computerized clinical reminders in a large healthcare delivery network. Medinfo. 2004;11(1):111–114.Google Scholar
  63. 63.
    Mayo-Smith MF, Agrawal A. Factors associated with improved completion of computerized clinical reminders across a large healthcare system. Int J Med Informat. 2007;76(10):710–716.CrossRefGoogle Scholar

Copyright information

© Society of General Internal Medicine 2012

Authors and Affiliations

  • Tim A. Holt
    • 1
    • 2
    Email author
  • Margaret Thorogood
    • 2
  • Frances Griffiths
    • 2
  1. 1.Department of Primary Care Health SciencesUniversity of OxfordOxfordUK
  2. 2.Health Sciences Research InstituteUniversity of WarwickCoventryUK

Personalised recommendations