Abstract
BACKGROUND: Measuring and reporting patients’ experiences with health plans has been routine for several years. There is now substantial interest in measuring patients’ experiences with individual physicians, but numerous concerns remain.
OBJECTIVE: The Massachusetts Ambulatory Care Experiences Survey Project was a statewide demonstration project designed to test the feasibility and value of measuring patients’ experiences with individual primary care physicians and their practices.
DESIGN: Cross-sectional survey administered to a statewide sample by mail and telephone (May–August 2002).
PATIENTS: Adult patients from 5 commercial health plans and Medicaid sampled from the panels of 215 generalist physicians at 67 practice sites (n=9,625).
MEASUREMENTS: Ambulatory Care Experiences Survey produces 11 summary measures of patients’ experiences across 2 domains: quality of physician-patient interactions and organizational features of care. Physician-level reliability was computed for all measures, and variance components analysis was used to determine the influence of each level of the system (physician, site, network organization, plan) on each measure. Risk of misclassifying individual physicians was evaluated under varying reporting frameworks.
RESULTS: All measures except 2 achieved physician-level reliability of at least 0.70 with samples of 45 patients per physician, and several exceeded 0.80. Physicians and sites accounted for the majority of system-related variance on all measures, with physicians accounting for the majority on all “interaction quality” measures (range: 61.7% to 83.9%) and sites accounting for the largest share on “organizational” measures (range: 44.8% to 81.1%). Health plans accounted for neglible variance (<3%) on all measures. Reporting frameworks and principles for assuring misclassification risk ≤ 2.5% were identified.
CONCLUSIONS: With considerable national attention on the importance of patient-centered care, this project demonstrates the feasibility of obtaining highly reliable measures of patients’ experiences with individual physicians and practices. The analytic findings underscore the validity and importance of looking beyond health plans to individual physicians and sites as we seek to improve health care quality.
Similar content being viewed by others
References
Epstein AM. Public release of performance data: a progress report from the front. JAMA. 2000;283:1884–6.
Lee TH, Meyer GS, Brennan TA. A middle ground on public accountability. N Engl J Med. 2004;350:2409–12.
Institute of Medicine. Crossing the Quality Chasm: A New Health System for the Twenty-First Century. Washington, DC: National Academy Press; 2001.
Solomon LS, Zaslavsky AM, Landon BE, Cleary PD. Variation in patient-reported quality among health care organizations. Health Care Fin Rev. 2002;23:1–16.
Landon BE, Wilson IB, Cleary PD. A conceptual model of the effects of health care organizations on the quality of medical care. JAMA. 1998;279:1377–82.
Rodriguez H, von Glahn T, Rogers W, Chang H, Fanjiang G, Safran D. Evaluating patients’ experiences with individual physicians: a randomized trial of mail, internet and interactive voice response (IVR) telephone administration of surveys. Med Care. 2006, in press.
Safran DG, Kosinski M, Tarlov AR, et al. The primary care assessment survey: tests of data quality and measurement performance. Med Care. 1998;36:728–39.
Hays RD, Shaul JA, Williams VS, et al. Psychometric properties of the CAHPS 1.0 survey measures. Consumer Assessment of Health Plans Study. Med Care. 1999;37:22–31.
Institute of Medicine. Primary Care: America’s Health in a New Era. Washington, DC: National Academy Press; 1996.
Dillman DA. Mail and Telephone Surveys: The Total Design Method. New York: John Wiley; 1978.
Fowler FJ Jr., Gallagher PM, Stringfellow VL, Zaslavsky AM, Thompson JW, Cleary PD. Using telephone interviews to reduce nonresponse bias to mail surveys of health plan members. Med Care. 2002;40:190–200.
Zaslavsky AM, Zaborski LB, Cleary PD. Factors affecting response rates to the consumer assessment of health plans study survey. Med Care. 2002;40:485–99.
Lasek RJ, Barkley W, Harper DL, Rosenthal GE. An evaluation of the impact of nonresponse bias on patient satisfaction surveys. Med Care. 1997;35:646–52.
Little R, Rubin D. Statistical Analysis with Missing Data. NY: Wiley and Sons; 1987.
Allen HM, Rogers WH. The consumer health plan value survey: round two. Health Aff (Millwood). 1997;16:156–66.
Brenner H. Alternative approaches for estimating prevalence in epidemiologic surveys with two waves of respondents. Am J Epidemiol. 1995;142:1236–45.
Herbert R, Bravo G, Korner-Bitensky N, Voyer L. Refusal and information bias associated with postal questionnaires and face-to-face interviews in very elderly subjects. J Clin Epidemiol. 1996;49:373–81.
Keeter S, Miller C, Kohut A, Groves RM, Presser S. Consequences of reducing nonresponse in a national telephone survey. Public Opin Q. 2000;64:125–48.
Gasquet I, Falissard B, Ravaud P. Impact of reminders and method of questionnaire distribution on patient response to mail-back satisfaction survey. J Clin Epidemiol. 2001;54:1174–80.
Friedman EM, Clusen NA, Hartzell M. Better Late? Characteristics of Late Responders to a Health Care Survey. No. PP03-52, 1–7. Washington, DC: Mathematica Policy Research, Inc; 2004.
Nunnelly J, Bernstein I. Psychometric Theory. 3rd ed. New York: McGraw-Hill; 1994.
SAS Institute Inc. SAS/STAT User’s Guide. Version 8. Cary, NC: SAS Institute Inc.; 1999.
Zaslavsky AM, Landon BE, Beaulieu ND, Cleary PD. How consumer assessments of managed care vary within and among markets. Inquiry. 2000;37:146–61.
Hofer TP, Hayward RA, Greenfield S, Wagner EH, Kaplan SH, Manning WG. The unreliability of individual physician “report cards” for assessing the costs and quality of care of a chronic disease. JAMA. 1999;281:2098–105.
Landon BE, Normand SL, Blumenthal D, Daley J. Physician clinical performance assessment: prospects and barriers. JAMA. 2003;290:1183–9.
Miller ME, Hui SL, Tierney WM, McDonald CJ. Estimating physician costliness. An empirical Bayes approach. Med Care. 1993;31:YS16–28.
Orav EJ, Wright EA, Palmer RH, Hargraves JL. Issues of variability and bias affecting multisite measurement of quality of care. Med Care. 1996;34:SS87–101.
Hayward RA, Manning WG Jr., McMahon LF Jr., Bernard AM. Do attending or resident physician practice styles account for variations in hospital resource use? Med Care. 1994;32:788–94.
DiMatteo MR. Enhancing patient adherence to medical recommendations. JAMA. 1994;271:79–83.
DiMatteo MR, Sherbourne CD, Hays RD, et al. Physicians’ characteristics influence patients’ adherence to medical treatment: results from the Medical Outcomes Study. Health Psychol. 1993;12:93–102.
Francis V, Korsch BM, Morris MJ. Gaps in doctor-patient communication: patients’ response to medical advice. N Engl J Med. 1969;280:535–40.
Wilson IB, Rogers WH, Chang H, Safran DG. Cost-related skipping of medications and other treatments among Medicare beneficiaries between 1998 and 2000: results of a national study. J Gen Intern Med. 2005;20:715–20.
Safran DG, Taira DA, Rogers WH, Kosinski M, Ware JE, Tarlov AR. Linking primary care performance to outcomes of care. J Fam Pract. 1998;47:213–20.
Greenfield S, Kaplan S, Ware JE Jr. Expanding patient involvement in care. Effects on patient outcomes. Ann Intern Med. 1985;102:520–8.
Greenfield S, Kaplan SH, Ware JEJ, Yano EM, Frank HJ. Patients’ participation in medical care: effects on blood sugar control and quality of life in diabetes. J Gen Intern Med. 1988;3:448–57.
Stewart MA. Effective physician-patient communication and health outcomes: a review [see comments]. CMAJ. 1995;152:1423–33.
Roter DL, Hall JA, Kern DE, Barker LR, Cole KA, Roca RP. Improving physicians’ intervewing skills and reducing patients’ emotional distress. A randomized clinical trial. Arch Intern Med. 1995;155:1877–84.
Safran DG, Montgomery JE, Chang H, Murphy J, Rogers WH. Switching doctors: predictors of voluntary disenrollment from a primary physician’s practice. J Fam Pract. 2001;50:130–6.
Levinson W, Roter DL, Mullooly JP, Dull V, Frankel RM. Physician-patient communication: the relationship with malpractice claims among primary care physicians and surgeons. JAMA. 1997;277:553–9.
Beckman HB, Markakis KM, Suchman AL, Frankel RM. The doctor-patient relationship and malpractice: lessons from plaintiff depositions. Arch Intern Med. 1994;154:1365–70.
Hickson GBC, Clayton EW, Entman SS, et al. Obstetrician’s prior malpractice experience and patients’ satisfaction with care. JAMA. 1994;272:1583–7.
Miller S. American Board of Medical Specialties: Implementing the CAHPS Ambulatory Care Survey: A Panel Discussion of Current and Planned Activities. Presented at CAHPS Across the Health Care Continuum: 9th National User Group Meeting. Baltimore, MD, 12-2-2004.
Karp M. Massachusetts Health Quality Partners: Implementing the CAHPS Ambulatory Care Survey: A Panel Discussion of Current and Planned Activities. Presented at CAHPS Across the Health Care Continuum: 9th National User Group Meeting. Baltimore, MD, 12-2-2004.
Damberg C. Pacific Business Group on Health; Implementing the CAHPS Ambulatory Care Survey: A Panel Discussion of Current and Planned Activities. Presented at CAHPS Across the Health Care Continuum: 9th National User Group Meeting. Baltimore, MD, 12-2-2004.
Centers for Medicare and Medicaid Services (CMS). CMS Doctor’s Office Quality Project Overview, 7-15-2004. http://www.cms.hhs.gov/quality/doq/DOQOverview.pdf
Safran DG. Defining the future of primary care: what can we learn from patients? Ann Intern Med. 2003;138:248–55.
Murphy J, Chang H, Montgomery J, Rogers WH, Safran DG. The quality of physician-patient relationships: patients’ experiences 1996–1999. J Fam Pract. 2001;50:123–9.
National Committee for Quality Assurance. State of Managed Care Quality 2000. Washington, DC, 2000.
National Committee for Quality Assurance. State of Managed Care Quality, 2001. Washington, DC, 2001.
Jencks SF, Huff ED, Cuerdon T. Change in the quality of care delivered to Medicare beneficiaries, 1998–1999 to 2000–2001. JAMA. 2003;289:305–12.
Author information
Authors and Affiliations
Corresponding author
Additional information
The authors have no conflicts of interest to report.
This research was supported by grants from the Commonwealth Fund and the Robert Wood Johns Foundation. The authors gratefully acknowledge members of our National Advisory Committee, our Massachusetts steering committee, and our project officers (Anne-Marie Audet, M.D., Steven Shoenbaum, M.D., and Michael Rothman, Ph.D.) for their invaluable advise and guidance throughout the project period. The authors also gratefully acknowledge Paul Kallaur, Nina Smith and their project staff at the Centers for the Study of Services (CSS) their technical expertise and commitment to excellence in obtaining the data from the study sample. Finally, we thank Ira B. Wilson, M.D., M.S. for comments on an earlier draft and Jamie Belli for technical assistance in preparing this manuscript.
Rights and permissions
About this article
Cite this article
Safran, D.G., Karp, M., Coltin, K. et al. Measuring patients’ experiences with individual primary care physicians. J GEN INTERN MED 21, 13–21 (2006). https://doi.org/10.1111/j.1525-1497.2005.00311.x
Received:
Revised:
Accepted:
Issue Date:
DOI: https://doi.org/10.1111/j.1525-1497.2005.00311.x