Skip to main content

Advertisement

SpringerLink
Log in

Comparison of risk adjustment measures based on self-report, administrative data, and pharmacy records to predict clinical outcomes

  • Published:
Health Services and Outcomes Research Methodology Aims and scope Submit manuscript

Abstract

Comparing clinical outcomes in observational studies often requires adjustment for comorbid disease. The objective of this study was to compare the performance of risk adjustment measures derived from different data sources to predict the clinical outcomes of mortality and hospitalization. We compared the predictive ability of self-reported comorbidity measures to those derived from administrative diagnosis codes and pharmacy data to predict all-cause mortality and hospitalizations in a large sample of veterans receiving care in the Veterans Affairs outpatient clinic setting. In logistic regression models to predict mortality adjusting for age and gender, the Seattle Index of Comorbidity, SF-36, Charlson Index, Diagnosis Cost Groups, and RxRisk had similar discriminatory power ranging between 0.73 and 0.74. The Adjusted Clinical Groups and Chronic Illness and Disability Payment System were less accurate in prediction mortality. Although all measures performed less well in predicting hospitalizations, administrative measures performed better than self-reported measures. We conclude that self-reported morbidity measures had similar performance to administrative and pharmacy measures to predict mortality in a larger outpatient sample, but under-performed these measures in predicting hospitalization. While models using self-report measures can typically only be run on subsamples of patients for which models using administrative and pharmacy measures can be run, models combining self-reported morbidity and other measures performed better than models with a single measure.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Abbreviations

ACG:

Adjusted Clinical Groups

DCG-HCC:

Diagnostic Cost Groups—Heirarchical Cost Categories

CDPS:

Chronic Illness and Disability Payment System

SF-36:

Short Form 36

PCS:

Physical Component Summary

MCS:

Mental Component Summary

SIC:

Seattle Index of Comorbidity

VA:

Veterans Affairs

ACQUIP:

Ambulatory Care Quality Improvement Project

BIRLS:

Benficiary Identification and Record Locator Sub-system

AUC:

Area under the receiver operator curve

References

  • Ash, A.S., Ellis, R.P., Pope, G.C., Ayanian, J.Z., Bates, D.W., Burstin, H., et al.: Using diagnoses to describe populations and predict costs. Health Care Financ Rev 21(3), 7–28 (2000)

    PubMed  CAS  Google Scholar 

  • Ash, A.S., Posner, M.A., Speckman, J., Franco, S., Yacht, A.C., Bramwell, L.: Using claims data to examine mortality trends following hospitalization for heart attack in Medicare. Health Res. Serv. 38(5), 1253–1262 (2003)

    Article  PubMed  Google Scholar 

  • Charlson, M.E., Pompei, P., Ales, K.L., MacKenzie, C.R.: A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J. Chronic. Dis. 40(5), 373–383 (1987)

    Article  PubMed  CAS  Google Scholar 

  • Clark, D.O., Von Korff, M., Saunders, K., Baluch, W.M., Simon, G.E.: A chronic disease score with empirically derived weights. Med Care 33(8), 783–795 (1995)

    Article  PubMed  CAS  Google Scholar 

  • D’Hoore, W., Sicotte, C., Tilquin, C.: Risk adjustment in outcome assessment: the Charlson comorbidity index. Methods Inf Med 32(5), 382–387 (1993)

    PubMed  CAS  Google Scholar 

  • DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L.: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3), 837–845 (1988)

    Article  PubMed  CAS  Google Scholar 

  • Deyo, R.A., Cherkin, D.C., Ciol, M.A.: Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J. Clin. Epidemiol. 45(6), 613–619 (1992)

    Article  PubMed  CAS  Google Scholar 

  • Ellis, R.P., Pope, G.C., Iezzoni, L., Ayanian, J.Z., Bates, D.W., Burstin, H., et al.: Diagnosis-based risk adjustment for Medicare capitation payments. Health Care Financ. Rev. 17(3), 101–128 (1996)

    PubMed  CAS  Google Scholar 

  • Extermann, M.: Measuring comorbidity in older cancer patients. Eur. J. Cancer 36(4), 453–471 (2000)

    Article  PubMed  CAS  Google Scholar 

  • Fan, V.S., Au, D., Heagerty, P., Deyo, R.A., McDonell, M.B., Fihn, S.D.: Validation of case-mix measures derived from self-reports of diagnoses and health. J. Clin. Epidemiol. 55(4), 371–380 (2002)

    Article  PubMed  Google Scholar 

  • Fihn, S.D., McDonell, M.B., Diehr, P., Anderson, S.M., Bradley, K.A., Au, D.H., et al.: Effects of sustained audit/feedback on self-reported health status of primary care patients. Am. J. Med. 116(4), 241–248 (2004)

    Article  PubMed  Google Scholar 

  • Fishman, P.A., Goodman, M.J., Hornbrook, M.C., Meenan, R.T., Bachman, D.J., O’Keeffe, Rosetti, M.C.: Risk adjustment using automated ambulatory pharmacy data: the RxRisk model. Med. Care 41(1), 84–99 (2003)

    Article  PubMed  Google Scholar 

  • Fleming, C., Fisher, E.S., Chang, C.H., Bubolz, T.A., Malenka, D.J.: Studying outcomes and hospital utilization in the elderly. The advantages of a merged data base for Medicare and Veterans Affairs hospitals. Med. Care 30(5), 377–391 (1992)

    Article  PubMed  CAS  Google Scholar 

  • Fowles, J.B., Weiner, J.P., Knutson, D., Fowler, E., Tucker, A.M., Ireland, M.: Taking health status into account when setting capitation rates: a comparison of risk-adjustment methods. Jama 276(16), 1316–1321 (1996)

    Article  PubMed  CAS  Google Scholar 

  • Ghali, W.A., Hall, R.E., Rosen, A.K., Ash, A.S., Moskowitz, M.A.: Searching for an improved clinical comorbidity index for use with ICD-9-CM administrative data. J. Clin. Epidemiol. 49(3), 273–278 (1996)

    Article  PubMed  CAS  Google Scholar 

  • Greenfield, S., Aronow, H.U., Elashoff, R.M., Watanabe, D.: Flaws in mortality data. The hazards of ignoring comorbid disease. Jama 260(15), 2253–2255 (1988)

    Article  PubMed  CAS  Google Scholar 

  • Hornbrook, M.C., Goodman, M.J.: Assessing relative health plan risk with the RAND-36 health survey. Inquiry 32(1), 56–74 (1995)

    PubMed  CAS  Google Scholar 

  • Hornbrook, M.C., Goodman, M.J.: Chronic disease, functional health status, and demographics: a multi-dimensional approach to risk adjustment. Health Res. Serv.31(3), 283–307 (1996)

    CAS  Google Scholar 

  • Hosmer, D., Lemeshow, S.: Applied Logistic Regression. 2nd edn, John Wiley & Sons, New York NY (2000)

    Book  Google Scholar 

  • Hynes, D.M., Cowper, D., Kerr, M., Kubal, J., Murphy, P.A.: Database, informatics support for QUERI: current systems and future needs. Quality Enhancement Research Initiative. Med. Care 38(6 Suppl 1), I114–128 (2000)

    PubMed  CAS  Google Scholar 

  • Hynes, D.M., Joseph G., Pfeil C.: Veterans Health Information Systems and Technology Architecture (VISTA) as a Research Tool. Virec Insights 3(1) (2002)

  • Iezzoni, L.I.: Risk Adjustment for Measuring Healthcare Outcomes. 2nd edn, Health Administration Press, Chicago Il (1997a)

    Google Scholar 

  • Iezzoni, L.I.: The risks of risk adjustment [see comments]. Jama 278(19), 1600–1607 (1997b)

    Article  PubMed  CAS  Google Scholar 

  • Ingber, M.J.: The current state of risk adjustment technology for capitation. J. Ambul. Care Manage. 21(4), 1–28 (1998)

    PubMed  CAS  Google Scholar 

  • Katz, J.N., Chang, L.C., Sangha, O., Fossel, A.H., Bates, D.W.: Can comorbidity be measured by questionnaire rather than medical record review? Med. Care 34(1), 73–84 (1996)

    Article  PubMed  CAS  Google Scholar 

  • Kazis, L.E., Ren, X.S., Lee, A., Skinner, K., Rogers, W., Clark, J., et al.: Health status in VA patients: results from the Veterans Health Study. Am. J. Med. Qual. 14(1), 28–38 (1999)

    PubMed  CAS  Google Scholar 

  • Kronick, R., Dreyfus, T., Lee, L., Zhou, Z.: Diagnostic risk adjustment for Medicaid: the disability payment system. Health Care Financ. Rev. 17(3), 7–33 (1996)

    PubMed  CAS  Google Scholar 

  • Kronick, R., Gilmer, T., Dreyfus, T., Lee, L.: Improving health-based payment for Medicaid beneficiaries: CDPS. Health Care Financ. Rev. 21(3), 29–64 (2000)

    PubMed  CAS  Google Scholar 

  • Librero, J., Peiro, S., Ordinana, R.: Chronic comorbidity and outcomes of hospital care: length of stay, mortality, and readmission at 30 and 365 days. J. Clin. Epidemiol. 52(3), 171–179 (1999)

    Article  PubMed  CAS  Google Scholar 

  • Maciejewski, M.L., Maynard, C.: Diabetes-related utilization and costs for inpatient and outpatient services in the veterans administration. Diabetes Care 27(Suppl 2), B69–73 (2004)

    PubMed  Google Scholar 

  • Maciejewski, M.L., Liu, C.F., Derleth, A., McDonell, M., Anderson, S., Fihn, S.D.: The performance of administrative and self-reported measures for risk adjustment of Veterans Affairs expenditures. Health Res. Serv. 40(3), 887–904 (2005)

    Article  Google Scholar 

  • Parker, J.P., McCombs, J.S., Graddy, E.A.: Can pharmacy data improve prediction of hospital outcomes? Comparisons with a diagnosis-based comorbidity measure. Med. Care 41(3), 407–419 (2003)

    Article  PubMed  Google Scholar 

  • Petersen, L.A., Pietz, K., Woodard, L.D., Byrne, M.: Comparison of the predictive validity of diagnosis-based risk adjusters for clinical outcomes. Med. Care 43(1), 61–67 (2005)

    PubMed  Google Scholar 

  • Pope, G.C., Adamache, K.W., Walsh, E.G., Khandker, R.K.: Evaluating alternative risk adjusters for medicare. Health Care Financ. Rev. 20(2), 109–129 (1998)

    Google Scholar 

  • Putnam, K.G., Buist, D.S., Fishman, P., Andrade, S.E., Boles, M., Chase, G.A., et al.: Chronic disease score as a predictor of hospitalization. Epidemiology 13(3), 340–346 (2002)

    Article  PubMed  Google Scholar 

  • Rius, C., Perez, G., Martinez, J.M., Bares, M., Schiaffino, A., Gispert, R., et al.: An adaptation of Charlson comorbidity index predicted subsequent mortality in a health survey. J. Clin. Epidemiol. 57(4), 403–408 (2004)

    Article  PubMed  Google Scholar 

  • Romano, P.S., Roos, L.L., Jollis, J.G.: Adapting a clinical comorbidity index for use with ICD-9-CM administrative data: differing perspectives. J. Clin. Epidemiol. 46(10), 1075–1079 (1993) discussion 1081–1090

    Article  PubMed  CAS  Google Scholar 

  • Rosen, A.K., Loveland, S., Anderson, J.J., Rothendler, J.A., Hankin, C.S., Rakovski, C.C., et al.: Evaluating diagnosis-based case-mix measures: how well do they apply to the VA population? Med. Care 39(7), 692–704 (2001)

    Article  PubMed  CAS  Google Scholar 

  • Rumsfeld, J.S., MaWhinney, S., McCarthy, M. Jr., Shroyer, A.L., VillaNueva, C.B., O’Brien, M., et al.: Health-related quality of life as a predictor of mortality following coronary artery bypass graft surgery. Participants of the Department of Veterans Affairs Cooperative Study Group on Processes, Structures, and Outcomes of Care in Cardiac Surgery. Jama 281(14), 1298–1303 (1999)

    Article  PubMed  CAS  Google Scholar 

  • Sales, A.E., Liu, C.F., Sloan, K.L., Malkin, J., Fishman, P.A., Rosen, A.K., et al.: Predicting costs of care using a pharmacy-based measure risk adjustment in a veteran population. Med. Care 41(6), 753–760 (2003)

    Article  PubMed  Google Scholar 

  • Schneeweiss, S., Seeger, J.D., Maclure, M., Wang, P.S., Avorn, J., Glynn, R.J.: Performance of comorbidity scores to control for confounding in epidemiologic studies using claims data. Am. J. Epidemiol. 154(9), 854–864 (2001)

    Article  PubMed  CAS  Google Scholar 

  • Selim, A.J., Berlowitz, D.R., Fincke, G., Rosen, A.K., Ren, X.S., Christiansen, C.L., et al.: Risk-adjusted mortality rates as a potential outcome indicator for outpatient quality assessments. Med. Care 40(3), 237–245 (2002)

    Article  PubMed  Google Scholar 

  • Selim, A.J., Fincke, G., Ren, X.S., Lee, A., Rogers, W.H., Miller, D.R., et al.: Comorbidity assessments based on patient report: results from the Veterans Health Study. J. Ambul. Care Manage. 27(3), 281–295 (2004)

    PubMed  Google Scholar 

  • Sloan, K.L., Sales, A.E., Liu, C.F., Fishman, P., Nichol, P., Suzuki, N.T., et al.: Construction and characteristics of the RxRisk-V: a VA-adapted pharmacy-based case-mix instrument. Med. Care 41(6), 761–774 (2003)

    Article  PubMed  Google Scholar 

  • Von Korff, M., Wagner, E.H., Saunders, K.: A chronic disease score from automated pharmacy data. J. Clin. Epidemiol. 45(2), 197–203 (1992)

    Article  Google Scholar 

  • Ware, J.: Comparison of methods for the scoring and statistical analysis of SF-36 health profile and summary measures: Summary of results from the medical outcomes study. Med. Care 33, AS264–AS279 (1995)

    Article  PubMed  Google Scholar 

  • Weiner, J.P., Starfield, B.H., Steinwachs, D.M., Mumford, L.M.: Development and application of a population-oriented measure of ambulatory care case-mix. Med. Care 29(5), 452–472 (1991)

    Article  PubMed  CAS  Google Scholar 

  • Weiner, J.P., Parente, S.T., Garnick, D.W., Fowles, J., Lawthers, A.G., Palmer, R.H.: Variation in office-based quality. A claims-based profile of care provided to Medicare patients with diabetes. Jama 273(19), 1503–1508 (1995)

    Article  PubMed  CAS  Google Scholar 

Download references

Grants and Acknowledgements

The research reported here was supported by the Department of Veterans Affairs, Health Services Research and Development Service Grants RCD 02-170-2, SDR 96-002 and IIR 99-376.

Author information

Authors and Affiliations

  1. Veterans Affairs Puget Sound Health Care System, Health Services Research and Development (152), 1660 S. Columbian Way, Seattle, WA, 98108-1597, USA

    Vincent S. Fan, Matthew L. Maciejewski, Chuan-Fen Liu, Mary B. McDonell & Stephan D. Fihn

  2. Department of Medicine, University of Washington, Seattle, WA, USA

    Vincent S. Fan & Stephan D. Fihn

  3. Department of Health Services, University of Washington, Seattle, WA, USA

    Matthew L. Maciejewski, Chuan-Fen Liu & Stephan D. Fihn

Authors
  1. Vincent S. Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matthew L. Maciejewski
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chuan-Fen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mary B. McDonell
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Stephan D. Fihn
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vincent S. Fan.

Additional information

The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fan, V.S., Maciejewski, M.L., Liu, CF. et al. Comparison of risk adjustment measures based on self-report, administrative data, and pharmacy records to predict clinical outcomes. Health Serv Outcomes Res Method 6, 21–36 (2006). https://doi.org/10.1007/s10742-006-0004-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10742-006-0004-1

Keywords

Search

Navigation