Evaluating Community-Based Translational Interventions Using Historical Controls: Propensity Score vs. Disease Risk Score Approach
Many community-based translations of evidence-based interventions are designed as one-arm studies due to ethical and other considerations. Evaluating the impacts of such programs is challenging. Here, we examine the effectiveness of the lifestyle intervention implemented by the Special Diabetes Program for Indians Diabetes Prevention (SDPI-DP) demonstration project, a translational lifestyle intervention among American Indian and Alaska Native communities. Data from the landmark Diabetes Prevention Program placebo group was used as a historical control. We compared the use of propensity score (PS) and disease risk score (DRS) matching to adjust for potential confounder imbalance between groups. The unadjusted hazard ratio (HR) for diabetes risk was 0.35 for SDPI-DP lifestyle intervention vs. control. However, when relevant diabetes risk factors were considered, the adjusted HR estimates were attenuated toward 1, ranging from 0.56 (95% CI 0.44–0.71) to 0.69 (95% CI 0.56–0.96). The differences in estimated HRs using the PS and DRS approaches were relatively small but DRS matching resulted in more participants being matched and smaller standard errors of effect estimates. Carefully employed, publicly available randomized clinical trial data can be used as a historical control to evaluate the intervention effectiveness of one-arm community translational initiatives. It is critical to use a proper statistical method to balance the distributions of potential confounders between comparison groups in this kind of evaluations.
KeywordsComparative effectiveness evaluation Disease risk score Prognostic score Propensity score Single-arm intervention Historical controls
The authors would like to express our gratitude to the Indian Health Service (IHS) as well as tribal and urban Indian health programs and participants involved in the SDPI-DP. The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the IHS. The DPP was conducted by the DPP Investigators and supported by the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK). This manuscript was not prepared in collaboration with the Investigators of the DPP study and does not necessarily reflect the opinions or views of the DPP study or the NIDDK. The authors would also like to thank Dr. James Hill for his valuable scientific suggestions and comments.
Grant programs participating in the Special Diabetes Program for Indians Diabetes Prevention Program are as follows: Confederated Tribes of the Chehalis Reservation, Cherokee Nation, Cheyenne River Sioux Tribe, the Chickasaw Nation, Coeur d’Alene Tribe, Colorado River Indian Tribes, Colville Confederated Tribes, Cow Creek Band of Umpqua Tribe, Klamath Tribes, and Coquille Tribe, Fond du Lac Reservation, Gila River Health Care, Haskell Health Center, Ho-Chunk Nation, Indian Health Board of Minneapolis, Indian Health Center of Santa Clara Valley, Native American Rehabilitation Association of the NW, Hunter Health, Kenaitze Indian Tribe IRA, Lawton IHS Service Unit, Menominee Indian Tribe of Wisconsin, Mississippi Band of Choctaw Indians, Norton Sound Health Corporation, Pine Ridge IHS Service Unit, Pueblo of San Felipe, Quinault Indian Nation, Rapid City IHS Diabetes Program, Red Lake Comprehensive Health Services, Rocky Boy Health Board, Seneca Nation of Indians, Sonoma County Indian Health Project, South East Alaska Regional Health Consortium, Southcentral Foundation, Trenton Indian Service Area, Tuba City Regional Health Care Corporation, United American Indian Involvement, Inc., United Indian Health Services, Inc., Warm Springs Health & Wellness Center, Winnebago Tribe of Nebraska, Zuni Pueblo.
Funding for the SDPI-DP project was provided by the Indian Health Service (HHSI242200400049C, S.M. Manson). Manuscript preparation was supported in part by American Diabetes Association (ADA #7-12-CT-36, L. Jiang), and National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) (1P30DK092923, S.M. Manson, and R21DK108187, L. Jiang).
Compliance with Ethical Standards
Conflict of Interest
The authors declare that they have no conflict of interest.
All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. The SDPI-DP protocol was approved by the institutional review board (IRB) of the University of Colorado and the National IHS IRB. The use of the DPP data from the NIDDK Central Repositories was approved by the University of California Irvine IRB.
All participants provided written informed consent and Health Insurance Portability and Accountability Act authorization.
- Campbell, D. T., & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research on teaching. Chicago: Rand McNally.Google Scholar
- Cefalu, W. T., Buse, J. B., Tuomilehto, J., Fleming, G. A., Ferrannini, E., Gerstein, H. C., et al. (2016). Update and next steps for real-world translation of interventions for type 2 diabetes prevention: Reflections from a diabetes care editors’ expert forum. Diabetes Care, 39, 1186–1201. https://doi.org/10.2337/dc16-0873.CrossRefPubMedPubMedCentralGoogle Scholar
- Chevreul, K., Brunn, M., Cadier, B., Nolte, E., & Durand-Zaleski, I. (2014). Evaluating structured care for diabetes: Can calibration on margins help to avoid overestimation of the benefits? An illustration from French diabetes provider networks using data from the ENTRED survey. Diabetes Care, 37, 1892–1899. https://doi.org/10.2337/dc13-2141.CrossRefPubMedGoogle Scholar
- Collaboration, N. C. D. R. F. (2015). Effects of diabetes definition on global surveillance of diabetes prevalence and diagnosis: A pooled analysis of 96 population-based studies with 331,288 participants. The Lancet Diabetes and Endocrinology, 3, 624–637. https://doi.org/10.1016/S2213-8587(15)00129-1.CrossRefGoogle Scholar
- D'Agostino, R. B., Sr., Vasan, R. S., Pencina, M. J., Wolf, P. A., Cobain, M., Massaro, J. M., & Kannel, W. B. (2008). General cardiovascular risk profile for use in primary care: The Framingham Heart Study. Circulation, 117, 743–753. https://doi.org/10.1161/CIRCULATIONAHA.107.699579.CrossRefPubMedGoogle Scholar
- Dunkley, A. J., Bodicoat, D. H., Greaves, C. J., Russell, C., Yates, T., Davies, M. J., & Khunti, K. (2014). Diabetes prevention in the real world: Effectiveness of pragmatic lifestyle interventions for the prevention of type 2 diabetes and of the impact of adherence to guideline recommendations: A systematic review and meta-analysis. Diabetes Care, 37, 922–933. https://doi.org/10.2337/dc13-2195.CrossRefPubMedGoogle Scholar
- Grodstein, F., Stampfer, M. J., Manson, J. E., Colditz, G. A., Willett, W. C., Rosner, B., . . . Hennekens, C. H. (1996). Postmenopausal estrogen and progestin use and the risk of cardiovascular disease. The New England Journal of Medicine, 335, 453–461. https://doi.org/10.1056/NEJM199608153350701.CrossRefPubMedGoogle Scholar
- Guo, S. Y., & Fraser, M. W. (2009). Propensity score analysis: Statistical methods and applications: SAGE Publications, Inc.Google Scholar
- Jiang, L., Manson, S. M., Beals, J., Henderson, W. G., Huang, H., Acton, K. J., et al. (2013). Translating the diabetes prevention program into American Indian and Alaska native communities: Results from the special diabetes program for Indians diabetes prevention demonstration project. Diabetes Care, 36, 2027–2034. https://doi.org/10.2337/dc12-1250.CrossRefPubMedPubMedCentralGoogle Scholar
- Klein, S., Allison, D. B., Heymsfield, S. B., Kelley, D. E., Leibel, R. L., Nonas, C., et al. (2007). Waist circumference and cardiometabolic risk: A consensus statement from shaping America’s health: Association for Weight Management and Obesity Prevention; NAASO, the Obesity Society; the American Society for Nutrition; and the American Diabetes Association. Diabetes Care, 30, 1647–1652. https://doi.org/10.2337/dc07-9921.CrossRefGoogle Scholar
- Lee, E. T., Howard, B. V., Wang, W., Welty, T. K., Galloway, J. M., Best, L. G., et al. (2006). Prediction of coronary heart disease in a population with high prevalence of diabetes and albuminuria: The strong heart study. Circulation, 113, 2897–2905. https://doi.org/10.1161/CIRCULATIONAHA.105.593178.CrossRefPubMedGoogle Scholar
- Mann, D. M., Bertoni, A. G., Shimbo, D., Carnethon, M. R., Chen, H., Jenny, N. S., & Muntner, P. (2010). Comparative validity of 3 diabetes mellitus risk prediction scoring models in a multiethnic US cohort: The multi-ethnic study of atherosclerosis. American Journal of Epidemiology, 171, 980–988. https://doi.org/10.1093/aje/kwq030.CrossRefPubMedPubMedCentralGoogle Scholar
- Meijnikman, A. S., De Block, C. E. M., Dirinck, E., Verrijken, A., Mertens, I., Corthouts, B., & Van Gaal, L. F. (2017). Not performing an OGTT results in significant underdiagnosis of (pre)diabetes in a high risk adult Caucasian population. International Journal of Obesity, 41, 1615–1620. https://doi.org/10.1038/ijo.2017.165.CrossRefPubMedGoogle Scholar
- Norris, S. L., Zhang, X., Avenell, A., Gregg, E., Bowman, B., Schmid, C. H., & Lau, J. (2005). Long-term effectiveness of weight-loss interventions in adults with pre-diabetes: A review. American Journal of Preventive Medicine, 28, 126–139. https://doi.org/10.1016/j.amepre.2004.08.006.CrossRefPubMedGoogle Scholar
- Prentice, R. L., Pettinger, M., & Anderson, G. L. (2005). Statistical issues arising in the Women’s Health Initiative. Biometrics, 61, 899–911; discussion 911-841. https://doi.org/10.1111/j.0006-341X.2005.454_1.x.
- Rosenbaum, P. R., & Rubin, D. B. (1985). Constructing a control-group using multivariate matched sampling methods that incorporate the propensity score. American Statistician, 39, 33–38.Google Scholar
- Sturmer, T., Schneeweiss, S., Brookhart, M. A., Rothman, K. J., Avorn, J., & Glynn, R. J. (2005). Analytic strategies to adjust confounding using exposure propensity scores and disease risk scores: Nonsteroidal antiinflammatory drugs and short-term mortality in the elderly. American Journal of Epidemiology, 161, 891–898. https://doi.org/10.1093/aje/kwi106.CrossRefPubMedPubMedCentralGoogle Scholar
- Wilson, P. W., Meigs, J. B., Sullivan, L., Fox, C. S., Nathan, D. M., & D'Agostino, R. B., Sr. (2007). Prediction of incident diabetes mellitus in middle-aged adults: The Framingham Offspring Study. Archives of Internal Medicine, 167, 1068–1074. https://doi.org/10.1001/archinte.167.10.1068.CrossRefPubMedGoogle Scholar
- Wyss, R., Ellis, A. R., Brookhart, M. A., Jonsson Funk, M., Girman, C. J., Simpson, R. J., Jr., & Sturmer, T. (2015). Matching on the disease risk score in comparative effectiveness research of new treatments. Pharmacoepidemiology and Drug Safety, 24, 951–961. https://doi.org/10.1002/pds.3810.CrossRefPubMedPubMedCentralGoogle Scholar
- Wyss, R., Hansen, B. B., Ellis, A. R., Gagne, J. J., Desai, R. J., Glynn, R. J., & Sturmer, T. (2017). The “dry-run” analysis: A method for evaluating risk scores for confounding control. American Journal of Epidemiology, 185, 842–852. https://doi.org/10.1093/aje/kwx032.CrossRefPubMedPubMedCentralGoogle Scholar