Estimating improvement in prediction with matched case–control designs
- 270 Downloads
When an existing risk prediction model is not sufficiently predictive, additional variables are sought for inclusion in the model. This paper addresses study designs to evaluate the improvement in prediction performance that is gained by adding a new predictor to a risk prediction model. We consider studies that measure the new predictor in a case–control subset of the study cohort, a practice that is common in biomarker research. We ask if matching controls to cases in regards to baseline predictors improves efficiency. A variety of measures of prediction performance are studied. We find through simulation studies that matching improves the efficiency with which most measures are estimated, but can reduce efficiency for some. Efficiency gains are less when more controls per case are included in the study. A method that models the distribution of the new predictor in controls appears to improve estimation efficiency considerably.
KeywordsClassification Diagnosis Medical decision making Receiver operating characteristic curve
The support for this research was provided by RO1-GM054438 and PO1-CA-053996. The authors thank Mr. Jing Fan for his contribution to the simulation studies.
- Breslow NE, Day NE (1980) Statistical methods in cancer research, vol 1. International Agency for Research on Cancer, LyonGoogle Scholar
- Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (2001) Executive summary of the Third Report of The National Cholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults (Adult Treatment Panel III). J Am Med Assoc 285(19):2486–2497Google Scholar
- Gu W, Pepe M (2009a) Measures to summarize and compare the predictive capacity of markers. Int J Biostat 5. doi: 10.2202/1557-4679.1188
- Janssens ACJW, Deng Y, Borsboom GJJM, Eijkemans MJC, Habemma JDF, Steyerberg EW (2005) A new logistic regression approach for the evaluation of diagnostic test results. Ann Intern Med 25(2):168–177Google Scholar
- Krijnen P, van Jaarsveld BC, Steyerberg EW, Man in’t Veld AJ, Schalekamp MADH, Habbema JDF (1998) A clinical prediction rule for renal artery stenosis. Stat Med 129(9):705–711Google Scholar
- Pepe MS, Kerr KF, Longton G, Wang Z (2013) Testing for improvement in prediction model performance. Stat Med. doi: 10.1002/sim.5727
- Pepe MS, Fan J, Seymour CW, Li C, Huang Y, Feng Z (2012) Biases introduced by choosing controls to match risk factors of cases in biomarker research. Clin Chem 58(8):1242–1251Google Scholar
- Pepe MS, Janes H (2013) Methods for evaluating prediction performance of biomarkers and tests. In MLT Lee, M Gail, G Satten, T Cai, A Gandy, R Pfeiffer (Ed.), Risk assessment and evaluation of predictions. SpringerGoogle Scholar
- Vickers AJ, Cronin AM, Begg CM (2011) One statistical test is sufficient for assessing new predictive markers. BMC Med Res Methodol 11(13). doi: 10.1186/1471-2288-11-13