Abstract
The number of risk prediction models has been increasingly developed, for estimating about breast cancer in individual women. However, those model performances are questionable. We therefore have conducted a study with the aim to systematically review previous risk prediction models. The results from this review help to identify the most reliable model and indicate the strengths and weaknesses of each model for guiding future model development. We searched MEDLINE (PubMed) from 1949 and EMBASE (Ovid) from 1974 until October 2010. Observational studies which constructed models using regression methods were selected. Information about model development and performance were extracted. Twenty-five out of 453 studies were eligible. Of these, 18 developed prediction models and 7 validated existing prediction models. Up to 13 variables were included in the models and sample sizes for each study ranged from 550 to 2,404,636. Internal validation was performed in four models, while five models had external validation. Gail and Rosner and Colditz models were the significant models which were subsequently modified by other scholars. Calibration performance of most models was fair to good (expected/observe ratio: 0.87–1.12), but discriminatory accuracy was poor to fair both in internal validation (concordance statistics: 0.53–0.66) and in external validation (concordance statistics: 0.56–0.63). Most models yielded relatively poor discrimination in both internal and external validation. This poor discriminatory accuracy of existing models might be because of a lack of knowledge about risk factors, heterogeneous subtypes of breast cancer, and different distributions of risk factors across populations. In addition the concordance statistic itself is insensitive to measure the improvement of discrimination. Therefore, the new method such as net reclassification index should be considered to evaluate the improvement of the performance of a new develop model.
Similar content being viewed by others
References
Parkin DM, Bray F, Ferlay J, Pisani P (2005) Global cancer statistics 2002. CA Cancer J Clinicians 55(2):74–108
Ferlay J, Bray F, Parkin DM, Pisani P eds (2001) Globalcan 2000. IARC Press, Lyon
VH BF (2002) IARC handbooks of cancer prevention. IARC Press, Lyon
Ries LAG EM, Kosary CL, et al. (2004) SEER cancer statistics review, 1975–2001. National Cancer Institute. http://seer.cancer.gov/csr/1975_2001/. Accessed Dec 22 2004
Steyerberg EW (ed) (2009) Clinical prediction models a practical approach to development, validation, and updating. Springer Science + Business Media LLC, New York
Boyle P, Mezzetti M, La Vecchia C, Franceschi S, Decarli A, Robertson C (2004) Contribution of three components to individual cancer risk predicting breast cancer risk in Italy. Eur J Cancer Prev 13(3):183–191
Chlebowski RT, Anderson GL, Lane DS, Aragaki AK, Rohan T, Yasmeen S, Sarto G, Rosenberg CA, Hubbell FA (2007) Predicting risk of breast cancer in postmenopausal women by hormone receptor status. J Natl Cancer Inst 99(22):1695–1705
Colditz GA, Rosner B (2000) Cumulative risk of breast cancer to age 70 years according to risk factor status: data from the Nurses’ Health Study. Am J Epidemiol 152(10):950–964
Colditz GA, Rosner BA, Chen WY, Holmes MD, Hankinson SE (2004) Risk factors for breast cancer according to estrogen and progesterone receptor status. J Natl Cancer Inst 96(3):218–228
Decarli A, Calza S, Masala G, Specchia C, Palli D, Gail MH (2006) Gail model for prediction of absolute risk of invasive breast cancer: independent evaluation in the Florence-European prospective investigation into cancer and nutrition cohort. J Natl Cancer Inst 98(23):1686–1693
Gail MH, Brinton LA, Byar DP, Corle DK, Green SB, Schairer C, Mulvihill JJ (1989) Projecting individualized probabilities of developing breast cancer for white females who are being examined annually. J Natl Cancer Inst 81(24):1879–1886
Lee EO, Ahn SH, You C, Lee DS, Han W, Choe KJ, Noh DY (2004) Determining the main risk factors and high-risk groups of breast cancer using a predictive model for breast cancer risk assessment in South Korea. Cancer Nurs 27(5):400–406
Rosner B, Colditz GA (1996) Nurses’ health study: log-incidence mathematical model of breast cancer incidence. J Natl Cancer Inst 88(6):359–364
Barlow WE, White E, Ballard-Barbash R, Vacek PM, Titus-Ernstoff L, Carney PA, Tice JA, Buist DS, Geller BM, Rosenberg R, Yankaskas BC, Kerlikowske K (2006) Prospective breast cancer risk prediction model for women undergoing screening mammography. J Natl Cancer Inst 98(17):1204–1214. doi:10.1093/jnci/djj331
Chen J, Pee D, Ayyagari R, Graubard B, Schairer C, Byrne C, Benichou J, Gail MH (2006) Projecting absolute invasive breast cancer risk in white women with a model that includes mammographic density. J Natl Cancer Inst 98(17):1215–1226
Gail MH, Costantino JP, Pee D, Bondy M, Newman L, Selvan M, Anderson GL, Malone KE, Marchbanks PA, McCaskill-Stevens W, Norman SA, Simon MS, Spirtas R, Ursin G, Bernstein L (2007) Projecting individualized absolute invasive breast cancer risk in African American women. J Natl Cancer Inst 99(23):1782–1792
Novotny J, Pecen L, Petruzelka L, Svobodnik A, Dusek L, Danes J, Skovajsova M (2006) Breast cancer risk assessment in the Czech female population—an adjustment of the original Gail model. Breast Cancer Res Treat 95(1):29–35
Rosner B, Colditz GA, Iglehart JD, Hankinson SE (2008) Risk prediction models with incomplete data with application to prediction of estrogen receptor-positive breast cancer: prospective data from the Nurses’ Health Study. Breast Cancer Res BCR 10(4):R55
Tamimi RM, Rosner B, Colditz GA Evaluation of a breast cancer risk prediction model expanded to include category of prior benign breast disease lesion. Cancer. doi: 10.1002/cncr.25386
Tice JA, Cummings SR, Smith-Bindman R, Ichikawa L, Barlow WE, Kerlikowske K (2008) Using clinical factors and mammographic breast density to estimate breast cancer risk: development and validation of a new predictive model. Ann Intern Med 148(5):337–347
Tice JA, Cummings SR, Ziv E, Kerlikowske K (2005) Mammographic breast density and the Gail model for breast cancer risk prediction in a screening population. Breast Cancer Res Treat 94(2):115–122
Tice JA, Miike R, Adduci K, Petrakis NL, King E, Wrensch MR (2005) Nipple aspirate fluid cytology and the Gail model for breast cancer risk assessment in a screening population. Cancer Epidemiol Biomarkers Prev 14(2):324–328
Ueda K, Tsukuma H, Tanaka H, Ajiki W, Oshima A (2003) Estimation of individualized probabilities of developing breast cancer for Japanese women. Breast cancer (Tokyo, Japan) 10(1):54–62
Bondy ML, Lustbader ED, Halabi S, Ross E, Vogel VG (1994) Validation of a breast cancer risk assessment model in women with a positive family history. J Natl Cancer Inst 86(8):620–625
Costantino JP, Gail MH, Pee D, Anderson S, Redmond CK, Benichou J, Wieand HS (1999) Validation studies for models projecting the risk of invasive and total breast cancer incidence. J Natl Cancer Inst 91(18):1541–1548
McKian KP, Reynolds CA, Visscher DW, Nassar A, Radisky DC, Vierkant RA, Degnim AC, Boughey JC, Ghosh K, Anderson SS, Minot D, Caudill JL, Vachon CM, Frost MH, Pankratz VS, Hartmann LC (2009) Novel breast tissue feature strongly associated with risk of breast cancer. J Clin Oncol 27(35):5893–5898
Rockhill B, Byrne C, Rosner B, Louie MM, Colditz G (2003) Breast cancer risk prediction with a log-incidence model: evaluation of accuracy. J Clin Epidemiol 56(9):856–861
Rockhill B, Spiegelman D, Byrne C, Hunter DJ, Colditz GA (2001) Validation of the Gail et al. Model of breast cancer risk prediction and implications for chemoprevention. J Natl Cancer Inst 93(5):358–366
Schonfeld SJ, Pee D, Greenlee RT, Hartge P, Lacey JV, Jr., Park Y, Schatzkin A, Visvanathan K, Pfeiffer RM (2010) Effect of changing breast cancer incidence rates on the calibration of the Gail model. J Clin Oncol 28(14):2411–2417. doi: 10.1200/JCO.2009.25.2767
Ulusoy C, Kepenekci I, Kose K, Aydintug S, Cam R (2010) Applicability of the Gail model for breast cancer risk assessment in Turkish female population and evaluation of breastfeeding as a risk factor. Breast Cancer Res Treat 120(2):419–424
Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21(1):128–138. doi:10.1097/EDE.0b013e3181c30fb2
Vickers AJ, Cronin AM (2010) Traditional statistical methods for evaluating prediction models are uninformative as to clinical value: towards a decision analytic framework. Semin Oncol 37(1):31–38. doi:10.1053/j.seminoncol.2009.12.004
Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27(2):157–172 discussion 207–112
Altman DG, Royston P (2000) What do we mean by validating a prognostic model? Stat Med 19(4):453–473
Peduzzi P, Concato J, Feinstein AR, Holford TR (1995) Importance of events per independent variable in proportional hazards regression analysis. II. Accuracy and precision of regression estimates. J Clin Epidemiol 48(12):1503–1510
Courvoisier DS, Combescure C, Agoritsas T, Gayet-Ageron A, Perneger TV (2011) Performance of logistic regression modeling: beyond the number of events per variable, the role of data structure. J Clin Epidemiol 64(9):993–1000. doi:10.1016/j.jclinepi.2010.11.012
Pepe MS, Janes H, Longton G, Leisenring W, Newcomb P (2004) Limitations of the odds ratio in gauging the performance of a diagnostic, prognostic, or screening marker. Am J Epidemiol 159(9):882–890
Haynes RB, Sackett DL, Guyatt GH (eds) (2006) Clinical epidemiology: how to do clinical practice research, 3rd edn. Lippincott Williams and Wilkins, Philadelphia
Breast cancer and hormone replacement therapy: collaborative reanalysis of data from 51 epidemiological studies of 52,705 women with breast cancer and 108,411 women without breast cancer. Collaborative Group on Hormonal Factors in Breast Cancer (1997). Lancet 350(9084):1047–1059
Ewertz M, Duffy SW, Adami HO, Kvale G, Lund E, Meirik O, Mellemgaard A, Soini I, Tulinius H (1990) Age at first birth, parity and risk of breast cancer: a meta-analysis of 8 studies from the Nordic countries. Int J Cancer 46(4):597–603
Pharoah PD, Day NE, Duffy S, Easton DF, Ponder BA (1997) Family history and the risk of breast cancer: a systematic review and meta-analysis. Int J Cancer 71(5):800–809
North RA, McCowan LME, Dekker GA, Poston L, Chan EHY, Stewart AW, Black MA, Taylor RS, Walker JJ, Baker PN, Kenny LC (2011) Clinical risk prediction for pre-eclampsia in nulliparous women: development of model in international prospective cohort. BMJ (Clinical research ed) 342:d1875s. doi: 10.1136/bmj.d1875
Tyrer J, Duffy SW, Cuzick J (2004) A breast cancer prediction model incorporating familial and personal risk factors. Stat Med 23(7):1111–1130
Amir E, Evans DG, Shenton A, Lalloo F, Moran A, Boggis C, Wilson M, Howell A (2003) Evaluation of breast cancer risk assessment packages in the family history evaluation and screening programme. J Med Genet 40(11):807–814
Cook NR (2007) Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 115(7):928–935. doi:10.1161/CIRCULATIONAHA.106.672402
Cook NR, Buring JE, Ridker PM (2006) The effect of including C-reactive protein in cardiovascular risk prediction models for women. Ann Intern Med 145(1):21–29
Janes H, Pepe MS, Gu W (2008) Assessing the value of risk predictions by using risk stratification tables. Ann Intern Med 149(10):751–760
McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM (2005) Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst 97(16):1180–1184. doi:10.1093/jnci/dji237
Acknowledgment
This study was supported by the Health Intervention and Technology Assessment Program, the Thai Health Promotion Foundation, the Health Systems Research Institute, the Bureau of Policy and Strategy of the Ministry of Public Health, and Thai Health-Global Link Initiative Project.
Conflict of interest
TA: received honorarium from the Health Intervention and Technology Assessment Program, Ministry of Public Health, Thailand; received travel grants for conference from the Health Intervention and Technology Assessment Program and Faculty of Graduate Studies, Mahidol University, Thailand. YT, CW, VK, and AT have no conflicts of interest to declare.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Anothaisintawee, T., Teerawattananon, Y., Wiratkapun, C. et al. Risk prediction models of breast cancer: a systematic review of model performances. Breast Cancer Res Treat 133, 1–10 (2012). https://doi.org/10.1007/s10549-011-1853-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10549-011-1853-z