Annals of Surgical Oncology

, Volume 25, Supplement 3, pp 916–917 | Cite as

ASO Author Reflections: Careful Development and Thoughtful Interpretation are Needed when Developing Online Prognostic Tools

  • Emily C. Zabor
ASO Author Reflections


Historically, cancer staging systems have been used to stratify patients into risk groups with fixed ranges of predicted survival outcome. However, there is often significant heterogeneity in outcomes within a stage, and there can also be substantial overlap across stages. In response, the use of nomograms and other prognostic models to obtain patient-specific outcome predictions based on a collection of variables has been increasing in popularity. Often, the resulting prediction models will be made available online so that both patients and doctors have easy access. In melanoma, three different prediction models are available online and, depending on a patient’s disease characteristics, the resulting predictions can vary in clinically meaningful ways.1, 2, 3 This study aimed to critically assess the development and deployment of online prognostic tools through a demonstration of the differences in input variables and modeling procedures used in these three online melanoma prediction tools.


The included variables and modeling strategies varied across the three online melanoma prediction tools.4 While variables such as ulceration, lesion site, and tumor thickness were included in some form in all three tools, other variables such as age, sex, and measures of nodal status and histologic features were not consistently incorporated. One tool generated final predictions from a series of stratified multivariable Cox regression models, whereas another used a single multivariable Cox regression model, and the third used a customized prediction model based on the probability of the spread of individual tumor cells. These differences led to discrepancies in both calibration, a measure of agreement between predictions and observed data, and discrimination, a measure of how accurately patients were classified as alive versus dead, when a single dataset was applied to all three tools.


The American Joint Committee on Cancer is currently working to host online probability models that would incorporate prognostic markers in addition to anatomic stage, and, as part of this effort, developed a checklist of 16 criteria that should be used to evaluate a prognostic model.5 The checklist highlights the importance for researchers to focus on the primary goal of a prognostic model, which is to provide both patients and doctors with accurate estimates of predicted survival at a specific timepoint to aid in decision making about treatment and follow-up. Careful development of prognostic models using this checklist, followed by model validation in an external dataset and thoughtful interpretation of the final model, will help to ensure that online prognostic tools will be a useful resource for both patients and doctors.



The author has no conflicts of interest to disclose.


  1. 1.
    Soong SJ, Ding S, Coit D, et al. Predicting survival outcome of localized melanoma: an electronic prediction tool based on the AJCC Melanoma Database. Ann Surg Oncol. 2010;17(8):2006–14.CrossRefGoogle Scholar
  2. 2.
    Callender GG, Gershenwald JE, Egger ME, et al. A novel and accurate computer model of melanoma prognosis for patients staged by sentinel lymph node biopsy: comparison with the American Joint Committee on Cancer model. J Am Coll Surg. 2012;214(4):608–617; discussion 617–609.CrossRefGoogle Scholar
  3. 3.
    Chen LL, Nolan ME, Silverstein MJ, et al. The impact of primary tumor size, lymph node status, and other prognostic factors on the risk of cancer death. Cancer. 2009;115(21):5071–83.CrossRefGoogle Scholar
  4. 4.
    Zabor EC, Coit D, Gershenwald JE, McMasters KM, Michaelson JS, Stromberg AJ, et al. Variability in predictions from online tools: a demonstration using internet-based melanoma predictors. Ann Surg Oncol. 2018;25(8):2172–77.CrossRefGoogle Scholar
  5. 5.
    Kattan MW, Hess KR, Amin MB, et al. American Joint Committee on Cancer acceptance criteria for inclusion of risk models for individualized prognosis in the practice of precision medicine. CA Cancer J Clin. 2016;66(5):370–4.CrossRefPubMedCentralGoogle Scholar

Copyright information

© Society of Surgical Oncology 2018

Authors and Affiliations

  1. 1.Department of Epidemiology and BiostatisticsMemorial Sloan Kettering Cancer CenterNew YorkUSA

Personalised recommendations