ASO Author Reflections: Careful Development and Thoughtful Interpretation are Needed when Developing Online Prognostic Tools
Historically, cancer staging systems have been used to stratify patients into risk groups with fixed ranges of predicted survival outcome. However, there is often significant heterogeneity in outcomes within a stage, and there can also be substantial overlap across stages. In response, the use of nomograms and other prognostic models to obtain patient-specific outcome predictions based on a collection of variables has been increasing in popularity. Often, the resulting prediction models will be made available online so that both patients and doctors have easy access. In melanoma, three different prediction models are available online and, depending on a patient’s disease characteristics, the resulting predictions can vary in clinically meaningful ways.1, 2, 3 This study aimed to critically assess the development and deployment of online prognostic tools through a demonstration of the differences in input variables and modeling procedures used in these three online melanoma prediction tools.
The included variables and modeling strategies varied across the three online melanoma prediction tools.4 While variables such as ulceration, lesion site, and tumor thickness were included in some form in all three tools, other variables such as age, sex, and measures of nodal status and histologic features were not consistently incorporated. One tool generated final predictions from a series of stratified multivariable Cox regression models, whereas another used a single multivariable Cox regression model, and the third used a customized prediction model based on the probability of the spread of individual tumor cells. These differences led to discrepancies in both calibration, a measure of agreement between predictions and observed data, and discrimination, a measure of how accurately patients were classified as alive versus dead, when a single dataset was applied to all three tools.
The American Joint Committee on Cancer is currently working to host online probability models that would incorporate prognostic markers in addition to anatomic stage, and, as part of this effort, developed a checklist of 16 criteria that should be used to evaluate a prognostic model.5 The checklist highlights the importance for researchers to focus on the primary goal of a prognostic model, which is to provide both patients and doctors with accurate estimates of predicted survival at a specific timepoint to aid in decision making about treatment and follow-up. Careful development of prognostic models using this checklist, followed by model validation in an external dataset and thoughtful interpretation of the final model, will help to ensure that online prognostic tools will be a useful resource for both patients and doctors.
The author has no conflicts of interest to disclose.