Abstract
Objective
Patient-reported outcome measures following elective lumbar fusion surgery demonstrate major heterogeneity. Individualized prediction tools can provide valuable insights for shared decision-making. We externally validated the spine surgical care and outcomes assessment programme/comparative effectiveness translational network (SCOAP-CERTAIN) model for prediction of 12-month minimum clinically important difference in Oswestry Disability Index (ODI) and in numeric rating scales for back (NRS-BP) and leg pain (NRS-LP) after elective lumbar fusion.
Methods
Data from a prospective registry were obtained. We calculated the area under the curve (AUC), calibration slope and intercept, and Hosmer–Lemeshow values to estimate discrimination and calibration of the models.
Results
We included 100 patients, with average age of 50.4 ± 11.4 years. For 12-month ODI, AUC was 0.71 while the calibration intercept and slope were 1.08 and 0.95, respectively. For NRS-BP, AUC was 0.72, with a calibration intercept of 1.02, and slope of 0.74. For NRS-LP, AUC was 0.83, with a calibration intercept of 1.08, and slope of 0.95. Sensitivity ranged from 0.64 to 1.00, while specificity ranged from 0.38 to 0.65. A lack of fit was found for all three models based on Hosmer–Lemeshow testing.
Conclusions
The SCOAP-CERTAIN tool can accurately predict which patients will achieve favourable outcomes. However, the predicted probabilities—which are the most valuable in clinical practice—reported by the tool do not correspond well to the true probability of a favourable outcome. We suggest that any prediction tool should first be externally validated before it is applied in routine clinical practice.
Graphic abstract
These slides can be retrieved under Electronic Supplementary Material.
Similar content being viewed by others
References
Steinmetz MP, Mroz T (2018) Value of adding predictive clinical decision tools to spine surgery. JAMA Surg. https://doi.org/10.1001/jamasurg.2018.0078
Khor S, Lavallee D, Cizik AM et al (2018) Development and validation of a prediction model for pain and functional outcomes after lumbar spine surgery. JAMA Surg. https://doi.org/10.1001/jamasurg.2018.0072
Siccoli A, de Wispelaere MP, Schröder ML, Staartjes VE (2019) Machine learning–based preoperative predictive analytics for lumbar spinal stenosis. Neurosurg Focus 46:E5. https://doi.org/10.3171/2019.2.FOCUS18723
Staartjes VE, de Wispelaere MP, Vandertop WP, Schröder ML (2018) Deep learning-based preoperative predictive analytics for patient-reported outcomes following lumbar diskectomy: feasibility of center-specific modeling. Spine J Off J North Am Spine Soc. https://doi.org/10.1016/j.spinee.2018.11.009
Janssen DMC, van Kuijk SMJ, d’Aumerie B, Willems P (2019) A prediction model of surgical site infection after instrumented thoracolumbar spine surgery in adults. Eur Spine J. https://doi.org/10.1007/s00586-018-05877-z
Janssen DMC, van Kuijk SMJ, d’Aumerie BB, Willems PC (2018) External validation of a prediction model for surgical site infection after thoracolumbar spine surgery in a Western European cohort. J Orthop Surg. https://doi.org/10.1186/s13018-018-0821-2
Senders JT, Staples PC, Karhade AV et al (2018) Machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg 109:476–486.e1. https://doi.org/10.1016/j.wneu.2017.09.149
Brusko GD, Kolcun JPG, Wang MY (2018) Machine-learning models: the future of predictive analytics in neurosurgery. Neurosurgery 83:E3–E4. https://doi.org/10.1093/neuros/nyy166
Cabitza F, Rasoini R, Gensini GF (2017) Unintended consequences of machine learning in medicine. JAMA 318:517–518. https://doi.org/10.1001/jama.2017.7797
Collins GS, Ogundimu EO, Le Manach Y (2015) Assessing calibration in an external validation study. Spine J 15:2446–2447. https://doi.org/10.1016/j.spinee.2015.06.043
Debray TPA, Vergouwe Y, Koffijberg H et al (2015) A new framework to enhance the interpretation of external validation studies of clinical prediction models. J Clin Epidemiol 68:279–289. https://doi.org/10.1016/j.jclinepi.2014.06.018
Tetreault LA, Côté P, Kopjar B et al (2015) A clinical prediction model to assess surgical outcome in patients with cervical spondylotic myelopathy: internal and external validations using the prospective multicenter AOSpine North American and international datasets of 743 patients. Spine J 15:388–397. https://doi.org/10.1016/j.spinee.2014.12.145
Riley RD, Ensor J, Snell KIE et al (2016) External validation of clinical prediction models using big datasets from e-health records or IPD meta-analysis: opportunities and challenges. BMJ 353:i3140. https://doi.org/10.1136/bmj.i3140
Collins GS, de Groot JA, Dutton S et al (2014) External validation of multivariable prediction models: a systematic review of methodological conduct and reporting. BMC Med Res Methodol 14:40. https://doi.org/10.1186/1471-2288-14-40
Collins GS, Ogundimu EO, Altman DG (2016) Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med 35:214–226. https://doi.org/10.1002/sim.6787
Staartjes VE, de Wispelaere MP, Schröder ML (2019) Improving recovery after elective degenerative spine surgery: 5-year experience with an enhanced recovery after surgery (ERAS) protocol. Neurosurg Focus 46:E7. https://doi.org/10.3171/2019.1.FOCUS18646
Schröder ML, Staartjes VE (2017) Revisions for screw malposition and clinical outcomes after robot-guided lumbar fusion for spondylolisthesis. Neurosurg Focus 42:E12. https://doi.org/10.3171/2017.3.FOCUS16534
Staartjes VE, Schröder ML (2018) Effectiveness of a decision-making protocol for the surgical treatment of lumbar stenosis with grade 1 degenerative spondylolisthesis. World Neurosurg 110:e355–e361. https://doi.org/10.1016/j.wneu.2017.11.001
Collins GS, Reitsma JB, Altman DG, Moons KGM (2015) Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 350:g7594
Staartjes VE, Siccoli A, de Wispelaere MP, Schröder ML (2018) Patient-reported outcomes unbiased by length of follow-up after lumbar degenerative spine surgery: do we need 2 years of follow-up? Spine J Off J North Am Spine Soc. https://doi.org/10.1016/j.spinee.2018.10.004
Van Hooff ML, Spruit M, Fairbank JCT et al (2015) The Oswestry Disability Index (version 2.1a): validation of a Dutch language version. Spine 40:E83–E90. https://doi.org/10.1097/BRS.0000000000000683
Steyerberg EW, Vickers AJ, Cook NR et al (2010) Assessing the performance of prediction models: a framework for some traditional and novel measures. Epidemiol Camb Mass 21:128–138. https://doi.org/10.1097/EDE.0b013e3181c30fb2
Hosmer DW, Lemeshow S, Sturdivant RX (2013) Assessing the fit of the model. In: Hosmer DW Jr, Lemeshow S, Sturdivant RX (eds) Applied logistic regression. Wiley, Hoboken, pp 153–225
Brier GW (1950) Verification of forecasts expressed in terms of probability. Mon Weather Rev 78:1–3. https://doi.org/10.1175/1520-0493(1950)078%3c0001:VOFEIT%3e2.0.CO;2
Van Hoorde K, Van Huffel S, Timmerman D et al (2015) A spline-based tool to assess and visualize the calibration of multiclass risk predictions. J Biomed Inform 54:283–293. https://doi.org/10.1016/j.jbi.2014.12.016
Core Team R (2018) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna
Falavigna A, Dozza DC, Teles AR et al (2017) Current status of worldwide use of patient-reported outcome measures (PROMs) in spine care. World Neurosurg 108:328–335. https://doi.org/10.1016/j.wneu.2017.09.002
Glassman SD, Schwab F, Bridwell KH et al (2009) Do 1-year outcomes predict 2-year outcomes for adult deformity surgery? Spine J Off J North Am Spine Soc 9:317–322. https://doi.org/10.1016/j.spinee.2008.06.450
van Niftrik CHB, van der Wouden F, Staartjes VE et al (2019) Machine learning algorithm identifies patients at high risk for early complications after intracranial tumor surgery: registry-based cohort study. Neurosurgery. https://doi.org/10.1093/neuros/nyz145
Durand WM, DePasse JM, Daniels AH (2018) Predictive modeling for blood transfusion after adult spinal deformity surgery: a tree-based machine learning approach. Spine 43:1058. https://doi.org/10.1097/BRS.0000000000002515
Ehlers AP, Roy SB, Khor S et al (2017) Improved risk prediction following surgery using machine learning algorithms. eGEMs. https://doi.org/10.13063/2327-9214.1278
Kim JS, Merrill RK, Arvind V et al (2018) Examining the ability of artificial neural networks machine learning models to accurately predict complications following posterior lumbar spine fusion. Spine. https://doi.org/10.1097/BRS.0000000000002442
van Rein EAJ, van der Sluijs R, Voskens FJ et al (2019) Development and validation of a prediction model for prehospital triage of trauma patients. JAMA Surg. https://doi.org/10.1001/jamasurg.2018.4752
Janssen KJM, Moons KGM, Kalkman CJ et al (2008) Updating methods improved the performance of a clinical prediction model in new patients. J Clin Epidemiol 61:76–86. https://doi.org/10.1016/j.jclinepi.2007.04.018
Niculescu-Mizil A, Caruana R (2005) Predicting Good Probabilities with Supervised Learning. In: Proceedings of the 22nd international conference on machine learning. ACM, New York, pp 625–632
Staartjes VE, Schröder ML (2018) Letter to the editor. Class imbalance in machine learning for neurosurgical outcome prediction: are our models valid? J Neurosurg Spine. https://doi.org/10.3171/2018.5.SPINE18543
Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP (2002) SMOTE: synthetic minority over-sampling technique. J Artif Intell Res 16:321–357. https://doi.org/10.1613/jair.953
Goldstein CL, Phillips FM, Rampersaud YR (2016) Comparative effectiveness and economic evaluations of open versus minimally invasive posterior or transforaminal lumbar interbody fusion: a systematic review. Spine 41(Suppl 8):S74–S89. https://doi.org/10.1097/BRS.0000000000001462
Goldstein CL, Macwan K, Sundararajan K, Rampersaud YR (2016) Perioperative outcomes and adverse events of minimally invasive versus open posterior lumbar fusion: meta-analysis and systematic review. J Neurosurg Spine 24:416–427. https://doi.org/10.3171/2015.2.SPINE14973
Schröder ML, de Wispelaere MP, Staartjes VE (2018) Are patient-reported outcome measures biased by method of follow-up? Evaluating paper-based and digital follow-up after lumbar fusion surgery. Spine J Off J North Am Spine Soc 2:2. https://doi.org/10.1016/j.spinee.2018.05.002
Funding
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
586_2019_6189_MOESM1_ESM.pptx
Supplement 1 Statistical Code. The code was executed in R Version 3.5.4 (The R Foundation for Statistical Computing, Vienna, Austria) on a machine running Windows 10 (Microsoft Corp., Redmond, WA, USA). (PPTX 1304 kb)
Rights and permissions
About this article
Cite this article
Quddusi, A., Eversdijk, H.A.J., Klukowska, A.M. et al. External validation of a prediction model for pain and functional outcome after elective lumbar spinal fusion. Eur Spine J 29, 374–383 (2020). https://doi.org/10.1007/s00586-019-06189-6
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00586-019-06189-6