End-to-End Deep Learning Vector Autoregressive Prognostic Models to Predict Disease Progression with Uneven Time Intervals

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12722)


We propose an end-to-end deep learning method combining implicit feature extraction and an autoregressive model to predict the future course of a disease or condition. By merging the feature extraction and autoregression into one deep learning model, we can simultaneously train both models together. Our novel approach begins by fine-tuning a pretrained convolutional neural network to extract features from previously obtained images of patients. A trainable autoregression mechanism then predicts the features of the future image and a fully connected layer gives a prognosis based on the predicted features. We utilize a novel time interval scaling, allowing the model to account for uneven time intervals and allowing us to choose the final time point that we wish to predict. Experiments on the Age-Related Eye Disease Study give a testing area under the receiver operating characteristic curve, sensitivity, and specificity of 0.966 (95% CI: 0.947, 0.984), 0.878 (0.810, 0.945), and 0.930 (0.914, 0.947), respectively. This shows that the model can predict progression with good performance.


Prognosis Autoregressive Deep learning 


  1. 1.
    Abadi, M., et al.: Tensorflow: Large-scale machine learning on heterogeneous systems (2015). Software available from tensorflow.orgGoogle Scholar
  2. 2.
    Age-Related Eye Disease Study Research Group: The Age-Related Eye Disease Study (AREDS): design implications. AREDS report no. 1. Control. Clin. Trials 20(6), 573–600 (1999).
  3. 3.
    Arcadu, F., Benmansour, F., Maunz, A., Willis, J., Haskova, Z., Prunotto, M.: Deep learning algorithm predicts diabetic retinopathy progression in individual patients. NPJ Digit. Med. 2, 92 (2019). Scholar
  4. 4.
    Arenja, N., et al.: Prognostic value of novel imaging parameters derived from standard cardiovascular magnetic resonance in high risk patients with systemic light chain amyloidosis. J. Cardiovasc. Magn. Reson. 21(1), 53 (2019). Scholar
  5. 5.
    Babenko, B., et al.: Predicting progression of age-related macular degeneration from fundus images using deep learning (2019). arXiv preprint arXiv:1904.05478
  6. 6.
    Banerjee, I., et al.: A deep-learning approach for prognosis of age-related macular degeneration disease using SD-OCT imaging biomarkers (2019). arXiv preprint arXiv:1902.10700
  7. 7.
    Bridge, J., Harding, S., Zheng, Y.: Development and validation of a novel prognostic model for predicting amd progression using longitudinal fundus images. BMJ Open Ophthalmol. 5(1) (2020).
  8. 8.
    Brier, G.W.: Verification of forecasts expressed in terms of probability. Monthly Weather Rev. 78(1), 1–3 (1950)CrossRefGoogle Scholar
  9. 9.
    Choi, H., Jin, K.H.: Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging. Behav. Brain Res. 344, 103–109 (2018). Scholar
  10. 10.
    Chollet, F., et al.: Keras (2015).
  11. 11.
    Cox, D.R.: Regression models and life-tables. J. Roy. Stat. Soc. Ser. B (Methodol.) 34(2), 187–220 (1972).
  12. 12.
    DeLong, E.R., DeLong, D.M., Clarke-Pearson, D.L.: Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3), 837–845 (1988)CrossRefGoogle Scholar
  13. 13.
    Du, Z., Hao, Y.: reportROC: An Easy Way to Report ROC Analysis (2019)., R package version 3.4
  14. 14.
    Grassmann, F., et al.: A deep learning algorithm for prediction of age-related eye disease study severity scale for age-related macular degeneration from color fundus photography. Ophthalmology 125(9), 1410–1420 (2018). Scholar
  15. 15.
    Gregor, K., Danihelka, I., Mnih, A., Blundell, C., Wierstra, D.: Deep autoregressive networks (2013). arXiv preprint arXiv:1310.8499
  16. 16.
    Harrell, F.E., Califf, R.M., Pryor, D.B., Lee, K.L., Rosati, R.A.: Evaluating the yield of medical tests. Jama 247(18), 2543–2546 (1982).
  17. 17.
    Hilario, A., et al.: A prognostic model based on preoperative MRI predicts overall survival in patients with diffuse gliomas. Am. J. Neuroradiol. 35(6), 1096 (2014). Scholar
  18. 18.
    Keane, P.A., Patel, P.J., Liakopoulos, S., Heussen, F.M., Sadda, S.R., Tufail, A.: Evaluation of age-related macular degeneration with optical coherence tomography. Surv. Ophthalmol 57(5), 389–414 (2012). Scholar
  19. 19.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). arXiv preprint arXiv:1412.6980
  20. 20.
    Kundu, S., Aulchenko, Y.S., Janssens, A.C.J.: PredictABEL: Assessment of Risk Prediction Models (2014)., R package version 1.2-2
  21. 21.
    Leening, M.J., Vedder, M.M., Witteman, J.C., Pencina, M.J., Steyerberg, E.W.: Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician’s guide. Ann. Internal Med. 160(2), 122–131 (2014). Scholar
  22. 22.
    Owen, C.G., Jarrar, Z., Wormald, R., Cook, D.G., Fletcher, A.E., Rudnicka, A.R.: The estimated prevalence and incidence of late stage age related macular degeneration in the uk. Brit. J. Ophthalmol. 96(5), 752 (2012). Scholar
  23. 23.
    Priestley, M.: Non-linear and Non-stationary Time Series Analysis. Academic Press, Cambridge (1988)Google Scholar
  24. 24.
    R Core Team: R: A language and environment for statistical computing. R Foundation for Statistical Computing (2019).
  25. 25.
    Robin, X., et al.: pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinf. 12(1), 77 (2011). Scholar
  26. 26.
    Russakovsky, O., et al.: Imagenet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015)MathSciNetCrossRefGoogle Scholar
  27. 27.
    de Sisternes, L., Simon, N., Tibshirani, R., Leng, T., Rubin, D.L.: Quantitative SD-OCT imaging biomarkers as indicators of age-related macular degeneration progression. Invest. Ophthalmol. Vision Sci. 55(11), 7093–103 (2014). Scholar
  28. 28.
    Sjoberg, D.D.: dca: Decision Curve Analysis (2020). R package version Scholar
  29. 29.
    Steyerberg, E.W., et al.: Prognosis research strategy (PROGRESS) 3: prognostic model research. PLoS Med. 10(2), e1001381–e1001381 (2013)Google Scholar
  30. 30.
    Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826Google Scholar
  31. 31.
    Turkowski, K.: Filters for common resampling tasks (1990)Google Scholar
  32. 32.
    Uria, B., Côté, M.A., Gregor, K., Murray, I., Larochelle, H.: Neural autoregressive distribution estimation. J. Mach. Learn. Res. 17(1), 7184–7220 (2016)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Van Rossum, G., Drake Jr, F.L.: Python tutorial. Centrum voor Wiskunde en Informatica Amsterdam (1995)Google Scholar
  34. 34.
    Vickers, A.J., Elkin, E.B.: Decision curve analysis: a novel method for evaluating prediction models. Med. Decis. Mak. 26(6), 565–574 (2006). Scholar
  35. 35.
    Williams, B.M., Burgess, P.I., Zheng, Y.: Chapter 13 - Drusen and macular degeneration, pp. 245–272. Academic Press, Cambridge (2019).
  36. 36.
    Yan, Q., et al.: Deep-learning-based prediction of late age-related macular degeneration progression. medRxiv, p. 19006171 (2019).
  37. 37.
    Youden, W.J.: Index for rating diagnostic tests. Cancer 3(1), 32–35 (1950)CrossRefGoogle Scholar
  38. 38.
    Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2021

Authors and Affiliations

  1. 1.Department of Eye and Vision Science, Institute of Life Course and Medical SciencesUniversity of LiverpoolLiverpoolUK

Personalised recommendations