Abstract
Neural models, with their ability to provide novel representations, have shown promising results in prediction tasks in healthcare. However, patient demographics, medical technology, and quality of care change over time. This often leads to drop in the performance of neural models for prospective patients, especially in terms of their calibration. The deep kernel learning (DKL) framework may be robust to such changes as it combines neural models with Gaussian processes, which are aware of prediction uncertainty. Our hypothesis is that out-of-distribution test points will result in probabilities closer to the global mean and hence prevent overconfident predictions. This in turn, we hypothesise, will result in better calibration on prospective data.
This paper investigates DKL’s behaviour when facing a temporal shift, which was naturally introduced when an information system that feeds a cohort database was changed. We compare DKL’s performance to that of a neural baseline based on recurrent neural networks. We show that DKL indeed produced superior calibrated predictions. We also confirm that the DKL’s predictions were indeed less sharp. In addition, DKL’s discrimination ability was even improved: its AUC was 0.746 \( (\pm \)0.014 std), compared to 0.739 (±0.028 std) for the baseline. The paper demonstrated the importance of including uncertainty in neural computing, especially for their prospective use.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Code is available at: https://github.com/mriosb08/dkl-temporal-shift.git.
- 2.
References
Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)
Cheng, L.F., Darnell, G., Chivers, C., Draugelis, M.E., Li, K., Engelhardt, B.E.: Sparse multi-output Gaussian processes for medical time series prediction. BMC Med. Inform. Decis. Making 20(152) (2020)
Cox, D.R.: Two further applications of a model for binary regression. Biometrika 45, 562–565 (1958)
Debray, T.P.A., Vergouwe, Y., Koffijberg, H., Nieboer, D., Steyerberg, E.W., Moons, K.G.M.: A new framework to enhance the interpretation of external validation studies of clinical prediction models. J. Clin. Epidemiol 68(3), 279–89 (2015)
Dürichen, R., Pimentel, M., Clifton, L., Schweikard, A., Clifton, D.: Multi-task Gaussian processes for multivariate physiological time-series analysis. IEEE Trans. BioMed. Eng. 62, 314–322 (2014)
Futoma, J., Hariharan, S., Heller, K.: Learning to detect sepsis with a multitask Gaussian process RNN classifier. In: International Conference on Machine Learning. JMLR.org (2017)
Gardner, J.R., Pleiss, G., Bindel, D., Weinberger, K.Q., Wilson, A.G.: GPyTorch: blackbox matrix-matrix Gaussian process inference with GPU acceleration. CoRR abs/1809.11165 (2018)
Ghassemi, M., et al.: A multivariate timeseries modeling approach to severity of illness assessment and forecasting in ICU with sparse, heterogeneous clinical data. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI 2015, pp. 446–453. AAAI Press (2015)
Harutyunyan, H., Khachatrian, H., Kale, D.C., Ver Steeg, G., Galstyan, A.: Multitask learning and benchmarking with clinical time series data. Sci. Data 6(1), 1–18 (2019)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Johnson, A.E., et al.: MIMIC-III, a freely accessible critical care database. Sci. Data 3, 160035 (2016)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization (2014). Cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego (2015)
Mackay, D.J.C.: Bayesian methods for adaptive models. Ph.D. thesis, USA (1992). uMI Order No. GAX92-32200
Minne, L., Eslami, S., de Keizer, N., de Jonge, E., de Rooij, S.E., Abu-Hanna, A.: Effect of changes over time in the performance of a customized SAPS-II model on the quality of care assessment. Intensive Care Med. 38, 40–46 (2012)
Murphy, A., Winkler, R.: A general framework for forecast verification. Mon. Weather Rev. 115, 1330–1338 (1987)
Nestor, B., et al.: Feature robustness in non-stationary health records: caveats to deployable model performance in common clinical machine learning tasks. In: Doshi-Velez, F., et al. (eds.) Proceedings of the 4th Machine Learning for Healthcare Conference. Proceedings of Machine Learning Research, Ann Arbor, Michigan, vol. 106, pp. 381–405. PMLR (2019)
Quiñonero Candela, J., Ramussen, C.E., Williams, C.K.I.: Approximation methods for Gaussian process regression. Technical report MSR-TR-2007-124 (2007)
Rajkomar, A., et al.: Scalable and accurate deep learning for electronic health records. CoRR abs/1801.07860 (2018)
Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, Cambridge (2005)
Roberts, S., Osborne, M., Ebden, M., Reece, S., Gibson, N., Aigrain, S.: Gaussian processes for timeseries modelling. Philos. Trans. R. Soc. (2012)
Shickel, B., Loftus, T.J., Ozrazgat-Baslanti, T., Ebadi, A., Bihorac, A., Rashidi, P.: DeepSOFA: a real-time continuous acuity score framework using deep learning. CoRR abs/1802.10238 (2018)
Wilson, A.G., Hu, Z., Salakhutdinov, R.R., Xing, E.P.: Stochastic variational deep kernel learning. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 2586–2594. Curran Associates, Inc. (2016)
Wilson, A.G., Hu, Z., Salakhutdinov, R., Xing, E.P.: Deep kernel learning. In: Gretton, A., Robert, C.C. (eds.) Proceedings of the 19th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, Cadiz, Spain, vol. 51, pp. 370–378. PMLR (2016)
Wilson, A.G., Nickisch, H.: Kernel interpolation for scalable structured Gaussian processes (KISS-GP). In: International Conference on Machine Learning. JMLR.org (2015)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Rios, M., Abu-Hanna, A. (2021). Deep Kernel Learning for Mortality Prediction in the Face of Temporal Shift. In: Tucker, A., Henriques Abreu, P., Cardoso, J., Pereira Rodrigues, P., Riaño, D. (eds) Artificial Intelligence in Medicine. AIME 2021. Lecture Notes in Computer Science(), vol 12721. Springer, Cham. https://doi.org/10.1007/978-3-030-77211-6_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-77211-6_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77210-9
Online ISBN: 978-3-030-77211-6
eBook Packages: Computer ScienceComputer Science (R0)