Abstract
Machine Learning offers the potential to revolutionise healthcare with recent work showing that machine-learned algorithms can achieve or exceed expert human performance. The adoption of such systems in the medical domain should not happen, however, unless sufficient assurance can be demonstrated. In this paper we consider the implicit assurance argument for state-of-the-art systems that uses machine-learnt models for clinical diagnosis, e.g. retinal disease diagnosis. Based upon an assessment of this implicit argument we identify a number of additional assurance considerations that would need to be addressed in order to create a compelling assurance case. We present an assurance case pattern that we have developed to explicitly address these assurance considerations. This pattern may also have the potential to be applied to a wide class of critical domains where ML is used in the decision making process.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Assurance Case Working Group [ACWG]: Goal Structuring Notation Community Standard version 2 (2018). https://scsc.uk/r141B:1?t=1. Accessed 13 Nov 2018
Azure-Taxonomy: How to choose algorithms for Azure Machine Learning Studio (2019). https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-choice. Accessed Feb 2019
Bourne, R.R., et al.: Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. Lancet Glob. Health 5(9), e888–e897 (2017)
Burton, S., Gauerhof, L., Heinzemann, C.: Making the case for safety of machine learning in highly automated driving. In: Tonetta, S., Schoitsch, E., Bitsch, F. (eds.) SAFECOMP 2017. LNCS, vol. 10489, pp. 5–16. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66284-8_1
De Fauw, J., et al.: Clinically applicable deep learning for diagnosis and referral in retinal disease. Nat. Med. 24(9), 1342 (2018)
Došilović, F.K., Brčić, M., Hlupić, N.: Explainable artificial intelligence: a survey. In: 41st International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), pp. 210–215. IEEE (2018)
EUROCAE WG-12, RTCA SC-205: Software Considerations in Airborne Systems and Equipment Certification. EUROCAE and RTCA (2012)
Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT press, Cambridge (2016)
Gu, T., Dolan-Gavitt, B., Garg, S.: BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. arXiv:1708.06733 (2017)
Habli, I., White, S., Sujan, M., Harrison, S., Ugarte, M.: What is the safety case for health IT? a study of assurance practices in England. Saf. Sci. 110, 324–335 (2018)
Hannun, A.Y., et al.: Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat. Med. 25(1), 65 (2019)
Hawkins, R., Habli, I., Kelly, T., McDermid, J.: Assurance cases and prescriptive software safety certification: a comparative study. Saf. Sci. 59, 55–71 (2013)
Hawkins, R., Kelly, T., Knight, J., Graydon, P.: A new approach to creating clear safety arguments. In: Dale, C., Anderson, T. (eds) Advances in Systems Safety, pp. 3–23. Springer, London (2011). https://doi.org/10.1007/978-0-85729-133-2_1
Kelly, T.: Reviewing assurance arguments-a step-by-step approach. In: Workshop on Assurance Cases for Security-the Metrics Challenge, Dependable Systems and Networks (DSN) (2007)
Komorowski, M., Celi, L.A., Badawi, O., Gordon, A.C., Faisal, A.A.: The artificial intelligence clinician learns optimal treatment strategies for sepsis in intensive care. Nat. Med. 24(11), 1716 (2018)
Maddox, T.M., Rumsfeld, J.S., Payne, P.R.: Questions for artificial intelligence in health care. JAMA 321(1), 31–32 (2018)
NHS Digital: Digital Health Safety Conference (2019). https://digital.nhs.uk/news-and-events/events/2019-events/digital-health-safety-conference-2019. Accessed 30 May 2019
Pan, S.J., Yang, Q., et al.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)
Picardi, C., Habli, I.: Perspectives on assurance case development for retinal disease diagnosis using deep learning. In: Riano, D., Wilk, S., ten Teije, A. (eds) Artificial Intelligence in Medicine. AIME 2019. LNCS, vol. 11526. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-21642-9_46
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., Lopez, A.M.: The SYNTHIA dataset: a large collection of synthetic images for semantic segmentation of urban scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3234–3243 (2016)
scikit-Taxonomy: scikit - Choosing the right estimator (2019). https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html. Accessed Feb 2019
Shneier, M., et al.: Repository of sensor data for autonomous driving research. In: Unmanned Ground Vehicle Technology, vol. 5083, pp. 390–396. International Society for Optics and Photonics (2003)
Shortliffe, E.H., Sepúlveda, M.J.: Clinical decision support in the era of artificial intelligence. JAMA 320(21), 2199–2200 (2018)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Topol, E.: The Topol Review: Preparing the healthcare workforce to deliver the digital future (2019). https://topol.hee.nhs.uk/. Accessed 27 Feb 2019
University of York: Goal Structuring Notation, November 2014. https://impact.ref.ac.uk/casestudies/CaseStudy.aspx?Id=43445. Accessed 03 Jan 2019
Wagstaff, K.: Machine Learning that Matters. arXiv preprint arXiv:1206.4656 (2012)
World Health Organisation (WHO): Health workforce (2019). https://www.who.int/gho/health_workforce/en. Accessed 27 Feb 2019
Acknowledgements
This work is funded by the Assuring Autonomy International Programme https://www.york.ac.uk/assuring-autonomy.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Picardi, C., Hawkins, R., Paterson, C., Habli, I. (2019). A Pattern for Arguing the Assurance of Machine Learning in Medical Diagnosis Systems. In: Romanovsky, A., Troubitsyna, E., Bitsch, F. (eds) Computer Safety, Reliability, and Security. SAFECOMP 2019. Lecture Notes in Computer Science(), vol 11698. Springer, Cham. https://doi.org/10.1007/978-3-030-26601-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-26601-1_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-26600-4
Online ISBN: 978-3-030-26601-1
eBook Packages: Computer ScienceComputer Science (R0)