Advertisement

Real-Time Prediction of Segmentation Quality

  • Robert RobinsonEmail author
  • Ozan Oktay
  • Wenjia Bai
  • Vanya V. Valindria
  • Mihir M. Sanghvi
  • Nay Aung
  • José M. Paiva
  • Filip Zemrak
  • Kenneth Fung
  • Elena Lukaschuk
  • Aaron M. Lee
  • Valentina Carapella
  • Young Jin Kim
  • Bernhard Kainz
  • Stefan K. Piechnik
  • Stefan Neubauer
  • Steffen E. Petersen
  • Chris Page
  • Daniel Rueckert
  • Ben Glocker
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11073)

Abstract

Recent advances in deep learning based image segmentation methods have enabled real-time performance with human-level accuracy. However, occasionally even the best method fails due to low image quality, artifacts or unexpected behaviour of black box algorithms. Being able to predict segmentation quality in the absence of ground truth is of paramount importance in clinical practice, but also in large-scale studies to avoid the inclusion of invalid data in subsequent analysis.

In this work, we propose two approaches of real-time automated quality control for cardiovascular MR segmentations using deep learning. First, we train a neural network on 12,880 samples to predict Dice Similarity Coefficients (DSC) on a per-case basis. We report a mean average error (MAE) of 0.03 on 1,610 test samples and 97% binary classification accuracy for separating low and high quality segmentations. Secondly, in the scenario where no manually annotated data is available, we train a network to predict DSC scores from estimated quality obtained via a reverse testing strategy. We report an \(\mathrm {MAE} = 0.14\) and 91% binary classification accuracy for this case. Predictions are obtained in real-time which, when combined with real-time segmentation methods, enables instant feedback on whether an acquired scan is analysable while the patient is still in the scanner. This further enables new applications of optimising image acquisition towards best possible analysis results.

Notes

Acknowledgements

RR is funded by KCL&Imperial EPSRC CDT in Medical Imaging (EP/L015226/1) and GlaxoSmithKline; VV by Indonesia Endowment for Education (LPDP) Indonesian Presidential PhD Scholarship; KF supported by The Medical College of Saint Bartholomew’s Hospital Trust. AL and SEP acknowledge support from NIHR Barts Biomedical Research Centre and EPSRC program grant (EP/P001009/ 1). SN and SKP are supported by the Oxford NIHR BRC and the Oxford British Heart Foundation Centre of Research Excellence. This project supported by the MRC (grant number MR/L016311/1). NA is supported by a Wellcome Trust Research Training Fellowship (203553/Z/Z). The authors SEP, SN and SKP acknowledge the British Heart Foundation (BHF) (PG/14/89/31194). BG received funding from the ERC under Horizon 2020 (grant agreement No. 757173, project MIRA, ERC-2017-STG).

References

  1. 1.
    Petersen, S.E., et al.: Reference ranges for cardiac structure and function using cardiovascular magnetic resonance (CMR) in Caucasians from the UK Biobank population cohort. J. Cardiovasc. Magn. Reson. 19(1), 18 (2017)CrossRefGoogle Scholar
  2. 2.
    Bosse, S., Maniry, D., Müller, K.R., Wiegand, T., Samek, W.: Deep neural networks for no-reference and full-reference image quality assessment. 1, 1–14 (2016)Google Scholar
  3. 3.
    Farzi, M., Pozo, J.M., McCloskey, E.V., Wilkinson, J.M., Frangi, A.F.: Automatic quality control for population imaging: a generic unsupervised approach. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 291–299. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-46723-8_34CrossRefGoogle Scholar
  4. 4.
    Zhong, E., Fan, W., Yang, Q., Verscheure, O., Ren, J.: Cross validation framework to choose amongst models and datasets for transfer learning. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010. LNCS (LNAI), vol. 6323, pp. 547–562. Springer, Heidelberg (2010).  https://doi.org/10.1007/978-3-642-15939-8_35CrossRefGoogle Scholar
  5. 5.
    Fan, W., Davidson, I.: Reverse testing. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD 2006, p. 147. ACM Press, New York (2006)Google Scholar
  6. 6.
    Kohlberger, T., Singh, V., Alvino, C., Bahlmann, C., Grady, L.: Evaluating segmentation error without ground truth. In: Ayache, N., Delingette, H., Golland, P., Mori, K. (eds.) MICCAI 2012. LNCS, vol. 7510, pp. 528–536. Springer, Heidelberg (2012).  https://doi.org/10.1007/978-3-642-33415-3_65CrossRefGoogle Scholar
  7. 7.
    Valindria, V.V., et al.: Reverse classification accuracy: predicting segmentation performance in the absence of ground truth. IEEE Trans. Med. Imaging 36, 1597–1606 (2017)CrossRefGoogle Scholar
  8. 8.
    Robinson, R., et al.: Automatic quality control of cardiac MRI segmentation in large-scale population imaging. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 720–727. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-66182-7_82CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Robert Robinson
    • 1
  • Ozan Oktay
    • 1
  • Wenjia Bai
    • 1
  • Vanya V. Valindria
    • 1
  • Mihir M. Sanghvi
    • 3
    • 4
  • Nay Aung
    • 3
    • 4
  • José M. Paiva
    • 3
  • Filip Zemrak
    • 3
    • 4
  • Kenneth Fung
    • 3
    • 4
  • Elena Lukaschuk
    • 5
  • Aaron M. Lee
    • 3
    • 4
  • Valentina Carapella
    • 5
  • Young Jin Kim
    • 5
    • 6
  • Bernhard Kainz
    • 1
  • Stefan K. Piechnik
    • 5
  • Stefan Neubauer
    • 5
  • Steffen E. Petersen
    • 3
    • 4
  • Chris Page
    • 2
  • Daniel Rueckert
    • 1
  • Ben Glocker
    • 1
  1. 1.BioMedIA Group, Department of ComputingImperial College LondonLondonUK
  2. 2.Research & DevelopmentGlaxoSmithKlineBrentfordUK
  3. 3.NIHR Barts Biomedical Research CentreQueen Mary University LondonLondonUK
  4. 4.Barts Heart CentreBarts Health NHS TrustLondonUK
  5. 5.Radcliffe Department of MedicineUniversity of OxfordOxfordUK
  6. 6.Severance HospitalYonsei University College of MedicineSeoulSouth Korea

Personalised recommendations