Medical & Biological Engineering & Computing

, Volume 56, Issue 9, pp 1699–1713 | Cite as

Patient-specific and global convolutional neural networks for robust automatic liver tumor delineation in follow-up CT studies

  • Refael VivantiEmail author
  • Leo Joskowicz
  • Naama Lev-Cohain
  • Ariel Ephrat
  • Jacob Sosna
Original Article


Radiological longitudinal follow-up of tumors in CT scans is essential for disease assessment and liver tumor therapy. Currently, most tumor size measurements follow the RECIST guidelines, which can be off by as much as 50%. True volumetric measurements are more accurate but require manual delineation, which is time-consuming and user-dependent. We present a convolutional neural networks (CNN) based method for robust automatic liver tumor delineation in longitudinal CT studies that uses both global and patient specific CNNs trained on a small database of delineated images. The inputs are the baseline scan and the tumor delineation, a follow-up scan, and a liver tumor global CNN voxel classifier built from radiologist-validated liver tumor delineations. The outputs are the tumor delineations in the follow-up CT scan. The baseline scan tumor delineation serves as a high-quality prior for the tumor characterization in the follow-up scans. It is used to evaluate the global CNN performance on the new case and to reliably predict failures of the global CNN on the follow-up scan. High-scoring cases are segmented with a global CNN; low-scoring cases, which are predicted to be failures of the global CNN, are segmented with a patient-specific CNN built from the baseline scan. Our experimental results on 222 tumors from 31 patients yield an average overlap error of 17% (std = 11.2) and surface distance of 2.1 mm (std = 1.8), far better than stand-alone segmentation. Importantly, the robustness of our method improved from 67% for stand-alone global CNN segmentation to 100%. Unlike other medical imaging deep learning approaches, which require large annotated training datasets, our method exploits the follow-up framework to yield accurate tumor tracking and failure detection and correction with a small training dataset.

Graphical abstract

Flow diagram of the proposed method. In the offline mode (orange), a global CNN is trained as a voxel classifier to segment liver tumor as in [31]. The online mode (blue) is used for each new case. The input is baseline scan with delineation and the follow-up CT scan to be segmented. The main novelty is the ability to predict failures by trying the system on the baseline scan and the ability to correct them using the patient-specific CNN


Liver tumor segmentation Follow-up CT scans Longitudinal studies Deep learning Convolutional neural networks 



This work was partially supported by Grant 53681 from the Israel Ministry of Science, Technology and Space entitled: METASEG: a new medical image segmentation paradigm for clinical decision support and big data radiology.

Compliance with ethical standards

Conflict of interest

The authors declare that they have no conflict of interest.

Protection of human and animal rights statement

No animals or humans were involved in this research. All scans were anonymized before delivery to the researchers.

Supplementary material

11517_2018_1803_MOESM1_ESM.docx (15 kb)
ESM 1 (DOCX 14.9 kb)


  1. 1.
    Bilello M, Gokturk SB, Desser T, Napel S, Jeffrey RB, Beaulieu CF (2004) Automatic detection and classification of hypo-dense hepatic lesion on contrast-enhanced venous-phase CT. Med Phys 31(9):2584–2593CrossRefPubMedGoogle Scholar
  2. 2.
    Bourquain H, Schenk A, Link F, Preim B, Peitgen OH (2002) Hepavision2a software assistant for preoperative planning in living related liver transplantation and oncologic liver surgery. Proc 16th Conf Comp Assist Radiol Surg, 341–6Google Scholar
  3. 3.
    Chen EL (1998) An automatic diagnostic system for CT liver image classification. IEEE Trans Biomed Eng 45(6):783–794CrossRefPubMedGoogle Scholar
  4. 4.
    Coghlin C, Murray GI (2010) Current and emerging concepts in tumor metastasis. J Pathol 222(1):1–15CrossRefPubMedGoogle Scholar
  5. 5.
    Cohen AB, Diamant I, Klang E, Amitai M, Greenspan H (2014) Automatic detection and segmentation of liver metastatic lesions on serial CT examinations. Proc SPIE Med Imag Conf, 903519-27Google Scholar
  6. 6.
    Deng X, Du G (2008) Proc. MICCAI workshop on 3D segmentation in the clinic: a grand challenge II—liver tumor segmentationGoogle Scholar
  7. 7.
    Eisenhauer E, Therasse P et al (2009) New response evaluation criteria in solid tumours: revised RECIST guideline (version 1.1). Eu J Cancer 45(2):228–247CrossRefGoogle Scholar
  8. 8.
    Freiman M, Eliassaf O, Taieb Y, Joskowicz L, Azraq Y, Sosna J (2008) An iterative Bayesian approach for nearly automatic liver segmentation: algorithm and validation. Int J Comp Aided Radiol Surg 3:439–446CrossRefGoogle Scholar
  9. 9.
    Freiman M, Cooper O, Lischinski D, Joskowicz L (2010) Liver tumors segmentation from CTA images using voxels classification and affinity constraint propagation. Int J Comput Assist Radiol Surg 6(2):247–255CrossRefPubMedGoogle Scholar
  10. 10.
    Greenspan H, van Ginneken B, Summers RM (2016) Guest editorial deep learning in medical imaging: overview and future promise of an exciting new technique. IEEE Trans Med Imaging 35(5):1153–1159CrossRefGoogle Scholar
  11. 11.
    Hassouna MS, Farag AA (2007) Multistencils fast marching methods: a highly accurate solution to the eikonal equation on cartesian domains. IEEE Trans Pattern Analy Mach Intell 29(9):1563–1574CrossRefGoogle Scholar
  12. 12.
    Hong JS, Kaneto T, Sekiguchi R, Park KH (2001) Automatic liver tumor detection from CT. IEEE Trans Inf Syst 84(6):741–748Google Scholar
  13. 13.
    Jia Y (2013) Caffe: An open source convolutional architecture for fast feature embedding.
  14. 14.
    Klein S, Staring M, Murphy K, Viergever MA, Pluim JP (2010) Elastix: a toolbox for intensity-based medical image registration. IEEE Trans Med Imaging 29(1):196–205CrossRefPubMedGoogle Scholar
  15. 15.
    Lewis RL (2007) Liver and biliary tract tumors. Cecil medicine, 23rd edn. Saunders Elsevier, Philadelphia (Ch. 206)Google Scholar
  16. 16.
    Li Y, Hara S, Shimura K (2006) A machine learning approach for locating boundaries of liver tumors in ct images. Proc 18th Int Conf Pattern Recogn, 400–3Google Scholar
  17. 17.
    Li W, Jia F, Hu Q (2015) Automatic segmentation of liver tumor in CT images with deep convolutional neural networks. J. of Comput Commun 3(11):146CrossRefGoogle Scholar
  18. 18.
    Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. Proc IEEE Conf Comput Vis Pattern Recog :3431–40Google Scholar
  19. 19.
    Mala K, Sadasivam V, Avlagappan S (2007) Neural network based texture analysis of liver tumors from computed tomography images. Int J Biomed Sci 2:33–40Google Scholar
  20. 20.
    Masuda Y, Tateyama T, Xiong W, Zhou J, Wakamiya M, Kanasaki S, Furukawa A, Chen YW (2011) Liver tumor detection in CT images by adaptive contrast enhancement and the EM/MPM algorithm. 18th IEEE Int Conf Image Process :1421–1424Google Scholar
  21. 21.
    Militzer A, Tietjen C, Hornegger J (2013) Learning a prior model for automatic liver lesion segmentation in follow-up CT images. Dept Informatik technical reports, CS-2013-03, ISSN 2191-5008Google Scholar
  22. 22.
    Miller AB, Hoogstraten B, Staquet M, Winkler A (2011) Reporting results of cancer treatment. Cancer 47(1):207–214CrossRefGoogle Scholar
  23. 23.
    Moltz J, Bornemann L, Dicken V, Peitgen H (2008) Segmentation of liver metastases in CT scans by adaptive thresholding and morphological processing. Proc. MICCAI workshop on 3D segmentation in the clinic: a grand challenge II—liver tumor segmentationGoogle Scholar
  24. 24.
    Moltz JH, Schwier M, Peitgen HO (2009) A general framework for automatic detection of matching lesions in follow-up CT. Proc IEEE Int Symp Biomed Imag, :843–6Google Scholar
  25. 25.
    Pescia D, Paragios N, Chemouny S (2008) Automatic detection of liver tumors. Prof. 5th IEEE Int Symp Biomed Imaging :672–67Google Scholar
  26. 26.
    Schmidt G, Binnig G, Kietzmann M, Kim J (2008) Cognition network technology for a fully automated 3D segmentation of liver tumors. Proc. MICCAI workshop on 3D segmentation in the clinic: a grand challenge II—liver tumor segmentationGoogle Scholar
  27. 27.
    Shimizu A, Narihira T, Furukawa D, Kobatake H, Nawano S, Shinozaki K (2008) Ensemble segmentation using AdaBoost with application to liver lesion extraction from a CT volume. Proc. MICCAI workshop on 3D segmentation in the clinic: a grand challenge II—liver tumor segmentationGoogle Scholar
  28. 28.
    Smeets D, Stijnen B, Loeckx D, Dobbelaer B, Suetens P (2008) Segmentation of liver metastases using a level set method with spiral-scanning technique and supervised fuzzy pixel classification. Proc. MICCAI workshop on 3D segmentation in the clinic: a grand challenge II—liver tumor segmentationGoogle Scholar
  29. 29.
    Stawiaski J, Decenciere E, Bidault F (2008) Interactive liver tumor segmentation using graph-cuts and watershed. Proc. MICCAI workshop on 3D segmentation in the clinic: a grand challenge II—liver tumor segmentationGoogle Scholar
  30. 30.
    Vivanti R, Joskowicz L, Karaaslan OA, Sosna J (2015a) Automatic lung tumor segmentation with leaks removal in follow-up CT studies. Int J Comp Assist Radiol Surg. to appear
  31. 31.
    Vivanti R, Ephrat A, Joskowicz L, Lev-Cohain N, Sosna J, (2015b) Automatic liver tumor segmentation in follow-up CT studies using convolutional neural networks. 1st International Workshop on Patch-based Techniques in Medical Imaging, MICCAI workshop Springer International Publishing, 49Google Scholar
  32. 32.
    Weizman L, Ben-Sira L, Joskowicz L, Precel R, Constantini S, Ben-Bashat D (2010) Automatic segmentation and components classification of optic pathway gliomas in MRI. Med Image Comput Comput Assist Interv 103–110Google Scholar
  33. 33.
    Wong D, Liu J, Fengshou Y, Tian Q, Xiong W, Zhou J, Wang SC (2008) A semi-automated method for liver tumor segmentation based on 2D region growing with knowledge-based constraints. Proc MICCAI Work 41(43):159Google Scholar
  34. 34.
    Zhou J, Xiong W, Tian Q, Qi Y, Liu J, Leow WK, Han T, Venkatesh SK, Wang SC (2008) Semi-automatic segmentation of 3D liver tumors from CT scans using voxel classification and propagational learning. Proc MICCAI Work 41:43–51Google Scholar

Copyright information

© International Federation for Medical and Biological Engineering 2018

Authors and Affiliations

  • Refael Vivanti
    • 1
    Email author
  • Leo Joskowicz
    • 1
  • Naama Lev-Cohain
    • 2
  • Ariel Ephrat
    • 1
  • Jacob Sosna
    • 2
  1. 1.School of Computer Science and EngineeringThe Hebrew University of JerusalemJerusalemIsrael
  2. 2.Department of RadiologyHadassah Hebrew University Medical CenterJerusalemIsrael

Personalised recommendations