Multimedia Tools and Applications

, Volume 77, Issue 22, pp 29669–29686 | Cite as

Multi-task neural networks for joint hippocampus segmentation and clinical score regression

  • Liang Cao
  • Long Li
  • Jifeng Zheng
  • Xin Fan
  • Feng Yin
  • Hui Shen
  • Jun ZhangEmail author


Feature representations extracted from hippocampus in magnetic resonance (MR) images are widely used in computer-aided Alzheimer’s disease (AD) diagnosis, and thus accurate segmentation for the hippocampus has been remaining an active research topic. Previous studies for hippocampus segmentation require either human annotation which is tedious and error-prone or pre-processing MR images via time-consuming non-linear registration. Although many automatic segmentation approaches have been proposed, their performance is often limited by the small size of hippocampus and complex confounding information around the hippocampus. In particular, human-engineered features extracted from segmented hippocampus regions (e.g., the volume of the hippocampus) are essential for brain disease diagnosis, while these features are independent of diagnosis models, leading to sub-optimal performance. To address these issues, we propose a multi-task deep learning (MDL) method for joint hippocampus segmentation and clinical score regression using MR images. The prominent advantages of our MDL method lie on that we don’t need any time-consuming non-linear registration for pre-processing MR images, and features generated by MDL are consistent with subsequent diagnosis models. Specifically, we first align all MR images onto a standard template, followed by a patch extraction process to approximately locate hippocampus regions in the template space. Using image patches as input data, we develop a multi-task convolutional neural network (CNN) for joint hippocampus segmentation and clinical score regression. The proposed CNN network contains two subnetworks, including 1) a U-Net with a Dice-like loss function for hippocampus segmentation, and 2) a convolutional neural network with a mean squared loss function for clinical regression. Note that these two subnetworks share a part of network parameters, to exploit the inherent association between these two tasks. We evaluate the proposed method on 407 subjects with MRI data from baseline Alzheimer’s Disease Neuroimaging Initiative (ADNI) database. The experimental results suggest that our MDL method achieves promising results in both tasks of hippocampus segmentation and clinical score regression, compared with several state-of-the-art methods.


Hippocampus segmentation Clinical score regression Brain disease diagnosis Convolutional neural network 



This study was supported by National Natural Science Foundation of China (Nos. 61703301, 61573023) one more support grant and University Science and Technology Project of Shandong Province (No. J17KA086).


  1. 1.
    Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: A system for large-scale machine learning. In: Proceedings of the 12th USENIX symposium on operating systems design and implementationGoogle Scholar
  2. 2.
    Ahmed O B, Mizotin M, Benois-Pineau J, Allard M, Catheline G, Amar C B, Initiative A D N et al (2015) Alzheimer’s disease diagnosis on structural MR images using circular harmonic functions descriptors on hippocampus and posterior cingulate cortex. Comput Med Imaging Graph 44:13–25CrossRefGoogle Scholar
  3. 3.
    Ashburner J (2007) A fast diffeomorphic image registration algorithm. NeuroImage 38(1):95–113CrossRefGoogle Scholar
  4. 4.
    Barnes J, Boyes R, Lewis E, Schott J, Frost C, Scahill R, Fox N (2007) Automatic calculation of hippocampal atrophy rates using a hippocampal template and the boundary shift integral. Neurobiol Aging 28(11):1657–1663CrossRefGoogle Scholar
  5. 5.
    Boyd S, Vandenberghe L (2004) Convex optimization. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  6. 6.
    Cao X, Gao Y, Yang J, Wu G, Shen D (2016) Learning-based multimodal image registration for prostate cancer radiation therapy. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 1–9Google Scholar
  7. 7.
    Cao X, Yang J, Gao Y, Guo Y, Wu G, Shen D (2017) Dual-core steered non-rigid registration for multi-modal images via bi-directional image synthesis. Medical Image AnalysisGoogle Scholar
  8. 8.
    Carmichael O T, Aizenstein H A, Davis S W, Becker J T, Thompson P M, Meltzer C C, Liu Y (2005) Atlas-based hippocampus segmentation in Alzheimer’s disease and mild cognitive impairment. NeuroImage 27(4):979–990CrossRefGoogle Scholar
  9. 9.
    Chang C C, Lin C J (2011) Libsvm: A library for support vector machines. ACM Trans Intell Syst Technol 2(3):27CrossRefGoogle Scholar
  10. 10.
    Chincarini A, Bosco P, Calvini P, Gemme G, Esposito M, Olivieri C, Rei L, Squarcia S, Rodriguez G, Bellotti R et al (2011) Local MRI analysis approach in the diagnosis of early and prodromal Alzheimer’s disease. NeuroImage 58 (2):469–480CrossRefGoogle Scholar
  11. 11.
    Clark K A, Woods R P, Rottenberg D A, Toga A W, Mazziotta J C (2006) Impact of acquisition protocols and processing streams on tissue segmentation of T1 weighted MR images. NeuroImage 29(1):185–202CrossRefGoogle Scholar
  12. 12.
    Coupé P, Manjón J V, Fonov V, Pruessner J, Robles M, Collins D L (2011) Patch-based segmentation using expert priors: Application to hippocampus and ventricle segmentation. NeuroImage 54(2):940–954CrossRefGoogle Scholar
  13. 13.
    Coupé P, Eskildsen S F, Manjón J V, Fonov V S, Collins D L (2012) Simultaneous segmentation and grading of anatomical structures for patient’s classification: Application to Alzheimer’s disease. NeuroImage 59(4):3736–3747CrossRefGoogle Scholar
  14. 14.
    Dagher A, Owen A M, Boecker H, Brooks D J (2001) The role of the striatum and hippocampus in planning: A PET activation study in Parkinson’s disease. Brain 124(5):1020–1032CrossRefGoogle Scholar
  15. 15.
    Dill V, Franco A R, Pinho M S (2015) Automated methods for hippocampus segmentation: The evolution and a review of the state of the art. Neuroinformatics 13 (2):133CrossRefGoogle Scholar
  16. 16.
    Folstein M F, Folstein S E, McHugh P R (1975) “mini-mental state”. A practical method for grading the cognitive state of patients for the clinician. J Psychiatr Res 12(3):189–198CrossRefGoogle Scholar
  17. 17.
    Hao Y, Wang T, Zhang X, Duan Y, Yu C, Jiang T, Fan Y (2014) Local label learning (LLL) for subcortical structure segmentation: Application to hippocampus segmentation. Hum Brain Mapp 35 (6):2674–2697CrossRefGoogle Scholar
  18. 18.
    Heckemann R A, Hajnal J V, Aljabar P, Rueckert D, Hammers A (2006) Automatic anatomical brain MRI segmentation combining label propagation and decision fusion. NeuroImage 33(1):115–126CrossRefGoogle Scholar
  19. 19.
    Hyman B T, Van Hoesen G W, Damasio A R, Barnes C L (1984) Alzheimer’s disease: Cell-specific pathology isolates the hippocampal formation. Science 225:1168–1171CrossRefGoogle Scholar
  20. 20.
    Iglesias J E, Sabuncu M R (2015) Multi-atlas segmentation of biomedical images: A survey. Med Image Anal 24(1):205–219CrossRefGoogle Scholar
  21. 21.
    Jack C R, Petersen R C, O’Brien P C, Tangalos E G (1992) MR-based hippocampal volumetry in the diagnosis of Alzheimer’s disease. Neurology 42(1):183–183CrossRefGoogle Scholar
  22. 22.
    Jack C R, Bernstein M A, Fox N C, Thompson P, Alexander G, Harvey D, Borowski B, Britson P J, L Whitwell J, Ward C (2008) The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. J Magn Reson Imaging 27(4):685–691CrossRefGoogle Scholar
  23. 23.
    Jin K, Peel A L, Mao X O, Xie L, Cottrell B A, Henshall D C, Greenberg D A (2004) Increased hippocampal neurogenesis in Alzheimer’s disease. Proc Natl Acad Sci 101(1):343–347CrossRefGoogle Scholar
  24. 24.
    Kwak K, Yoon U, Lee D K, Kim G H, Seo S W, Na D L, Shim H J, Lee J M (2013) Fully-automated approach to hippocampus segmentation using a graph-cuts algorithm combined with atlas-based segmentation and morphological opening. Magn Reson Imaging 31(7):1190–1196CrossRefGoogle Scholar
  25. 25.
    Lian C, Ruan S, Denoeux T (2015) An evidential classifier based on feature selection and two-step classification strategy. Pattern Recogn 48(7):2318–2327CrossRefGoogle Scholar
  26. 26.
    Lian C, Ruan S, Denœux T, Jardin F, Vera P (2016) Selecting radiomic features from FDG-PET images for cancer treatment outcome prediction. Med Image Anal 32:257–268CrossRefGoogle Scholar
  27. 27.
    Lindner C, Thiagarajah S, Wilkinson J, Consortium T, Wallis G, Cootes T (2013) Fully automatic segmentation of the proximal femur using random forest regression voting. IEEE Trans Med Imaging 32(8):1462–1472CrossRefGoogle Scholar
  28. 28.
    Liu M, Zhang D, Shen D (2015) View-centralized multi-atlas classification for Alzheimer’s disease diagnosis. Hum Brain Mapp 36(5):1847–1865CrossRefGoogle Scholar
  29. 29.
    Liu M, Zhang D, Chen S, Xue H (2016) Joint binary classifier learning for ECOC-based multi-class classification. IEEE Trans Pattern Anal Mach Intell 38 (11):2335–2341CrossRefGoogle Scholar
  30. 30.
    Liu M, Zhang D, Shen D (2016) Relationship induced multi-template learning for diagnosis of Alzheimer’s disease and mild cognitive impairment. IEEE Trans Med Imaging 35(6):1463–1474CrossRefGoogle Scholar
  31. 31.
    Liu M, Zhang J, Yap P T, Shen D (2017) View-aligned hypergraph learning for alzheimer’s disease diagnosis with incomplete multi-modality data. Med Image Anal 36:123–134CrossRefGoogle Scholar
  32. 32.
    Liu M, Zhang J, Adeli E, Shen D (2018) Landmark-based deep multi-instance learning for brain disease diagnosis. Med Image Anal 43:157–168CrossRefGoogle Scholar
  33. 33.
    Moodley K, Minati L, Contarino V, Prioni S, Wood R, Cooper R, D’incerti L, Tagliavini F, Chan D (2015) Diagnostic differentiation of mild cognitive impairment due to Alzheimer’s disease using a hippocampus-dependent test of spatial memory. Hippocampus 25(8):939–951CrossRefGoogle Scholar
  34. 34.
    Pohl K M, Bouix S, Nakamura M, Rohlfing T, McCarley R W, Kikinis R, Grimson W E L, Shenton M E, Wells W M (2007) A hierarchical algorithm for MR brain image parcellation. IEEE Trans Med Imaging 26(9):1201–1212CrossRefGoogle Scholar
  35. 35.
    Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. arXiv:150504597
  36. 36.
    Sled J G, Zijdenbos A P, Evans A C (1998) A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE Trans Med Imaging 17(1):87–97CrossRefGoogle Scholar
  37. 37.
    Tulving E, Markowitsch H J (1998) Episodic and declarative memory: Role of the hippocampus. Hippocampus 8(3):198–204CrossRefGoogle Scholar
  38. 38.
    Zandifar A, Fonov V, Coupé P, Pruessner J, Collins DL, Initiative ADN et al (2017) A comparison of accurate automatic hippocampal segmentation methods. NeuroImageGoogle Scholar
  39. 39.
    Zarpalas D, Gkontra P, Daras P, Maglaveras N (2014) Accurate and fully automatic hippocampus segmentation using subject-specific 3D optimal local maps into a hybrid active contour model. IEEE J Trans Eng Health Med 2:1–16CrossRefGoogle Scholar
  40. 40.
    Zhang J, Liang J, Zhao H (2013) Local energy pattern for texture classification using self-adaptive quantization thresholds. IEEE Trans Image Process 22(1):31–42MathSciNetCrossRefGoogle Scholar
  41. 41.
    Zhang J, Gao Y, Wang L, Tang Z, Xia J J, Shen D (2016) Automatic craniomaxillofacial landmark digitization via segmentation-guided partially-joint regression forest model and multiscale statistical features. IEEE Trans Biomed Eng 63(9):1820–1829CrossRefGoogle Scholar
  42. 42.
    Zhang J, Gao Y, Park SH, Zong X, Lin W, Shen D (2017) Structured learning for 3D perivascular spaces segmentation using vascular features. IEEE Transactions on Biomedical EngineeringGoogle Scholar
  43. 43.
    Zhang J, Liu M, An L, Gao Y, Shen D (2017) Alzheimer’s disease diagnosis using landmark-based features from longitudinal structural MR images. IEEE Journal of Biomedical and Health InformaticsGoogle Scholar
  44. 44.
    Zhang J, Liu M, Shen D (2017) Detecting anatomical landmarks from limited medical imaging data using two-stage task-oriented deep neural networks. IEEE Trans Image Process 26(10):4753–4764MathSciNetCrossRefGoogle Scholar
  45. 45.
    Zhu X, Suk H I, Shen D (2014) A novel matrix-similarity based loss function for joint regression and classification in AD diagnosis. NeuroImage 100:91–105CrossRefGoogle Scholar
  46. 46.
    Zhu X, Suk H I, Wang L, Lee SW, Shen D (2015) A novel relational regularization feature selection method for joint regression and classification in AD diagnosis. Medical Image analysisGoogle Scholar
  47. 47.
    Zhu Y, Zhu X, Kim M, Shen D, Wu G (2016) Early diagnosis of Alzheimer’s disease by joint feature selection and classification on temporally structured support vector machine. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 264–272Google Scholar
  48. 48.
    Zhu Y, Zhu X, Zhang H, Gao W, Shen D, Wu G (2016) Reveal consistent spatial-temporal patterns from dynamic functional connectivity for autism spectrum disorder identification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pp 106–114Google Scholar
  49. 49.
    Zhu Y, Zhu X, Kim M, Kaufer D, Wu G (2017) A novel dynamic hyper-graph inference framework for computer assisted diagnosis of neuro-diseases. In: International Conference on Information Processing in Medical Imaging, Springer, pp 158–169Google Scholar
  50. 50.
    Zou K H, Warfield S K, Bharatha A, Tempany C M, Kaus M R, Haker S J, Wells W M, Jolesz F A, Kikinis R (2004) Statistical validation of image segmentation quality based on a spatial overlap index 1: Scientific reports. Acad Radiol 11(2):178–189CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  • Liang Cao
    • 1
  • Long Li
    • 1
  • Jifeng Zheng
    • 2
  • Xin Fan
    • 3
  • Feng Yin
    • 1
  • Hui Shen
    • 1
  • Jun Zhang
    • 4
    Email author
  1. 1.Taian Tumor Prevention and Treatment HospitalTaianChina
  2. 2.Taian Maternity and Child Care HospitalTaianChina
  3. 3.Taian Institute of Science and Technology InformationTaianChina
  4. 4.Department of RadiologyDuke UniversityDurhamUSA

Personalised recommendations