Fast and Precise Hippocampus Segmentation Through Deep Convolutional Neural Network Ensembles and Transfer Learning

  • Dimitrios AtaloglouEmail author
  • Anastasios Dimou
  • Dimitrios Zarpalas
  • Petros Daras
Original Article


Automatic segmentation of the hippocampus from 3D magnetic resonance imaging mostly relied on multi-atlas registration methods. In this work, we exploit recent advances in deep learning to design and implement a fully automatic segmentation method, offering both superior accuracy and fast result. The proposed method is based on deep Convolutional Neural Networks (CNNs) and incorporates distinct segmentation and error correction steps. Segmentation masks are produced by an ensemble of three independent models, operating with orthogonal slices of the input volume, while erroneous labels are subsequently corrected by a combination of Replace and Refine networks. We explore different training approaches and demonstrate how, in CNN-based segmentation, multiple datasets can be effectively combined through transfer learning techniques, allowing for improved segmentation quality. The proposed method was evaluated using two different public datasets and compared favorably to existing methodologies. In the EADC-ADNI HarP dataset, the correspondence between the method’s output and the available ground truth manual tracings yielded a mean Dice value of 0.9015, while the required segmentation time for an entire MRI volume was 14.8 seconds. In the MICCAI dataset, the mean Dice value increased to 0.8835 through transfer learning from the larger EADC-ADNI HarP dataset.


Hippocampus segmentation Convolutional neural networks Deep learning Error correction Transfer learning Magnetic resonance imaging 



The authors gratefully acknowledge the support of NVIDIA Corporation with the donation of the GPU used for this research.

Compliance with Ethical Standards

Conflict of interests

The authors declare no conflicts of interest.


  1. Ahdidan, J., Raji, C.A., DeYoe, E.A., Mathis, J., Noe, K., Rimestad, J., Kjeldsen, T.K., Mosegaard, J., Becker, J.T., Lopez, O. (2016). Quantitative neuroimaging software for clinical assessment of hippocampal volumes on MR imaging. Journal of Alzheimer’s Disease, 49(3), 723–732.PubMedGoogle Scholar
  2. Aljabar, P., Heckemann, R.A., Hammers, A., Hajnal, J.V., Rueckert, D. (2009). Multi-atlas based segmentation of brain images: atlas selection and its effect on accuracy. NeuroImage, 46(3), 726–738.PubMedGoogle Scholar
  3. Bateman, R.J., Xiong, C., Benzinger, T.L., Fagan, A.M., Goate, A., Fox, N.C., Marcus, D.S., Cairns, N.J., Xie, X., Blazey, T.M., et al. (2012). Clinical and biomarker changes in dominantly inherited Alzheimer’s disease. New England Journal of Medicine, 367(9), 795–804.PubMedGoogle Scholar
  4. Bernasconi, N., Bernasconi, A., Caramanos, Z., Antel, S., Andermann, F., Arnold, D. (2003). Mesial temporal damage in temporal lobe epilepsy: a volumetric MRI study of the hippocampus, amygdala and parahippocampal region. Brain, 126(2), 462–469.PubMedGoogle Scholar
  5. Boccardi, M., Bocchetta, M., Morency, F.C., Collins, D.L., Nishikawa, M., Ganzola, R., Grothe, M.J., Wolf, D., Redolfi, A., Pievani, M., et al. (2015). Training labels for hippocampal segmentation based on the EADC-ADNI harmonized hippocampal protocol. Alzheimer’s & Dementia, 11(2), 175–183.Google Scholar
  6. de Brébisson, A., & Montana, G. (2015). Deep neural networks for anatomical brain segmentation. arXiv:150202445.
  7. Bremner, J.D., Narayan, M., Anderson, E.R., Staib, L.H., Miller, H.L., Charney, D.S. (2000). Hippocampal volume reduction in major depression. American Journal of Psychiatry, 157(1), 115– 118.PubMedGoogle Scholar
  8. Brosch, T., Tang, L.Y., Yoo, Y., Li, D.K., Traboulsee, A., Tam, R. (2016). Deep 3D convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation. IEEE Transactions on Medical Imaging, 35(5), 1229–1239.PubMedGoogle Scholar
  9. Carreira, J., Agrawal, P., Fragkiadaki, K., Malik, J. (2016). Human pose estimation with iterative error feedback. In: Proceedings of the IEEE conference on computer vision and pattern recognition, IEEE, pp. 4733–4742.Google Scholar
  10. Chen, H., Dou, Q., Yu, L., Heng, P.A. (2016). Voxresnet: Deep voxelwise residual networks for volumetric brain segmentation. arXiv:160805895.
  11. Chen, Y., Shi, B., Wang, Z., Sun, T., Smith, C.D., Liu, J. (2017). Accurate and consistent hippocampus segmentation through convolutional LSTM and view ensemble. In: International workshop on machine learning in medical imaging, Springer, pp. 88–96.Google Scholar
  12. Chetlur, S., Woolley, C., Vandermersch, P., Cohen, J., Tran, J., Catanzaro, B., Shelhamer, E. (2014). cuDNN: Efficient primitives for deep learning. arXiv:14100759.
  13. Chincarini, A., Sensi, F., Rei, L., Gemme, G., Squarcia, S., Longo, R., Brun, F., Tangaro, S., Bellotti, R., Amoroso, N., et al. (2016). Integrating longitudinal information in hippocampal volume measurements for the early detection of Alzheimer’s disease. NeuroImage, 125, 834–847.PubMedGoogle Scholar
  14. Choi, H., & Jin, K.H. (2016). Fast and robust segmentation of the striatum using deep convolutional neural networks. Journal of Neuroscience Methods, 274, 146–153.PubMedGoogle Scholar
  15. Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O. (2016). 3d u-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention, Springer, pp. 424–432.Google Scholar
  16. Collins, D.L., & Pruessner, J.C. (2010). Towards accurate, automatic segmentation of the hippocampus and amygdala from MRI by augmenting ANIMAL with a template library and label fusion. NeuroImage, 52(4), 1355–1366.PubMedGoogle Scholar
  17. Collobert, R., Kavukcuoglu, K., Farabet, C. (2011). Torch7: A matlab-like environment for machine learning. In: BigLearn, NIPS workshop, EPFL-CONF-192376.Google Scholar
  18. Coupé, P, Manjón, JV, Fonov, V., Pruessner, J., Robles, M., Collins, D.L. (2010). Nonlocal patch-based label fusion for hippocampus segmentation. In: International conference on medical image computing and computer assisted intervention, Springer, pp. 129–136.Google Scholar
  19. Dolz, J., Desrosiers, C., Ayed, I.B. (2017). 3D fully convolutional networks for subcortical segmentation in MRI: A large-scale study. NeuroImage.Google Scholar
  20. Du, A., Schuff, N., Amend, D., Laakso, M., Hsu, Y., Jagust, W., Yaffe, K., Kramer, J., Reed, B., Norman, D., et al. (2001). Magnetic resonance imaging of the entorhinal cortex and hippocampus in mild cognitive impairment and Alzheimer’s disease. Journal of Neurology. Neurosurgery & Psychiatry, 71(4), 441–447.Google Scholar
  21. Fischl, B., Salat, D.H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., Van Der Kouwe, A., Killiany, R., Kennedy, D., Klaveness, S., et al. (2002). Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33(3), 341–355.PubMedGoogle Scholar
  22. Gidaris, S., & Komodakis, N. (2017). Detect, replace, refine: Deep structured prediction for pixel wise labeling. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5248–5257.Google Scholar
  23. Giraud, R., Ta, V.T., Papadakis, N., Manjón, J V, Collins, D.L., Coupé, P, Alzheimer’s Disease Neuroimaging Initiative, et al. (2016). An optimized patchmatch for multi-scale and multi-feature label fusion. NeuroImage, 124, 770–782.PubMedGoogle Scholar
  24. Glorot, X., & Bengio, Y. (2010). Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the 13th international conference on artificial intelligence and statistics, pp. 249–256.Google Scholar
  25. Harrison, P.J. (2004). The hippocampus in schizophrenia: a review of the neuropathological evidence and its pathophysiological implications. Psychopharmacology, 174(1), 151–162.PubMedGoogle Scholar
  26. Havaei, M., Davy, A., Warde-Farley, D., Biard, A., Courville, A., Bengio, Y., Pal, C., Jodoin, P.M., Larochelle, H. (2017). Brain tumor segmentation with deep neural networks. Medical Image Analysis, 35, 18–31.PubMedGoogle Scholar
  27. He, K., Zhang, X., Ren, S., Sun, J. (2016). Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778.Google Scholar
  28. Inglese, P., Amoroso, N., Boccardi, M., Bocchetta, M., Bruno, S., Chincarini, A., Errico, R., Frisoni, G., Maglietta, R., Redolfi, A., et al. (2015). Multiple RF classifier for the hippocampus segmentation: Method and validation on EADC-ADNI harmonized hippocampal protocol. Physica Medica, 31(8), 1085–1091.PubMedGoogle Scholar
  29. Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:150203167.
  30. Jack, C.R., Bernstein, M.A., Fox, N.C., Thompson, P., Alexander, G., Harvey, D., Borowski, B., Britson, P.J., L Whitwell, J., Ward, C., et al. (2008). The Alzheimer’s disease neuroimaging initiative (ADNI): MRI methods. Journal of Magnetic Resonance Imaging, 27(4), 685–691.PubMedGoogle Scholar
  31. Jack, C.R., Barkhof, F., Bernstein, M.A., Cantillon, M., Cole, P.E., DeCarli, C., Dubois, B., Duchesne, S., Fox, N.C., Frisoni, G.B., et al. (2011). Steps to standardization and validation of hippocampal volumetry as a biomarker in clinical trials and diagnostic criterion for Alzheimer’s disease. Alzheimer’s & Dementia, 7 (4), 474– 485.Google Scholar
  32. Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B. (2017). Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation. Medical Image Analysis, 36, 61–78.PubMedGoogle Scholar
  33. Kingma, D., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv:14126980.
  34. Kleesiek, J., Urban, G., Hubert, A., Schwarz, D., Maier-Hein, K., Bendszus, M., Biller, A. (2016). Deep MRI brain extraction: a 3D convolutional neural network for skull stripping. NeuroImage, 129, 460–469.PubMedGoogle Scholar
  35. Krizhevsky, A., Sutskever, I., Hinton, G.E. (2012). ImageNet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp. 1097–1105.Google Scholar
  36. Kushibar, K., Valverde, S., Gonzalez-Villa, S., Bernal, J., Cabezas, M., Oliver, A., Llado, X. (2017). Automated sub-cortical brain structure segmentation combining spatial and deep convolutional features. arXiv:170909075.
  37. Landman, B., & Warfield, S. (2012). MICCAI 2012 Workshop on Multi-Atlas Labeling. CreateSpace Independent Publishing Platform, ISBN: 1479126187.Google Scholar
  38. Langerak, T.R., van der Heide, U.A., Kotte, A.N., Viergever, M.A., Van Vulpen, M., Pluim, J.P. (2010). Label fusion in atlas-based segmentation using a selective and iterative method for performance level estimation (SIMPLE). IEEE Transactions on Medical Imaging, 29(12), 2000–2008.PubMedGoogle Scholar
  39. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.Google Scholar
  40. Ledig, C., Heckemann, R.A., Hammers, A., Lopez, J.C., Newcombe, V.F., Makropoulos, A., Lötjönen, J, Menon, D.K., Rueckert, D. (2015). Robust whole-brain segmentation: application to traumatic brain injury. Medical Image Analysis, 21(1), 40–58.PubMedGoogle Scholar
  41. Leung, K.K., Barnes, J., Ridgway, G.R., Bartlett, J.W., Clarkson, M.J., Macdonald, K., Schuff, N., Fox, N.C., Ourselin, S., Alzheimer’s Disease Neuroimaging Initiative, et al. (2010). Automated cross-sectional and longitudinal hippocampal volume measurement in mild cognitive impairment and Alzheimer’s disease. NeuroImage, 51(4), 1345–1359.PubMedPubMedCentralGoogle Scholar
  42. Li, K., Hariharan, B., Malik, J. (2016). Iterative instance segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3659–3667.Google Scholar
  43. Long, J., Shelhamer, E., Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431–3440.Google Scholar
  44. Maglietta, R., Amoroso, N., Boccardi, M., Bruno, S., Chincarini, A., Frisoni, G.B., Inglese, P., Redolfi, A., Tangaro, S., Tateo, A., et al. (2016). Automated hippocampal segmentation in 3D MRI using random undersampling with boosting algorithm. Pattern Analysis and Applications, 19(2), 579–591.PubMedGoogle Scholar
  45. Marcus, D.S., Wang, T.H., Parker, J., Csernansky, J.G., Morris, J.C., Buckner, R.L. (2007). Open Access Series of Imaging Studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. Journal of Cognitive Neuroscience, 19(9), 1498–1507.PubMedGoogle Scholar
  46. Mehta, R., Majumdar, A., Sivaswamy, J. (2017). BrainSegNet: a convolutional neural network architecture for automated segmentation of human brain structures. Journal of Medical Imaging, 4(2), 024003.PubMedPubMedCentralGoogle Scholar
  47. Milletari, F., Ahmadi, S.A., Kroll, C., Plate, A., Rozanski, V., Maiostre, J., Levin, J., Dietrich, O., Ertl-Wagner, B., Bötzel, K, et al. (2017). Hough-CNN: deep learning for segmentation of deep brain regions in MRI and ultrasound. Computer Vision and Image Understanding, 164, 92–102.Google Scholar
  48. Moeskops, P., Viergever, M.A., Mendrik, A.M. (2016). de Vries LS, Benders MJ, Iṡgum I. Automatic segmentation of MR brain images with a convolutional neural network. IEEE Transactions on Medical Imaging, 35(5), 1252–1261.PubMedGoogle Scholar
  49. Morra, J.H., Tu, Z., Apostolova, L.G., Green, A.E., Toga, A.W., Thompson, P.M. (2010). Comparison of AdaBoost and support vector machines for detecting Alzheimer’s disease through automated hippocampal segmentation. IEEE Transactions on Medical Imaging, 29(1), 30.PubMedGoogle Scholar
  50. Nair, V., & Hinton, G.E. (2010). Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814.Google Scholar
  51. Patenaude, B., Smith, S.M., Kennedy, D.N., Jenkinson, M. (2011). A bayesian model of shape and appearance for subcortical brain segmentation. Neuroimage, 56(3), 907–922.PubMedPubMedCentralGoogle Scholar
  52. Pereira, S., Pinto, A., Alves, V., Silva, C.A. (2016). Brain tumor segmentation using convolutional neural networks in MRI images. IEEE Transactions on Medical Imaging, 35(5), 1240–1251.PubMedGoogle Scholar
  53. Platero, C., & Tobar, M.C. (2017). Combining a patch-based approach with a non-rigid registration-based label fusion method for the hippocampal segmentation in Alzheimer’s disease. Neuroinformatics, 15(2), 165–183.PubMedGoogle Scholar
  54. Razavian, A.S., Azizpour, H., Sullivan, J., Carlsson, S. (2014). CNN features off-the-shelf: an astounding baseline for recognition (Vol. 2014, pp. 512–519). IEEE Conference on: IEEE.Google Scholar
  55. Ronneberger, O., Fischer, P., Brox, T. (2015). U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention, Springer, pp. 234–241.Google Scholar
  56. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 115(3), 211–252.Google Scholar
  57. Scoville WB, & Milner B. (2000). Loss of recent memory after bilateral hippocampal lesions. The Journal of Neuropsychiatry and Clinical Neurosciences, 12(1), 103–a.PubMedGoogle Scholar
  58. Sdika, M. (2010). Combining atlas based segmentation and intensity classification with nearest neighbor transform and accuracy weighted vote. Medical Image Analysis, 14(2), 219–226.PubMedGoogle Scholar
  59. Shakeri, M., Tsogkas, S., Ferrante, E., Lippe, S., Kadoury, S., Paragios, N., Kokkinos, I. (2016). Sub-cortical brain structure segmentation using F-CNN’s. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI), IEEE, pp. 269–272.Google Scholar
  60. Shen, D., Moffat, S., Resnick, S.M., Davatzikos, C. (2002). Measuring size and shape of the hippocampus in MR images using a deformable shape model. NeuroImage, 15(2), 422–434.PubMedGoogle Scholar
  61. Sled, J.G., Zijdenbos, A.P., Evans, A.C. (1998). A nonparametric method for automatic correction of intensity nonuniformity in MRI data. IEEE Transactions on Medical Imaging, 17(1), 87–97.PubMedGoogle Scholar
  62. Smith, S.M. (2002). Fast robust automated brain extraction. Human Brain Mapping, 17(3), 143–155.PubMedGoogle Scholar
  63. Sun, C., Shrivastava, A., Singh, S., Gupta, A. (2017). Revisiting unreasonable effectiveness of data in deep learning era. In: 2017 IEEE international conference on computer vision (ICCV). IEEE, pp. 843–852.Google Scholar
  64. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A. (2017). Inception-v4, Inception-ResNet and the impact of residual connections on learning. In: AAAI conference on artificial intelligence.Google Scholar
  65. Tong, T., Wolz, R., Coupé, P, Hajnal, J.V., Rueckert, D., Initiative, A.D.N., et al. (2013). Segmentation of MR images via discriminative dictionary learning and sparse coding: application to hippocampus labeling. NeuroImage, 76, 11–23.PubMedGoogle Scholar
  66. Wachinger, C., Reuter, M., Klein T. (2017). DeepNAT, Deep convolutional neural network for segmenting neuroanatomy. NeuroImage.Google Scholar
  67. Wang, H., & Yushkevich, P.A. (2013). Multi-atlas segmentation with joint label fusion and corrective learning an open source implementation. Frontiers in Neuroinformatics 7.Google Scholar
  68. Wang, H., Das, S.R., Suh, J.W., Altinay, M., Pluta, J., Craige, C., Avants, B., Yushkevich, P.A., Alzheimer’s Disease Neuroimaging Initiative, et al. (2011). A learning-based wrapper method to correct systematic errors in automatic image segmentation: consistently improved performance in hippocampus, cortex and brain segmentation. NeuroImage, 55(3), 968–985.PubMedPubMedCentralGoogle Scholar
  69. Wang, H., Suh, J.W., Das, S.R., Pluta, J.B., Craige, C., Yushkevich, P.A. (2013). Multi-atlas segmentation with joint label fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(3), 611–623.PubMedGoogle Scholar
  70. Wang, L., Wang, L., Lu, H., Zhang, P., Ruan, X. (2016). Saliency detection with recurrent fully convolutional networks. In: European conference on computer vision, Springer, pp. 825–841.Google Scholar
  71. Wang, Y., Ma, G., Wu, X., Zhou, J. (2018). Patch-based label fusion with structured discriminant embedding for hippocampus segmentation. Neuroinformatics, 1–13.Google Scholar
  72. Yang, J., Staib, L.H., Duncan, J.S. (2004). Neighbor-constrained segmentation with level set based 3-D deformable models. IEEE Transactions on Medical Imaging, 23(8), 940–948.PubMedPubMedCentralGoogle Scholar
  73. Yosinski, J., Clune, J., Bengio, Y., Lipson, H. (2014). How transferable are features in deep neural networks? In: Advances in neural information processing systems, pp. 3320–3328.Google Scholar
  74. Zarpalas, D., Gkontra, P., Daras, P., Maglaveras, N. (2014a). Accurate and fully automatic hippocampus segmentation using subject-specific 3D optimal local maps into a hybrid active contour model. IEEE Journal of Translational Engineering in Health and Medicine, 2, 1–16.Google Scholar
  75. Zarpalas, D., Gkontra, P., Daras, P., Maglaveras, N. (2014b). Gradient-based reliability maps for ACM-based segmentation of hippocampus. IEEE Transactions on Biomedical Engineering, 61(4), 1015–1026.PubMedGoogle Scholar
  76. Zhang, W., Li, R., Deng, H., Wang, L., Lin, W., Ji, S., Shen, D. (2015). Deep convolutional neural networks for multi-modality isointense infant brain image segmentation. NeuroImage, 108, 214– 224.PubMedPubMedCentralGoogle Scholar
  77. Zhu, H., Cheng, H., Yang, X., Fan, Y., Initiative, A.D.N., et al. (2017). Metric learning for multi-atlas based segmentation of hippocampus. Neuroinformatics, 15(1), 41–50.PubMedPubMedCentralGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Information Technologies Institute (ITI)Centre for Research and Technology HELLASThessalonikiGreece

Personalised recommendations