, Volume 16, Issue 3–4, pp 339–349 | Cite as

Large-scale Exploration of Neuronal Morphologies Using Deep Learning and Augmented Reality

  • Zhongyu Li
  • Erik Butler
  • Kang Li
  • Aidong Lu
  • Shuiwang Ji
  • Shaoting Zhang
Original Article


Recently released large-scale neuron morphological data has greatly facilitated the research in neuroinformatics. However, the sheer volume and complexity of these data pose significant challenges for efficient and accurate neuron exploration. In this paper, we propose an effective retrieval framework to address these problems, based on frontier techniques of deep learning and binary coding. For the first time, we develop a deep learning based feature representation method for the neuron morphological data, where the 3D neurons are first projected into binary images and then learned features using an unsupervised deep neural network, i.e., stacked convolutional autoencoders (SCAEs). The deep features are subsequently fused with the hand-crafted features for more accurate representation. Considering the exhaustive search is usually very time-consuming in large-scale databases, we employ a novel binary coding method to compress feature vectors into short binary codes. Our framework is validated on a public data set including 58,000 neurons, showing promising retrieval precision and efficiency compared with state-of-the-art methods. In addition, we develop a novel neuron visualization program based on the techniques of augmented reality (AR), which can help users take a deep exploration of neuron morphologies in an interactive and immersive manner.


Neuron morphology Large-scale retrieval Deep learning Binary coding Augmented reality 



This work is partially supported by the National Science Foundation under grant ABI-1661280, ABI-1661289, and CNS-1629913.


  1. Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. NIPS, 19, 153.Google Scholar
  2. Cannon, R.C., Turner, D.A., Pyapali, G.K., & Wheal, H.V. (1998). An on-line archive of reconstructed hippocampal neurons. Journal of Neuroscience Methods, 84(1), 49–54.CrossRefPubMedGoogle Scholar
  3. Conjeti, S., Katouzian, A., Kazi, A., Mesbah, S., Beymer, D., Syeda-Mahmood, T.F., & Navab, N. (2016a). Metric hashing forests. Medical image analysis, 34, 13–29.Google Scholar
  4. Conjeti, S., Mesbah, S., Negahdar, M., Rautenberg, P.L., Zhang, S., Navab, N., & Katouzian, A. (2016b). Neuron-miner: an advanced tool for morphological search and retrieval in neuroscientific image databases. Neuroinformatics, 14(4), 369–385.Google Scholar
  5. Costa, L.D.F., Zawadzki, K., Miazaki, M., Viana, M.P., & Taraskin, S. (2010). Unveiling the neuromorphological space. Frontiers in Computational Neuroscience, 4, 150–163.CrossRefPubMedCentralGoogle Scholar
  6. Costa, M., Manton, J.D., Ostrovsky, A.D., Prohaska, S., & Jefferis, G.S. (2016). NBLAST: Rapid, sensitive comparison of neuronal structure and construction of neuron family databases. Neuron, 91(2), 293–311.CrossRefPubMedPubMedCentralGoogle Scholar
  7. Gong, Y., Lazebnik, S., Gordo, A., & Perronnin, F. (2013). Iterative quantization: a procrustean approach to learning binary codes for large-scale image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(12), 2916–2929.CrossRefPubMedGoogle Scholar
  8. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770–778).Google Scholar
  9. Hinton, G.E., & Salakhutdinov, R.R. (2006). Reducing the dimensionality of data with neural networks. Science, 313(5786), 504–507.CrossRefPubMedGoogle Scholar
  10. Ioffe, S., & Szegedy, C. (2015). Batch normalization: accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning (ICML) (pp. 448–456).Google Scholar
  11. Jain, A., Nandakumar, K., & Ross, A. (2005). Score normalization in multimodal biometric systems. Pattern recognition, 38(12), 2270–2285.CrossRefGoogle Scholar
  12. Ji, S. (2013). Computational genetic neuroanatomy of the developing mouse brain: dimensionality reduction, visualization, and clustering. BMC bioinformatics, 14(1), 222.CrossRefPubMedPubMedCentralGoogle Scholar
  13. Krizhevsky, A., Sutskever, I., & Hinton, G.E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS) (pp. 1097–1105).Google Scholar
  14. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444.CrossRefPubMedGoogle Scholar
  15. LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278–2324.CrossRefGoogle Scholar
  16. Li, R., Zeng, T., Peng, H., & Ji, S. (2017a). Deep learning segmentation of optical microscopy images improves 3-D neuron reconstruction. IEEE Transactions on Medical Imaging, 36(7), 1533–1541.Google Scholar
  17. Li, Z., Fang, R., Shen, F., Katouzian, A., & Zhang, S. (2017b). Indexing and mining large-scale neuron databases using maximum inner product search. Pattern Recognition, 63, 680–688.Google Scholar
  18. Li, Z., Metaxas, D.N., Lu, A., & Zhang, S. (2017c). Interactive exploration for continuously expanding neuron databases. Methods, 115, 100–109.Google Scholar
  19. Li, Z., Shen, F., Fang, R., Conjeti, S., Katouzian, A., & Zhang, S. (2016). Maximum inner product search for morphological retrieval of large-scale neuron data.. In International Symposium on Biomedical Imaging (ISBI) (pp. 602–606).Google Scholar
  20. Li, Z., Zhang, X., Mller, H., & Zhang, S. (2018). Large-scale retrieval for medical image analytics: A comprehensive review. Medical Image Analysis, 43, 66–84.CrossRefPubMedGoogle Scholar
  21. Liu, J., Zhang, S., Liu, W., Deng, C., Zheng, Y., & Metaxas, D.N. (2017). Scalable mammogram retrieval using composite anchor graph hashing with iterative quantization. IEEE Transactions on Circuits and Systems for Video Technology, 27(11), 2450–2460.CrossRefGoogle Scholar
  22. Liu, W., Wang, J., Ji, R., Jiang, Y.G., & Chang, S.F. (2012). Super vised hashing with kernels. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2074–2081).Google Scholar
  23. Liu, W., Wang, J., Kumar, S., & Chang, S.F. (2011). Hashing with graphs. In International Conference on Machine Learning (ICML) (pp. 1–8).Google Scholar
  24. Masci, J., Meier, U., Ciresan, D., & Schmidhuber, J. (2011). Stacked convolutional auto-encoders for hierarchical feature extraction. ICANN, 52–59.Google Scholar
  25. Mesbah, S., Conjeti, S., Kumaraswamy, A., Rautenberg, P., Navab, N., & Katouzian, A. (2015). Hashing forests for morphological search and retrieval in neuroscientific image databases. In International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) (pp. 52–59).Google Scholar
  26. Mukherjee, S., Basu, S., Condron, B., & Acton, S.T. (2013). Tree2Tree2: neuron tracing in 3D. In International Symposium on Biomedical Imaging (ISBI) (pp. 448–451).Google Scholar
  27. Nair, V., & Hinton, G.E. (2010). Rectified linear units improve restricted boltzmann machines. In International Conference on Machine Learning (ICML) (pp. 807–814).Google Scholar
  28. Peng, H., Ruan, Z., Long, F., Simpson, J.H., & Myers, E.W. (2010). V3D enables real-time 3D visualization and quantitative analysis of large-scale biological image data sets. Nature biotechnology, 28(4), 348–353.CrossRefPubMedPubMedCentralGoogle Scholar
  29. Salakhutdinov, R. (2015). Learning deep generative models. Annual Review of Statistics and Its Application, 2, 361–385.CrossRefGoogle Scholar
  30. Scorcioni, R., Polavaram, S., & Ascoli, G.A. (2008). L-Measure: a web-accessible tool for the analysis, comparison and search of digital reconstructions of neuronal morphologies. Nature protocols, 3(5), 866–876.CrossRefPubMedPubMedCentralGoogle Scholar
  31. Shen, F., Liu, W., Zhang, S., Yang, Y., & Shen, H.T. (2015). Learning binary codes for maximum inner product search. In IEEE International Conference on Computer Vision (ICCV) (pp. 4148–4156).Google Scholar
  32. Shen, F., Yang, Y., Liu, L., Liu, W., Tao, D., & Shen, H.T. (2017). Asymmetric binary coding for image search. IEEE Transactions on Multimedia, 19(9), 2022–2032.CrossRefGoogle Scholar
  33. Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556.
  34. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR) (pp. 1–9).Google Scholar
  35. Wan, Y., Long, F., Qu, L., Xiao, H., Hawrylycz, M., Myers, E.W., & Peng, H. (2015). BlastNeuron for automated comparison, retrieval and clustering of 3D neuron morphologies. Neuroinformatics, 13(4), 487–499.CrossRefPubMedGoogle Scholar
  36. Wang, J., Liu, W., Kumar, S., & Chang, S.F. (2016). Learning to hash for indexing big dataa survey. Proceedings of the IEEE, 104(1), 34–57.CrossRefGoogle Scholar
  37. Weiss, Y., Torralba, A., & Fergus, R. (2009). Spectral hashing. In Advances in Neural Information Processing Systems (NIPS) (pp. 1753–1760).Google Scholar
  38. Wu, G., Jia, H., Wang, Q., & Shen, D. (2011). SharpMean: groupwise registration guided by sharp mean image and tree-based registration. NeuroImage, 56(4), 1968–1981.CrossRefPubMedPubMedCentralGoogle Scholar
  39. Yan, C., Zhang, Y., Dai, F., Wang, X., Li, L., & Dai, Q. (2014a). Parallel deblocking filter for HEVC on many-core processor. Electronics Letters, 50(5), 367–368.Google Scholar
  40. Yan, C., Zhang, Y., Xu, J., Dai, F., Li, L., Dai, Q., & Wu, F. (2014b). A highly parallel framework for HEVC coding unit partitioning tree decision on many-core processors. IEEE Signal Processing Letters, 21(5), 573–576.Google Scholar
  41. Yan, C., Zhang, Y., Xu, J., Dai, F., Zhang, J., Dai, Q., & Wu, F. (2014c). Efficient parallel framework for HEVC motion estimation on many-core processors. IEEE Transactions on Circuits and Systems for Video Technology, 24(12), 2077–2089.Google Scholar
  42. Yu, G., & Yuan, J. (2014). Scalable forest hashing for fast similarity search. In IEEE International Conference on Multimedia and Expo (ICME) (pp. 1–6).Google Scholar
  43. Zeiler, M.D., Taylor, G.W., & Fergus, R. (2011). Adaptive deconvolutional networks for mid and high level feature learning. In IEEE International Conference on Computer Vision (ICCV) (pp. 2018–2025).Google Scholar
  44. Zhang, S., Yang, M., Cour, T., Yu, K., & Metaxas, D.N. (2015a). Query specific rank fusion for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(4), 803–815.Google Scholar
  45. Zhang, X., Dou, H., Ju, T., Xu, J., & Zhang, S. (2016). Fusing heterogeneous features from stacked sparse autoencoder for histopathological image analysis. IEEE journal of biomedical and health informatics, 20(5), 1377–1383.CrossRefPubMedGoogle Scholar
  46. Zhang, X., Liu, W., Dundar, M., Badve, S., & Zhang, S. (2015b). Towards large-scale histopathological image analysis: Hashing-based image retrieval. IEEE Transactions on Medical Imaging, 34(2), 496–506.Google Scholar
  47. Zhang, X., Xing, F., Su, H., Yang, L., & Zhang, S. (2015c). High-throughput histopathological image analysis via robust cell segmentation and hashing. Medical image analysis, 26(1), 306–315.Google Scholar
  48. Zhou, Z., Liu, X., Long, B., & Peng, H. (2016). TReMAP: automatic 3D neuron reconstruction based on tracing, reverse mapping and assembling of 2D projections. Neuroinformatics, 14(1), 41–50.CrossRefPubMedGoogle Scholar
  49. Zhou, Z., Sorensen, S., Zeng, H., Hawrylycz, M., & Peng, H. (2015). Adaptive image enhancement for tracing 3D morphologies of neurons and brain vasculatures. Neuroinformatics, 13(2), 153–166.CrossRefPubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of North Carolina at CharlotteCharlotteUSA
  2. 2.Department of Industrial and Systems EngineeringThe State University of New JerseyPiscatawayUSA
  3. 3.School of Electrical Engineering and Computer ScienceWashington State UniversityPullmanUSA

Personalised recommendations