, Volume 16, Issue 3–4, pp 325–337 | Cite as

Automated Pathogenesis-Based Diagnosis of Lumbar Neural Foraminal Stenosis via Deep Multiscale Multitask Learning

  • Zhongyi Han
  • Benzheng WeiEmail author
  • Stephanie Leung
  • Ilanit Ben Nachum
  • David Laidley
  • Shuo LiEmail author
Original Article


Pathogenesis-based diagnosis is a key step to prevent and control lumbar neural foraminal stenosis (LNFS). It conducts both early diagnosis and comprehensive assessment by drawing crucial pathological links between pathogenic factors and LNFS. Automated pathogenesis-based diagnosis would simultaneously localize and grade multiple spinal organs (neural foramina, vertebrae, intervertebral discs) to diagnose LNFS and discover pathogenic factors. The automated way facilitates planning optimal therapeutic schedules and relieving clinicians from laborious workloads. However, no successful work has been achieved yet due to its extreme challenges since 1) multiple targets: each lumbar spine has at least 17 target organs, 2) multiple scales: each type of target organ has structural complexity and various scales across subjects, and 3) multiple tasks, i.e., simultaneous localization and diagnosis of all lumbar organs, are extremely difficult than individual tasks. To address these huge challenges, we propose a deep multiscale multitask learning network (DMML-Net) integrating a multiscale multi-output learning and a multitask regression learning into a fully convolutional network. 1) DMML-Net merges semantic representations to reinforce the salience of numerous target organs. 2) DMML-Net extends multiscale convolutional layers as multiple output layers to boost the scale-invariance for various organs. 3) DMML-Net joins a multitask regression module and a multitask loss module to prompt the mutual benefit between tasks. Extensive experimental results demonstrate that DMML-Net achieves high performance (0.845 mean average precision) on T1/T2-weighted MRI scans from 200 subjects. This endows our method an efficient tool for clinical LNFS diagnosis.


Neural foraminal stenosis Multiscale learning Multitask learning Deep learning 



This work was made possible through support from Natural Science Foundation of Shandong Province in China (ZR2015FM010), Project of Shandong Province Higher Educational Science and Technology Program in China (No. J15LN20), Project of Shandong Province Traditional Chinese Medicine Technology Development Program in China (2015-026, 2017-001), and Project of Shandong Province Medical and Health Technology Development Program in China (No. 2016WS0577).

Compliance with Ethical Standards

Conflict of interests

The authors declare that they have no conflict of interest.


  1. Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro C., Corrado, G.S., Davis, A., Dean, J., Devin, M., et al (2016). Tensorflow: large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:160304467.
  2. Alomari, R.S., Corso, J.J., Chaudhary, V. (2011). Labeling of lumbar discs using both pixel- and object-level features with a two-level probabilistic model. IEEE Transactions on Medical Imaging, 30(1), 1–10. Scholar
  3. Ando, R.K. (2006). (2006). Applying alternating structure optimization to word sense disambiguation. In Proceedings of the tenth conference on computational natural language learning, association for computational linguistics (pp. 77–84).Google Scholar
  4. Ando, RK, & Zhang, T. (2005). A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6, 1817–1853.Google Scholar
  5. Baxter, J. et al. (2000). A model of inductive bias learning. Journal of Artificial Intelligence Research (JAIR), 12(149–198), 3.Google Scholar
  6. Bell, S., Lawrence Zitnick, C., Bala, K., Girshick, R. (2016). Inside-outside net: detecting objects in context with skip pooling and recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2874–2883).Google Scholar
  7. Ben-David, S, & Schuller, R. (2003). Exploiting task relatedness for multiple task learning. In Learning theory and kernel machines (pp. 567–580). Springer.Google Scholar
  8. Cai, Y., Osman, S., Sharma, M., Landis, M., Li, S. (2015). Multi-modality vertebra recognition in arbitrary views using 3d deformable hierarchical model. IEEE Transactions on Medical Imaging, 34(8), 1676–1693. Scholar
  9. Cai, Y., Leungb, S., Warringtonb, J., Pandeyb, S., Shmuilovichb, O., Lib, S. (2017). Direct spondylolisthesis identification and measurement in mr/ct using detectors trained by articulated parameterized spine model. In Proc. of SPIE (Vol. 10133, pp. 1013,319–1).Google Scholar
  10. Chen, X, & Gupta, A. (2017). An implementation of faster rcnn with study for region sampling. arXiv preprint arXiv:
  11. Cinotti, G., De Santis, P., Nofroni, I., Postacchini, F. (2002). Stenosis of lumbar intervertebral foramen: anatomic study on predisposing factors. Spine, 27(3), 223–229.CrossRefPubMedGoogle Scholar
  12. Corso, J.J., Raja’S, A., Chaudhary, V. (2008). Lumbar disc localization and labeling with a probabilistic model on both pixel and object features. In International conference on medical image computing and computer-assisted intervention (pp. 202–210). Springer.Google Scholar
  13. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A. (2010). The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2), 303–338.CrossRefGoogle Scholar
  14. Evgeniou, T., Pontil, M., Toubia, O. (2007). A convex optimization approach to modeling consumer heterogeneity in conjoint estimation. Marketing Science, 26(6), 805–818.CrossRefGoogle Scholar
  15. Ghosha, S., Raja’S, A., Chaudharya, V., Dhillonb, G. (2011). Automatic lumbar vertebra segmentation from clinical ct for wedge compression fracture diagnosis. Work, 9, 11.Google Scholar
  16. Girshick, R. (2015). Fast r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 1440–1448).Google Scholar
  17. Hariharan, B, Arbeláez, P, Girshick, R, Malik, J. (2015). Hypercolumns for object segmentation and fine-grained localization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 447–456).Google Scholar
  18. Hasegawa, T., An, H.S., Haughton, V.M., Nowicki, B.H. (1995). Lumbar foraminal stenosis: critical heights of the intervertebral discs and foramina. a cryomicrotome study in cadavera. Journal of Bone and Joint Surgery (American), 77(1), 32–38.CrossRefGoogle Scholar
  19. He, X., Yin, Y., Sharma, M., Brahm, G., Mercado, A., Li, S. (2016). Automated diagnosis of neural foraminal stenosis using synchronized superpixels representation. In MICCAI (2) (pp. 335–343). Springer.Google Scholar
  20. He, X., Landisa, M., Leunga, S., Warringtona, J., Shmuilovicha, O., Lia, S. (2017a). Automated grading of lumbar disc degeneration via supervised distance metric learning. In Proc. of SPIE Vol (Vol. 10134, pp. 1013,443-1).Google Scholar
  21. He, X., Lum, A., Sharma, M., Brahm, G., Mercado, A., Li, S. (2017b). Automated segmentation and area estimation of neural foramina with boundary regression model. Pattern Recognition, 63, 625–641.Google Scholar
  22. He, X., Zhang, H., Landis, M., Sharma, M., Warrington, J., Li, S. (2017c). Unsupervised boundary delineation of spinal neural foramina using a multi-feature and adaptive spectral segmentation. Medical Image Analysis, 36, 22–40.Google Scholar
  23. Hoang Ngan Le, T., Zheng, Y., Zhu, C., Luu, K., Savvides, M. (2016). Multiple scale faster-rcnn approach to driver’s cell-phone usage and hands on steering wheel detection. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 46–53).Google Scholar
  24. Hoiem, D., Chodpathumwan, Y., Dai, Q. (2012). Diagnosing error in object detectors. Computer Vision–ECCV, 2012, 340–353.Google Scholar
  25. Huang, S.H., Chu, Y.H., Lai, S.H., Novak, C.L. (2009). Learning-based vertebra detection and iterative normalized-cut segmentation for spinal mri. IEEE Transactions on Medical Imaging, 28(10), 1595–1605.CrossRefPubMedGoogle Scholar
  26. Jamaludin, A., Kadir, T., Zisserman, A. (2017). Spinenet: automated classification and evidence visualization in spinal mris. Medical Image Analysis, 41, 63–73., special Issue on the 2016 Conference on Medical Image Computing and Computer Assisted Intervention (Analog to MICCAI 2015).CrossRefPubMedGoogle Scholar
  27. Kaneko, Y., Matsumoto, M., Takaishi, H., Nishiwaki, Y., Momoshima, S., Toyama, Y. (2012). Morphometric analysis of the lumbar intervertebral foramen in patients with degenerative lumbar scoliosis by multidetector-row computed tomography. European Spine Journal, 21(12), 2594–2602.CrossRefPubMedPubMedCentralGoogle Scholar
  28. Kelm, B.M., Wels, M., Zhou, S.K., Seifert, S., Suehling, M., Zheng, Y., Comaniciu, D. (2013). Spine detection in ct and mr using iterated marginal space learning. Medical Image Analysis, 17(8), 1283–1292.CrossRefGoogle Scholar
  29. Kim, S., Lee, J.W., Chai, J.W., Yoo, H.J., Kang, Y., Seo, J., Ahn, J.M., Kang, H.S. (2015). A new mri grading system for cervical foraminal stenosis based on axial t2-weighted images. Korean Journal of Radiology, 16(6), 1294–1302.CrossRefPubMedPubMedCentralGoogle Scholar
  30. Klinder, T., Wolz, R., Lorenz, C., Franz, A., Ostermann, J. (2008). Spine segmentation using articulated shape models. Medical Image Computing and Computer-Assisted Intervention–MICCAI, 2008, 227–234.Google Scholar
  31. Law, M.W., Tay, K., Leung, A., Garvin, G.J., Li, S. (2013a). Intervertebral disc segmentation in mr images using anisotropic oriented flux. Medical Image Analysis, 17(1), 43–61.
  32. Law, M.W.K., Garvin, G.J., Tummala, S., Tay, K., Leung, A.E., Li, S. (2013b). Gradient competition anisotropy for centerline extraction and segmentation of spinal cords (pp. 49–61). Berlin: Springer.Google Scholar
  33. Lee, S., Lee, J.W., Yeom, J.S., Kim, K.J., Kim, H.J., Chung, S.K., Kang, H.S. (2010). A practical mri grading system for lumbar foraminal stenosis. American Journal of Roentgenology, 194(4), 1095–1098.CrossRefPubMedGoogle Scholar
  34. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C. (2016). Ssd: single shot multibox detector. In European conference on computer vision (pp. 21–37). Springer.Google Scholar
  35. Luo, W., Li, Y., Urtasun, R., Zemel, R. (2016). Understanding the effective receptive field in deep convolutional neural networks. In Advances in neural information processing systems (pp. 4898–4906).Google Scholar
  36. Panjabi, M.M., Maak, T.G., Ivancic, P.C., Ito, S. (2006). Dynamic intervertebral foramen narrowing during simulated rear impact. Spine, 31(5), E128–E134.CrossRefPubMedGoogle Scholar
  37. Park, H.J., Kim, S., Lee, S.Y., Park, N.H., Rho, M.H., Hong, H.P., Kwag, H.J., Kook, S.H., Choi, S.H. (2012). Clinical correlation of a new mr imaging method for assessing lumbar foraminal stenosis. American Journal of Neuroradiology, 33(5), 818–822.CrossRefPubMedGoogle Scholar
  38. Peng, Z., Zhong, J., Wee, W., Lee, J.H. (2006). Automated vertebra detection and segmentation from the whole spine mr images. In 27th annual international conference of the engineering in medicine and biology society, 2005. IEEE-EMBS 2005 (pp. 2527–2530). IEEE.Google Scholar
  39. Rajaee, S.S., Bae, H.W., Kanim, L.E., Delamarter, R.B. (2012). Spinal fusion in the united states: analysis of trends from 1998 to 2008. Spine, 37(1), 67–76.CrossRefPubMedGoogle Scholar
  40. Raja’S, A., Corso, J.J., Chaudhary, V., Dhillon, G. (2011). Toward a clinical lumbar cad: herniation diagnosis. International Journal of Computer Assisted Radiology and Surgery, 6(1), 119–126.CrossRefGoogle Scholar
  41. Ren, S., He, K., Girshick, R., Sun, J. (2015). Faster r-cnn: towards real-time object detection with region proposal networks. In Advances in neural information processing systems (pp. 91–99).Google Scholar
  42. Shi, R., Sun, D., Qiu, Z., Weiss, K.L. (2007). An efficient method for segmentation of mri spine images. In IEEE/ICME international conference on complex medical engineering, 2007. CME 2007 (pp. 713–717). IEEE.Google Scholar
  43. Simonyan, K, Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:
  44. Ṡtern, D, Likar, B, Pernuṡ, F, Vrtovec, T. (2009). Automated detection of spinal centrelines, vertebral bodies and intervertebral discs in ct and mr images of lumbar spine. Physics in Medicine and Biology, 55(1), 247.CrossRefGoogle Scholar
  45. Sun, X., Wu, P., Hoi, S.C. (2017). Face detection using deep learning: an improved faster rcnn approach. arXiv preprint arXiv:
  46. Torralba, A., Murphy, K.P., Freeman, W.T. (2004). Sharing features: efficient boosting procedures for multiclass object detection. In Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004 (Vol. 2, pp. II–762–II–769).
  47. Wan, S., Chen, Z., Zhang, T., Zhang, B., Wong, K.K. (2016). Bootstrapping face detection with hard negative examples. arXiv preprint arXiv:
  48. Wang, Q., Lu, L., Wu, D., El-Zehiry, N., Zheng, Y., Shen, D., Zhou, K.S. (2015a). Automatic segmentation of spinal canals in ct images via iterative topology refinement. IEEE Transactions on Medical Imaging, 34(8), 1694–1704.
  49. Wang, Z, Zhen, X, Tay, K, Osman, S, Romano, W, Li, S. (2015b). Regression segmentation for m3 spinal images. IEEE Transactions on Medical Imaging, 34(8), 1640–1648.Google Scholar
  50. Yan, C., Zhang, Y., Xu, J., Dai, F., Li, L., Dai, Q., Wu, F. (2014a). A highly parallel framework for hevc coding unit partitioning tree decision on many-core processors. IEEE Signal Processing Letters, 21(5), 573–576.
  51. Yan, C., Zhang, Y., Xu, J., Dai, F., Zhang, J., Dai, Q., Wu, F. (2014b). Efficient parallel framework for hevc motion estimation on many-core processors. IEEE Transactions on Circuits and Systems for Video Technology, 24(12), 2077–2089.
  52. Yan, C., Xie, H., Liu, S., Yin, J., Zhang, Y., Dai, Q. (2017a). Effective uyghur language text detection in complex background images for traffic prompt identification. IEEE Transactions on Intelligent Transportation Systems, PP(99), 1–10.
  53. Yan, C., Xie, H., Yang, D., Yin, J., Zhang, Y., Dai, Q. (2017b). Supervised hash coding with deep neural network for environment perception of intelligent vehicles. IEEE Transactions on Intelligent Transportation Systems, PP(99), 1–12.
  54. Yao, J., Burns, J.E., Forsberg, D., Seitel, A., Rasoulian, A., Abolmaesumi, P., Hammernik, K., Urschler, M., Ibragimov, B., Korez, R., et al (2016). A multi-center milestone study of clinical vertebral ct segmentation. Computerized Medical Imaging and Graphics, 49, 16–28.CrossRefPubMedPubMedCentralGoogle Scholar
  55. Zhan, Y., Maneesh, D., Harder, M., Zhou, X.S. (2012). Robust mr spine detection using hierarchical learning and local articulated model. In International conference on medical image computing and computer-assisted intervention (pp. 141–148). Springer.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.College of Science and TechnologyShandong University of Traditional Chinese MedicineJinanChina
  2. 2.Computational Medicine LabShandong University of Traditional Chinese MedicineJinanChina
  3. 3.Department of Medical ImagingWestern UniversityLondonCanada
  4. 4.Digital Image Group (DIG)LondonCanada

Personalised recommendations