Advertisement

A Deep Convolutional Neural Network for Annotation of Magnetic Resonance Imaging Sequence Type

  • Sara RanjbarEmail author
  • Kyle W. Singleton
  • Pamela R. Jackson
  • Cassandra R. Rickertsen
  • Scott A. Whitmire
  • Kamala R. Clark-Swanson
  • J. Ross Mitchell
  • Kristin R. Swanson
  • Leland S. Hu
Article

Abstract

The explosion of medical imaging data along with the advent of big data analytics has launched an exciting era for clinical research. One factor affecting the ability to aggregate large medical image collections for research is the lack of infrastructure for automated data annotation. Among all imaging modalities, annotation of magnetic resonance (MR) images is particularly challenging due to the non-standard labeling of MR image types. In this work, we aimed to train a deep neural network to annotate MR image sequence type for scans of brain tumor patients. We focused on the four most common MR sequence types within neuroimaging: T1-weighted (T1W), T1-weighted post-gadolinium contrast (T1Gd), T2-weighted (T2W), and T2-weighted fluid-attenuated inversion recovery (FLAIR). Our repository contains images acquired using a variety of pulse sequences, sequence parameters, field strengths, and scanner manufacturers. Image selection was agnostic to patient demographics, diagnosis, and the presence of tumor in the imaging field of view. We used a total of 14,400 two-dimensional images, each visualizing a different part of the brain. Data was split into train, validation, and test sets (9600, 2400, and 2400 images, respectively) and sets consisted of equal-sized groups of image types. Overall, the model reached an accuracy of 99% on the test set. Our results showed excellent performance of deep learning techniques in predicting sequence types for brain tumor MR images. We conclude deep learning models can serve as tools to support clinical research and facilitate efficient database management.

Keywords

Artificial intelligence Deep learning Magnetic resonance imaging Sequence type Automated annotation Image database 

Notes

Funding Information

This study received support from the James S. McDonnell Foundation, the Ivy Foundation, the Mayo Clinic, the Zicarelli Foundation, and the NIH (R01 NS060752, R01 CA164371, U54 CA210180, U54 CA143970, U54 CA193489, U01 CA220378).

References

  1. 1.
    Commission, M. P. A, Book AD, others: Healthcare spending and the Medicare program. Washington, DC: MedPAC, 2012Google Scholar
  2. 2.
    Smith-Bindman R, Miglioretti DL, Larson EB: Rising use of diagnostic medical imaging in a large integrated health system. Health Aff 27:1491–1502, 2008CrossRefGoogle Scholar
  3. 3.
    Nitz WR: MR imaging: Acronyms and clinical applications. Eur. Radiol. 9:979–997, 1999CrossRefGoogle Scholar
  4. 4.
    MRI sequences acronyms. IMAIOS Available at: https://www.imaios.com/en/e-Courses/e-MRI/MRI-Sequences/Sequences-acronyms. (Accessed: 30th April 2019)Google Scholar
  5. 5.
    Enlarge, C. T. T. O. & Ge, G. GRE Acronyms.Google Scholar
  6. 6.
    Swanson KR, Rockne RC, Claridge J, Chaplain MA, Alvord EC, Anderson ARA: Quantifying the role of angiogenesis in malignant progression of gliomas: in silico modeling integrates imaging and histology. Cancer Res. 71:7366–7375, 2011CrossRefGoogle Scholar
  7. 7.
    Jackson PR, Juliano J, Hawkins-Daarud A, Rockne RC, Swanson KR: Patient-specific mathematical neuro-oncology: Using a simple proliferation and invasion tumor model to inform clinical practice. Bull. Math. Biol. 77:846–856, 2015CrossRefGoogle Scholar
  8. 8.
    Hawkins-Daarud A, Rockne R, Corwin D, Anderson ARA, Kinahan P, Swanson KR: In silico analysis suggests differential response to bevacizumab and radiation combination therapy in newly diagnosed glioblastoma. J. R. Soc. Interface 12:20150388, 2015CrossRefGoogle Scholar
  9. 9.
    Rockne RC. et al.: A patient-specific computational model of hypoxia-modulated radiation resistance in glioblastoma using 18F-FMISO-PET. J. R. Soc. Interface 12, 2015CrossRefGoogle Scholar
  10. 10.
    Alfonso JCL, Talkenberger K, Seifert M, Klink B, Hawkins-Daarud A, Swanson KR, Hatzikirou H, Deutsch A: The biology and mathematical modelling of glioma invasion: a review. J. R. Soc. Interface 14:20170490, 2017CrossRefGoogle Scholar
  11. 11.
    Rayfield CA, Grady F, de Leon G, Rockne R, Carrasco E, Jackson P, Vora M, Johnston SK, Hawkins-Daarud A, Clark-Swanson KR, Whitmire S, Gamez ME, Porter A, Hu L, Gonzalez-Cuyar L, Bendok B, Vora S, Swanson KR: Distinct phenotypic clusters of glioblastoma growth and response kinetics predict survival. JCO Clinical Cancer Informatics 2:1–14, 2018CrossRefGoogle Scholar
  12. 12.
    Randall EC, Emdal KB, Laramy JK, Kim M, Roos A, Calligaris D, Regan MS, Gupta SK, Mladek AC, Carlson BL, Johnson AJ, Lu FK, Xie XS, Joughin BA, Reddy RJ, Peng S, Abdelmoula WM, Jackson PR, Kolluri A, Kellersberger KA, Agar JN, Lauffenburger DA, Swanson KR, Tran NL, Elmquist WF, White FM, Sarkaria JN, Agar NYR: Integrated mapping of pharmacokinetics and pharmacodynamics in a patient-derived xenograft model of glioblastoma. Nat. Commun. 9:4904, 2018CrossRefGoogle Scholar
  13. 13.
    Swanson KR, Rostomily RC, Alvord, Jr EC: A mathematical modelling tool for predicting survival of individual patients following resection of glioblastoma: A proof of principle. Br. J. Cancer 98:113–119, 2008CrossRefGoogle Scholar
  14. 14.
    Massey SC, Rockne RC, Hawkins-Daarud A, Gallaher J, Anderson ARA, Canoll P, Swanson KR: Simulating PDGF-Driven Glioma Growth and Invasion in an Anatomically Accurate Brain Domain. Bull. Math. Biol. 80:1292–1309, 2018CrossRefGoogle Scholar
  15. 15.
    Johnston SK. et al.: ENvironmental Dynamics Underlying Responsive Extreme Survivors (ENDURES) of Glioblastoma: a Multi-disciplinary Team-based. Multifactorial Analytical Approach. bioRxiv 461236, 2018. doi: https://doi.org/10.1101/461236
  16. 16.
    Barnholtz-Sloan JS, Swanson KR: Sex differences in GBM revealed by analysis of patient imaging, transcriptome, and survival data. Sci. Transl. Med, 2019Google Scholar
  17. 17.
    Massey SC et al.: Image-based metric of invasiveness predicts response to adjuvant temozolomide for primary glioblastoma. bioRxiv 509281, 2019. doi: https://doi.org/10.1101/509281
  18. 18.
    Brown RW et al.: Magnetic Resonance Imaging: Physical Principles and Sequence Design. (John Wiley & Sons, 2014Google Scholar
  19. 19.
    Cha S: Neuroimaging in neuro-oncology. Neurotherapeutics 6:465–477, 2009CrossRefGoogle Scholar
  20. 20.
    Armstrong TS, Cohen MZ, Weinberg J, Gilbert MR: Imaging techniques in neuro-oncology. Semin. Oncol. Nurs. 20:231–239, 2004CrossRefGoogle Scholar
  21. 21.
    Krizhevsky A, Sutskever I, Hinton GE: ImageNet classification with deep convolutional neural networks. in Advances in Neural Information Processing Systems 25 (eds. Pereira, F., Burges, C. J. C., Bottou, L. & Weinberger, K. Q.) 1097–1105 (Curran Associates, Inc., 2012Google Scholar
  22. 22.
    Mohsen H, El-Dahshan E-SA, El-Horbaty E-SM, Salem A-BM: Classification using deep learning neural networks for brain tumors. Future Computing and Informatics Journal 3:68–71, 2018CrossRefGoogle Scholar
  23. 23.
    Huynh BQ, Li H, Giger ML: Digital mammographic tumor classification using transfer learning from deep convolutional neural networks. Journal of Medical Imaging 3:034501, 2016CrossRefGoogle Scholar
  24. 24.
    Chen Y-JY-J, Hua K-L, Hsu C-H, Cheng W-H, Hidayati SC: Computer-aided classification of lung nodules on computed tomography images via deep learning technique. OncoTargets and Therapy, 2015.  https://doi.org/10.2147/ott.s80733
  25. 25.
    Simonyan K, Zisserman A: Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv [cs.CV], 2014Google Scholar
  26. 26.
    Chollet F, Others. Keras. 2015Google Scholar
  27. 27.
    Abadi, M. et al.: Tensorflow: A system for large-scale machine learning. in 12th ${USENIX} Symposium on Operating Systems Design and Implementation ({OSDI}$ 16) 265–283, 2016Google Scholar
  28. 28.
    Kingma DP, Adam BJ: A method for stochastic optimization. arXiv [cs.LG], 2014Google Scholar
  29. 29.
    Pedregosa F et al.: Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 12:2825–2830, 2011Google Scholar
  30. 30.
    Gong, Y., Jia, Y., Leung, T., Toshev, A. & Ioffe, S. Deep convolutional ranking for multilabel image annotation. arXiv [cs.CV], 2013Google Scholar
  31. 31.
    Wu, F., Wang Z., Zhang Z., Yang Y., Luo J., Zhu W., Zhuang Y. Weakly Semi-Supervised Deep Learning for Multi-Label Image Annotation. IEEE Transactions on Big Data 1, 109–122 (2015), 109, 122.CrossRefGoogle Scholar
  32. 32.
    Wu J, Yu Y, Huang C, Yu K: Deep multiple instance learning for image classification and auto-annotation. in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 3460–3469, 2015Google Scholar
  33. 33.
    Murthy VN, Maji S, Manmatha R: Automatic Image Annotation Using Deep Learning Representations. in Proceedings of the 5th ACM on International Conference on Multimedia Retrieval 603–606 (ACM), 2015.Google Scholar
  34. 34.
    Ojha U, Adhikari U, Singh DK: Image annotation using deep learning: A review. 2017 International Conference on Intelligent Computing and Control (I2C2), 2017.  https://doi.org/10.1109/i2c2.2017.8321819
  35. 35.
    Hamm CA, Wang CJ, Savic LJ, Ferrante M, Schobert I, Schlachter T, Lin MD, Duncan JS, Weinreb JC, Chapiro J, Letzen B: Deep learning for liver tumor diagnosis part I: development of a convolutional neural network classifier for multi-phasic MRI. Eur. Radiol. 29:3338–3347, 2019.  https://doi.org/10.1007/s00330-019-06205-9 CrossRefPubMedGoogle Scholar
  36. 36.
    Hermessi, H., Mourali, O. & Zagrouba, E. Deep feature learning for soft tissue sarcoma classification in MR images via transfer learning. Expert Systems with Applications 120, (2019), 116, 127.Google Scholar
  37. 37.
    Margeta J, Criminisi A, Cabrera Lozoya R, Lee DC, Ayache N: Fine-tuned convolutional neural nets for cardiac MRI acquisition plane recognition. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization 5:339–349, 2017Google Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2019

Authors and Affiliations

  • Sara Ranjbar
    • 1
    • 2
    Email author
  • Kyle W. Singleton
    • 1
    • 2
  • Pamela R. Jackson
    • 1
    • 2
  • Cassandra R. Rickertsen
    • 1
    • 2
  • Scott A. Whitmire
    • 1
    • 2
  • Kamala R. Clark-Swanson
    • 1
    • 2
  • J. Ross Mitchell
    • 3
  • Kristin R. Swanson
    • 1
    • 2
  • Leland S. Hu
    • 1
    • 4
  1. 1.Mathematical NeuroOncology Lab, Precision Neurotherapeutics Innovation ProgramMayo ClinicPhoenixUSA
  2. 2.Department of NeurosurgeryMayo ClinicPhoenixUSA
  3. 3.Department of Biostatistics and BioinformaticsH. Lee Moffitt Cancer Center and Research InstituteTampaUSA
  4. 4.Department of RadiologyMayo ClinicPhoenixUSA

Personalised recommendations