Hello World Deep Learning in Medical Imaging
There is recent popularity in applying machine learning to medical imaging, notably deep learning, which has achieved state-of-the-art performance in image analysis and processing. The rapid adoption of deep learning may be attributed to the availability of machine learning frameworks and libraries to simplify their use. In this tutorial, we provide a high-level overview of how to build a deep neural network for medical image classification, and provide code that can help those new to the field begin their informatics projects.
KeywordsDeep learning Machine learning Artificial neural networks Medical imaging
Machine learning has sparked tremendous interest over the past few years, particularly deep learning, a branch of machine learning that employs multi-layered neural networks. Deep learning has done remarkably well in image classification and processing tasks, mainly owing to convolutional neural networks (CNN) . Their use became popularized after Drs. Krizhevsky and Hinton used a deep CNN called AlexNet  to win the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC), an international competition for object detection and classification, consisting of 1.2 million everyday color images .
The goal of this paper is to provide a high-level introduction into practical machine learning for purposes of medical image classification. A variety of tutorials exist explaining steps to use CNNs, but the medical literature currently lacks a step-by-step source for those practitioners new to the field in need of instructions and code to build and test a network. There are many different libraries and machine learning frameworks available, including Caffe, MXNet, Tensorflow, Theano, Torch and PyTorch, which have facilitated machine learning research and application development . In this tutorial, we chose to use the Tensorflow framework  (Tensorflow 1.4, Google LLC, Mountain View, CA) as it is currently the most actively used  and the Keras library (Keras v 2.12, https://keras.io/), which a high-level application programming interface that simplifies working with Tensorflow, although one could use other frameworks as well. Currently, Keras also supports Theano, Microsoft Cognitive Toolkit (CNTK), and soon MXNet.
We hope that this tutorial will spark interest and provide a basic starting point for those interested in machine learning in regard to medical imaging. This tutorial assumes basic understanding of CNNs, some Python programming language (Python 3.6, Python Software Foundation, Wilmington DE), and is more of a practical introduction to using the libraries and frameworks. The tutorial will also highlight some important concepts but due to space constraints not cover everything in full detail.
For larger datasets, you will want a computer that contains a graphics processing unit (GPU) that supports the CUDA® Deep Neural Network library (cuDNN) designed for Nvidia GPUs (Nvidia Corp., Santa Clara, CA). This will tremendously speed up training (up to 75 times faster than a CPU) depending on the model of the GPU . However, for smaller datasets, training on a standard central processing unit (CPU) is fine.
This tutorial is performed on a computer containing an Nvidia 1080ti GPU, dual-xeon E5-2670 Intel CPUs, and 64 gb RAM. However, you could perform this experiment on a typical laptop using the CPU only.
A common machine learning classification problem is to differentiate between two categories (e.g., abdominal and chest radiographs). Typically, one would use a larger sample of cases for a machine learning task, but for this tutorial, our dataset consists of 75 images, split roughly in half, with 37 of the abdomen and 38 of the chest. The data is derived from OpenI, a searchable online repository of medical images from published PubMed Central articles, hosted by the National Institutes of Health (https://openi.nlm.nih.gov). For your convenience, we hosted the images on the following SIIM Github repository: https://github.com/ImagingInformatics/machine-learning. These images are in PNG (Portable Network Graphics) format and ready to be utilized by any machine learning framework. For handling Digital Imaging and Communications in Medicine (DICOM) images, a Python library such as PyDicom (http://pydicom.readthedocs.io/en/stable/index.html) may be used to import the images and convert them into a numpy array for use within the Tensorflow framework. With other frameworks such as Caffe, it may be easier to convert the DICOM files to either PNG or Joint Photographic Experts Group (JPEG) format prior to use.
First, randomly divide your images into training and validation. In this example, we put 65 cases into training and 10 into validation. More information regarding principles of splitting and evaluating your model, including more robust methodologies such as cross-validation, are referenced here [8, 9].
Setting Up Your Environment
For this example, we will assume you are running this on your laptop or workstation. You will need a computer running Tensorflow, Keras, and Jupyter Notebook (http://jupyter.org/), an open-source web application that permits creation and sharing of documents with text and live code . To make things easier, there is a convenient SIIM docker that has Tensorflow, Keras, and Jupyterlab already installed available at https://github.com/ImagingInformatics/machine-learning/tree/master/docker-keras-tensorflow-python3-jupyter.
Build the Model
There are many available gradient descent optimization algorithms, which minimize a particular objective function . In the example, we use the Adam  optimizer with commonly used settings (Fig. 4).
More About Transfer Learning
In machine learning, transfer learning refers to application of a process suited for one specific task to a different problem . For example, a machine learning algorithm trained to recognize every day color images, such as animals, could be used to classify radiographs. The idea is that all images share similar features such as edges and blobs, which aids transfer learning. In addition, deep neural networks often require large datasets (in the millions) to properly train. As such, starting with weights from pretrained networks will often perform better than random initialization if using small datasets [14, 15, 16]. In medical imaging classification tasks, this is often the case, as it may be difficult to annotate a large dataset to train from scratch.
There are many strategies for transfer learning, which include freezing layers and training on later layers, and using a low learning rate. Some of this optimization is frequently done by trial and error, so you may have to experiment with different options. For this tutorial, we remove the final (top) fully connected layers of the pretrained Inception V3 model that was intended for a 1000-class problem in ImageNet, and insert a few additional layers with random initialization (Fig. 4), so they can learn from the new medical data provided. We then fine-tune the entire model using a very low learning rate (0.0001), as not to rapidly perturb the weights that are already relatively well optimized.
Image Preprocessing and Augmentation
Training the Model
Evaluating the Trained Model
With only 65 training cases, the power of transfer learning and deep neural networks, we built an accurate classifier that can differentiate chest vs. abdominal radiographs with a small amount of code. The availability of frameworks and high-level libraries has made machine learning more accessible in medical imaging. We hope that this tutorial provides a foundation for those interested in starting with machine learning informatics projects in medical imaging.
The Jupyter Ipython Notebook containing the code to run this tutorial is available on the public SIIM Github repositiory: https://github.com/ImagingInformatics/machine-learning, under “HelloWorldDeepLearning.” A live interactive demo to the model is available at https://public.md.ai/hub/models/public.
- 2.Krizhevsky A, Sutskever I, Hinton GE: Imagenet classification with deep convolutional neural networks. Adv Neural Inf Proces Syst 1097–1105, 2012Google Scholar
- 4.Erickson BJ, Korfiatis P, Akkus Z, Kline T, Philbrick K: Toolkits and Libraries for Deep Learning. J Digit Imaging 17:1–6, 2017Google Scholar
- 5.Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S. Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv preprint arXiv:1603.04467, 2016Google Scholar
- 6.Intel AI Academy. https://software.intel.com/en-us/articles/hands-on-ai-part-5-select-a-deep-learning-framework. Accessed 2, January 2018
- 7.Johnson, J. CNN Benchmarks. https://github.com/jcjohnson/cnn-benchmarks. Accessed January 2, 2018
- 10.Kluyver T, Ragan-Kelley B, Pérez F, Granger BE, Bussonnier M, Frederic J, Kelley K, Hamrick JB, Grout J, Corlay S, Ivanov P: Jupyter Notebooks-a publishing format for reproducible computational workflows. In ELPUB 87-90, 2016Google Scholar
- 11.Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition: 2818–2826, 2016Google Scholar
- 12.Ngiam J, Coates A, Lahiri A, Prochnow B, Le QV, Ng AY. On optimization methods for deep learning. Proceedings of the 28th international conference on machine learning: 265–272, 2011Google Scholar
- 13.Kingma D, Ba JA: A method for stochastic optimization. arXiv preprint arXiv 1412:6980, 2014Google Scholar
- 17.Wu R, Yan S, Shan Y, Dang Q, Sun G: Deep image: Scaling up image recognition. arXiv preprint arXiv 1501:02876, 2015Google Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.