Cancer Metastasis Detection via Spatially Structured Deep Network

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10265)


Metastasis detection of lymph nodes in Whole-slide Images (WSIs) plays a critical role in the diagnosis of breast cancer. Automatic metastasis detection is a challenging issue due to the large variance of their appearances and the size of WSIs. Recently, deep neural networks have been employed to detect cancer metastases by dividing the WSIs into small image patches. However, most existing works simply treat these patches independently and do not consider the structural information among them. In this paper, we propose a novel deep neural network, namely Spatially Structured Network (Spatio-Net) to tackle the metastasis detection problem in WSIs. By integrating the Convolutional Neural Network (CNN) with the 2D Long-Short Term Memory (2D-LSTM), our Spatio-Net is able to learn the appearances and spatial dependencies of image patches effectively. Specifically, the CNN encodes each image patch into a compact feature vector, and the 2D-LSTM layers provide the classification results (i.e., normal or tumor), considering its dependencies on other relevant image patches. Moreover, a new loss function is designed to constrain the structure of the output labels, which further improves the performance. Finally, the metastasis positions are obtained by locating the regions with high tumor probabilities in the resulting accurate probability map. The proposed method is validated on hundreds of WSIs, and the accuracy is significantly improved, in comparison with a state-of-the-art baseline that does not have the spatial dependency constraint.


Structure Constraint Image Patch Hide State Convolutional Neural Network Deep Neural Network 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Apou, G., Naegel, B., Forestier, G., Feuerhake, F., Wemmert, C.: Efficient region-based classification for whole slide images. In: Battiato, S., Coquillart, S., Pettré, J., Laramee, R.S., Kerren, A., Braz, J. (eds.) VISIGRAPP 2014. CCIS, vol. 550, pp. 239–256. Springer, Cham (2015). doi: 10.1007/978-3-319-25117-2_15 CrossRefGoogle Scholar
  2. 2.
    Doyle, S., Agner, S., Madabhushi, A., Feldman, M., Tomaszewski, J.: Automated grading of breast cancer histopathology using spectral clustering with textural and architectural image features. In: 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, pp. 496–499. IEEE (2008)Google Scholar
  3. 3.
    Geçer, B.: Detection and classification of breast cancer in whole slide histopathology images using deep convolutional networks. Ph.D. thesis, Bilkent University (2016)Google Scholar
  4. 4.
    Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016, in preparation).
  5. 5.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar
  6. 6.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  7. 7.
    Hou, L., Samaras, D., Kurc, T.M., Gao, Y., Davis, J.E., Saltz, J.H.: Patch-based convolutional neural network for whole slide tissue image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2424–2433 (2016)Google Scholar
  8. 8.
    Howlader, N., Noone, A., Krapcho, M., Garshell, J., Neyman, N., Altekruse, S., Kosary, C., Yu, M., Ruhl, J., Tatalovich, Z., et al.: SEER Cancer Statistics Review, 1975–2010. National Cancer Institute, Bethesda (2013)Google Scholar
  9. 9.
    Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T.: Caffe: convolutional architecture for fast feature embedding. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 675–678. ACM (2014)Google Scholar
  10. 10.
    Kalchbrenner, N., Danihelka, I., Graves, A.: Grid long short-term memory. arXiv preprint arXiv:1507.01526 (2015)
  11. 11.
    Kong, B., Zhan, Y., Shin, M., Denny, T., Zhang, S.: Recognizing end-diastole and end-systole frames via deep temporal regression network. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9902, pp. 264–272. Springer, Cham (2016). doi: 10.1007/978-3-319-46726-9_31 CrossRefGoogle Scholar
  12. 12.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)Google Scholar
  13. 13.
    Liang, X., Shen, X., Feng, J., Lin, L., Yan, S.: Semantic object parsing with graph LSTM. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9905, pp. 125–143. Springer, Cham (2016). doi: 10.1007/978-3-319-46448-0_8 CrossRefGoogle Scholar
  14. 14.
    Liang, X., Shen, X., Xiang, D., Feng, J., Lin, L., Yan, S.: Semantic object parsing with local-global long short-term memory. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3185–3193 (2016)Google Scholar
  15. 15.
    Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)Google Scholar
  16. 16.
    van den Oord, A., Kalchbrenner, N., Kavukcuoglu, K.: Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759 (2016)
  17. 17.
    Peng, Z., Zhang, R., Liang, X., Lin, L.: Geometric scene parsing with hierarchical LSTM. arXiv preprint arXiv:1604.01931 (2016)
  18. 18.
    Shiraishi, J., Li, Q., Suzuki, K., Engelmann, R., Doi, K.: Computer-aided diagnostic scheme for the detection of lung nodules on chest radiographs: localized search method based on anatomical classification. Med. Phys. 33(7), 2642–2653 (2006)CrossRefGoogle Scholar
  19. 19.
    Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)Google Scholar
  20. 20.
    Wang, D., Khosla, A., Gargeya, R., Irshad, H., Beck, A.H.: Deep learning for identifying metastatic breast cancer. arXiv preprint arXiv:1606.05718 (2016)
  21. 21.
    Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 818–833. Springer, Cham (2014). doi: 10.1007/978-3-319-10590-1_53 Google Scholar
  22. 22.
    Zhang, X., Liu, W., Dundar, M., Badve, S., Zhang, S.: Towards large-scale histopathological image analysis: hashing-based image retrieval. IEEE Trans. Med. Imaging 34(2), 496–506 (2015)CrossRefGoogle Scholar
  23. 23.
    Zhang, X., Xing, F., Su, H., Yang, L., Zhang, S.: High-throughput histopathological image analysis via robust cell segmentation and hashing. Med. Image Anal. 26(1), 306–315 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Computer ScienceUNC CharlotteCharlotteUSA
  2. 2.CuraCloud CorporationSeattleUSA

Personalised recommendations