Multiple Instance Learning for Heterogeneous Images: Training a CNN for Histopathology

  • Heather D. CoutureEmail author
  • J. S. Marron
  • Charles M. Perou
  • Melissa A. Troester
  • Marc Niethammer
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11071)


Multiple instance (MI) learning with a convolutional neural network enables end-to-end training in the presence of weak image-level labels. We propose a new method for aggregating predictions from smaller regions of the image into an image-level classification by using the quantile function. The quantile function provides a more complete description of the heterogeneity within each image, improving image-level classification. We also adapt image augmentation to the MI framework by randomly selecting cropped regions on which to apply MI aggregation during each epoch of training. This provides a mechanism to study the importance of MI learning. We validate our method on five different classification tasks for breast tumor histology and provide a visualization method for interpreting local image classifications that could lead to future insights into tumor heterogeneity.



This work was supported by a grant from the UNC Lineberger Comprehensive Cancer Center funded by the University Cancer Research Fund (LCCC2017T204), NCI Breast SPORE program (P50-CA58223), and the Breast Cancer Research Foundation.


  1. 1.
    Andrews, S., Tsochantaridis, I., Hofmann, T.: Support vector machines for multiple-instance learning. In: NIPS, pp. 561–568 (2002)Google Scholar
  2. 2.
    Broadhurst, R.E.: Compact appearance in object populations using quantile function based distribution families. Ph.D. thesis, The University of North Carolina at Chapel Hill (2008)Google Scholar
  3. 3.
    Hiley, C.T., Swanton, C.: Spatial and temporal cancer evolution: causes and consequences of tumour diversity. Clin. Med. 14(Suppl–6), s33–s37 (2014)CrossRefGoogle Scholar
  4. 4.
    Hou, L., Samaras, D., et al.: Patch-based convolutional neural network for whole slide tissue image classification. In: CVPR (2016)Google Scholar
  5. 5.
    Jia, Z., Huang, X., Chang, E.I.C., Xu, Y.: Constrained deep weak supervision for histopathology image segmentation. arXiv preprint: 1701.00794 (2017)Google Scholar
  6. 6.
    Kandemir, M., Hamprecht, F.A.F.: Computer-aided diagnosis from weak supervision: a benchmarking study. Comput. Med. Imaging Graph. 42, 44–50 (2014)CrossRefGoogle Scholar
  7. 7.
    Kraus, O.Z., Ba, J.L., Frey, B.J.: Classifying and segmenting microscopy images with deep multiple instance learning. Bioinformatics 32(12), i52–i59 (2016)CrossRefGoogle Scholar
  8. 8.
    Krizhevsky, A., Sutskever, I., Hinton, G.: Imagenet classification with deep convolutional neural networks. In: NIPS, pp. 1106–1114 (2012)Google Scholar
  9. 9.
    McGranahan, N., Swanton, C.: Biological and therapeutic impact of intratumor heterogeneity in cancer evolution. Cancer Cell 27(1), 15–26 (2015)CrossRefGoogle Scholar
  10. 10.
    Niethammer, M., Borland, D., Marron, J.S., Woosley, J., Thomas, N.E.: Appearance normalization of histology slides. In: Wang, F., Yan, P., Suzuki, K., Shen, D. (eds.) MLMI 2010. LNCS, vol. 6357, pp. 58–66. Springer, Heidelberg (2010). Scholar
  11. 11.
    Parker, J.S., Mullins, M.: Supervised risk predictor of breast cancer based on intrinsic subtypes. J. Clin. Oncol. 27(8), 1160–1167 (2009)CrossRefGoogle Scholar
  12. 12.
    Sun, M., Han, T.X., Liu, M.C., Khodayari-Rostamabad, A.: Multiple instance learning convolutional neural networks for object recognition. In: ICPR (2016)Google Scholar
  13. 13.
    Troester, M., Sun, X., et al.: Racial differences in PAM50 subtypes in the Carolina breast cancer study. J. Natl. Cancer Inst. 110(2), 176–182 (2018)CrossRefGoogle Scholar
  14. 14.
    Vanwinckelen, G., do Tragante, O.V., Fierens, D., Blockeel, H.: Instance-level accuracy versus bag-level accuracy in multi-instance learning. Data Min. Knowl. Discov. 30(2), 313–341 (2016)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Wang, X., Yan, Y., Tang, P., Bai, X., Liu, W.: Revisiting Multiple Instance Neural Networks. Pattern Recognit. 74, 15–24 (2018)CrossRefGoogle Scholar
  16. 16.
    Xu, Y., Zhu, J.Y.: Weakly supervised histopathology cancer image segmentation and classification. Med. Image Anal. 18(3), 591–604 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Heather D. Couture
    • 1
    Email author
  • J. S. Marron
    • 2
    • 3
  • Charles M. Perou
    • 2
    • 4
  • Melissa A. Troester
    • 2
    • 5
  • Marc Niethammer
    • 1
    • 6
  1. 1.Department of Computer ScienceUniversity of North CarolinaChapel HillUSA
  2. 2.Lineberger Comprehensive Cancer CenterUniversity of North CarolinaChapel HillUSA
  3. 3.Department of Statistics and Operations ResearchUniversity of North CarolinaChapel HillUSA
  4. 4.Department of GeneticsUniversity of North CarolinaChapel HillUSA
  5. 5.Department of EpidemiologyUniversity of North CarolinaChapel HillUSA
  6. 6.Biomedical Research Imaging CenterUniversity of North CarolinaChapel HillUSA

Personalised recommendations