Journal of Digital Imaging

, Volume 29, Issue 4, pp 496–506 | Cite as

Localization of Diagnostically Relevant Regions of Interest in Whole Slide Images: a Comparative Study

  • Ezgi Mercan
  • Selim Aksoy
  • Linda G. Shapiro
  • Donald L. Weaver
  • Tad T. Brunyé
  • Joann G. Elmore


Whole slide digital imaging technology enables researchers to study pathologists’ interpretive behavior as they view digital slides and gain new understanding of the diagnostic medical decision-making process. In this study, we propose a simple yet important analysis to extract diagnostically relevant regions of interest (ROIs) from tracking records using only pathologists’ actions as they viewed biopsy specimens in the whole slide digital imaging format (zooming, panning, and fixating). We use these extracted regions in a visual bag-of-words model based on color and texture features to predict diagnostically relevant ROIs on whole slide images. Using a logistic regression classifier in a cross-validation setting on 240 digital breast biopsy slides and viewport tracking logs of three expert pathologists, we produce probability maps that show 74 % overlap with the actual regions at which pathologists looked. We compare different bag-of-words models by changing dictionary size, visual word definition (patches vs. superpixels), and training data (automatically extracted ROIs vs. manually marked ROIs). This study is a first step in understanding the scanning behaviors of pathologists and the underlying reasons for diagnostic errors.


Digital pathology Medical image analysis Computer vision Region of interest Whole slide imaging 



The research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under award numbers R01 CA172343, R01 CA140560, and KO5 CA104699. The content is solely the responsibility of the authors and does not necessarily represent the views of the National Cancer Institute or the National Institutes of Health. The authors wish to thank Ventana Medical Systems, Inc., a member of the Roche Group, for the use of iScan Coreo Au™ whole slide imaging system and HD View SL for the source code used to build our digital viewer. For a full description of HD View SL, please see Selim Aksoy is supported in part by the Scientific and Technological Research Council of Turkey Grant 113E602.

Compliance with Ethical Standards

The study was approved by the institutional review boards at Dartmouth College, Fred Hutchinson Cancer Research Center, Providence Health and Services Oregon, University of Vermont, University of Washington, and Bilkent University. Informed consent was obtained electronically from pathologists.


  1. 1.
    Brunyé TT, Carney PA, Allison KH, Shapiro LG, Weaver DL, Elmore JG: Eye Movements as an Index of Pathologist Visual Expertise: A Pilot Study. van Diest PJ, ed. PLoS One 98: e103447, 2014Google Scholar
  2. 2.
    Lesgold A, Rubinson H, Feltovich P, Glaser R, Klopfer D, Wang Y: Expertise in a complex skill: Diagnosing x-ray pictures. Nat Exp 311–342: 1988Google Scholar
  3. 3.
    Vink JP, Van Leeuwen MB, Van Deurzen CHM, De Haan G: Efficient nucleus detector in histopathology images. J Microsc 2492:124–135, 2013CrossRefGoogle Scholar
  4. 4.
    Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J: Mitosis detection in breast cancer histology images with deep neural networks. Lecture Notes in Computer Science (including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8150 LNCS., 411–418, 2013Google Scholar
  5. 5.
    Irshad H, Jalali S, Roux L, Racoceanu D, Hwee LJ, Le NG, Capron F: Automated mitosis detection using texture, SIFT features and HMAX biologically inspired approach. J Pathol Inform 4(Suppl):S12, 2013CrossRefPubMedPubMedCentralGoogle Scholar
  6. 6.
    Irshad H, Roux L, Racoceanu D: Multi-channels statistical and morphological features based mitosis detection in breast cancer histopathology. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 6091–6094, 2013Google Scholar
  7. 7.
    Wan T, Liu X, Chen J, Qin Z: Wavelet-based statistical features for distinguishing mitotic and non-mitotic cells in breast cancer histopathology. Image Processing (ICIP), 2014 I.E. International Conference on, 2290–2294, 2014Google Scholar
  8. 8.
    Chekkoury A, Khurd P, Ni J, Bahlmann C, Kamen A, Patel A, Grady L, Singh M, Groher M, Navab N, Krupinski E, Johnson J, Graham A, Weinstein R: Automated malignancy detection in breast histopathological images. Pelc NJ, Haynor DR, van Ginneken B, Holmes III DR, Abbey CK, Boonn W, Bosch JG, Doyley MM, Liu BJ, Mello-Thoms CR, Wong KH, Novak CL, Ourselin S, Nishikawa RM, Whiting BR, eds., SPIE Medical Imaging, 831515–831515 - 13, 2012Google Scholar
  9. 9.
    DiFranco MD, O’Hurley G, Kay EW, Watson RWG, Cunningham P: Ensemble based system for whole-slide prostate cancer probability mapping using color texture features. Comput Med Imaging Graph 357–8:629–645, 2011CrossRefGoogle Scholar
  10. 10.
    Dong F, Irshad H, Oh E-Y, Lerwill MF, Brachtel EF, Jones NC, Knoblauch NW, Montaser-Kouhsari L, Johnson NB, Rao LKF, Faulkner-Jones B, Wilbur DC, Schnitt SJ, Beck AH: Computational Pathology to Discriminate Benign from Malignant Intraductal Proliferations of the Breast. PLoS One 912, e114885, 2014CrossRefGoogle Scholar
  11. 11.
    Doyle S, Agner S, Madabhushi A, Feldman M, Tomaszewski J: Automated grading of breast cancer histopathology using spectral clustering with textural and architectural image features. 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Proceedings, ISBI, 496–499, 2008Google Scholar
  12. 12.
    Doyle S, Feldman M, Tomaszewski J, Madabhushi A: A boosted Bayesian multiresolution classifier for prostate cancer detection from digitized needle biopsies. IEEE Trans Biomed Eng 595:1205–1218, 2012CrossRefGoogle Scholar
  13. 13.
    Jafari-Khouzani K, Soltanian-Zadeh H: Multiwavelet grading of pathological images of prostate. IEEE Trans Biomed Eng 506:697–704, 2003CrossRefGoogle Scholar
  14. 14.
    Khurd P, Grady L, Kamen A, Gibbs-Strauss S, Genega EM, Frangioni J V.: Network cycle features: Application to computer-aided Gleason grading of prostate cancer histopathological images. Proceedings - International Symposium on Biomedical Imaging, 1632–1636, 2011Google Scholar
  15. 15.
    Kong J, Shimada H, Boyer K, Saltz J, Gurcan M: Image analysis for automated assessment of grade of neuroblastic differentiation. 2007 4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro - Proceedings, 61–64, 2007Google Scholar
  16. 16.
    Kong J, Sertel O, Shimada H, Boyer KL, Saltz JH, Gurcan MN: Computer-aided evaluation of neuroblastoma on whole-slide histology images: Classifying grade of neuroblastic differentiation. Pattern Recognit 426:1080–1092, 2009CrossRefGoogle Scholar
  17. 17.
    Sertel O, Kong J, Catalyurek UV, Lozanski G, Saltz JH, Gurcan MN: Histopathological image analysis using model-based intermediate representations and color texture: Follicular lymphoma grading. J Signal Process Syst 551–3:169–183, 2009CrossRefGoogle Scholar
  18. 18.
    Basavanhally A, Ganesan S, Feldman M, Shih N, Mies C, Tomaszewski J, Madabhushi A: Multi-Field-of-View Framework for Distinguishing Tumor Grade in ER #002B; Breast Cancer from Entire Histopathology Slides. Biomed Eng IEEE Trans 608:2089–2099, 2013CrossRefGoogle Scholar
  19. 19.
    Beck AH, Sangoi AR, Leung S, Marinelli RJ, Nielsen TO, van de Vijver MJ, West RB, van de Rijn M, Koller D: Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci Transl Med 3108:108ra113, 2011Google Scholar
  20. 20.
    Cooper LAD, Kong J, Gutman DA, Dunn WD, Nalisnik M, Brat DJ: Novel genotype-phenotype associations in human cancers enabled by advanced molecular platforms and computational analysis of whole slide images. Lab Invest 954:366–376, 2015CrossRefGoogle Scholar
  21. 21.
    Fuchs TJ, Wild PJ, Moch H, Buhmann JM: Computational pathology analysis of tissue microarrays predicts survival of renal clear cell carcinoma patients. Med Image Comput Comput Assist Interv 11(Pt 2):1–8, 2008PubMedGoogle Scholar
  22. 22.
    Kong J, Cooper LAD, Wang F, Gao J, Teodoro G, Scarpace L, Mikkelsen T, Schniederjan MJ, Moreno CS, Saltz JH, Brat DJ: Machine-based morphologic analysis of glioblastoma using whole-slide pathology images uncovers clinically relevant molecular correlates. PLoS One. 811: 2013Google Scholar
  23. 23.
    Chang H, Fontenay GV, Han J, Cong G, Baehner FL, Gray JW, Spellman PT, Parvin B: Morphometic analysis of TCGA glioblastoma multiforme. BMC Bioinforma 121:484, 2011CrossRefGoogle Scholar
  24. 24.
    Bahlmann C, Patel A, Johnson J, Chekkoury A, Khurd P, Kamen A, Grady L, Ni J, Krupinski E, Graham A, Weinstein R: Automated detection of diagnostically relevant regions in H&E stained digital pathology slides. Prog Biomed Opt Imaging - Proc SPIE 8315: 2012Google Scholar
  25. 25.
    Gutiérrez R, Gómez F, Roa-Peña L, Romero E: A supervised visual model for finding regions of interest in basal cell carcinoma images. Diagn Pathol 626, 2011Google Scholar
  26. 26.
    Huang CH, Veillard A, Roux L, Loménie N, Racoceanu D: Time-efficient sparse analysis of histopathological whole slide images. Comput Med Imaging Graph 357–8:579–591, 2011CrossRefGoogle Scholar
  27. 27.
    Romo D, Romero E, González F: Learning regions of interest from low level maps in virtual microscopy. Diagn Pathol 6(Suppl 1):S22, 2011CrossRefPubMedPubMedCentralGoogle Scholar
  28. 28.
    Mercan E, Aksoy S, Shapiro LG, Weaver DL, Brunye T, Elmore JG: Localization of Diagnostically Relevant Regions of Interest in Whole Slide Images. Pattern Recognit (ICPR), 2014 22nd Int Conf 1179–1184, 2014Google Scholar
  29. 29.
    Kothari S, Phan JH, Young AN, Wang MD: Histological image feature mining reveals emergent diagnostic properties for renal cancer. Proceedings - 2011 I.E. International Conference on Bioinformatics and Biomedicine, BIBM 2011, 422–425, 2011Google Scholar
  30. 30.
    Tabesh A, Teverovskiy M, Pang HY, Kumar VP, Verbel D, Kotsianti A, Saidi O: Multifeature prostate cancer diagnosis and gleason grading of histological images. IEEE Trans Med Imaging 2610:1366–1378, 2007CrossRefGoogle Scholar
  31. 31.
    Gunduz-Demir C, Kandemir M, Tosun AB, Sokmensuer C: Automatic segmentation of colon glands using object-graphs. Med Image Anal 141:1–12, 2010CrossRefGoogle Scholar
  32. 32.
    Yuan Y, Failmezger H, Rueda OM, Ali HR, Gräf S, Chin S-F, Schwarz RF, Curtis C, Dunning MJ, Bardwell H, Johnson N, Doyle S, Turashvili G, Provenzano E, Aparicio S, Caldas C, Markowetz F: Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling. Sci Transl Med 4157:157ra143, 2012Google Scholar
  33. 33.
    Lu C, Mahmood M, Jha N, Mandal M: Automated segmentation of the melanocytes in skin histopathological images. IEEE J Biomed Heal Inform 172:284–296, 2013Google Scholar
  34. 34.
    Martins F, de Santiago I, Trinh A, Xian J, Guo A, Sayal K, Jimenez-Linan M, Deen S, Driver K, Mack M, Aslop J, Pharoah PD, Markowetz F, Brenton JD: Combined image and genomic analysis of high-grade serous ovarian cancer reveals PTEN loss as a common driver event and prognostic classifier. Genome Biol 1512: 2014Google Scholar
  35. 35.
    Naik S, Doyle S, Agner S, Madabhushi A, Feldman M, Tomaszewski J: Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology. 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Proceedings, ISBI, 284–287, 2008Google Scholar
  36. 36.
    Mokhtari M, Rezaeian M, Gharibzadeh S, Malekian V: Computer aided measurement of melanoma depth of invasion in microscopic images. Micron 6140–48: 2014Google Scholar
  37. 37.
    Lu C, Mandal M: Automated segmentation and analysis of the epidermis area in skin histopathological images. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS 5355–5359, 2012Google Scholar
  38. 38.
    Itti L, Koch C: Computational modelling of visual attention. Nat Rev Neurosci 23:194–203, 2001CrossRefGoogle Scholar
  39. 39.
    He DC, Wang L: Texture unit, texture spectrum, and texture analysis. IEEE Trans Geosci Remote Sens 284:509–512, 1990Google Scholar
  40. 40.
    Tamura H, Mori S, Yamawaki T: Textural Features Corresponding to Visual Perception. IEEE Trans Syst Man Cybern 86: 1978Google Scholar
  41. 41.
    Oster NV, Carney PA, Allison KH, Weaver DL, Reisch LM, Longton G, Onega T, Pepe M, Geller BM, Nelson HD, Ross TR, Tosteson ANA, Elmore JG: Development of a diagnostic test set to assess agreement in breast pathology: practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS). BMC Womens Health 131:3, 2013CrossRefGoogle Scholar
  42. 42.
    Feng S, Weaver D, Carney P, Reisch L, Geller B, Goodwin A, Rendi M, Onega T, Allison K, Tosteson A, Nelson H, Longton G, Pepe M, Elmore J: A Framework for Evaluating Diagnostic Discordance in Pathology Discovered During Research Studies. Arch Pathol Lab Med 1387:955–961, 2014CrossRefGoogle Scholar
  43. 43.
    Allison KH, Reisch LM, Carney PA, Weaver DL, Schnitt SJ, O’Malley FP, Geller BM, Elmore JG: Understanding diagnostic variability in breast pathology: Lessons learned from an expert consensus review panel. Histopathology 652:240–251, 2014CrossRefGoogle Scholar
  44. 44.
    Elmore JG, Longton GM, Carney PA, Geller BM, Onega T, Tosteson ANA, Nelson HD, Pepe MS, Allison KH, Schnitt SJ, O’Malley FP, Weaver DL: Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 31311:1122–1132, 2015CrossRefGoogle Scholar
  45. 45.
    Sivic J, Zisserman A: Efficient visual search of videos cast as text retrieval. IEEE Trans Pattern Anal Mach Intell 314:591–606, 2009CrossRefGoogle Scholar
  46. 46.
    Ruifrok AC, Johnston DA: Quantification of histochemical staining by color deconvolution. Anal Quant Cytol Histol 234:291–299, 2001Google Scholar
  47. 47.
    Ren X, Malik J: Learning a classification model for segmentation. Proc Ninth IEEE Int Conf Comput Vis 2003Google Scholar
  48. 48.
    Bejnordi BE, Litjens G, Hermsen M, Karssemeijer N, van der Laak JAWM: A multi-scale superpixel classification approach to the detection of regions of interest in whole slide histopathology images. 9420., 94200H - 94200H - 6, 2015Google Scholar
  49. 49.
    Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 3411:2274–2281, 2012CrossRefGoogle Scholar

Copyright information

© Society for Imaging Informatics in Medicine 2016

Authors and Affiliations

  • Ezgi Mercan
    • 1
  • Selim Aksoy
    • 2
  • Linda G. Shapiro
    • 1
  • Donald L. Weaver
    • 3
  • Tad T. Brunyé
    • 4
  • Joann G. Elmore
    • 5
  1. 1.Department of Computer Science & Engineering, Paul G. Allen Center for ComputingUniversity of WashingtonSeattleUSA
  2. 2.Department of Computer EngineeringBilkent UniversityAnkaraTurkey
  3. 3.Department of PathologyUniversity of VermontBurlingtonUSA
  4. 4.Department of PsychologyTufts UniversityMedfordUSA
  5. 5.Department of MedicineUniversity of WashingtonSeattleUSA

Personalised recommendations