Skip to main content

Advertisement

SpringerLink
Log in

Localization of Diagnostically Relevant Regions of Interest in Whole Slide Images: a Comparative Study

  • Published:
Journal of Digital Imaging Aims and scope Submit manuscript

Abstract

Whole slide digital imaging technology enables researchers to study pathologists’ interpretive behavior as they view digital slides and gain new understanding of the diagnostic medical decision-making process. In this study, we propose a simple yet important analysis to extract diagnostically relevant regions of interest (ROIs) from tracking records using only pathologists’ actions as they viewed biopsy specimens in the whole slide digital imaging format (zooming, panning, and fixating). We use these extracted regions in a visual bag-of-words model based on color and texture features to predict diagnostically relevant ROIs on whole slide images. Using a logistic regression classifier in a cross-validation setting on 240 digital breast biopsy slides and viewport tracking logs of three expert pathologists, we produce probability maps that show 74 % overlap with the actual regions at which pathologists looked. We compare different bag-of-words models by changing dictionary size, visual word definition (patches vs. superpixels), and training data (automatically extracted ROIs vs. manually marked ROIs). This study is a first step in understanding the scanning behaviors of pathologists and the underlying reasons for diagnostic errors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Brunyé TT, Carney PA, Allison KH, Shapiro LG, Weaver DL, Elmore JG: Eye Movements as an Index of Pathologist Visual Expertise: A Pilot Study. van Diest PJ, ed. PLoS One 98: e103447, 2014

  2. Lesgold A, Rubinson H, Feltovich P, Glaser R, Klopfer D, Wang Y: Expertise in a complex skill: Diagnosing x-ray pictures. Nat Exp 311–342: 1988

  3. Vink JP, Van Leeuwen MB, Van Deurzen CHM, De Haan G: Efficient nucleus detector in histopathology images. J Microsc 2492:124–135, 2013

    Article  Google Scholar 

  4. Cireşan DC, Giusti A, Gambardella LM, Schmidhuber J: Mitosis detection in breast cancer histology images with deep neural networks. Lecture Notes in Computer Science (including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8150 LNCS., 411–418, 2013

  5. Irshad H, Jalali S, Roux L, Racoceanu D, Hwee LJ, Le NG, Capron F: Automated mitosis detection using texture, SIFT features and HMAX biologically inspired approach. J Pathol Inform 4(Suppl):S12, 2013

    Article  PubMed  PubMed Central  Google Scholar 

  6. Irshad H, Roux L, Racoceanu D: Multi-channels statistical and morphological features based mitosis detection in breast cancer histopathology. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, 6091–6094, 2013

  7. Wan T, Liu X, Chen J, Qin Z: Wavelet-based statistical features for distinguishing mitotic and non-mitotic cells in breast cancer histopathology. Image Processing (ICIP), 2014 I.E. International Conference on, 2290–2294, 2014

  8. Chekkoury A, Khurd P, Ni J, Bahlmann C, Kamen A, Patel A, Grady L, Singh M, Groher M, Navab N, Krupinski E, Johnson J, Graham A, Weinstein R: Automated malignancy detection in breast histopathological images. Pelc NJ, Haynor DR, van Ginneken B, Holmes III DR, Abbey CK, Boonn W, Bosch JG, Doyley MM, Liu BJ, Mello-Thoms CR, Wong KH, Novak CL, Ourselin S, Nishikawa RM, Whiting BR, eds., SPIE Medical Imaging, 831515–831515 - 13, 2012

  9. DiFranco MD, O’Hurley G, Kay EW, Watson RWG, Cunningham P: Ensemble based system for whole-slide prostate cancer probability mapping using color texture features. Comput Med Imaging Graph 357–8:629–645, 2011

    Article  Google Scholar 

  10. Dong F, Irshad H, Oh E-Y, Lerwill MF, Brachtel EF, Jones NC, Knoblauch NW, Montaser-Kouhsari L, Johnson NB, Rao LKF, Faulkner-Jones B, Wilbur DC, Schnitt SJ, Beck AH: Computational Pathology to Discriminate Benign from Malignant Intraductal Proliferations of the Breast. PLoS One 912, e114885, 2014

    Article  Google Scholar 

  11. Doyle S, Agner S, Madabhushi A, Feldman M, Tomaszewski J: Automated grading of breast cancer histopathology using spectral clustering with textural and architectural image features. 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Proceedings, ISBI, 496–499, 2008

  12. Doyle S, Feldman M, Tomaszewski J, Madabhushi A: A boosted Bayesian multiresolution classifier for prostate cancer detection from digitized needle biopsies. IEEE Trans Biomed Eng 595:1205–1218, 2012

    Article  Google Scholar 

  13. Jafari-Khouzani K, Soltanian-Zadeh H: Multiwavelet grading of pathological images of prostate. IEEE Trans Biomed Eng 506:697–704, 2003

    Article  Google Scholar 

  14. Khurd P, Grady L, Kamen A, Gibbs-Strauss S, Genega EM, Frangioni J V.: Network cycle features: Application to computer-aided Gleason grading of prostate cancer histopathological images. Proceedings - International Symposium on Biomedical Imaging, 1632–1636, 2011

  15. Kong J, Shimada H, Boyer K, Saltz J, Gurcan M: Image analysis for automated assessment of grade of neuroblastic differentiation. 2007 4th IEEE International Symposium on Biomedical Imaging: From Nano to Macro - Proceedings, 61–64, 2007

  16. Kong J, Sertel O, Shimada H, Boyer KL, Saltz JH, Gurcan MN: Computer-aided evaluation of neuroblastoma on whole-slide histology images: Classifying grade of neuroblastic differentiation. Pattern Recognit 426:1080–1092, 2009

    Article  Google Scholar 

  17. Sertel O, Kong J, Catalyurek UV, Lozanski G, Saltz JH, Gurcan MN: Histopathological image analysis using model-based intermediate representations and color texture: Follicular lymphoma grading. J Signal Process Syst 551–3:169–183, 2009

    Article  Google Scholar 

  18. Basavanhally A, Ganesan S, Feldman M, Shih N, Mies C, Tomaszewski J, Madabhushi A: Multi-Field-of-View Framework for Distinguishing Tumor Grade in ER #002B; Breast Cancer from Entire Histopathology Slides. Biomed Eng IEEE Trans 608:2089–2099, 2013

    Article  Google Scholar 

  19. Beck AH, Sangoi AR, Leung S, Marinelli RJ, Nielsen TO, van de Vijver MJ, West RB, van de Rijn M, Koller D: Systematic analysis of breast cancer morphology uncovers stromal features associated with survival. Sci Transl Med 3108:108ra113, 2011

    Google Scholar 

  20. Cooper LAD, Kong J, Gutman DA, Dunn WD, Nalisnik M, Brat DJ: Novel genotype-phenotype associations in human cancers enabled by advanced molecular platforms and computational analysis of whole slide images. Lab Invest 954:366–376, 2015

    Article  Google Scholar 

  21. Fuchs TJ, Wild PJ, Moch H, Buhmann JM: Computational pathology analysis of tissue microarrays predicts survival of renal clear cell carcinoma patients. Med Image Comput Comput Assist Interv 11(Pt 2):1–8, 2008

    PubMed  Google Scholar 

  22. Kong J, Cooper LAD, Wang F, Gao J, Teodoro G, Scarpace L, Mikkelsen T, Schniederjan MJ, Moreno CS, Saltz JH, Brat DJ: Machine-based morphologic analysis of glioblastoma using whole-slide pathology images uncovers clinically relevant molecular correlates. PLoS One. 811: 2013

  23. Chang H, Fontenay GV, Han J, Cong G, Baehner FL, Gray JW, Spellman PT, Parvin B: Morphometic analysis of TCGA glioblastoma multiforme. BMC Bioinforma 121:484, 2011

    Article  Google Scholar 

  24. Bahlmann C, Patel A, Johnson J, Chekkoury A, Khurd P, Kamen A, Grady L, Ni J, Krupinski E, Graham A, Weinstein R: Automated detection of diagnostically relevant regions in H&E stained digital pathology slides. Prog Biomed Opt Imaging - Proc SPIE 8315: 2012

  25. Gutiérrez R, Gómez F, Roa-Peña L, Romero E: A supervised visual model for finding regions of interest in basal cell carcinoma images. Diagn Pathol 626, 2011

  26. Huang CH, Veillard A, Roux L, Loménie N, Racoceanu D: Time-efficient sparse analysis of histopathological whole slide images. Comput Med Imaging Graph 357–8:579–591, 2011

    Article  Google Scholar 

  27. Romo D, Romero E, González F: Learning regions of interest from low level maps in virtual microscopy. Diagn Pathol 6(Suppl 1):S22, 2011

    Article  PubMed  PubMed Central  Google Scholar 

  28. Mercan E, Aksoy S, Shapiro LG, Weaver DL, Brunye T, Elmore JG: Localization of Diagnostically Relevant Regions of Interest in Whole Slide Images. Pattern Recognit (ICPR), 2014 22nd Int Conf 1179–1184, 2014

  29. Kothari S, Phan JH, Young AN, Wang MD: Histological image feature mining reveals emergent diagnostic properties for renal cancer. Proceedings - 2011 I.E. International Conference on Bioinformatics and Biomedicine, BIBM 2011, 422–425, 2011

  30. Tabesh A, Teverovskiy M, Pang HY, Kumar VP, Verbel D, Kotsianti A, Saidi O: Multifeature prostate cancer diagnosis and gleason grading of histological images. IEEE Trans Med Imaging 2610:1366–1378, 2007

    Article  Google Scholar 

  31. Gunduz-Demir C, Kandemir M, Tosun AB, Sokmensuer C: Automatic segmentation of colon glands using object-graphs. Med Image Anal 141:1–12, 2010

    Article  Google Scholar 

  32. Yuan Y, Failmezger H, Rueda OM, Ali HR, Gräf S, Chin S-F, Schwarz RF, Curtis C, Dunning MJ, Bardwell H, Johnson N, Doyle S, Turashvili G, Provenzano E, Aparicio S, Caldas C, Markowetz F: Quantitative image analysis of cellular heterogeneity in breast tumors complements genomic profiling. Sci Transl Med 4157:157ra143, 2012

    Google Scholar 

  33. Lu C, Mahmood M, Jha N, Mandal M: Automated segmentation of the melanocytes in skin histopathological images. IEEE J Biomed Heal Inform 172:284–296, 2013

    Google Scholar 

  34. Martins F, de Santiago I, Trinh A, Xian J, Guo A, Sayal K, Jimenez-Linan M, Deen S, Driver K, Mack M, Aslop J, Pharoah PD, Markowetz F, Brenton JD: Combined image and genomic analysis of high-grade serous ovarian cancer reveals PTEN loss as a common driver event and prognostic classifier. Genome Biol 1512: 2014

  35. Naik S, Doyle S, Agner S, Madabhushi A, Feldman M, Tomaszewski J: Automated gland and nuclei segmentation for grading of prostate and breast cancer histopathology. 2008 5th IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Proceedings, ISBI, 284–287, 2008

  36. Mokhtari M, Rezaeian M, Gharibzadeh S, Malekian V: Computer aided measurement of melanoma depth of invasion in microscopic images. Micron 6140–48: 2014

  37. Lu C, Mandal M: Automated segmentation and analysis of the epidermis area in skin histopathological images. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS 5355–5359, 2012

  38. Itti L, Koch C: Computational modelling of visual attention. Nat Rev Neurosci 23:194–203, 2001

    Article  Google Scholar 

  39. He DC, Wang L: Texture unit, texture spectrum, and texture analysis. IEEE Trans Geosci Remote Sens 284:509–512, 1990

    Google Scholar 

  40. Tamura H, Mori S, Yamawaki T: Textural Features Corresponding to Visual Perception. IEEE Trans Syst Man Cybern 86: 1978

  41. Oster NV, Carney PA, Allison KH, Weaver DL, Reisch LM, Longton G, Onega T, Pepe M, Geller BM, Nelson HD, Ross TR, Tosteson ANA, Elmore JG: Development of a diagnostic test set to assess agreement in breast pathology: practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS). BMC Womens Health 131:3, 2013

    Article  Google Scholar 

  42. Feng S, Weaver D, Carney P, Reisch L, Geller B, Goodwin A, Rendi M, Onega T, Allison K, Tosteson A, Nelson H, Longton G, Pepe M, Elmore J: A Framework for Evaluating Diagnostic Discordance in Pathology Discovered During Research Studies. Arch Pathol Lab Med 1387:955–961, 2014

    Article  Google Scholar 

  43. Allison KH, Reisch LM, Carney PA, Weaver DL, Schnitt SJ, O’Malley FP, Geller BM, Elmore JG: Understanding diagnostic variability in breast pathology: Lessons learned from an expert consensus review panel. Histopathology 652:240–251, 2014

    Article  Google Scholar 

  44. Elmore JG, Longton GM, Carney PA, Geller BM, Onega T, Tosteson ANA, Nelson HD, Pepe MS, Allison KH, Schnitt SJ, O’Malley FP, Weaver DL: Diagnostic concordance among pathologists interpreting breast biopsy specimens. JAMA 31311:1122–1132, 2015

    Article  Google Scholar 

  45. Sivic J, Zisserman A: Efficient visual search of videos cast as text retrieval. IEEE Trans Pattern Anal Mach Intell 314:591–606, 2009

    Article  Google Scholar 

  46. Ruifrok AC, Johnston DA: Quantification of histochemical staining by color deconvolution. Anal Quant Cytol Histol 234:291–299, 2001

    Google Scholar 

  47. Ren X, Malik J: Learning a classification model for segmentation. Proc Ninth IEEE Int Conf Comput Vis 2003

  48. Bejnordi BE, Litjens G, Hermsen M, Karssemeijer N, van der Laak JAWM: A multi-scale superpixel classification approach to the detection of regions of interest in whole slide histopathology images. 9420., 94200H - 94200H - 6, 2015

  49. Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Süsstrunk S: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans Pattern Anal Mach Intell 3411:2274–2281, 2012

    Article  Google Scholar 

Download references

Acknowledgments

The research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under award numbers R01 CA172343, R01 CA140560, and KO5 CA104699. The content is solely the responsibility of the authors and does not necessarily represent the views of the National Cancer Institute or the National Institutes of Health. The authors wish to thank Ventana Medical Systems, Inc., a member of the Roche Group, for the use of iScan Coreo Au™ whole slide imaging system and HD View SL for the source code used to build our digital viewer. For a full description of HD View SL, please see http://hdviewsl.codeplex.com/. Selim Aksoy is supported in part by the Scientific and Technological Research Council of Turkey Grant 113E602.

Author information

Authors and Affiliations

  1. Department of Computer Science & Engineering, Paul G. Allen Center for Computing, University of Washington, 185 Stevens Way, Seattle, WA, 98195, USA

    Ezgi Mercan & Linda G. Shapiro

  2. Department of Computer Engineering, Bilkent University, Bilkent, 06800, Ankara, Turkey

    Selim Aksoy

  3. Department of Pathology, University of Vermont, Burlington, VT, 05405, USA

    Donald L. Weaver

  4. Department of Psychology, Tufts University, Medford, MA, 02155, USA

    Tad T. Brunyé

  5. Department of Medicine, University of Washington, Seattle, WA, 98195, USA

    Joann G. Elmore

Authors
  1. Ezgi Mercan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Selim Aksoy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linda G. Shapiro
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Donald L. Weaver
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tad T. Brunyé
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Joann G. Elmore
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ezgi Mercan.

Ethics declarations

The study was approved by the institutional review boards at Dartmouth College, Fred Hutchinson Cancer Research Center, Providence Health and Services Oregon, University of Vermont, University of Washington, and Bilkent University. Informed consent was obtained electronically from pathologists.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mercan, E., Aksoy, S., Shapiro, L.G. et al. Localization of Diagnostically Relevant Regions of Interest in Whole Slide Images: a Comparative Study. J Digit Imaging 29, 496–506 (2016). https://doi.org/10.1007/s10278-016-9873-1

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10278-016-9873-1

Keywords

Search

Navigation