Skip to main content
SpringerLink
Log in

Applying Machine Learning for Integration of Multi-Modal Genomics Data and Imaging Data to Quantify Heterogeneity in Tumour Tissues

  • Protocol
  • First Online:
Book cover Artificial Neural Networks

Part of the book series: Methods in Molecular Biology ((MIMB,volume 2190))

Abstract

With rapid advances in experimental instruments and protocols, imaging and sequencing data are being generated at an unprecedented rate contributing significantly to the current and coming big biomedical data. Meanwhile, unprecedented advances in computational infrastructure and analysis algorithms are realizing image-based digital diagnosis not only in radiology and cardiology but also oncology and other diseases. Machine learning methods, especially deep learning techniques, are already and broadly implemented in diverse technological and industrial sectors, but their applications in healthcare are just starting. Uniquely in biomedical research, a vast potential exists to integrate genomics data with histopathological imaging data. The integration has the potential to extend the pathologist’s limits and boundaries, which may create breakthroughs in diagnosis, treatment, and monitoring at molecular and tissue levels. Moreover, the applications of genomics data are realizing the potential for personalized medicine, making diagnosis, treatment, monitoring, and prognosis more accurate. In this chapter, we discuss machine learning methods readily available for digital pathology applications, new prospects of integrating spatial genomics data on tissues with tissue morphology, and frontier approaches to combining genomics data with pathological imaging data. We present perspectives on how artificial intelligence can be synergized with molecular genomics and imaging to make breakthroughs in biomedical and translational research for computer-aided applications.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Protocol
USD 49.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ramilowski JA, Goldberg T, Harshbarger J, Kloppman E, Lizio M, Satagopam VP, Itoh M, Kawaji H, Carninci P, Rost B, Forrest ARR (2015) A draft network of ligand-receptor-mediated multicellular signalling in human. Nat Commun 6(1):7866

    Article  CAS  PubMed  Google Scholar 

  2. Puram SV, Tirosh I, Parikh AS, Patel AP, Yizhak K, Gillespie S, Rodman C, Luo CL, Mroz EA, Emerick KS, Deschler DG, Varvares MA, Mylvaganam R, Rozenblatt-Rosen O, Rocco JW, Faquin WC, Lin DT, Regev A, Bernstein BE (2017) Single-cell transcriptomic analysis of primary and metastatic tumor ecosystems in head and neck cancer. Cell 171(7):1611–1624.e24

    Google Scholar 

  3. Boutros C, Tarhini A, Routier E, Lambotte O, Ladurie FL, Carbonnel F, Izzeddine H, Marabelle A, Champiat S, Berdelou A, Lanoy E, Texier M, Libenciuc C, Eggermont AMM, Soria JC, Mateus C, Robert C (2016) Safety profiles of anti-CTLA-4 and anti-PD-1 antibodies alone and in combination. Nat Rev Clin Oncol 13(8):473–486. http://search.proquest.com/docview/1806076231/. Accessed 7 Dec 2019

  4. Galon J, Costes A, Sanchez-Cabo F, Kirilovsky A, Mlecnik B, Lagorce-Pagès C, Tosolini M, Camus M, Berger A, Wind P, Zinzindohoué F, Bruneval P, Cugnenc PH, Trajanoski Z, Fridman WH, Pagès F (2006) Type, density, and location of immune cells within human colorectal tumors predict clinical outcome. Science 313(5795):1960

    Article  CAS  PubMed  Google Scholar 

  5. Kivisaari A, Kähäri VM (2013) Squamous cell carcinoma of the skin: emerging need for novel biomarkers. World J Clin Oncol 4(4):85

    Article  PubMed  PubMed Central  Google Scholar 

  6. Uhlen M, Zhang C, Lee S, Sjöstedt E, Fagerberg L, Bidkhori G, Benfeitas R, Arif M, Liu Z, Edfors F, Sanli K, Von Feilitzen K, Oksvold P, Lundberg E, Hober S, Nilsson P, Mattsson J, Schwenk JM, Brunnström H, Glimelius B, Sjöblom T, Edqvist PH, Djureinovic D, Micke P, Lindskog C, Mardinoglu A, Ponten F (2017) A pathology atlas of the human cancer transcriptome. Science 357(6352):pii: eaan2507

    Google Scholar 

  7. Ståhl PL, Salmén F, Vickovic S, Lundmark A, Navarro JF, Magnusson J, Giacomello S, Asp M, Westholm JO, Huss M, Mollbrink A, Linnarsson S, Codeluppi S, Borg k, Pontén F, Costea PI, Sahlén P, Mulder J, Bergmann O, Lundeberg J, Frisén J (2016) Visualization and analysis of gene expression in tissue sections by spatial transcriptomics. Science 353(6294):78

    Google Scholar 

  8. Lecun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436–444. http://search.proquest.com/docview/1684430894/. Accessed 7 Dec 2019

  9. Schmidhuber J (2015) Deep learning in neural networks: an overview. Neural Netw 61:85–117

    Article  PubMed  Google Scholar 

  10. Libbrecht MW, Noble WS (2015) Machine learning applications in genetics and genomics. Nat Rev Gen 16(6):321–332

    Article  CAS  Google Scholar 

  11. Hu Z, Tang J, Wang Z, Zhang K, Zhang L, Sun Q (2018) Deep learning for image-based cancer detection and diagnosis − A survey. Pattern Recognition 83:134–149

    Google Scholar 

  12. Eraslan G, Avsec Ž, Gagneur J, Theis FJ (2019) Deep learning: new computational modelling techniques for genomics. Nat Rev Genet 20(7):389–403

    Google Scholar 

  13. Wei JW, Tafe LJ, Linnik YA, Vaickus LJ, Tomita N, Hassanpour S (2019) Pathologist-level classification of histologic patterns on resected lung adenocarcinoma slides with deep neural networks. Scientific Reports 9(1):3358

    Google Scholar 

  14. Sun D, Wang M, Li A (2019) A multimodal deep neural network for human breast cancer prognosis prediction by integrating multi-dimensional data. IEEE/ACM Trans Comput Biol Bioinform 16(3):841–850

    Article  Google Scholar 

  15. Ching T, Zhu X, Garmire L (2018) Cox-nnet: an artificial neural network method for prognosis prediction of high-throughput omics data. Plos Comput Biol 14(4):e1006076

    Article  PubMed  PubMed Central  Google Scholar 

  16. Krizhevsky A, Sutskever I, Hinton G (2017) ImageNet classification with deep convolutional neural networks. Commun ACM 60(6):84–90

    Article  Google Scholar 

  17. Xing F, Xie Y, Yang L (2016) An automatic learning-based framework for robust nucleus segmentation. IEEE Trans Med Imag 35(2):550–566

    Article  Google Scholar 

  18. Shen D, Wu G, Suk HI (2017) Deep learning in medical image analysis. Annu Rev Biomed Eng 19:221–248

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Simon O, Yacoub R, Jain S, Tomaszewski J, Sarder P (2018) Multi-radial LBP features as a tool for rapid glomerular detection and assessment in whole slide histopathology images. Sci Rep 8(1):2032–2032

    Article  PubMed  PubMed Central  Google Scholar 

  20. Cheng JZ, Ni D, Chou YH, Qin J, Tiu CM, Chang YC, Huang CS, Shen D, Chen CM (2016) Computer-aided diagnosis with deep learning architecture: applications to breast lesions in US images and pulmonary nodules in CT scans. Sci Rep 6(1):24454

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Cruz-Roa A, Gilmore H, Basavanhally A, Feldman M, Ganesan S, Shih N, Tomaszewski J, González F, Madabhushi A (2017) Accurate and reproducible invasive breast cancer detection in whole-slide images: a deep learning approach for quantifying tumor extent. Sci Rep 7(1):46450. http://search.proquest.com/docview/1903454183/. Accessed 7 Dec 2019

  22. Sirinukunwattana K, Ahmed Raza SE, Tsang YW, Snead DRJ, Cree IA, Rajpoot NM (2016) Locality sensitive deep learning for detection and classification of nuclei in routine colon cancer histology images. IEEE Trans Med Imag 35(5):1196–1206

    Article  Google Scholar 

  23. Ertosun M, Rubin D (2015) Automated grading of gliomas using deep learning in digital pathology images: a modular approach with ensemble of convolutional neural networks. AMIA Annu Symp Proc 2015:1899–1908

    PubMed  PubMed Central  Google Scholar 

  24. Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S (2016) Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imag 35(5):1207–1216

    Article  Google Scholar 

  25. Coudray N, Ocampo PS, Sakellaropoulos T, Narula N, Snuderl M, Fenyo D, Moreira AL (2018) Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning (report). Nat Med 24(10):1559

    Article  CAS  PubMed  Google Scholar 

  26. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639):115

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  27. Pan SJ, Yang Q (2010) A survey on transfer learning. IEEE Trans Knowl Data Eng 22(10):1345–1359

    Article  Google Scholar 

  28. Tang B, Pan Z, Yin K, Khateeb A (2019) Recent advances of deep learning in bioinformatics and computational biology. Frontiers in Genetics 10(214)

    Google Scholar 

  29. Zeng K, Yu J, Wang R, Li C, Tao D (2017) Coupled deep autoencoder for single image super-resolution. IEEE Trans Cybern 47(1):27–37

    Article  PubMed  Google Scholar 

  30. Al-Stouhi S, Reddy CK (2016) Transfer Learning for Class Imbalance Problems with Inadequate Data. Knowl Inf Syst 48 (1):201–228

    Google Scholar 

  31. Nagpal K, Foote D, Liu Y, Chen PHC, Wulczyn E, Tan F, Olson N, Smith JL, Mohtashamian A, Wren JH, et al (2019) Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer. NPJ Digit Med 2(1):48

    Article  PubMed  PubMed Central  Google Scholar 

  32. Isaksson J, Arvidsson I, Åaström K, Heyden A (2017) Semantic segmentation of microscopic images of h&e stained prostatic tissue using CNN. In: 2017 international joint conference on neural networks (IJCNN). IEEE, Piscataway, pp 1252–1256

    Chapter  Google Scholar 

  33. Khan UAH, Stürenberg C, Gencoglu O, Sandeman K, Heikkinen T, Rannikko A, Mirtti T (2019) Improving prostate cancer detection with breast histopathology images. arXiv:190305769

    Google Scholar 

  34. Källén H, Molin J, Heyden A, Lundström C, Åström K (2016) Towards grading Gleason score using generically trained deep convolutional neural networks. In: 2016 IEEE 13th international symposium on biomedical imaging (ISBI). IEEE, Piscataway, pp 1163–1167

    Chapter  Google Scholar 

  35. Sermanet P, Eigen D, Zhang X, Mathieu M, Fergus R, LeCun Y (2013) Overfeat: integrated recognition, localization and detection using convolutional networks. arXiv:13126229

    Google Scholar 

  36. Arvaniti E, Claassen M (2018) Coupling weak and strong supervision for classification of prostate cancer histopathology images. arXiv:181107013

    Google Scholar 

  37. Arvaniti E, Fricker KS, Moret M, Rupp N, Hermanns T, Fankhauser C, Wey N, Wild PJ, Rueschoff JH, Claassen M (2018) Automated Gleason grading of prostate cancer tissue microarrays via deep learning. Sci Rep 8(1):12054

    Google Scholar 

  38. Campanella G, Silva VWK, Fuchs TJ (2018) Terabyte-scale deep multiple instance learning for classification and localization in pathology. arXiv:180506983

    Google Scholar 

  39. Way GP, Greene CS (2018) Bayesian deep learning for single-cell analysis. Nature Methods 15(12):1009–1010

    Google Scholar 

  40. Chaudhary K, Poirion O, Lu L, Huang S, Travers C, Garmire L (2018) Multi-modal meta-analysis of 1494 hepatocellular carcinoma samples reveals vast impacts of consensus driver genes on phenotypes. BioRxiv. http://search.proquest.com/docview/2071227297/. Accessed 7 Dec 2019

  41. Zhang C, Song J, Pei Z, Jiang J (2016) An imbalanced data classification algorithm of de-noising auto-encoder neural network based on smote. EDP Sciences, Les Ulis, vol 56. http://search.proquest.com/docview/1786240651/. Accessed 7 Dec 2019

  42. Way G, Greene C (2017) Extracting a biologically relevant latent space from cancer transcriptomes with variational autoencoders. BioRxiv. http://search.proquest.com/docview/2071245134/. Accessed 7 Dec 2019

  43. Lin C, Jain S, Kim H, Bar-Joseph Z (2017) Using neural networks for reducing the dimensions of single-cell RNA-seq data. Nucleic Acids Res 45(17):e156–e156. http://search.proquest.com/docview/1947096259/. Accessed 7 Dec 2019

  44. Danaee P, Ghaeini R, Hendrix DA (2017) A deep learning approach for cancer detection and relevant gene identification. In: Pacific symposium on biocomputing 2017. World Scientific, Singapore, pp 219–229

    Chapter  Google Scholar 

  45. Caruana R, Lou Y, Gehrke J, Koch P, Sturm M, Elhadad N (2015) Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. ACM, New York, pp 1721–1730

    Chapter  Google Scholar 

  46. Doshi-Velez F, Kim B (2017) Towards a rigorous science of interpretable machine learning. arXiv:1702.08608

    Google Scholar 

  47. Montavon G, Samek W, Müller KR (2018) Methods for interpreting and understanding deep neural networks. Digit Signal Process 73:1–15

    Article  Google Scholar 

  48. Erhan D, Bengio Y, Courville A, Vincent P (2009) Visualizing higher-layer features of a deep network. University of Montreal 1341(3):1

    Google Scholar 

  49. Nguyen AM, Yosinski J, Clune J (2016) Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks. ArXiv abs/1602.03616

    Google Scholar 

  50. Simonyan K, Vedaldi A, Zisserman A (2014) Deep inside convolutional networks: visualising image classification models and saliency maps. In: Workshop at international conference on learning representations

    Google Scholar 

  51. Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A (2016) Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2921–2929

    Google Scholar 

  52. Dabkowski P, Gal Y (2017) Real time image saliency for black box classifiers. In: Proceedings of the 31st international conference on neural information processing systems, Curran Associates, NIPS’17, pp 6970–6979. http://dl.acm.org/citation.cfm?id=3295222.3295440. Accessed 7 Dec 2019

  53. Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer, Berlin, pp 818–833

    Google Scholar 

  54. Milletari F, Navab N, Ahmadi SA (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision (3DV). IEEE, Piscataway, pp 565–571

    Chapter  Google Scholar 

  55. Komura D, Ishikawa S (2018) Machine learning methods for histopathological image analysis. Comput Struct Biotechnol J 16:34–42. https://doi.org/10.1016/j.csbj.2018.01.001. http://www.sciencedirect.com/science/article/pii/S2001037017300867. Accessed 7 Dec 2019

  56. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542(7639):115–118

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  57. Tan X, Su A, Tran M, Nguyen Q (2019) Spacell: integrating tissue morphology and spatial gene expression to predict disease cells. bioRxiv (Accepted Bioinformatics) https://doi.org/10.1101/837211. 837211

  58. Janda M, Soyer HP (2019) Can clinical decision making be enhanced by artificial intelligence? British Journal of Dermatology 180(2):247–248

    Google Scholar 

  59. Topol EJ (2019) High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine 25(1):44–56

    Google Scholar 

  60. Hekler A, Utikal JS, Enk AH, Solass W, Schmitt M, Klode J, Schadendorf D, Sondermann W, Franklin C, Bestvater F, Flaig MJ, Krahl D, von Kalle C, Fröhling S, Brinker TJ (2019) Deep learning outperformed 11 pathologists in the classification of histopathological melanoma images. Eur J Cancer 118:91–96. https://doi.org/10.1016/j.ejca.2019.06.012. http://www.sciencedirect.com/science/article/pii/S0959804919303806. Accessed 7 Dec 2019

  61. Navarro JF, Sjostrand J, Salmen F, Lundeberg J, Stahl PL (2017) ST Pipeline: an automated pipeline for spatial mapping of unique transcripts. Bioinformatics 33(16):2591–2593

    Google Scholar 

  62. Dries R, Zhu Q, Dong R, Eng C-HL, Li H, Liu K, Fu Y, Zhao T, Sarkar A, Bao F et al (2020) Giotto, a toolbox for integrative analysis and visualization of spatial expression data. bioRxiv:701680

    Google Scholar 

Download references

Acknowledgements

We thank members in Nguyen’s Biomedical Machine Learning Lab for helpful discussion. This work has been supported by the Australian Research Council (ARC DECRA DE190100116), the University of Queensland, and the Genome Innovation Hub.

Author information

Authors and Affiliations

  1. Division of Genetics and Genomics, Institute for Molecular Bioscience, The University of Queensland, Brisbane, QLD, Australia

    Xiao Tan, Andrew T. Su, Hamideh Hajiabadi, Minh Tran & Quan Nguyen

Authors
  1. Xiao Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew T. Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hamideh Hajiabadi
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Minh Tran
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Quan Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Quan Nguyen .

Editor information

Editors and Affiliations

  1. Chemistry, Oxford University, Oxford, UK

    Hugh Cartwright

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Science+Business Media, LLC, part of Springer Nature

About this protocol

Check for updates. Verify currency and authenticity via CrossMark

Cite this protocol

Tan, X., Su, A.T., Hajiabadi, H., Tran, M., Nguyen, Q. (2021). Applying Machine Learning for Integration of Multi-Modal Genomics Data and Imaging Data to Quantify Heterogeneity in Tumour Tissues. In: Cartwright, H. (eds) Artificial Neural Networks. Methods in Molecular Biology, vol 2190. Humana, New York, NY. https://doi.org/10.1007/978-1-0716-0826-5_10

Download citation

  • DOI: https://doi.org/10.1007/978-1-0716-0826-5_10

  • Published:

  • Publisher Name: Humana, New York, NY

  • Print ISBN: 978-1-0716-0825-8

  • Online ISBN: 978-1-0716-0826-5

  • eBook Packages: Springer Protocols

Publish with us

Policies and ethics

Search

Navigation