Advertisement

Why imaging data alone is not enough: AI-based integration of imaging, omics, and clinical data

  • Andreas Holzinger
  • Benjamin Haibe-Kains
  • Igor JurisicaEmail author
Original Article
Part of the following topical collections:
  1. Advanced Image Analyses (Radiomics and Artificial Intelligence)

Abstract

Artificial intelligence (AI) is currently regaining enormous interest due to the success of machine learning (ML), and in particular deep learning (DL). Image analysis, and thus radiomics, strongly benefits from this research. However, effectively and efficiently integrating diverse clinical, imaging, and molecular profile data is necessary to understand complex diseases, and to achieve accurate diagnosis in order to provide the best possible treatment. In addition to the need for sufficient computing resources, suitable algorithms, models, and data infrastructure, three important aspects are often neglected: (1) the need for multiple independent, sufficiently large and, above all, high-quality data sets; (2) the need for domain knowledge and ontologies; and (3) the requirement for multiple networks that provide relevant relationships among biological entities. While one will always get results out of high-dimensional data, all three aspects are essential to provide robust training and validation of ML models, to provide explainable hypotheses and results, and to achieve the necessary trust in AI and confidence for clinical applications.

Keywords

Precision medicine Artificial intelligence Machine learning Decision support Integrative computational biology Network-based analysis Radiomics 

Notes

Funding

This study was funded in part by Ontario Research Fund (34876) to IJ. B.H.K. was supported by the Gattuso-Slaight Personalized Cancer Medicine Fund at Princess Margaret Cancer Centre and the Artificial Intelligence and Microbiome Program supported by the Princess Margaret Cancer Foundation.

Compliance with Ethical Standards

Conflict of interest

Author AH declares no conflict of interest. Author BHK declares no conflict of interest. Author IJ declares no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

References

  1. 1.
    Marcus G. 2018. Deep learning: A critical appraisal. arXiv:1801.00631.
  2. 2.
    Zhang Q, Yang LT, Chen Z, Li P. A survey on deep learning for big data. Inf Fus 2018;42:146–57.Google Scholar
  3. 3.
    Komaroff AL. The variability and inaccuracy of medical data. Proc IEEE 1979;67(9):1196–207.Google Scholar
  4. 4.
    Kadmon G, Pinchover M, Weissbach A, Kogan Hazan S, Nahum E. Case not closed: prescription errors 12 years after computerized physician order entry implementation. J Pediatr 2017;190 :236–40.e2.  https://doi.org/10.1016/j.jpeds.2017.08.013.CrossRefPubMedGoogle Scholar
  5. 5.
    Holzinger A, Jurisica I. Knowledge discovery and data mining in biomedical informatics: the future is in integrative, interactive machine learning solutions. Interactive knowledge discovery and data mining in biomedical informatics: state-of-the-art and future challenges. Lecture notes in computer science LNCS 8401. In: Holzinger A and Jurisica I, editors. Springer; 2014. p. 1–18.Google Scholar
  6. 6.
    Wang B, et al. Similarity network fusion for aggregating data types on genomic scale. Nat Methods 2014;11 (3):3377.Google Scholar
  7. 7.
    Holzinger A. Biomedical informatics: discovering knowledge in big data. New York: Springer; 2014.Google Scholar
  8. 8.
    Hutson M. Artificial intelligence faces reproducibility crisis. Science 2018;359(6377):725–6.PubMedGoogle Scholar
  9. 9.
    Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility mean? Sci Transl Med 2016; 8(341):341ps12.PubMedGoogle Scholar
  10. 10.
    Turkay C, Jeanquartier F, Holzinger A, Hauser H. On computationally-enhanced visual analysis of heterogeneous data and its application in biomedical informatics. Interactive knowledge discovery and data mining: state-of-the-art and future challenges in biomedical informatics. Lecture notes in computer science LNCS 8401. Springer; 2014. p. 117–140.Google Scholar
  11. 11.
    Collins FS, Varmus H. A new initiative on precision medicine. England J Med 2015;372(9):793–5.Google Scholar
  12. 12.
    Schiphof D, Oei EH, Hofman A, Waarsing JH, Weinans H, Bierma-Zeinstra SM. Sensitivity and associations with pain and body weight of an MRI definition of knee osteoarthritis compared with radiographic Kellgren and Lawrence criteria: a population-based study in middle-aged females. Osteoarthritis Cartilage 2014;22(3): 440–6.PubMedGoogle Scholar
  13. 13.
    Mlecnik B, Bindea G, Kirilovsky A, Angell HK, Obenauf AC, Tosolini M, Church SE, Maby P, Vasaturo A, Angelova M, Fredriksen T, Mauger S, Waldner M, Berger A, Speicher MR, Pages F, Valge-Archer V, Galon J. The tumor microenvironment and immunoscore are critical determinants of dissemination to distant metastasis. Sci Transl Med 2016;8(327):327ra26.PubMedGoogle Scholar
  14. 14.
    Sanduleanu S, Woodruff HC, de Jong EEC, van Timmeren JE, Jochems A, Dubois L, Lambin P. Tracking tumor biology with radiomics: a systematic review utilizing a radiomics quality score. Radiotherapy and Oncology: Journal of the European Society for Therapeutic Radiology and Oncology 2018;127(3):349– 60.Google Scholar
  15. 15.
    Vargas HA, Veeraraghavan H, Micco M, Nougaret S, Lakhman Y, Meier AA, Sosa R, Soslow RA, Levine DA, Weigelt B, Aghajanian C, Hricak H, Deasy J, Snyder A, Sala E. A novel representation of inter-site tumour heterogeneity from pre-treatment computed tomography textures classifies ovarian cancers by clinical outcome. Eur Radiol 2017;27(9):3991– 4001.PubMedPubMedCentralGoogle Scholar
  16. 16.
    Yin Q, Hung SC, Rathmell WK, Shen L, Wang L, Lin W, Fielding JR, Khandani AH, Woods ME, Milowsky MI, Brooks SA, Wallen EM, Shen D. Integrative radiomics expression predicts molecular subtypes of primary clear cell renal cell carcinoma. Clin Radiol 2018;73(9):782–91.PubMedPubMedCentralGoogle Scholar
  17. 17.
    Liao Xin, Bo Cai, Tian Bin, Luo Yilin, Song Wen, Li Yinglong. 2019. Machine-learning based radiogenomics analysis of MRI features and metagenes in glioblastoma multiforme patients with different survival time. J Cell Mol Med, 4.Google Scholar
  18. 18.
    Danala G, Thai T, Gunderson CC, Moxley KM, Moore K, Mannel RS, Liu H, Zheng B, Qiu Y. Applying quantitative CT image feature analysis to predict response of ovarian cancer patients to chemotherapy. Acad Radiol 2017;24(10):1233–9.PubMedPubMedCentralGoogle Scholar
  19. 19.
    Cook GJR, Azad G, Owczarczyk K, Siddique M, Goh V. Challenges and promises of PET radiomics. Int J Radiat Oncol Biol Phys 2018;102(4):1083–89.PubMedPubMedCentralGoogle Scholar
  20. 20.
    Lin G, Lai CH, Yen TC. Emerging molecular imaging techniques in gynecologic oncology. PET Clin 2018;13(2):289– 99.PubMedGoogle Scholar
  21. 21.
    Xu Y, Hosny A, Zeleznik R, Parmar C, Coroller T, Franco I, Mak RH, Aerts HJWL. 2019. Deep learning predicts lung cancer treatment response from serial medical imaging. Clinical Cancer Research.Google Scholar
  22. 22.
    Baessler B, Weiss K, Pinto Dos Santos D. Robustness and reproducibility of radiomics in magnetic resonance imaging: a phantom study. Invest Radiol 2019;54(4):221–8.PubMedGoogle Scholar
  23. 23.
    Lau SK, Boutros PC, Pintilie M, Blackhall FH, Zhu CQ, Strumpf D, Johnston MR, Darling G, Keshavjee S, Waddell TK, Liu N, Lau D, Penn LZ, Shepherd FA, Jurisica I, Der SD, Tsao MS. Three-gene prognostic classifier for early-stage non small-cell lung cancer. J Clin Oncol 2007;25 (35):5562–9.PubMedGoogle Scholar
  24. 24.
    Shedden K, Taylor JM, Enkemann SA, Tsao MS, Yeatman TJ, Gerald WL, Eschrich S, Jurisica I, Giordano TJ, Misek DE, Chang AC, Zhu CQ, Strumpf D, Hanash S, Shepherd FA, Ding K, Seymour L, Naoki K, Pennell N, Weir B, Verhaak R, Ladd-Acosta C, Golub T, Gruidl M, Sharma A, Szoke J, Zakowski M, Rusch V, Kris M, Viale A, Motoi N, Travis W, Conley B, Seshan VE, Meyerson M, Kuick R, Dobbin KK, Lively T, Jacobson JW, Beer DG. Gene expression-based survival prediction in lung adenocarcinoma: a multi-site, blinded validation study. Nat Med 2008;14(8):822–7.PubMedPubMedCentralGoogle Scholar
  25. 25.
    Boutros PC, Lau SK, Pintilie M, Liu N, Shepherd FA, Der SD, Tsao MS, Penn LZ, Jurisica I. Prognostic gene signatures for non-small-cell lung cancer. Proc Natl Acad Sci USA 2009;106(8):2824–8.PubMedGoogle Scholar
  26. 26.
    Beaulieu-Jones BK, Greene CS. Reproducibility of computational workflows is automated using continuous analysis. Nat Biotechnol 2017;34(4):342–6.Google Scholar
  27. 27.
    Munafò MR, Nosek BA, Bishop VMD, Button SB, Chambers CD, du Sert N, Simonsohn U, Wagenmakers E-J, Ware JJ, Ioannidis JPA. A manifesto for reproducible science. Nat Hum Behav 2017; 1(1):s41562–016–0021.Google Scholar
  28. 28.
    Lamb J, Crawford ED, Peck D, Modell JW, Blat IC, Wrobel MJ, Lerner J, Brunet J-P, Subramanian A, Ross KN, Reich M, Hieronymus H, Wei G, Armstrong SA, Haggarty SJ, Clemons PA, Wei R, Carr SA, Lander ES, Golub TR. The connectivity map: using gene-expression signatures to connect small molecules, genes, and disease. Science 2006;313(5795):1929–35.Google Scholar
  29. 29.
    Sun R, Limkin EJ, Vakalopoulou M, Dercle L, Champiat S, Han SR, Verlingue L, Brandao D, Lancia A, Ammari S, Hollebecque A, Scoazec JY, Marabelle A, Massard C, Soria JC, Robert C, Paragios N, Deutsch E, Ferte C. A radiomics approach to assess tumour-infiltrating cd8 cells and response to anti-pd-1 or anti-pd-l1 immunotherapy: an imaging biomarker, retrospective multicohort study. Lancet Oncol 2018;19(9):1180–91.PubMedPubMedCentralGoogle Scholar
  30. 30.
    Aerts HJ, Grossmann P, Tan Y, Oxnard GR, Rizvi N, Schwartz LH, Zhao B. Defining a radiomic response phenotype: a pilot study using targeted therapy in NSCLC. Sci Rep 2016;6:33860.PubMedPubMedCentralGoogle Scholar
  31. 31.
    Tokar T, Pastrello C, Ramnarine VR, Zhu CQ, Craddock KJ, Pikor LA, Vucic EA, Vary S, Shepherd FA, Tsao MS, Lam WL, Jurisica I. Differentially expressed microRNAs in lung adenocarcinoma invert effects of copy number aberrations of prognostic genes. Oncotarget 2018;9(10):9137–55.PubMedPubMedCentralGoogle Scholar
  32. 32.
    Viswanath SE, Tiwari P, Lee G, Madabhushi A. Dimensionality reduction-based fusion approaches for imaging and non-imaging biomedical data: concepts, workflow, and use-cases. BMC Med Imaging 2017;17(1):2.PubMedPubMedCentralGoogle Scholar
  33. 33.
    Ingalhalikar M, Smith A, Parker D, Satterthwaite TD, Elliott MA, Ruparel K, Hakonarson H, Gur RE, Gur RC, Verma R. Sex differences in the structural connectome of the human brain. Proc Natl Acad Sci USA 2014;111(2):823–8.PubMedGoogle Scholar
  34. 34.
    Bigler ED. Systems biology, neuroimaging, neuropsychology, neuroconnectivity and traumatic brain injury. Front Syst Neurosci 2016;10:55.PubMedPubMedCentralGoogle Scholar
  35. 35.
    Cui LB, Liu L, Wang HN, Wang LX, Guo F, Xi YB, Liu TT, Li C, Tian P, Liu K, Wu WJ, Chen YH, Qin W, Yin H. Disease definition for schizophrenia by functional connectivity using radiomics strategy. Schizophrenia Bull 2018;44(5):1053–9.Google Scholar
  36. 36.
    Jeong H, Mason SP, Barabasi AL, Oltvai ZN. Lethality and centrality in protein networks. Nature 2001;411(6833):41–2.PubMedGoogle Scholar
  37. 37.
    Przulj N, Wigle DA, Jurisica I. Functional topology in a network of protein interactions. Bioinformatics 2004;20(3):340–8.PubMedGoogle Scholar
  38. 38.
    Ye P, Peyser BD, Pan X, Boeke JD, Spencer FA, Bader JS. Gene function prediction from congruent synthetic lethal interactions in yeast. Molec Syst Biol 2005;1:1.Google Scholar
  39. 39.
    Sharan R, Ulitsky I, Shamir R. 2007. Network-based prediction of protein function. Mol Syst Biol, 3.PubMedPubMedCentralGoogle Scholar
  40. 40.
    Ulitsky I, Shamir R. 2007. Identification of functional modules using network topology and high-throughput data. BMC Syst Biol, 1.Google Scholar
  41. 41.
    Chuang HY, Lee E, Liu YT, Lee D, Ideker T. 2007. Network-based classification of breast cancer metastasis. Mol Syst Biol, 3.PubMedPubMedCentralGoogle Scholar
  42. 42.
    Fortney K, Kotlyar M, Jurisica I. Inferring the functions of longevity genes with modular subnetwork biomarkers of Caenorhabditis elegans aging. Genome Biol 2010;11(2):R13.PubMedPubMedCentralGoogle Scholar
  43. 43.
    Kotlyar M, Fortney K, Jurisica I. Network-based characterization of drug-regulated genes, drug targets, and toxicity. Methods 2012;57(4):499–507. Protein-Protein Interactions.PubMedGoogle Scholar
  44. 44.
    Hinze L, Pfirrmann M, Karim S, Degar J, McGuckin C, Vinjamur D, Sacher J, Stevenson KE, Neuberg DS, Orellana E, Stanulla M, Gregory RI, Bauer DE, Wagner FF, Stegmaier K, Gutierrez A. Synthetic lethality of Wnt pathway activation and asparaginase in drug-resistant acute leukemias. Cancer Cell 2019;35(4):664–676.e667.PubMedGoogle Scholar
  45. 45.
    Wong SWH, Pastrello C, Kotlyar M, Faloutsos C, Jurisica I. Sdregion: fast spotting of changing communities in biological networks. Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining. New York: ACM; 2018. p. 867–875.Google Scholar
  46. 46.
    Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RG, Granton P, Zegers CM, Gillies R, Boellard R, Dekker A, Aerts HJ. Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 2012;48(4):441–6.PubMedPubMedCentralGoogle Scholar
  47. 47.
    Ger RB, Zhou S, Chi PM, Lee HJ, Layman RR, Jones AK, Goff DL, Fuller CD, Howell RM, Li H, Stafford RJ, Court LE, Mackin DS. Comprehensive investigation on controlling for CT imaging variabilities in radiomics studies. Sci Rep 2018;8(1):13047.PubMedPubMedCentralGoogle Scholar
  48. 48.
    Jorgensen P, Nelson B, Robinson MD, Chen Y, Andrews B, Tyers M, Boone C. High-resolution genetic mapping with ordered arrays of Saccharomyces cerevisiae deletion mutants. Genetics 2002;162 (3):1091–9.PubMedPubMedCentralGoogle Scholar
  49. 49.
    Suthers PF, Zomorrodi A, Maranas CD. 2009. Genome-scale gene/reaction essentiality and synthetic lethality analysis, Vol. 5.PubMedPubMedCentralGoogle Scholar
  50. 50.
    Chan EM, Shibue T, McFarland JM, Gaeta B, Ghandi M, Dumont N, Gonzalez A, McPartlan JS, Li T, Zhang Y, Liu JB, Lazaro J-B, Gu P, Piett CG, Apffel A, Ali SO, Deasy R, Keskula P, Ng RWS, Roberts EA, Reznichenko E, Leung L, Alimova M, Schenone M, Islam M, Maruvka YE, Liu Y, Roper J, Raghavan S, Giannakis M, Tseng Y-Y, Nagel ZD, D’Andrea A, Root DE, Boehm JS, Getz G, Chang S, Golub TR, Tsherniak A, Vazquez F, Bass AJ. Wrn helicase is a synthetic lethal target in microsatellite unstable cancers. Nature 2019;568(7753): 551–6.PubMedPubMedCentralGoogle Scholar
  51. 51.
    Holzinger A, Langs G, Denk H, Zatloukal K, Mueller H. 2019. Causability and explainability of AI in medicine. Wiley Interdisciplinary Reviews, Data Mining and Knowledge Discovery.Google Scholar
  52. 52.
    Krittanawong C, Zhang HJ, Wang Z, Aydar M, Kitai T. Artificial intelligence in precision cardiovascular medicine. J Am Coll Cardiol 2017;69(21):2657–64.PubMedGoogle Scholar
  53. 53.
    Hamet P, Tremblay J. Artificial intelligence in medicine. Metabolism 2017;69:S36–40.Google Scholar
  54. 54.
    Robertson S, Azizpour H, Smith K, Hartman J. Digital image analysis in breast pathology—from image processing techniques to artificial intelligence. Transl Res 2018;194:19–35.PubMedGoogle Scholar
  55. 55.
    Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019;25 (1):44–56.PubMedGoogle Scholar
  56. 56.
    Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts H. Artificial intelligence in radiology. Nat Rev Cancer 2018;18(8):500–510.PubMedPubMedCentralGoogle Scholar
  57. 57.
    Kang J, Rancati T, Lee S, Oh J, Kerns SL, Scott JG, Schwartz R, Kim S, Rosenstein BS. Machine learning and radiogenomics: lessons learned and future directions. Frontiers Oncol 2018;8:228.Google Scholar
  58. 58.
    Sun R, Limkin E, Vakalopoulou M, Dercle L, Champiat S, Han S, Verlingue L, Brandao D, Lancia A, Ammari S, Hollebecque A, Scoazec J-Y, Marabelle A, Massard C, Soria J-C, Robert C, Paragios N, Deutsch E, Ferté C. A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study. Lancet Oncol 2018;19(9):1180–91.PubMedPubMedCentralGoogle Scholar
  59. 59.
    Iorio F, Knijnenburg TA, Vis DJ, Bignell GR, Menden MP, Schubert M, Aben N, Gonçalves E, Barthorpe S, Lightfoot H, Cokelaer T, Greninger P, van Dyk E, Chang H, de Silva H, Heyn H, Deng X, Egan RK, Liu Q, Mironenko T, Mitropoulos X, Richardson L, Wang J, Zhang T, Moran S, Sayols S, Soleimani M, Tamborero D, López-Bigas N, Ross-Macdonald P, Esteller M, Gray NS, Haber DA, Stratton MR, Benes CH, Wessels LFA, Saez-Rodriguez J, McDermott U, Garnett MJ. A landscape of pharmacogenomic interactions in cancer. Cell 2016;166(3):740–54.PubMedPubMedCentralGoogle Scholar
  60. 60.
    Seashore-Ludlow B, Rees MG, Cheah JH, Cokol M, Price EV, Coletti ME, Jones V, Bodycombe NE, Soule CK, Gould J, Alexander B, Li A, Montgomery P, Wawer MJ, Kuru N, Kotz JD, Hon S-YC, Munoz B, Liefeld T, Dancik V, Bittker JA, Palmer M, Bradner JE, Shamji AF, Clemons PA, Schreiber SL. Harnessing connectivity in a large-scale small-molecule sensitivity dataset. Cancer Discovery.Google Scholar
  61. 61.
    Haverty PM, Lin E, Tan J, Yu Y, Lam B, Lianoglou S, Neve RM, Martin S, Settleman J, Yauch RL, Bourgon R. Reproducible pharmacogenomic profiling of cancer cell line panels. Nature 2016; 533(7603):333–7.PubMedGoogle Scholar
  62. 62.
    Gao H, Korn JM, Ferretti S, Monahan JE, Wang Y, Singh M, Zhang C, Schnell C, Yang G, Zhang Y, Balbin AO, Barbe S, Cai H, Casey F, Chatterjee S, Chiang DY, Chuai S, Cogan SM, Collins SD, Dammassa E, Ebel N, Embry M, Green J, Kauffmann A, Kowal C, Leary RJ, Lehar J, Liang Y, Loo Alice, Lorenzana E III, Robert E, McLaughlin ME, Merkin J, Meyer R, Naylor TL, Patawaran M, Reddy A, Röelli C, Ruddy DA, Salangsang F, Santacroce F, Singh AP, Tang Y, Tinetto W, Tobler S, Velazquez R, Venkatesan K, Arx F, Wang H, Wang Z, Wiesmann M, Wyss D, Xu F, Bitter H, Atadja P, Lees E, Hofmann F, Li E, Keen N, Cozens R, Jensen M, Pryer NK, Williams JA, Sellers WR. High-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response. Nat Med 2015;21(11):1318–25.PubMedGoogle Scholar
  63. 63.
    LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521(7553):436.PubMedPubMedCentralGoogle Scholar
  64. 64.
    Chartrand G, Cheng PM, Vorontsov E, Drozdzal M, Turcotte S, Pal CJ, Kadoury S, An T. Deep learning A primer for radiologists. Radiographics Rev Publ Radiological Soc North Am Inc 2017;37(7): 2113–31.Google Scholar
  65. 65.
    Sharp G, Fritscher KD, Pekar V, Peroni M, Shusharina N, Veeraraghavan H, Yang J. Vision 20/20: perspectives on automated image segmentation for radiotherapy. Med Phys 2014;41(5):050902.PubMedPubMedCentralGoogle Scholar
  66. 66.
    Cardenas CE, Yang J, Anderson BM, Court LE, Brock KB. Advances in auto-segmentation. Semin Radiat Oncol 2019;29(3):185–97. Adaptive Radiotherapy and Automation.PubMedGoogle Scholar
  67. 67.
    Kreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF, Forshee R, Walderhaug M, Botsis T. 2017. Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review. J Biomed Inform, 73.Google Scholar
  68. 68.
    Asim M, Wasim M, Khan M, Mahmood W, Abbasi H. 2018. A survey of ontology learning techniques and applications. Database, 2018.Google Scholar
  69. 69.
    Deshpande PR, Rajan S, Sudeepthi LB, Abdul CN. Patient-reported outcomes: a new era in clinical research. Perspectives Clin Res 2011;2(4):137–44.Google Scholar
  70. 70.
    Loh E. Medicine and the rise of the robots: a qualitative review of recent advances of artificial intelligence in health. BMJ Leader 2018;2(2):59–63.Google Scholar
  71. 71.
    Dias D, Cunha J. Wearable health Devices—Vital sign monitoring, systems and technologies. Sensors 2018; 18(8):2414.Google Scholar
  72. 72.
    Guyon I, Bennett K, Cawley G, Escalante HJ, Escalera S, Ho TK, Macia N, Ray B, Saeed M, Statnikov A. Design of the 2015 ChaLearn AutoML Challenge. 2015 International joint conference on neural networks (IJCNN). IEEE; 2015. p. 1–8.Google Scholar
  73. 73.
    Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D. Mastering the game of go with deep neural networks and tree search. Nature 2016;529(7587):484–9.PubMedPubMedCentralGoogle Scholar
  74. 74.
    Stickel C, Ebner M, Steinbach-Nordmann S, Searle G, Holzinger A. Emotion detection: application of the valence arousal space for rapid biological usability testing to enhance universal access. Universal access in human–computer interaction. Addressing diversity, lecture notes in computer science, LNCS 5614. In: Stephanidis C, editors. Springer; 2009. p. 615–24.Google Scholar
  75. 75.
    Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017;542(7639):115–8.Google Scholar
  76. 76.
    Holzinger A. Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inform 2016;3(2):119–31.PubMedPubMedCentralGoogle Scholar
  77. 77.
    Holzinger A, Stocker C, Ofner B, Prohaska G, Brabenetz A, Hofmann-Wellenhof R. Combining HCI, natural language processing, and knowledge discovery - potential of IBM content analytics as an assistive technology in the biomedical domain. Springer lecture notes in computer science LNCS 7947. Springer; 2013.Google Scholar
  78. 78.
    Holzinger A, Plass M, Holzinger K, Crisan GC, Pintea C-M, Palade V. Towards interactive machine learning (IML): applying ant colony algorithms to solve the traveling salesman problem with the human-in-the-loop approach. Springer lecture notes in computer science LNCS 9817. Springer; 2016. p. 81–95.Google Scholar

Copyright information

© Springer-Verlag GmbH Germany, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Institute for Medical Informatics / StatisticsMedical University GrazGrazAustria
  2. 2.Princess Margaret Cancer CentreUniversity Health NetworkTorontoCanada
  3. 3.Departments of Medical Biophysics and Computer ScienceUniversity of TorontoTorontoCanada
  4. 4.Krembil Research InstituteUHNTorontoCanada
  5. 5.Institute of NeuroimmunologySlovak Academy of SciencesBratislavaSlovakia

Personalised recommendations