Abstract
Artificial intelligence models can produce powerful predictive computer vision tools for healthcare. However, their development simultaneously requires computational skill as well as biomedical expertise. This barrier often impedes the wider utilization of AI in professional environments since biomedical experts often lack software development skills. We present the first development environment where a user with no prior training can build near-expert level convolutional neural network classifiers on real-world datasets. Our key contribution is a simplified environment in virtual reality where the user can build, compute, and critique a model. Through a controlled user study, we show that our software enables biomedical researchers and healthcare professionals with no AI development experience to build AI models with near-expert performance. We conclude that the potential role for AI in the biomedical domain can be realized more effectively by making its development more intuitive for non-technical domain experts using novel modes of interaction.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Code availability
References
Azure Machine Learning (2019) Microsoft Azure. Retrieved December 26, 2019 from https://azure.microsoft.com/en-us/services/machine-learning/
Bejnordi B, Mitko V, Paul Johannes van D, Bram van G, Nico K, Geert L, Jeroen A. W. M. van der L, Meyke H, Quirine FM, Maschenka B, Oscar G, Nikolaos S, Marcory CRF van D, Peter B, Francisco B, Andrew HB, Dayong W, Aditya K, Rishab G, Humayun I, Aoxiao Z, Qi D, Quanzheng L, Hao C, Huang-Jing L, Pheng-Ann H, Christian H, Elia B, Quincy W, Ugur H, Mustafa Ü, Rengul C-A, Matt B, Vitali K, Alexei V, Oren K, Muhammad S, Nasir R, Ruqayya A, Korsuk S, Talha Q, Yee-Wah T, David T, Jonas A, Peter H, Mira V, Kimmo K, Leena L, Pekka R, Kaisa L, Shadi A, Bharti M, Ami G, Stefanie D, Nassir N, Seiryo W, Shigeto S, Yoichi T, Hideo M, Hady AP, Vassili K, Alexander K, Vitali L, Gloria B, Fernandez-Carrobles MM, Ismael S, Oscar D, Daniel R, Rui V (2017) Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318(22) 2199–2210. https://doi.org/10.1001/jama.2017.14585
Blei DM (2014) Build, compute, critique, repeat: data analysis with latent variable models. Ann Rev Stat Appl 1(1):203–232. https://doi.org/10.1146/annurev-statistics-022513-115657
Bowman DA, McMahan RP (2007) Virtual reality: how much immersion is enough? Computer 40(7):36–43. https://doi.org/10.1109/MC.2007.257
Box George EP (1976) Science and statistics. J Am Stat Assoc 71(356):791–799. https://doi.org/10.2307/2286841
Cohen G, Afshar S, Tapson J, André van S (2017). EMNIST: an extension of MNIST to handwritten letters. arXiv:1702.05373 [cs] (February 2017). Retrieved December 27, 2019 from arXiv:1702.05373
ConvNetJS (2016) Deep Learning in your browser. Retrieved May 12, 2019 from https://cs.stanford.edu/people/karpathy/convnetjs/
Dede C (2009) Immersive interfaces for engagement and learning. Science 323:66–69. https://doi.org/10.1126/science.1167311
Deep Cognition (2017) DeepCognition.ai. Retrieved December 26, 2019 from https://deepcognition.ai/
Design (2020)Material design. Retrieved May 25, 2019 from https://material.io/design/
Dice LR (1945) Measures of the amount of ecologic association between species. Ecology 26:297–302. https://doi.org/10.2307/1932409
Dietz S, Henrich C (2014) Texting as a distraction to learning in college students. Comput Hum Behav 36:163–167. https://doi.org/10.1016/j.chb.2014.03.045
Esteva A, Kuprel B, Novoa RA, Ko J, Swetter Susan M, Blau Helen M, Thrun S (2017) Dermatologist-level classification of skin cancer with deep neural networks. Nature 542:115–118. https://doi.org/10.1038/nature21056
Everett NM (2018) Intuitive design, eight steps to an intuitive UI. Black Watch Publishing
Ghadai S, Balu A, Sarkar S, Krishnamurthy A (2018) Learning localized features in 3D CAD models for manufacturability analysis of drilled holes. Comput Aided Geom Des 62:263–275. https://doi.org/10.1016/j.cagd.2018.03.024
Harley AW (2015) An interactive node-link visualization of convolutional neural networks. in advances in visual computing. In: George B, Richard B, Bahram P, Darko K, Ioannis P, Rogerio F, Tim M, Mark E, Regis K, Eric R, Zhao Y, Gunther W (eds) Springer, Cham, pp 867–877. https://doi.org/10.1007/978-3-319-27857-5_77
Hernik J, Jaworska E (2018) The effect of enjoyment on learning, pp 508–514. https://doi.org/10.21125/inted.2018.1087
HIPS/Spearmint (2020) Harvard intelligent probabilistic systems group. Retrieved January 9, 2020 from https://github.com/HIPS/Spearmint
Kearnes S, Kevin M, Marc B, Vijay P, Patrick R (2016) Molecular graph convolutions: moving beyond fingerprints. J Comput Aided Mol Des 30(8):595–608. https://doi.org/10.1007/s10822-016-9938-8
Keras.js (2018) Run Keras models in the browser. Retrieved May 12, 2019 from https://transcranial.github.io/keras-js/#/
keras-team/keras (2019) Keras. Retrieved December 31, 2019 from https://github.com/keras-team/keras
Kim YM, Rhiu I, Yun MH (2019) A systematic review of a virtual reality system from the perspective of user experience. Int J Hum Comput Interact. https://doi.org/10.1080/10447318.2019.1699746
Kindermans P-J, Schütt KT, Alber M, Müller K-R, Erhan D, Kim B, Dühne S (2017) Learning how to explain neural networks: PatternNet and PatternAttribution. arXiv:1705.05598 [cs, stat] (May 2017). Retrieved May 11, 2019 from arXiv:1705.05598
Kingma DP, Ba J (2017) Adam: a method for stochastic optimization. arXiv:1412.6980 [cs] (January 2017). Retrieved January 8, 2020 from arXiv:1412.6980
Kolb DA (2014) Experiential learning: experience as the source of learning and development. FT Press
Le H, Dimitris S, Tahsin K, Gao Y, Davis James E, Saltz Joel H (2016) Patch-based convolutional neural network for whole slide tissue image classification. Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit 2016:2424–2433. https://doi.org/10.1109/CVPR.2016.266
Lecun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proce IEEE 86(11):2278–2324. https://doi.org/10.1109/5.726791
Lisha L, Kevin J, Giulia D, Afshin R, AmeetT (2018) A novel bandit-based approach to hyperparameter optimization, hyperband, p 52
Liu Y, Kohlberger T, Norouzi M, Dahl GE, Smith JL, Mohtashamian A, Olson N, Peng LH, Hipp JD, Stumpe MC (2018) Artificial intelligence–based breast cancer nodal metastasis detection: insights into the black box for pathologists. Arch Pathol Lab Med (October 2018), arpa.2018-0147-OA. https://doi.org/10.5858/arpa.2018-0147-OA
Lobe (2019) Deep learning made simple. Retrieved November 14, 2019 from https://lobe.ai
Ma B, Jain E, Entezari A (2017) 3D Saliency from eye tracking with tomography. In: Burch M, Chuang L, Fisher B, Schmidt A, Weiskopf D (eds) Eye tracking and visualization, mathematics and visualization. Springer, Cham, pp 185–198. https://doi.org/10.1007/978-3-319-47024-5_11
Matthews BW (1975) Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim Biophys Acta (BBA) Protein Struct 405:442–451. https://doi.org/10.1016/0005-2795(75)90109-9
Meissler N, Wohlan A, Hochgeschwender N, Schreiber A (2019) Using visualization of convolutional neural networks in virtual reality for machine learning newcomers. In: 2019 IEEE international conference on artificial intelligence and virtual reality (AIVR), pp 152–1526. https://doi.org/10.1109/AIVR46125.2019.00031
Miotto R, Wang F, Wang S, Jiang X, Dudley JT (2018) Deep learning for healthcare: review, opportunities and challenges. Brief Bioinform 19(6):1236–1246. https://doi.org/10.1093/bib/bbx044
Naumann A, Hurtienne J, Habakuk Israel J, Mohs C,Christof Kindsmüller M, Meyer HA, Hu S ßlein (2007) Intuitive use of user interfaces: defining a vague concept. In: Engineering psychology and cognitive ergonomics (lecture notes in computer science). Springer, Berlin, pp 128–136. https://doi.org/10.1007/978-3-540-73331-7_14
Netzer Y, Wang T, Coates A, Bissacco A, Wu B, Ng AY (2011) Reading digits in natural images with unsupervised feature learning. In: NIPS workshop on deep learning and unsupervised feature learning 2011. Retrieved August 11, 2020 from http://ufldl.stanford.edu/housenumbers/nips2011_housenumbers.pdf
Neural Network Console (2020) Retrieved December 26, 2019 from https://dl.sony.com/
Neural network modeler (2019) IBM Watson. Retrieved December 26, 2019 from https://dataplatform.cloud.ibm.com/docs/content/wsj/analyze-data/ml-canvas-nnd-nodes.html
NVIDIA DIGITS (2015) NVIDIA Developer. Retrieved December 26, 2019 from https://developer.nvidia.com/digits
Obuchowski NA, Beiden SV, Berbaum KS, Hillis SL, Ishwaran H, Song HH, Wagner RF (2004) Multireader, multicase receiver operating characteristic analysis: an empirical comparison of five methods1. Acad Radiol 11:980–995. https://doi.org/10.1016/j.acra.2004.04.014
Ounkomol C, Seshamani S, Maleckar MM, Collman F, Johnson GR (2018) Label-free prediction of three-dimensional fluorescence images from transmitted-light microscopy. Nat Methods 15(11):917. https://doi.org/10.1038/s41592-018-0111-2
Reddy ND, Vo M, Narasimhan SG (2019) Occlusion-net: 2D/3D occluded keypoint localization using graph networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2019, pp 7326–7335
Sacks R, Perlman A, Barak R (2013) Construction safety training using immersive virtual reality. Constr Manag Econ 31:9. https://doi.org/10.1080/01446193.2013.828844
Schreiber A, Bock M (2019) Visualization and exploration of deep learning Networks in 3D and virtual reality. In HCI international (2019) posters (communications in computer and information science). Springer, Cham 206–211. https://doi.org/10.1007/978-3-030-23528-4_29
Smilkov D, Carter S (2016) Tensorflow—neural network playground. http://playground.tensorflow.org
Sørensen TJ (1948) A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on Danish commons. I kommission hos E, Munksgaard, København
Springenberg JT, Dosovitskiy A, Brox T, Riedmiller M (2015) Striving for simplicity: the all convolutional net. arXiv:1412.6806 [cs] (April 2015). Retrieved January 9, 2020 from arXiv:1412.6806
TensorSpace.js (2019) Retrieved May 11, 2019 from https://tensorspace.org/index.html
Um ER, Plass JL, Hayward EO, Homer BD (2012) Emotional design in multimedia learning. J Educ Psychol 104(2):485-498. https://doi.org/10.1037/a0026609
Vardhana M, Arunkumar N, Lasrado S, Abdulhay E, Ramirez-Gonzalez G (2018) Convolutional neural network for bio-medical image segmentation with hardware acceleration. Cognit Syst Res 50:10–14. https://doi.org/10.1016/j.cogsys.2018.03.005
Veeling B (2019) The PatchCamelyon (PCam) deep learning classification benchmark.: basveeling/pcam. Retrieved May 31, 2019 from https://github.com/basveeling/pcam
Veeling BS, Linmans J, Winkens J, Cohen T, Welling M (2018) Rotation Equivariant CNNs for Digital Pathology. arXiv:1806.03962 [cs, stat] (June 2018). Retrieved January 7, 2020 from arXiv:1806.03962
Xiao H, Rasul K, Vollgraf R (2017) Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747 [cs, stat] (September 2017). Retrieved December 27, 2019 from arXiv:1708.07747
Yang C, Rangarajan A, Ranka S (2018) Visual explanations from deep 3D convolutional neural networks for Alzheimer’s disease classification. AMIA Annu Symp Proc 2018:1571–1580
Zureick AH, Burk-Rafel J, Purkiss JA, Hortsch M (2018) The interrupted learner: How distractions during live and video lectures influence learning outcomes. Anat Sci Educ 11(4):366–376. https://doi.org/10.1002/ase.1754
Acknowledgements
The software is a derivative of work from the UT Southwestern hackathon, U-HACK Med 2018, and has continued development under the same Principal Investigator (Murat Can Çobanoğlu) and lead developer (Kevin VanHorn). The project was originally proposed by Murat Can Çobanoğlu, with preliminary draft code submitted to the NCBI-Hackathons GitHub under the MIT License. We thank hackathon contributors Meyer Zinn (UT Southwestern Medical Center), Xiaoxian Jing (Southern Methodist University), Siddharth Agarwal (University of Texas Arlington), and Michael Dannuzio (University of Texas at Dallas) for their initial work in design and development. We further thank Meyer Zinn for continued review of the manuscript and for his design of the initial RPC framework which was instrumental in the development of the work. We thank all our user study participants and thank the administration of the Lyda Hill Department of Bioinformatics for their patience and guidance.
Funding
Lyda Hill Department of Bioinformatics startup funds awarded to M.C.C.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest/Competing interests
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Supplementary file2 (DOCX 124226 KB)
Rights and permissions
About this article
Cite this article
VanHorn, K., Çobanoğlu, M.C. Democratizing AI in biomedical image classification using virtual reality. Virtual Reality 26, 159–171 (2022). https://doi.org/10.1007/s10055-021-00550-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10055-021-00550-1