Augmented Telemedicine Platform for Real-Time Remote Medical Consultation
Current telemedicine systems for remote medical consultation are based on decades old video-conferencing technology. Their primary role is to deliver video and voice communication between medical providers and to transmit vital signs of the patient. This technology, however, does not provide the expert physician with the same hands-on experience as when examining a patient in person. Virtual and Augmented Reality (VR and AR) on the other hand have the capacity to enhance the experience and communication between healthcare professionals in geographically distributed locations. By transmitting RGB+D video of the patient, the expert physician can interact with this real-time 3D representation in novel ways. Furthermore, the use of AR technology at the patient side has potential to improve communication by providing clear visual instructions to the caregiver. In this paper, we propose a framework for 3D real-time communication that combines interaction via VR and AR. We demonstrate the capabilities of our framework on a prototype system consisting of a depth camera, projector and 3D display. The system is used to analyze the network performance and data transmission quality of the multimodal streaming in a remote scenario.
KeywordsTelepresence Virtual reality Augmented reality Real-time multimedia streaming 3D interaction
This work was partially supported by the following sources: National Science Foundation (NSF) grant #1427260, Office of Naval Research (ONR) grant #N00014-09-1-0230, and Siemens Fellowship (#20150859).
- 3.Flodgren, G., Rachas, A., Farmer, A.J., Inzitari, M., Shepperd, S.: Interactive telemedicine: effects on professional practice and health care outcomes. The Cochrane Library (2015)Google Scholar
- 9.Söderholm, H.M., Sonnenwald, D.H., Cairns, B., Manning, J.E., Welch, G.F., Fuchs, H.: The potential impact of 3D telepresence technology on task performance in emergency trauma care. In: Proceedings of the 2007 International ACM Conference on Supporting Group Work, pp. 79–88 (2007)Google Scholar
- 11.Irizarry, D., Wadman, M.C., Bernhagen, M.A., Miljkovic, N., Boedeker, B.H.: Using the battlefield telemedicine system (BTS) to train deployed medical personnel in complicated medical tasks-a proof of concept. In: MMVR, pp. 215–217 (2012)Google Scholar
- 14.Welch, G., Fuchs, H., Cairns, B., Mayer-Patel, K., Sonnenwald, D.H., Ilie, A., Noland, M., Noel, V., Yang, H.: Improving, expanding and extending 3D telepresence. In: International Workshop on Advanced Information Processing for Ubiquitous Networks, 15th International Conference on Artifical Reality and Telexistence (ICAT 2005) (2005)Google Scholar
- 16.Protocol buffers (2016). https://developers.google.com/protocol-buffers
- 17.Maeda, K.: Performance evaluation of object serialization libraries in XML, JSON and binary formats. In: 2012 Second International Conference on Digital Information and Communication Technology and Its Applications (DICTAP), pp. 177–182. IEEE (2012)Google Scholar
- 18.Kinect for windows (2016). https://developer.microsoft.com/en-us/windows/kinect
- 19.Kinect projector toolkit (2016). https://github.com/genekogan/KinectProjectorToolkit
- 20.Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 20th International Conference on Pattern Recognition (ICPR), pp. 2366–2369. IEEE (2010)Google Scholar