Augmented Telemedicine Platform for Real-Time Remote Medical Consultation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10132)


Current telemedicine systems for remote medical consultation are based on decades old video-conferencing technology. Their primary role is to deliver video and voice communication between medical providers and to transmit vital signs of the patient. This technology, however, does not provide the expert physician with the same hands-on experience as when examining a patient in person. Virtual and Augmented Reality (VR and AR) on the other hand have the capacity to enhance the experience and communication between healthcare professionals in geographically distributed locations. By transmitting RGB+D video of the patient, the expert physician can interact with this real-time 3D representation in novel ways. Furthermore, the use of AR technology at the patient side has potential to improve communication by providing clear visual instructions to the caregiver. In this paper, we propose a framework for 3D real-time communication that combines interaction via VR and AR. We demonstrate the capabilities of our framework on a prototype system consisting of a depth camera, projector and 3D display. The system is used to analyze the network performance and data transmission quality of the multimodal streaming in a remote scenario.


Telepresence Virtual reality Augmented reality Real-time multimedia streaming 3D interaction 



This work was partially supported by the following sources: National Science Foundation (NSF) grant #1427260, Office of Naval Research (ONR) grant #N00014-09-1-0230, and Siemens Fellowship (#20150859).


  1. 1.
    Goldzweig, C.L., Towfigh, A., Maglione, M., Shekelle, P.G.: Costs and benefits of health information technology: new trends from the literature. Health Aff. (Millwood) 28, w282–w293 (2009)CrossRefGoogle Scholar
  2. 2.
    Kvedar, J., Coye, M.J., Everett, W.: Connected health: a review of technologies and strategies to improve patient care with telemedicine and telehealth. Health Aff. (Millwood) 33, 194–199 (2014)CrossRefGoogle Scholar
  3. 3.
    Flodgren, G., Rachas, A., Farmer, A.J., Inzitari, M., Shepperd, S.: Interactive telemedicine: effects on professional practice and health care outcomes. The Cochrane Library (2015)Google Scholar
  4. 4.
    Augestad, K.M., Lindsetmo, R.O.: Overcoming distance: video-conferencing as a clinical and educational tool among surgeons. World J. Surg. 33, 1356–1365 (2009)CrossRefGoogle Scholar
  5. 5.
    Markarian, G., Mihaylova, L., Tsitserov, D.V., Zvikhachevskaya, A.: Video distribution techniques over WiMAX networks for m-health applications. IEEE Trans. Inf. Technol. Biomed. 16, 24–30 (2012)CrossRefGoogle Scholar
  6. 6.
    Pande, A., Mohapatra, P., Zambreno, J.: Securing multimedia content using joint compression and encryption. IEEE MultiMedia 20, 50–61 (2013)CrossRefGoogle Scholar
  7. 7.
    Latifi, R., Weinstein, R., Porter, J., Ziemba, M., Judkins, D., Ridings, D., Nassi, R., Valenzuela, T., Holcomb, M., Leyva, F.: Telemedicine and telepresence for trauma and emergency care management. Scand. J. Surg. 96, 281–289 (2007)CrossRefGoogle Scholar
  8. 8.
    Boniface, K.S., Shokoohi, H., Smith, E.R., Scantlebury, K.: Tele-ultrasound and paramedics: real-time remote physician guidance of the focused assessment with sonography for trauma examination. Am. J. Emerg. Med. 29, 477–481 (2011)CrossRefGoogle Scholar
  9. 9.
    Söderholm, H.M., Sonnenwald, D.H., Cairns, B., Manning, J.E., Welch, G.F., Fuchs, H.: The potential impact of 3D telepresence technology on task performance in emergency trauma care. In: Proceedings of the 2007 International ACM Conference on Supporting Group Work, pp. 79–88 (2007)Google Scholar
  10. 10.
    Garcia, P.: Telemedicine for the battlefield: present and future technologies. In: Rosen, J., Hannaford, B., Satava, R.M. (eds.) Surgical Robotics, pp. 33–68. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  11. 11.
    Irizarry, D., Wadman, M.C., Bernhagen, M.A., Miljkovic, N., Boedeker, B.H.: Using the battlefield telemedicine system (BTS) to train deployed medical personnel in complicated medical tasks-a proof of concept. In: MMVR, pp. 215–217 (2012)Google Scholar
  12. 12.
    Ling, G.S., Rhee, P., Ecklund, J.M.: Surgical innovations arising from the Iraq and Afghanistan wars. Annu. Rev. Med. 61, 457–468 (2010)CrossRefGoogle Scholar
  13. 13.
    Satava, R.M.: Virtual reality and telepresence for military medicine. Comput. Biol. Med. 25, 229–236 (1995)CrossRefGoogle Scholar
  14. 14.
    Welch, G., Fuchs, H., Cairns, B., Mayer-Patel, K., Sonnenwald, D.H., Ilie, A., Noland, M., Noel, V., Yang, H.: Improving, expanding and extending 3D telepresence. In: International Workshop on Advanced Information Processing for Ubiquitous Networks, 15th International Conference on Artifical Reality and Telexistence (ICAT 2005) (2005)Google Scholar
  15. 15.
    Jang-Jaccard, J., Nepal, S., Celler, B., Yan, B.: WebRTC-based video conferencing service for telehealth. Computing 98, 169–193 (2016)MathSciNetCrossRefGoogle Scholar
  16. 16.
  17. 17.
    Maeda, K.: Performance evaluation of object serialization libraries in XML, JSON and binary formats. In: 2012 Second International Conference on Digital Information and Communication Technology and Its Applications (DICTAP), pp. 177–182. IEEE (2012)Google Scholar
  18. 18.
  19. 19.
    Kinect projector toolkit (2016).
  20. 20.
    Hore, A., Ziou, D.: Image quality metrics: PSNR vs. SSIM. In: 20th International Conference on Pattern Recognition (ICPR), pp. 2366–2369. IEEE (2010)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.University of California at BerkeleyBerkeleyUSA

Personalised recommendations