Keywords

1 Introduction

Telemedicine refers to clinical and healthcare services that are provided at a distance via the use of telecommunication and information technologies. To date, telemedicine has been applied in many areas of medical care, such as emergency medicine, remote diagnosis, surgery, rehabilitation, and many others. It has been well documented that telemedicine can improve access to medical services and their quality for rural or poor communities [1]. Telemedicine is also being used in critical care to connect patients in the field with medical professionals at trauma centers. Based on the means of interaction between providers or provider and patient, the telemedicine systems can be roughly divided into two categories: (1) synchronous or real-time systems and (2) asynchronous or store-and-forward systems [2]. The advantage of the synchronous approach is in its ability to immediately address the care and provide support for time-critical clinical decisions. The asynchronous approach on the other hand entails of exchanging relevant information over the network, analyzing it offline, and then providing a medical decision. This approach most often does not entail direct interaction with the patient but is rather focused on interpretation of medical data (e.g. radiology imaging, ultrasound, EKG, etc.). In the remainder of this paper we therefore focus on the synchronous or real-time telemedicine.

Real-time telemedicine systems have only in recent years been adopted in emergency medicine. In civilian and military scenarios, remote support is particularly critical in the so-called “golden hour” of trauma, where the efficiency of care greatly affects the outcome. Telemedicine can be applied at various stages of emergency care from triage and transport to consultation with specialists in the hospital. Typical solutions include two-way videoconferencing with single or multiscale views of the room and patient, alongside vital signs, medical imaging, and interviews with all participants, including the patient and family [2]. This vast information space, however, can be overwhelming and can limit critical decision-making due to the information bottlenecks and technology-imposed limitations on communication between specialist and patient-side care provider. Although the video information and vital signs are helpful, the feedback information provided to the emergency responders is typically only conveyed verbally, limiting the capacity of provided remote support. There is a need to deliver the information more concisely while minimizing the effort to retrieve relevant information generated locally or remotely at any given moment. In this paper, we propose a novel augmented telemedicine platform based on the emerging technologies of 3D teleimmersion, virtual reality (VR) and augmented reality (AR) that will address some of these issues and introduce disruptive capabilities in all the stages of collection, representation, transmission, and user interaction.

2 Background

Telemedicine technology began in the late 1960’s and early 1970’s in the NASA space program to monitor biometric data and provide guided medical treatment by non-physicians for astronauts during space flights [3]. This technology was subsequently applied to deliver healthcare to remote Indian Papago Tribe in Arizona as part of the project called STARPAHC (Space Technology Applied to Rural Papago Advanced Health Care) in the 1970’s [4]. Telemedicine was first deployed in a disaster in the aftermath of 1985 earthquake in Mexico City and later on during several other events around the world, such as 1988 earthquake in Armenia [5, 6]. This technology was early on embraced also by the United States Department of Defense (DoD) during 1980’s and 1990’s to provide support during global military missions. Although many bases had on-site clinics, the telemedicine made much broader range of physicians and specialists available to deployed soldiers. During the Persian Gulf War, telecommunication technologies were integrated into mobile health units providing transmission of CT images via satellite and international telephone network. Similar technology was also used during the military interventions in Somalia and Bosnia, where commercially available technology was used to transmit x-rays, ultrasound, CT scans, and other medical imaging data, and full-motion videos to remote field hospitals for diagnostic support [5]. Despite the advances in communication and video technology, wider use of telemedicine in subsequent wars in Iraq and Afghanistan was hindered by the lack of satellite bandwidth [7]. Instead, medical personnel used email to send pictures of casualties, wounds, and medical records. Many of the doctors still prefer this “store-and-forward” methods, possibly also due to lack of familiarity with the alternative real-time technologies.

With the emergence of internet-based communications in the late 1990’s, telemedicine became more widely used in civilian practice. The internet provided a cost-efficient form of communication independent of the hardware. Initially used for teleconsultation between specialists, telemedicine started to be employed in emergency medical care to connect small and rural hospitals with specialty centers to address emergency situations. One of the first large centers that provided real-time consultation services was the University of California Davis Health System with its California Tele-health Network [8]. Today there are 14 Telehealth Resource Centers across the United States that focus on advancing effective use of telemedicine [9]. Telemedicine has also become more prevalent in emergency care to reduce overcrowding of emergency rooms and to provide support to rural areas. Recent meta-review of 38 reports on the emergency telemedicine in the US by Ward et al. [10] found overall positive results in terms of improving clinical processes, outcomes, and throughput.

With advances in mobile communications, the telemedicine technology can also be deployed in ambulances. Such systems can assist paramedics during triage and provide pre-hospital treatment under direct supervision of an expert physician by transmitting vital signs and video imaging to the care center [11]. In another example, Boniface et al. [12] performed a feasibility study of remotely guiding paramedics to obtain ultrasound imaging in the field for immediate diagnosis.

3 Current Limitations

Current real-time telemedicine technology, whether in military or civilian domain, faces several limitations. Upmost is the information overload as the data from various sources, such as video, vital signs, medical records, have to be interpreted concurrently by the expert/specialist for critical decision making. This often entails looking at several monitors or even using a camera to transmit vital signs information captured on the screen.

Next, instructions for the guided medical evaluation or procedure need to be conveyed to first responder, typically via audio communication. Although the modern telemedicine systems often include two-way video communication, the video is primarily used only to transmit the view of the patient. While some systems support pan-tilt-zoom cameras, if a different camera position is required, the expert has to instruct the responder to reposition the camera in order to obtain the desired view. When communicating with the responder, the expert cannot directly use gestures to provide instructions of the procedures since the video feedback is dislocated from patient’s bedside and typically displayed on an external monitor.

Using only 2D video to display patient condition further limits the information available to the physician. By scanning and transmitting 3D surface information could provide additional cues to perceive the extent of the trauma as the expert may be able to detect bruising, swollen tissue, etc. In addition, combing the video feedback with other modalities could improve initial information needed to make informed decisions regarding emergency care and transportation.

4 Virtual and Augmented Reality

Use of virtual reality in telemedicine was first envisioned by Satava [13] who proposed using remote sensors, intelligent systems, telepresence surgery, and virtual reality surgical simulations, to improve combat casualty care in battlefield scenarios. In his view of the modern medical battlefield, he proposed using streaming video, vital signs information, and medical imaging transmitted to the telemedicine unit who would instruct the combat medic in the field. Before the evacuation, the critical care could be provided via remote surgery using an instrumented Trauma Pod, where the surgeon could combine live video data with VR models from a remote, safe location. Although significant advances have been made to date, the remote surgery using telepresence has only been accomplished on shorter distances in few cases due to network limitations in coverage and latency. The virtual reality has, on the other hand, been applied to considerable degree in training simulations to improve emergency care. In further review of VR technologies we limit ourselves only to those that can enhance real-time interaction and communication between geographically distributed individuals.

Recent advances in sensor technologies and miniaturization have made possible a robust and minimally intrusive digital acquisition of high quality video data (e.g., HD, 4K video), 360-degree video, real time 3D surface reconstruction (e.g., Microsoft Kinect, Intel RealSense), and acquisition of vital signs with various wearable sensors. Furthermore, modern smartphones and tablets, which are equipped with an array of sensors, high quality video camera, audio capabilities, and high bandwidth wireless network connectivity can be used in future telemedicine applications to provide better data acquisition. Visualization and interaction with collected data can be further enhanced by moving away from traditional computer displays to more immersive 3D stationary or wearable displays (i.e., head mounted displays). Another opportunity to reduce the information overload and provide the vital signs at the reach of the hand is to use augmented reality wearable displays which can show information overlain on the real world environment. The use of aforementioned technologies in remote collaboration for emergency telemedicine, however, requires a different paradigm of interaction.

Although the quality of experience with regard to visual and audio transmission has significantly improved over the years, the 2D video technology has several inherent drawbacks that cannot be mitigated easily. These include partial loss of non-verbal cues such as gestures and eye contact, which have been shown to increase trust, collaboration and productivity in video-enabled interactions [14]. In addition to the loss of non-verbal cues, there is a disconnect between the users and the content. In case of the telemedicine applications, the physician primarily conveys information back to the responder via verbal instructions. Video-based telepresence experience can be enhanced by 3D telepresence and 3D teleimmersion where remote interaction takes place in a virtual reality environment [15]. 3D teleimmersion technology for remote interaction has been explored by several researchers in the past decade (see the review in [16]), including our group at University of California, Berkeley. In our demonstration systems that we have developed over the years, we have investigated various aspects of interaction, networking, data compression, rendering, and others, while working with multi-disciplinary groups of users, including medical doctors. One of the lessons learned addressing the technological barriers includes the need to use low-complexity and off-the-shelf technologies that can be easily deployed and automatically calibrated in a real-world environment. In the proposed framework, we thus plan to facilitate real-time interaction between the physician and the emergency responder through the virtual and augmented reality technology that is already commercially available.

5 Augmented Telemedicine Platform

In this section we describe a novel telemedicine platform that addresses the existing technology gaps and breaks the information bottlenecks in current emergency telemedicine systems. The proposed platform shown in Fig. 1 consists of several compelling new features. First, a real-time patient data acquisition and visualization module captures patient’s external trauma and vital signals in the field and precisely renders the signals for remote physicians via 3D teleimmersion. Second, an AR module allows physicians and first responders to examine patient data and collaborate in simple medical procedures via gestures and annotations. Finally, a telemedicine communication module enables secure and robust deployment of the above two modules over long distance through wireless network.

Fig. 1.
figure 1

Diagram of the augmented telemedicine system for remote interaction between a physician and an emergency responder.

Full size image

In an example scenario, the augmented telemedicine platform can connect an ambulance with a local trauma center. A 3D image acquisition module is installed inside the ambulance. Once a patient is placed into the ambulance, the module scans the patient’s body with millimeter accuracy. The acquired 3D representation of the patient accurately captures external trauma and the visual appearance of the injury. To examine for internal bleeding or hidden injuries, the first responder can use ultrasound to scan the patient’s body and augment the patient’s 3D avatar in real time. During the procedure, measured medical data of the patient are transmitted to the trauma center in real time. The attending physician at the trauma center can then review the patient’s full-body medical condition through a virtual reality 3D display device and recommend treatments before and for the duration of the transportation until the patient arrives at the hospital. During this interaction, the physician is able to annotate the 3D avatar in the virtual environment to provide additional information. The annotations can be virtually attached to the patient’s body by registering them to detected visual features on the skin or clothing. The first responder is subsequently able to view these annotations in real time via an augmented reality display device or a dynamic projection onto the body part of interest. The annotations can provide additional cues to guide the triage and can be retrieved when the patient arrives at the hospital. In this example, the augmented telemedicine platform is used to provide more seamless two-way interaction between the physician and first responder that cannot be easily replicated with traditional video communication.

In the reminder of this section we provide further details on each module of the proposed platform:

3D data acquisition and visualization. To improve the visual information that can be obtained by standard 2D cameras, we propose to use one or more RGB+D cameras that can generate the depth information of the captured scene in addition to high resolution texture. The depth information can be used to reconstruct a 3D model of the entire body or a body part. The principle of 3D acquisition can be based on multi-view stereo, stereo reconstruction with structured light or time-of-flight (ToF). Each of these technologies has certain advantages and disadvantages [17], which are beyond the scope of this paper. In our prototype implementation we use Microsoft Kinect sensor, which provides accurate (with errors between 2–4 mm [18]) real-time 3D reconstruction based on ToF with 30 Hz acquisition rate.

Figure 2 shows the output of the Kinect camera observing a mockup setup of an individual on a stretcher. The obtained depth and color data can be used to obtain the real-time 3D avatar representation of a patient’s body and movement within the field of view of the acquisition system. The 3D acquisition from a stationary unit can be further combined with a handheld or adjustable scanner to provide a close-up and more detailed view of the injury. After the acquisition, the RGB+D video data are compressed and sent over the network. On the receiving end, the imagery data are converted into a 3D surface represented by a textured mesh (Fig. 3). This allows the viewer to select any viewpoint to examine the area of interest.

Fig. 2.
figure 2

RGB texture (left) and depth data output (right) generated by Kinect for Xbox One.

Full size image
Fig. 3.
figure 3

Reconstructed 3D mesh with annotations rendered from an arbitrary point of view (left) and user interacting with the 3D data on zSpace system (right).

Full size image

Immersive interaction. We will develop more intuitive virtual interaction methods that allow medical professionals to examine the patient remotely and also interact with the provider at the patient side. In particular, we are investigating remote collaboration with a stationary solution via zSpace virtual reality display technologyFootnote 1 and a wearable solution via upcoming Microsoft HoloLens technology or other wearable AR glasses. The primary role of the display system is to receive the data from the 3D acquisition module and render them in real time as a 3D surface to be viewed by the medical professionals. The display system also provides tools to annotate the data virtually using a stylus. The advantage of using a stationary 3D display, such as zSpace, as opposed to more immersive wearable head mounted display, such as Oculus Rift, is in a more comfortable interaction that does not have adverse effects linked to head mounted displays (e.g., motion sickness, eye discomfort and other temporary visual changes). The zSpace display system also includes a stylus that can be used to directly interact with the 3D data, create annotations and gesture inputs. The annotations and gestures are then transmitted back to the responder in the field and displayed as augmented reality features that can be either projected onto the patient from a mounted projector or viewed via an AR display, such as the Microsoft HoloLens. The interaction module with the aforementioned features is aimed to guide the first responders during the initial triage and to provide critical care to the patient during transport.

Telemedicine communication. The robustness and reliability of the communication is critical during any remote interaction. As mentioned several studies have identified the availability of the network and bandwidth restrictions as one of the limitations of applying real-time telemedicine in the field. With the advances in mobile networking and increased coverage of 4G/LTE networks, it is now possible to connect ambulances in most urban and rural areas to a local trauma center without dedicated satellite connection. Our approach aims to leverage the open source real-time peer-to-peer communication standard WebRTC, which is becoming increasingly popular to securely stream video, audio and other multimedia data between different devices. In our past work we have investigated the use of WebRTC to stream Kinect data for telerehabilitation [19]. One of the main features of WebRTC is the ability to connect clients through various firewalls while being able to dynamically adapt the multimedia transmission quality according to the available network bandwidth. We are currently investigating new compression methods for RGB+D video data that can be efficiently transmitted and adapted to the network while preserving important details in the captured data. Furthermore, the WebRTC connection will also be used to transmit vital signs data, medical imaging, and interaction data, such as gestures and annotation, in order to synchronize the two clients.

Medical imaging integration. For future work, we intend to further augment the visualization capabilities of the telemedicine platform with the integration of selected medical imaging modalities with the immersive telemedicine platform. In particular, we will integrate a portable ultrasound imaging module and overlay the ultrasound image on the patient’s avatar that the physician will be able to review in real time. By having real time information available, the physician will be able to direct the first responder to perform the ultrasound examination with not just voice cues but also visual annotations. A key technology gap in this area is visual 3D localization (registration) with the patient body.

6 Discussion and Conclusion

The proposed platform combines off-the-shelf AR and VR technology to create a more interactive experience and improve the communication between first responders and emergency care physicians. As we develop this platform, we are interested in answering the following research questions:

  • What is the required 3D resolution of the data acquisition to assess the extent of trauma remotely and add value to the current video technology? What is the minimal amount of information needed?

  • Importance of certain features of the proposed platform to streamline the collaboration (e.g. real time 3D visualization, use of annotations, integration with vital signs data, etc.)

  • How does network latency, jitter and overall connectivity affect the quality of experience (QoE)?

  • How will communication with the VR/AR system change the interaction between the first responders and physician in time-critical scenarios?

  • How to perform 3D reconstruction, automatic registration and data fusion of multiple depth-sensing cameras, each with different resolutions?

  • How to track features of deformable objects to allow for real time registration of annotations to the human body?

  • What methods are available to provide some form of effective haptic feedback to the remote user?

Some of the research questions will be examined during the development phase, where we will test the network connectivity and its effects on video resolution and interaction in terms of bandwidth and transmission delays/jitter. This information will be used to build in the mechanisms to adapt the 3D video transmission (e.g., reducing frame rate, increasing video compression ratio, etc.). We will work with our medical collaborators to understand the importance of various features for the specific emergency scenarios. Furthermore, we will collect user-feedback and collaboration performance metric [20] related to the interaction with the system (e.g., movement of the stylus, changes in viewpoint, user-interface interactions, etc.) and between users (e.g., use of annotations, references to visual markers, use of audio communication, etc.). This information will be used to further improve the system in future iterations.

We envision that the proposed augmented telemedicine system combining AR and VR can bring about disruptive new capabilities in telemedicine to achieve better patient triage and prehospital emergency care, combat casualty care in the battlefield, or during natural disasters. An important role of the augmented telemedicine platform will also be in the training of first responders and emergency care physicians. By taking advantage of AR technology, it will be possible to create various simulation scenarios with patient actors or mannequins to practice different situations. Using remote training capabilities of this platform, the first responders could be engaged in a training simulation scenario while waiting for a new on-site call.

Although the focus of this paper was on the emergency care, the proposed technology can be applied to other areas of telemedicine to facilitate more effective care by connecting patients with general and specialized providers. These include telerehabilitation services, occupational therapy, patient education, clinical trials, and others. Reducing the cost of transportation is an integral part of reducing the total cost of the current healthcare system.