Realization of a Cognitive Supervisory System for Laser Microsurgery

Abstract

Based on the models presented in Chaps. 4 and 5, here we describe the development of a prototypical supervisory system for laser microsurgery.

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Based on the models presented in Chaps. 4 and 5, here we describe the development of a prototypical supervisory system for laser microsurgery. The objective is to prove the concept formulated in Chap. 3 of a system capable of monitoring (i) the superficial temperature of tissues and (ii) the laser cutting depth during a surgical intervention.

As mentioned in the introduction of this dissertation, the work presented here is part of a broader research effort carried out at the Istituto Italiano di Tecnologia (IIT), in the scope of the European project $\upmu$RALP. This research recently culminated in the development of a novel computer-assisted surgical platform for laryngeal laser procedures, called the “$\upmu$RALP Surgical System”. The Cognitive Supervisor is implemented as a software component and integrated to this platform.

This chapter begins with a description of the $\upmu$RALP Surgical System and the hardware components it involves. Then, a technical description of the Cognitive Supervisory System is given, with a focus on its integration within the $\upmu$RALP System. Finally, we report on a preliminary evaluation study, aiming to verify the system’s performance in an operational context. Laser trials were performed, in which users’ performance in tasks involving incisions to a pre-specified depth were assessed.

6.1 Introduction: The $\upmu$RALP Surgical System

The $\upmu$RALP System is an experimental surgical system for laryngeal laser procedures. The system’s concept is illustrated in Fig. 6.1: this is inspired by the setup currently in use for transoral laser microsurgeries (TLM) (which was described in Sect. 3.1). The $\upmu$RALP system represents a revised, computer-assisted version of traditional TLM systems.

In the following, we give a description of the hardware and software components of the system.

Fig. 6.1

The $\upmu$RALP system concept. Here, the surgeon visualizes the surgical site through a virtual microscope device based on modified head-mounted display (HMD). These device is amenable to the display of virtual overlays that can be used to graphically display additional information, e.g. an intraoperative ablation plan. The position of the laser beam is controlled with a commercial input device (stylus and graphics tablet). An endoscopic device (b) brings the laser beam in proximity of the surgical site [1] (image courtesy of Dennis Kundrat, Leibniz Universität Hannover). The endoscope is equipped with a microscopic mirror that enables laser motion. The X-Ray image in (c) shows the insertion of the endoscope in the larynx (sagittal plane).

6.1.1 Hardware Components

The $\upmu$RALP system comprises the following components:

Flexible Endoscope

This device, whose first prototype is described in [1], is inserted in the larynx of the patient; it delivers the laser to the surgical field through an internal laser fiber. An actuated micro-mirror on the distal tip provides the means for laser motion [2]. Visualization of the surgical field is enabled by a pair of miniaturized high-definition cameras mounted on tip.

Surgeon Console

The console comprises a laser control device (a WACOM Bamboo Pen and Touch graphics tablet) and a virtual microscope, i.e. a stereoscopic head-mount display (SONY HMZ-T2 - 720p) that presents the stereoscopic video stream produced by the endoscope [3]. The commands imparted by the surgeon are mapped into corresponding laser beam trajectories on the surgical site. Motion scaling allows clinicians to work at a convenient scale, while their actions are adapted to the minuscule size of the target.

The console offers two distinct operating modes, called virtual scalpel and intraoperative planning. The former allows real-time user control of both laser aiming and activation; the latter modality enables the planning of laser incision/ablation patterns that, once approved, are automatically executed by the system. Plans are defined and previewed in the virtual microscope, through the superimposition of virtual overlays. Both operating modes are supplemented with the possibility to define safety region overlays, i.e. regions within the surgical area where the laser is either (i) allowed to operate or (ii) automatically inhibited, e.g. because it contains delicate structures that must be preserved.

Fig. 6.2

Architecture of the $\upmu$RALP System

Configuration Interface

This is an external display that provides supplementary visualization of the surgical field for other members of the surgical team. Through this device, an assistant can perform tasks such as system configuration or selection of the operating mode.

Control Computer

A control computer coordinates the interaction among all the components of the system, as the schematic depiction in Fig. 6.2 shows. The computer is in charge of processing user commands and verify their execution. If a hardware fault is detected, the computer may disable the laser output through a safety interlock, thus preventing potential patient injury.

6.1.2 Software Architecture

The $\upmu$RALP system adopts a distributed software architecture: different components are managed by distinct software modules, each instantiated as a separate Operating System process. These processes interact through a message passing scheme, with the Robot Operating System (ROS) [4] being used as communication middleware. The inclusion of a new module in the system requires the specification of its interface, i.e. the information it exchanges with other processed of the system.

6.2 System Implementation

This section describes the realization and integration of the Cognitive Supervisory System within the $\upmu$RALP Surgical platform. The Cognitive Supervisory System is intended to complement the clinician’s perception of the state of tissues during the laser incision process, through the visualization of estimated values of the laser cutting depth and the superficial temperature of the tissue. The estimations rely on the mathematical models derived in Chaps. 4 and 5.

Here, the models are encapsulated into shared libraries and instantiated by a ROS node called the Cognitive Supervisor (CS). This module is responsible for feeding the models with the required input and for distributing their output to other software components. The interconnection of this module in the $\upmu$RALP software architecture is shown in Fig. 6.3. The CS is enabled by the laser activation signal emitted by the laser controller (ROS Actionlib). During the application of laser power on tissue, the supervisor module produces two distinct messages; these contain estimations of the current depth of incision and tissue temperature. This information is made available to the user interface through publication on specific ROS topics. To produce the estimations mentioned above, the CS needs to know the laser dosimetry parameters in use (laser power, scanning frequency, delivery mode). These parameters are set by an assistant through the configuration interface and made available under the form of ROS parameters.

Fig. 6.3

Coordination of the software processes involved in the on-line estimation of laser cutting depth/thermal state of tissue. The estimations are calculated by the Cognitive Supervisor (CS), based on mathematical models that are loaded from file. The laser parameters required to calculate the estimations are received from the User Interface. The CS is synchronized with the activation signal produced by the Laser Controller. When the laser is activated, the CS delivers the output of the models to the User Interface, by means of a continuous flow of messages

6.2.1 Software Architecture

The class diagram of Fig. 6.4 illustrates the software architecture on which the Cognitive Supervisor is based. An App class encapsulates the data structures required by the module: this includes instances of the ROS publishers and subscribers required to establish connections with other ROS nodes in system.

Fig. 6.4

Class diagram of Cognitive Supervisor module

The Cognitive Supervisor implements a simple finite state machine with two states, namely Idle and LaserOn. This is modeled in the software architecture through the implementation of a State design pattern [5]. State transitions are handled in the main application loop, on the basis of the laser activation signal, which is polled at a frequency of 100 Hz.

The PublishInfo() method is routinely called in the main loop of the application. Exploiting the dynamic polymorphism offered by C++, this method performs different operations, according to the state of the application. The Idle class implements a null behavior [5]. By contrast, the LaserOn class performs the following sequence of operations:

• 1. estimation of the superficial temperature of tissue

• 2. estimation of the laser cutting depth

• 3. publishing the above information to the user interface

In its attributes, the LaserOn class stores the values of the laser parameters in use (laser power, scanning frequency, delivery mode). Estimations of temperature and cutting depth are demanded to classes defined in separate libraries (DepthModel and TemperatureModel).

When in the LaserOn state, the CS publishes messages containing the current estimation for the depth of incision and the superficial temperature of tissues. The specification of these messages—in the ROS message description language—is reported in Listings 6.1 and 6.2.

The cog_map message contains a temp attribute: this is a two-dimensional matrix (size: 40$\,\times \,$60 elements) representing the superficial temperature of tissue in the surroundings of the incision. Additional attributes are provided with the message: these provide a spatial mapping between the temperature surface estimated by the cognitive supervisor and the camera view of the $\upmu$RALP system. The purpose of these attributes is to provide the user interface with the information required to correctly overlap the temperature surface with the images of the surgical site.

Fig. 6.5

On-line monitoring of tissue temperature and laser cutting depth. A number indicates the depth of incision in millimeters. The superficial temperature of tissue is represented through a color map which is superimposed on the surgical visor. The laser activation indicator, on the left, turns red if the laser is on

6.2.2 Integration with the Surgical Console

The surgeon console was customized to display the information produced by the CS (see Fig. 6.5). Two widgets (graphical control elements) were added to the surgical viewer, which show (i) the superficial temperature of tissues and (ii) the progression of the incision depth. The former is represented by a colored overlay (heat map), while the latter is indicated numerically (in millimeters). These widgets can be enabled/disabled by the assistant through the configuration interface.

6.3 Towards Assistive Technologies for Laser Microsurgery

The implementation of the Cognitive Supervisory System described in this chapter provides the groundwork for further research in the domain of computer-assisted laser surgery. The system is capable of providing useful information regarding the state of tissues during laser surgery. This enables the creation of assistive technologies potentially resulting in safer and more accurate laser resections.

Here, we present a proof-of-concept study, that exploits the depth estimation capabilities of the CS to guide users during laser cutting. Tree volunteers were involved in this study. They were all members of the scientific staff at the Istituto Italiano di Tecnologia, and had limited or no prior experience with laser incision of soft tissue. Participants were asked to perform a simple task involving control of the laser exposure time to create a specific depth of incision (d$^*$ = 0.85 mm). The incision path was pre-programmed through software and kept constant throughout the experiment. Laser parameters were configured as follows: P = 3W, Continuous Wave, $t_s$ = 0.1s, $\omega _s$ = 10 Hz, l = 4.6 mm.

Each participant performed a total of six trials. In the first three trials they were required to accomplish the task only relying on their visual perception. During the three subsequent trials, they were supported with on-line estimation of depth.

Results

A comparison of the incision depths produced with and without the support of the on-line estimation system is shown in Fig. 6.6a. None of the users obtained incisions closer than 0.25 mm to the assigned target depth (0.85 mm) during the non-assisted trials. Conversely, with the support of the on-line estimation system, users produced incisions much closer to the assigned target. Deviations from the target depth are summarized in Table 6.1.

Fig. 6.6

Results of user trials. The plot on the left (a) shows the user control of incision depth with and without the support of the on-line estimation system. Measures of spread (standard deviation) are graphically represented for each sequence of trials. (b) shows the user control of exposure time during assisted trials (on-line estimation of depth was enabled during these trials). The target exposure time (3.36 s) is plotted for reference. Reproduced from [6] with kind permission from John Wiley & Sons, Ltd

The exposure times employed by users for each of these trials are represented in Fig. 6.6b. According to the forward model of depth (see Sect. 5.3), an exposure time of $t_{\textit{exp}}$ = 3.36 s is required to cut a depth of 0.85 mm. From the figure it can be observed that, with the support of the online estimation system, users produced exposure times close to this reference value. Mean deviations from the target exposure time were 0.12, 0.04 and 0.10 s for each user, respectively.

Table 6.1 Deviations (RMSE) from the assigned incision depth obtained by users with and without the assistance of the on-line depth estimation

At the present time, no other real-time assistive system exists to monitor and inform surgeons about the effects of the laser on tissues during the execution of incisions. Results of these trials suggest that on-line estimation of depth is beneficial to users who have no prior experience with laser operations. Providing depth estimations during the execution of laser incisions enabled inexperienced users to perform precise tissue cutting.