Keywords

1 Motivation

Minimally-invasive procedures have been extensively studied in the last decades and new solutions for various applications are an active research field [2]. These include, among others, continuum robots for drilling in multi-port temporal bone surgery [6], flexible needles for soft tissue [4] or flexible endoscopes [7] and allow more precise interventions.

These approaches use instruments that share common constraints: they follow nonlinear curvature constrained trajectories, and rapid re-planning is necessary to ensure a continuous safe insertion. Consequently, pre- and intra-operative planning in \(SE(3)=\mathbb {R}^3\times SO(3)\) is necessary to compute feasible trajectories with maximum clearance to risk structures.

Isolated solutions for the underlying motion planning problems have been proposed for many applications: computation of implant channels in intracavitary brachytherapy or trajectory planning for bevel tip needles [3], planning access paths for temporal bone surgery [5] or needle planning for liver surgery [13]. Finding feasible trajectories then requires nonholonomic motion planning where sampling based algorithms like Rapidly-exploring Random Trees (RRT) are well suited for [8]. Steering bevel-tip needles in soft tissue has been extensively studied [1] and both RRTs [9] and sequential convex optimization [10] have been shown to compute feasible trajectories. Convex Optimization has also been used to plan for automated suturing [11]. Nonlinear drilling units have been proposed to create access paths in temporal bone surgery and Bidirectional-RRT (Bi-RRT) were used to interpolate between start and goal states in \(SE(3)\) [6].

Fig. 1.
figure 1

Exemplary clinical applications where flexible instruments can be used: (A) temporal bone surgery for cochlear implantation (B) cardiopulmonary endoscopy, (C) spine biopsy and (D) tumor treatment in the liver.

Full size image

However, these solution are tailored to their specific use case and do not discuss a general solution. In this paper, we propose a general motion planning formulation for nonlinear minimally-invasive interventions in OR 2.0. We extend the formulation of Bi-RRTs introduced earlier by us [6] that exploit variable curvature arcs or Bézier-splines as underlying steering functions. This extension is suitable to form a common motion planning problem for instruments that follow curvature constrained trajectories. In particular, we derive the individual specifications for four different clinical applications: temporal bone surgery, cardiopulmonary endoscopy, spine biopsy and liver tumor treatment (Fig. 1). Experiments on data sets of real patients are presented where our methods successfully plan trajectories for the respective interventions.

2 Materials and Methods

2.1 Clinical Challenges:

All mentioned interventions - though quite similar in motion planning - offer unique challenges.

Temporal Bone Surgery operates in a very small and dense environment compared to other setups. Numerous obstacles - nerves, blood vessels and the organs of the hearing and equilibrium senses - limit the free space and thus complicate motion planning. This raises special needs for the extension of the search tree as well as the collision detection.

In cardiopulmonary endoscopy trajectories have to be planned through tube-like structures. Motion planning algorithms have to find feasible paths through instead of around risk structures. Such narrow environments often need tailored algorithm for sufficiently fast planning [14].

Spine biopsy and liver tumor treatment provide environments where the spinal cord or branches of the hepatic artery and portal vein, respectively, form highly sensitive regions where precise planning is critical. In fact, Sun et al. [12] extended planning to Belief Spaces in order to limit uncertainty.

Additionally, an automatic procedure requires to continuously reevaluate the planned path. Given the latest sensory inputs, a new trajectory must be re-planned from the currently measured pose of the instrument to the target of the intervention. Depending on the success of a call to solve the motion planning problem, feedback must be given to the surgeon if the intervention can still be carried on or if it has to be canceled due to unavailability of feasible trajectories. Such feedback needs to come immediately to enforce a smooth intervention.

2.2 Problem Formulation

Planning is done in the special Euclidean group \(SE(3)=\mathbb {R}^3\times SO(3)\), to account for the instrument’s position (\(\mathbb {R}^3\)) and its orientation (SO(3)), the latter represented by quaternions. The configuration space \(C\subset ~SE(3)\) is then divided into an obstacle region \(C_{Obs}\subset C\) and the free space \(C_{free} = \left\{ q\in C\vert q\notin C_{Obs} \right\} \). Valid start and goal states of trajectories are defined via subsets of \(C_{free}\). Given, a set \(M\subset C_{free}\) and the quaternion metric \(\rho : SO(3)\times SO(3)\rightarrow \mathbb {R}\) (e.g. [8]),

$$\begin{aligned} \begin{aligned} \rho (h_1, h_2)&= \text {min}\left\{ \rho _s(h_1,h_2), \rho _s(h_1,-h_2)\right\} \\ \rho _s(h_1, h_2)&= cos^{-1}(a_1a_2+b_1b_2+c_1c_2+d_1d_2), \end{aligned} \end{aligned}$$
(1)

we define the approximated set \(\tilde{M}(\epsilon ,\phi )\) of M, \(\epsilon \in \mathbb {R}^+, \phi \in [0, \pi ]\) as,

$$\begin{aligned} \tilde{M}(\epsilon , \phi ) = \{&q(x,h)\in C_{free}~|~\exists ~\hat{q}(y,g)\in M : \Vert x-y\Vert _{\mathbb {R}^3}<\epsilon , \rho (h,g)<\phi \}. \end{aligned}$$
(2)

Given a number of clinically ideal configurations for trajectories, such sets resemble clinically acceptable states that lie in the vicinity of the position and observe only a small perturbation in orientation. Further constraints are given by the minimum distance \(d_{min}\) to risk structures, the instrument’s curvature constraint \(\kappa _{max}\) and the time \(T_{max}\), in which a feedback is required. The problem formulation for an individual intervention is than expressed as:

Given,

$$\begin{aligned}&M_I\subset C_{free}, \epsilon _I\in \mathbb {R}^+, \phi _I\in [0,\pi ]\end{aligned}$$
(3a)
$$\begin{aligned}&M_G\subset C_{free}, \epsilon _G\in \mathbb {R}^+, \phi _G\in [0,\pi ]\end{aligned}$$
(3b)
$$\begin{aligned}&d_{min} \in \mathbb {R}^+, \kappa _{max} \in \mathbb {R}^{0+}, T_{max}\in \mathbb {R}^+ \end{aligned}$$
(3c)

Task: Find a path \(\gamma (t) : [0,1] \rightarrow SE(3)\) satisfying

$$\begin{aligned}&\gamma (0)\in \tilde{M}_I(\epsilon _I, \phi _I) \end{aligned}$$
(4a)
$$\begin{aligned}&\gamma (1)\in \tilde{M}_G(\epsilon _G, \phi _G) \end{aligned}$$
(4b)
$$\begin{aligned}&\forall t\in (0,1): \left\| \gamma ''(t)\right\| < \kappa _{max} \end{aligned}$$
(4c)
$$\begin{aligned}&\forall t\in [0,1], o\in C_{Obs}: \left\| \gamma (t)-o\right\| _{\mathbb {R}^3} > d_{min} \end{aligned}$$
(4d)

or report that no path could be found in the available time \(T_{max}\).

Figure 2 shows examples of initial and goal regions, \(M_I, M_G\), for a multi-port cochlear access. For preoperative planning of potential access canals, a surgeon manually defines a set of initial states at the surface of the lateral skull base (blue arrows, left image). Three goal states are defined at the round window of the cochlea as the ideal end points of the three canals for multi-port access (orange arrows, right image). Once the intervention starts, re-planning of a feasible trajectory might be necessary. Here, the current pose of the drilling unit replaces the initial region (middle image, orange arrow) and one of the three goal states is fixed as the single target state.

Fig. 2.
figure 2

Different initial and goal regions for cochlear implantation. (Left) Multiple initial states at the skull’ surface (blue arrows). (Middle) A single initial state pointing in the robot’s current direction. (Right) Three precise goal states for a multi-port cochlear access. (Color figure online)

Full size image

Note: This definition extends our previous formulation [6] to individual approximations at both start and goal. With \(\kappa _{max}=0\) it is suitable for linear approaches. With \(\phi _I=\pi \) or \(\phi _G=\pi \) it falls back to more general cases where only the orientation at one end point of the trajectory is relevant.

2.3 Motion Planning

We use Bi-RRTs to solve the individual motion planning problems of the experiments (Fig. 3). Specifically, we show that two variants of Bi-RRTs - one based on circular arcs and 3D Dubins Paths, one based on Bézier-Splines [6] - are suitable for our clinical exemplary anatomies.

Fig. 3.
figure 3

Bidirectional RRTs grow two search trees - one from the initial region (blue), the other from the goal region (green) - and attempts to connect them in between. A successful connection results in a feasible nonlinear trajectory (orange). (Color figure online)

Full size image

3 Experimental Results

We considered four different scenarios as shown in Fig. 1: (A) Planning of three access canals for multi-port bone surgery at the Otobasis. (B) Trajectory planning for cardiopulmonary endoscopy. (C) Flexible needle path creation for spine biopsy. (D) Access to metastases in the liver. For each scenario, expert annotations on real CT data were used to create 3D models of the individual anatomies. The obstacle regions \(C_{Obs}\) were built from the relevant risk structures of these anatomies. Adequate definitions of the general problem definition are given in Table 1. Samples of successfully planned paths in \(SE(3)\) are shown as tubes in Fig. 4.

Table 1. Parameters for the different Problem Formulations.
Full size table

Otobasis Surgery: Our current drilling prototype has a curvature constraint of 0.05 mm\(^{-1}\). We considered a more flexible version to have more space for multi-port surgery. For a cochlear implant, deviations at the target should not exceed \(5^\circ \). However, as our methods allow planning with ideal orientation, we set \(\epsilon =\phi =0\) and successfully created three access canals to the cochlea with no misalignment using a bidirectional Spline-Based-RRT.

Cardiopulmonary Endoscopy: Trajectories were planned both with a Spline-Based-RRT and its bidirectional counterpart. We considered the use of flexible endoscopes with radius 1.0 mm. Experiments with different additional safety-distances to the vessel’s inner walls showed that planning was still possible with \(d_{min}=3.0\) mm as a combined distance of radius and safety-distance. To create paths with the simpler RRT, too, we allowed a small error in target location (2.0 mm) and a quite high deviation from the supposed orientation (\(45^\circ \)).

Fig. 4.
figure 4

Feasible paths for drilling units in Otobasis surgery (A), endoscopes in cardiopulmonary interventions (B), needles in spine biopsy and liver tumor removal (C,D).

Full size image

Spine Biopsy: Next, we planned for percutaneous needle insertion. The minimal distance to obstacles resembled a needle of radius 1.0 mm and a safety distance of 3.0 mm to keep away from vertebrae. The curvature constraint was set according to flexible needles currently used in research [13]. As such instruments move along circular arcs, trajectories were computed with the Bi-RRT that extends via circular arcs and attempts connection with 3D-Dubins-Paths.

Liver Tumor: Last, we planned a needle trajectory for tumor treatment in the liver. A potential tumor of spherical shape was placed within the liver (Fig. 4 D) and a path interpolating between a single initial state and a single goal state was computed. We again considered the use of flexible bevel-tip needles and thus chose the parameters and motion planner as in the spine biopsy experiment.

Except for the second scenario, feasible trajectories were found much quicker than the given 0.25 s. Thus, immediate feedback was possible for these scenarios. For the endoscopic access, we had to extend the given time for solving the motion planning problem. This can be explained by necessary adaptations for RRTs, when planning in narrow environments [14], a feature our planners still lack. Moreover, maximum clearance to organs in the near vicinity is often the most important clinical requirement. Thus, after successful planning, we computed distances to risk structures along the resulting paths by sampling points along the trajectories every 0.1 mm. Table 2 shows the minimum, median and maximum distances to obstacles for the four different scenarios. We observed, that the threshold \(d_{min}\) is often almost perfectly matched. This is expected, as RRTs extend their search trees randomly and no optimization is performed.

Table 2. Resulting distances to risk structures in mm.
Full size table

4 Discussion and Conclusion

Minimally invasive surgeries with flexible instruments offer safer, more adaptable or completely automated procedures for a variety of clinical applications. In this paper we propose a general formulation of the necessary trajectory planning step for OR 2.0. We evaluate the theoretical definition on four clinical applications and show that the general formulation is adaptable to these scenarios. Further, we show that the proposed planning algorithms, bidirectional Rapidly-exploring Random Trees, are suitable tools to quickly compute precise nonlinear trajectories for instruments of such applications.

Currently, our planning method is purely geometric and does not consider uncertainty of any kind. In future, we want to add an optimization step to address noisy sensor measurements or dynamic constraints such as soft tissue deformation. For percutaneous needle insertion, convex optimization [10] has been shown to be an adequate technique. We expect that an adjustment of this method to interpolation in \(SE(3)\) between start and goal states will further improve the proposed bidirectional approach by maximizing clearance to obstacles (Table 2). Interactive definition of the motion planning problem, subsequent trajectory planning and its visualizations shown in this paper were implemented in a custom planning tool. We also plan to publish this work as an open-source library to establish a general framework for nonlinear minimally-invasive interventions.