Skip to main content

Advertisement

Log in

3D position estimation of flexible instruments: marker-less and marker-based methods

International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose Endoscopic images can be used to allow accurate flexible endoscopic instrument control. This can be implemented using a pose estimation algorithm, which estimates the actual instrument pose from the endoscopic images.

Methods In this paper, two pose estimation algorithms are compared: a marker-less and a marker-based method. The marker-based method uses the positions of three markers in the endoscopic image to update the state of a kinematic model of the endoscopic instrument. The marker-less method works similarly, but uses the positions of three feature points instead of the positions of markers. The algorithms are evaluated inside a colon model. The endoscopic instrument is manually operated, while an X-ray imager is used to obtain a ground-truth reference position.

Results The marker-less method achieves an RMS error of 1.5, 1.6, and 1.8 mm in the horizontal, vertical, and away-from-camera directions, respectively. The marker-based method achieves an RMS error of 1.1, 1.7, and 1.5 mm in the horizontal, vertical, and away-from-camera directions, respectively. The differences between the two methods are not found to be statistically significant.

Conclusions The proposed algorithms are suitable to realize accurate robotic control of flexible endoscopic instruments, enabling the physician to perform advanced procedures in an intuitive way.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

References

  1. Abbott D, Becke C, Rothstein R, Peine W (2007) Design of an endoluminal NOTES robotic system. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS). San Diego, pp 410–416 doi:10.1109/IROS.2007.4399536

  2. Bardou B (2011) Développement et étude d’un système robotisé pour l’assistance à la chirurgie transluminale. Ph.D. thesis, Université de Strasbourg

  3. Bardou B, Nageotte F, Zanne P, De Mathelin M (2012) Improvements in the control of a flexible endoscopic system. In: Proceedings of the IEEE international conference on robotics and automation (ICRA). St. Paul, pp 3725–3732

  4. Bardou B, Nageotte F, Zanne P, de Mathelin M (2009) Design of a telemanipulated system for transluminal surgery. In: 31st annual international conference of the IEEE EMBS

  5. Bouguet JY Camera calibration toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib_doc

  6. Chaumette F, Hutchinson S (2006) Visual servo control. I. basic approaches. IEEE Robot Autom Mag 13(4):82–90. doi:10.1109/MRA.2006.250573

    Article  Google Scholar 

  7. Doignon C, Nageotte F, Maurin B, Krupa A (2008) Pose estimation and feature tracking for robot assisted surgery with medical imaging. In: Kragic D, Kyrki V (eds) Unifying perspectives in computational and robot vision. Springer, Berlin, pp 79–101

    Chapter  Google Scholar 

  8. Fisher R (1936) The use of multiple measurements in taxonomic problems. Ann Eugen 7(2):179–188

    Article  Google Scholar 

  9. Gonzalez RC, Woods RE (2002) Digital image processing. Prentice Hall, Englewood Cliffs

    Google Scholar 

  10. Harris J, Stocker H (1998) Handbook of mathematics and computational science. Springer, Berlin

    Book  Google Scholar 

  11. Kalloo AN et al (2004) Flexible transgastric peritoneoscopy: a novel approach to diagnostic and therapeutic interventions in the peritoneal cavity. Gastrointest Endosc 60(1):114–117. doi:10.1016/S0016-5107(04)01309-4

    Article  PubMed  Google Scholar 

  12. Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 18(1):50–60

    Google Scholar 

  13. Marchand É, Chaumette F (2002) Virtual visual servoing: a framework for real-time augmented reality. Eurographics 21(3):289–298

    Google Scholar 

  14. Marescaux J, Dallemagne B, Perretta S, Wattiez A, Mutter D, Coumaros D (2007) Surgery without scars: report of transluminal cholecystectomy in a human being. Arch Surg 142(9):823–826

    Article  PubMed  Google Scholar 

  15. Nakamura Y (1991) Advanced robotics, redundancy and optimization. Addison-Wesley, Reading

    Google Scholar 

  16. OpenGL: The industry standard for high performance graphics. http://www.opengl.org

  17. Reilink R, Stramigioli S, Misra S (2011) Three-dimensional pose reconstruction of flexible instruments from endoscopic images. In: IEEE/RSJ international conference intelligent robots and systems (IROS). San Francisco, pp 2076–2082

  18. Reilink R, Stramigioli S, Misra S (2012) Pose reconstruction of flexible instruments from endoscopic images using markers. In: Proceedings of the IEEE international conference on robotics and automation (ICRA). St. Paul, pp 2939–2943

  19. SciPy: scientific tools for Python. http://www.scipy.org

  20. Stramigioli S, Bruyninckx H (2001) Geometry and screw theory for robotics. Seoul

Download references

Acknowledgments

This research is conducted within the TeleFLEX project, which is funded by the Dutch Ministry of Economic Affairs and the Province of Overijssel, within the Pieken in de Delta (PIDON) initiative. The ANUBIS endoscopic instrument was provided by Karl Storz GmbH & Co. KG, Tuttlingen, Germany. The endoscope was provided by Olympus Corp., Tokyo, Japan.

Conflict of interest

This is to certify that the authors have no financial or personal relationships with other people or organizations that would inappropriately influence our work.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Rob Reilink.

Appendix

Appendix

Here we show the derivation of the analytical Jacobian \(\mathbf J _\mathbf f (\mathbf q )\) of the forward kinematics function \(f(\mathbf q )\) in (2). We define five frames on the instrument (Fig. 13). Frame \(\varPsi ^0\) is the camera frame, with the \(z\)-axis in the direction of the camera optical axis. Frame \(\varPsi ^1\) is located at the point where the instrument emerges from the endoscope, with the \(z\)-axis aligned with the instrument direction. Frame \(\varPsi ^2\) is at the end of the straight section, rotating with the instrument rotation \(q_2\). Frame \(\varPsi ^3\) is midway the bending section, and frame \(\varPsi ^4\) is at the end of the bending section.

Fig. 13
figure 13

Five frames are defined: frame \(\varPsi ^0\) and \(\varPsi ^1\) are fixed to the endoscope, while frame \(\varPsi ^2\), \(\varPsi ^3\) and \(\varPsi ^4\) are fixed along the instrument. \(q_1,\, q_2\), and \(q_3\) denote the three DOFs: insertion, rotation, and bending, respectively

We first derive the unit twists of frames \(\varPsi ^2,\, \varPsi ^3\), and \(\varPsi ^4\) associated with each of the three DOFs. We denote the motion of frame \(\varPsi ^l\) with respect to frame \(\varPsi ^m\), expressed in frame \(\varPsi ^k\) as the infinitesimal twist \(\mathbf T ^{k,m}_{l}\). We denote the unit twist of frame \(\varPsi ^l\) associated with \(q_j\), with respect to frame \(\varPsi ^0\), expressed in frame \(\varPsi ^0\) as \(\hat{\mathbf{T }}_{l,j}\). From the unit twists, the Jacobian \(\mathbf J _\mathbf f (\mathbf q )\) is derived.

Straight section

The pose of frame \(\varPsi ^2\), located at the end of the straight section, is defined by \(q_1\) and \(q_2\), which are a translation along the \(z\)-axis of frame \(\varPsi ^1\) and a rotation around the same axis, respectively. Thus, the pose of frame \(\varPsi ^2\) with respect to frame \(\varPsi ^1\) is given by:

$$\begin{aligned} ^1_2\mathbf H = \begin{bmatrix}\mathbf R _\mathbf z (q_2)&\quad \begin{matrix}0\\ 0\\ q_1 \end{matrix}\\ \quad 0\ 0\ 0&\quad 1\end{bmatrix}\ , \end{aligned}$$
(16)

where \(\mathbf R _\mathbf z (\cdot )\) denotes the 3-by-3 rotation matrix around the \(z\)-axis. The pose of frame \(\varPsi ^1\) with respect to frame \(\varPsi ^0\) is determined by the geometry of the endoscope and is thus fixed.

The motion of frame \(\varPsi ^2\) with respect to frame \(\varPsi ^0\) is described by the infinitesimal twist:

$$\begin{aligned} \mathbf T ^{0,0}_{2} = \hat{\mathbf{T }}_{2,1} \dot{q}_1 + \hat{\mathbf{T }}_{2,2} \dot{q}_2 \ , \end{aligned}$$
(17)

where \(\hat{\mathbf{T }}_{2,1}\) and \(\hat{\mathbf{T }}_{2,2}\) represent a translation along the \(z\)-axis of frame \(\varPsi ^1\) and a rotation around that \(z\)-axis, respectively. They are:

$$\begin{aligned} \hat{\mathbf{T }}_{2,1}&= \mathrm Ad _{_1^0\mathbf H } \begin{bmatrix}0&0&0&0&0&1\end{bmatrix} ^\mathrm T \end{aligned}$$
(18)
$$\begin{aligned} \hat{\mathbf{T }}_{2,2}&= \mathrm Ad _{_1^0\mathbf H } \begin{bmatrix}0&0&1&0&0&0\end{bmatrix} ^\mathrm T , \end{aligned}$$
(19)

where \(\mathrm Ad _{_1^0\mathbf H }\) denotes the Adjoint operator that changes the coordinates of the twist from frame \(\varPsi ^1\) to frame \(\varPsi ^0\).

Bending section

The bending section is modeled as a constant curvature. It can be defined by a finite twist around axis \(\varvec{\omega }=\begin{bmatrix}0&\omega&0\end{bmatrix}^\mathrm T \) (Fig. 13), where \(\omega \) is the angle of the arc. The axis \(\varvec{\omega }\) is in the \(y\)-direction of frame \(\varPsi ^2\), located at \(\begin{bmatrix}\rho&0&0\end{bmatrix}^\mathrm T \) in frame \(\varPsi ^2\), where \(\rho \) denotes the curve radius. The chord length, denoted \(\ell \), is given by \(\ell =\omega \rho \). \(q_3\) is defined as \(q_3:=\omega \). This results in the finite twist describing the bending section:

$$\begin{aligned} \mathbf S ^{2,2}_{4} = \begin{bmatrix}\varvec{\omega }\\ \begin{matrix}\rho \\ 0\\ 0\end{matrix} \quad \wedge \quad \varvec{\omega }\end{bmatrix} = \begin{bmatrix}0\\q_3\\0\\0\\0\\\ell \end{bmatrix}, \end{aligned}$$
(20)

where \(\mathbf S ^{2,2}_{4}\) denotes the finite twist of frame \(\varPsi ^4\) with respect to frame \(\varPsi ^2\) expressed in frame \(\varPsi ^2\). The infinitesimal twist \(\mathbf T ^{2,2}_{4}\) can be derived from the finite twist \(\mathbf S ^{2,2}_{4}\) using the definition of the twist in matrix form (denoted by the tilde: \(\tilde{\mathbf{T }}^{k,m}_{l}\)):

$$\begin{aligned} \tilde{\mathbf{T }}^{2,2}_{4}&:= ^{2}_{4}\dot{\mathbf{H }}\, ^{4}_{2}\mathbf{H }\end{aligned}$$
(21)
$$\begin{aligned}&= \frac{\partial \, ^{2}_{4}\mathbf H }{\partial q_{3}}\dot{q}_{3} \exp \left(\tilde{\mathbf{S }}^{2,2}_{4} \right)\end{aligned}$$
(22)
$$\begin{aligned}&= \left[\begin{array}{llll} 0&0&1&\dfrac{\ell }{{q_3}^2} (-1 + \cos q_3 ) \\ 0&0&0&0\\ -1&0&0&\dfrac{\ell }{{q_3}^2}(q_3-\sin q_3)\\ 0&0&0&0\\ \end{array}\right]\dot{q}_3. \end{aligned}$$
(23)

The unit twist \(\hat{\mathbf{T }}_{4,3}\) is found by writing (23) in vector form, and transforming it to frame \(\varPsi ^0\):

$$\begin{aligned} \hat{\mathbf{T }}_{4,3} = \mathrm Ad _{_2^0\mathbf H } \begin{bmatrix}0\\1\\0\\ \frac{\ell }{{q_3}^2} (-1 + \cos q_3 ) \\ 0 \\ \frac{\ell }{{q_3}^2}(q_3-\sin q_3)\end{bmatrix} \end{aligned}$$
(24)

Since frame \(\varPsi ^3\) is located midway the bending section, unit twist \(\hat{\mathbf{T }}_{3,3}\) is found by substituting \(\ell \) by \(\frac{\ell }{2}\) in (24).

The velocity of a point \(\mathbf p _i\), that is fixed to frame \(\varPsi ^l\), is [20] as follows:

$$\begin{aligned} \dot{\mathbf{p }}_i=\tilde{\mathbf{T }}^{0,0}_{l} \mathbf p _i\ , \end{aligned}$$
(25)

with respect to frame \(\varPsi ^0\) and expressed in frame \(\varPsi ^0\). Since point \(A\) (Fig. 2) is fixed to frame \(\varPsi ^3\), and point \(B\) and \(C\) are fixed to frame \(\varPsi ^4\), the Jacobian \(\mathbf J _\mathbf f \) is as follows:

$$\begin{aligned} \mathbf J _\mathbf f =\begin{bmatrix} \tilde{\hat{\mathbf{T }}}_{3,1} \mathbf p _A&\tilde{\hat{\mathbf{T }}}_{3,2} \mathbf p _A&\tilde{\hat{\mathbf{T }}}_{3,3} \mathbf p _A\\ \tilde{\hat{\mathbf{T }}}_{4,1} \mathbf p _B&\tilde{\hat{\mathbf{T }}}_{4,2} \mathbf p _B&\tilde{\hat{\mathbf{T }}}_{4,3} \mathbf p _B\\ \tilde{\hat{\mathbf{T }}}_{4,1} \mathbf p _C&\tilde{\hat{\mathbf{T }}}_{4,2} \mathbf p _C&\tilde{\hat{\mathbf{T }}}_{4,3} \mathbf p _C\end{bmatrix}. \end{aligned}$$
(26)

Note that \(\tilde{\hat{\mathbf{T }}}_{3,1} = \tilde{\hat{\mathbf{T }}}_{4,1} = \tilde{\hat{\mathbf{T }}}_{2,1} \,\text{ and}\, \tilde{\hat{\mathbf{T }}}_{3,2} = \tilde{\hat{\mathbf{T }}}_{4,2} = \tilde{\hat{\mathbf{T }}}_{2,2}\) since the poses of frame \(\varPsi ^3\) and \(\varPsi ^4\) with respect to frame \(\varPsi ^2\) are independent of \(q_1\) and \(q_2\).

Rights and permissions

Reprints and permissions

About this article

Cite this article

Reilink, R., Stramigioli, S. & Misra, S. 3D position estimation of flexible instruments: marker-less and marker-based methods. Int J CARS 8, 407–417 (2013). https://doi.org/10.1007/s11548-012-0795-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-012-0795-1

Keywords

Navigation