Abstract
Purpose Endoscopic images can be used to allow accurate flexible endoscopic instrument control. This can be implemented using a pose estimation algorithm, which estimates the actual instrument pose from the endoscopic images.
Methods In this paper, two pose estimation algorithms are compared: a marker-less and a marker-based method. The marker-based method uses the positions of three markers in the endoscopic image to update the state of a kinematic model of the endoscopic instrument. The marker-less method works similarly, but uses the positions of three feature points instead of the positions of markers. The algorithms are evaluated inside a colon model. The endoscopic instrument is manually operated, while an X-ray imager is used to obtain a ground-truth reference position.
Results The marker-less method achieves an RMS error of 1.5, 1.6, and 1.8 mm in the horizontal, vertical, and away-from-camera directions, respectively. The marker-based method achieves an RMS error of 1.1, 1.7, and 1.5 mm in the horizontal, vertical, and away-from-camera directions, respectively. The differences between the two methods are not found to be statistically significant.
Conclusions The proposed algorithms are suitable to realize accurate robotic control of flexible endoscopic instruments, enabling the physician to perform advanced procedures in an intuitive way.
References
Abbott D, Becke C, Rothstein R, Peine W (2007) Design of an endoluminal NOTES robotic system. In: Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (IROS). San Diego, pp 410–416 doi:10.1109/IROS.2007.4399536
Bardou B (2011) Développement et étude d’un système robotisé pour l’assistance à la chirurgie transluminale. Ph.D. thesis, Université de Strasbourg
Bardou B, Nageotte F, Zanne P, De Mathelin M (2012) Improvements in the control of a flexible endoscopic system. In: Proceedings of the IEEE international conference on robotics and automation (ICRA). St. Paul, pp 3725–3732
Bardou B, Nageotte F, Zanne P, de Mathelin M (2009) Design of a telemanipulated system for transluminal surgery. In: 31st annual international conference of the IEEE EMBS
Bouguet JY Camera calibration toolbox for Matlab. http://www.vision.caltech.edu/bouguetj/calib_doc
Chaumette F, Hutchinson S (2006) Visual servo control. I. basic approaches. IEEE Robot Autom Mag 13(4):82–90. doi:10.1109/MRA.2006.250573
Doignon C, Nageotte F, Maurin B, Krupa A (2008) Pose estimation and feature tracking for robot assisted surgery with medical imaging. In: Kragic D, Kyrki V (eds) Unifying perspectives in computational and robot vision. Springer, Berlin, pp 79–101
Fisher R (1936) The use of multiple measurements in taxonomic problems. Ann Eugen 7(2):179–188
Gonzalez RC, Woods RE (2002) Digital image processing. Prentice Hall, Englewood Cliffs
Harris J, Stocker H (1998) Handbook of mathematics and computational science. Springer, Berlin
Kalloo AN et al (2004) Flexible transgastric peritoneoscopy: a novel approach to diagnostic and therapeutic interventions in the peritoneal cavity. Gastrointest Endosc 60(1):114–117. doi:10.1016/S0016-5107(04)01309-4
Mann HB, Whitney DR (1947) On a test of whether one of two random variables is stochastically larger than the other. Ann Math Stat 18(1):50–60
Marchand É, Chaumette F (2002) Virtual visual servoing: a framework for real-time augmented reality. Eurographics 21(3):289–298
Marescaux J, Dallemagne B, Perretta S, Wattiez A, Mutter D, Coumaros D (2007) Surgery without scars: report of transluminal cholecystectomy in a human being. Arch Surg 142(9):823–826
Nakamura Y (1991) Advanced robotics, redundancy and optimization. Addison-Wesley, Reading
OpenGL: The industry standard for high performance graphics. http://www.opengl.org
Reilink R, Stramigioli S, Misra S (2011) Three-dimensional pose reconstruction of flexible instruments from endoscopic images. In: IEEE/RSJ international conference intelligent robots and systems (IROS). San Francisco, pp 2076–2082
Reilink R, Stramigioli S, Misra S (2012) Pose reconstruction of flexible instruments from endoscopic images using markers. In: Proceedings of the IEEE international conference on robotics and automation (ICRA). St. Paul, pp 2939–2943
SciPy: scientific tools for Python. http://www.scipy.org
Stramigioli S, Bruyninckx H (2001) Geometry and screw theory for robotics. Seoul
Acknowledgments
This research is conducted within the TeleFLEX project, which is funded by the Dutch Ministry of Economic Affairs and the Province of Overijssel, within the Pieken in de Delta (PIDON) initiative. The ANUBIS endoscopic instrument was provided by Karl Storz GmbH & Co. KG, Tuttlingen, Germany. The endoscope was provided by Olympus Corp., Tokyo, Japan.
Conflict of interest
This is to certify that the authors have no financial or personal relationships with other people or organizations that would inappropriately influence our work.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Here we show the derivation of the analytical Jacobian \(\mathbf J _\mathbf f (\mathbf q )\) of the forward kinematics function \(f(\mathbf q )\) in (2). We define five frames on the instrument (Fig. 13). Frame \(\varPsi ^0\) is the camera frame, with the \(z\)-axis in the direction of the camera optical axis. Frame \(\varPsi ^1\) is located at the point where the instrument emerges from the endoscope, with the \(z\)-axis aligned with the instrument direction. Frame \(\varPsi ^2\) is at the end of the straight section, rotating with the instrument rotation \(q_2\). Frame \(\varPsi ^3\) is midway the bending section, and frame \(\varPsi ^4\) is at the end of the bending section.
We first derive the unit twists of frames \(\varPsi ^2,\, \varPsi ^3\), and \(\varPsi ^4\) associated with each of the three DOFs. We denote the motion of frame \(\varPsi ^l\) with respect to frame \(\varPsi ^m\), expressed in frame \(\varPsi ^k\) as the infinitesimal twist \(\mathbf T ^{k,m}_{l}\). We denote the unit twist of frame \(\varPsi ^l\) associated with \(q_j\), with respect to frame \(\varPsi ^0\), expressed in frame \(\varPsi ^0\) as \(\hat{\mathbf{T }}_{l,j}\). From the unit twists, the Jacobian \(\mathbf J _\mathbf f (\mathbf q )\) is derived.
Straight section
The pose of frame \(\varPsi ^2\), located at the end of the straight section, is defined by \(q_1\) and \(q_2\), which are a translation along the \(z\)-axis of frame \(\varPsi ^1\) and a rotation around the same axis, respectively. Thus, the pose of frame \(\varPsi ^2\) with respect to frame \(\varPsi ^1\) is given by:
where \(\mathbf R _\mathbf z (\cdot )\) denotes the 3-by-3 rotation matrix around the \(z\)-axis. The pose of frame \(\varPsi ^1\) with respect to frame \(\varPsi ^0\) is determined by the geometry of the endoscope and is thus fixed.
The motion of frame \(\varPsi ^2\) with respect to frame \(\varPsi ^0\) is described by the infinitesimal twist:
where \(\hat{\mathbf{T }}_{2,1}\) and \(\hat{\mathbf{T }}_{2,2}\) represent a translation along the \(z\)-axis of frame \(\varPsi ^1\) and a rotation around that \(z\)-axis, respectively. They are:
where \(\mathrm Ad _{_1^0\mathbf H }\) denotes the Adjoint operator that changes the coordinates of the twist from frame \(\varPsi ^1\) to frame \(\varPsi ^0\).
Bending section
The bending section is modeled as a constant curvature. It can be defined by a finite twist around axis \(\varvec{\omega }=\begin{bmatrix}0&\omega&0\end{bmatrix}^\mathrm T \) (Fig. 13), where \(\omega \) is the angle of the arc. The axis \(\varvec{\omega }\) is in the \(y\)-direction of frame \(\varPsi ^2\), located at \(\begin{bmatrix}\rho&0&0\end{bmatrix}^\mathrm T \) in frame \(\varPsi ^2\), where \(\rho \) denotes the curve radius. The chord length, denoted \(\ell \), is given by \(\ell =\omega \rho \). \(q_3\) is defined as \(q_3:=\omega \). This results in the finite twist describing the bending section:
where \(\mathbf S ^{2,2}_{4}\) denotes the finite twist of frame \(\varPsi ^4\) with respect to frame \(\varPsi ^2\) expressed in frame \(\varPsi ^2\). The infinitesimal twist \(\mathbf T ^{2,2}_{4}\) can be derived from the finite twist \(\mathbf S ^{2,2}_{4}\) using the definition of the twist in matrix form (denoted by the tilde: \(\tilde{\mathbf{T }}^{k,m}_{l}\)):
The unit twist \(\hat{\mathbf{T }}_{4,3}\) is found by writing (23) in vector form, and transforming it to frame \(\varPsi ^0\):
Since frame \(\varPsi ^3\) is located midway the bending section, unit twist \(\hat{\mathbf{T }}_{3,3}\) is found by substituting \(\ell \) by \(\frac{\ell }{2}\) in (24).
The velocity of a point \(\mathbf p _i\), that is fixed to frame \(\varPsi ^l\), is [20] as follows:
with respect to frame \(\varPsi ^0\) and expressed in frame \(\varPsi ^0\). Since point \(A\) (Fig. 2) is fixed to frame \(\varPsi ^3\), and point \(B\) and \(C\) are fixed to frame \(\varPsi ^4\), the Jacobian \(\mathbf J _\mathbf f \) is as follows:
Note that \(\tilde{\hat{\mathbf{T }}}_{3,1} = \tilde{\hat{\mathbf{T }}}_{4,1} = \tilde{\hat{\mathbf{T }}}_{2,1} \,\text{ and}\, \tilde{\hat{\mathbf{T }}}_{3,2} = \tilde{\hat{\mathbf{T }}}_{4,2} = \tilde{\hat{\mathbf{T }}}_{2,2}\) since the poses of frame \(\varPsi ^3\) and \(\varPsi ^4\) with respect to frame \(\varPsi ^2\) are independent of \(q_1\) and \(q_2\).
Rights and permissions
About this article
Cite this article
Reilink, R., Stramigioli, S. & Misra, S. 3D position estimation of flexible instruments: marker-less and marker-based methods. Int J CARS 8, 407–417 (2013). https://doi.org/10.1007/s11548-012-0795-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-012-0795-1