Skip to main content
SpringerLink
Log in

Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery

  • Topics
  • Recent advances in visualization and imaging in HBP sciences
  • Published:
Journal of Hepato-Biliary-Pancreatic Sciences

Abstract

Background

We applied a new concept of “image overlay surgery” consisting of the integration of virtual reality (VR) and augmented reality (AR) technology, in which dynamic 3D images were superimposed on the patient’s actual body surface and evaluated as a reference for surgical navigation in gastrointestinal, hepatobiliary and pancreatic surgery.

Methods

We carried out seven surgeries, including three cholecystectomies, two gastrectomies and two colectomies. A Macintosh and a DICOM workstation OsiriX were used in the operating room for image analysis. Raw data of the preoperative patient information obtained via MDCT were reconstructed to volume rendering and projected onto the patient's body surface during the surgeries. For accurate registration, OsiriX was first set to reproduce the patient body surface, and the positional coordinates of the umbilicus, left and right nipples, and the inguinal region were fixed as physiological markers on the body surface to reduce the positional error.

Results

The registration process was non-invasive and markerlesss, and was completed within 5 min. Image overlay navigation was helpful for 3D anatomical understanding of the surgical target in the gastrointestinal, hepatobiliary and pancreatic anatomies. The surgeon was able to minimize movement of the gaze and could utilize the image assistance without interfering with the forceps operation, reducing the gap from the VR. Unexpected organ injury could be avoided in all procedures. In biliary surgery, the projected virtual cholangiogram on the abdominal wall could advance safely with identification of the bile duct. For early gastric and colorectal cancer, the small tumors and blood vessels, which usually could not be found on the gastric serosa by laparoscopic view, were simultaneously detected on the body surface by carbon dioxide-enhanced MDCT. This provided accurate reconstructions of the tumor and involved lymph node, directly linked with optimization of the surgical procedures.

Conclusions

Our non-invasive markerless registration using physiological markers on the body surface reduced logistical efforts. The image overlay technique is a useful tool when highlighting hidden structures, giving more information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Tang SL, Kwoh CK, Teo MY, Sing NW, Ling KV. Augmented reality systems for medical applications. IEEE Eng Med Biol Mag. 1998;17:49–58.

    Article  CAS  PubMed  Google Scholar 

  2. Marescaux J, Rubino F, Arenas M, Mutter D, Soler L. Augmented-reality-assisted laparoscopic adrenalectomy. JAMA. 2004;292:2214–5.

    Article  CAS  PubMed  Google Scholar 

  3. Shuhaiber JH. Augmented reality in surgery. Arch Surg. 2004;139:170–4.

    Article  PubMed  Google Scholar 

  4. Milgram P, Kishino AF. Taxonomy of mixed reality visual displays. IEICE Trans Inform Syst 1994; E77-D:1321–9.

  5. Rosset A, Spadola L, Ratib O, Osiri X. An open-source software for navigating in multidimensional DICOM images. J Digit Imaging. 2004;17:205–16. (Epub ahead of print).

    Article  PubMed  Google Scholar 

  6. Rosset C, Rosset A, Ratib O. General consumer communication tools for improved image management and communication in medicine. J Digit Imaging. 2005;18:270–9.

    Article  PubMed  Google Scholar 

  7. Rosset A, Spadola L, Pysher L, Ratib O. Informatics in radiology (infoRAD): navigating the fifth dimension: innovative interface for multidimensional multimodality image navigation. Radiographics. 2006;26:299–308.

    Article  PubMed  Google Scholar 

  8. Bauernschmitt R, Feuerstein M, Traub J, Schirmbeck EU, Klinker G, Lange R. Optimal port placement and enhanced guidance in robotically assisted cardiac surgery. Surg Endosc. 2007;21:684–7. (Epub 2006 Dec 16).

    Article  CAS  PubMed  Google Scholar 

  9. Fichtinger G, Deguet A, Masamune K, Balogh E, Fischer GS, Mathieu H, et al. Image overlay guidance for needle insertion in CT scanner. IEEE Trans Biomed Eng. 2005;52:2–1415.

    Article  Google Scholar 

  10. Das M, Sauer F, Schoepf UJ, Khamene A, Vogt SK, Schaller S, et al. Augmented reality visualization for CT-guided interventions: system description, feasibility, and initial evaluation in an abdominal phantom. Radiology. 2006;240:230–5. (Epub 2006 May 23).

    Article  PubMed  Google Scholar 

  11. Nicolau SA, Pennec X, Soler L, Ayache N. A complete augmented reality guidance system for liver punctures: first clinical evaluation. Med Image Comput Comput Assist Interv Int Conf Med Image Comput Comput Assist Interv. 2005;8:539–47.

    CAS  Google Scholar 

  12. Deutschmann H, Steininger P, Nairz O, Kopp P, Merz F, Wurstbauer K, et al. “Augmented reality” in conventional simulation by projection of 3-D structures into 2-D images: a comparison with virtual methods. Strahlenther Onkol. 2008;184:93–9.

    Article  PubMed  Google Scholar 

  13. Iseki H, Masutani Y, Iwahara M, Tanikawa T, Muragaki Y, Taira T, et al. Volumegraph (overlaid three-dimensional image-guided navigation): clinical application of augmented reality in neurosurgery. Stereotact Funct Neurosurg. 1997;68:18–24.

    Article  CAS  PubMed  Google Scholar 

  14. Masutani Y, Dohi T, Yamane F, Iseki H, Takakura K. Augmented reality visualization system for intravascular neurosurgery. Comput Aided Surg. 1998;3:239–47.

    Article  CAS  PubMed  Google Scholar 

  15. Lovo EE, Quintana JC, Puebla MC, Torrealba G, Santos JL, Lira IH, et al. A novel, inexpensive method of image coregistration for applications in image-guided surgery using augmented reality. Neurosurgery. 2007;60:366–71. discussion 371–2.

    Article  PubMed  Google Scholar 

  16. Ackerman JD, Keller K, Fuchs H. Real-time anatomical 3D image extraction for laparoscopic surgery. Stud Health Technol Inform. 2001;81:18–22.

    CAS  PubMed  Google Scholar 

  17. Kawamata T, Iseki H, Shibasaki T, Hori T. Endoscopic augmented reality navigation system for endonasal transsphenoidal surgery to treat pituitary tumors: technical note. Neurosurgery. 2002;50:1393–7.

    Article  PubMed  Google Scholar 

  18. Falk V, Mourgues F, Vieville T, Jacobs S, Holzhey D, Walther T, et al. Augmented reality for intraoperative guidance in endoscopic coronary artery bypass grafting. Surg Technol Int. 2005;14:231–5.

    PubMed  Google Scholar 

  19. Ukimura O, Gill IS. Imaging-assisted endoscopic surgery: Cleveland clinic experience. J Endourol. 2008;22:803–10.

    Article  PubMed  Google Scholar 

  20. Nakamoto M, Nakada K, Sato Y, Konishi K, Hashizume M, Tamura S. Intraoperative magnetic tracker calibration using a magneto-optic hybrid tracker for 3-D ultrasound-based navigation in laparoscopic surgery. IEEE Trans Med Imaging. 2008;27:255–70.

    Article  PubMed  Google Scholar 

  21. Yuan ML, Ong SK, Nee AY. Registration using natural features for augmented reality systems. IEEE Trans Vis Comput Graph. 2006;12:569–80.

    Article  CAS  PubMed  Google Scholar 

  22. Marmulla R, Lüth T, Mühling J, Hassfeld S. Markerless laser registration in image-guided oral and maxillofacial surgery. J Oral Maxillofac Surg. 2004;62:845–51.

    Article  PubMed  Google Scholar 

  23. Comport AI, Marchand E, Pressigout M, Chaumette F. Real-time markerless tracking for augmented reality: the virtual visual serving framework. IEEE Trans Vis Comput Graph. 2006;12:615–28.

    Article  PubMed  Google Scholar 

  24. Marmulla R, Mühling J, Wirtz CR, Hassfeld S. High-resolution laser surface scanning for patient registration in cranial computer-assisted surgery. Minim Invasive Neurosurg. 2004;47:72–8.

    Article  CAS  PubMed  Google Scholar 

  25. Uenohara M, Kanade T. Vision-based object registration for real-time image overlay. Comput Biol Med. 1995;25:249–60.

    Article  CAS  PubMed  Google Scholar 

  26. Sugimoto M, Yasuda H, Koda K, Suzuki M, Yamazaki M, Tezuka T, et al. Evaluation for transvaginal and transgastric NOTES cholecystectomy in human and animal natural orifice translumenal endoscopic surgery. J Hepatobiliary Pancreat Surg. 2009;16:255–60. (Epub 2009 Apr 10).

    Article  PubMed  Google Scholar 

  27. Sugimoto M. Natural orifice translumenal endoscopic surgery (NOTES) for innovation in hepatobiliary and pancreatic surgery: preface. J Hepatobiliary Pancreat Surg. 2009;16:247–8. (Epub 2009 Apr 14).

    Article  PubMed  Google Scholar 

  28. Sugimoto M, Yasuda H, Koda K, Suzuki M, Yamazaki M, Tezuka T, et al. Rendezvous gastrotomy technique using direct percutaneous endoscopic gastrostomy for transgastric cholecystectomy in hybrid natural orifice translumenal endoscopic surgery. J Hepatobiliary Pancreat Surg. 2009. doi:10.1007/s00534-009-0143-1. (Epub 2009 Jul 15).

  29. Vosburgh KG, San José Estépar R. Natural orifice transluminal endoscopic surgery (NOTES): an opportunity for augmented reality guidance. Stud Health Technol Inform. 2007;125:485–90.

    PubMed  Google Scholar 

  30. Estépar RS, Stylopoulos N, Ellis R, Samset E, Westin CF, Thompson C, et al. Towards scarless surgery: an endoscopic ultrasound navigation system for transgastric access procedures. Comput Aided Surg. 2007;12:311–24.

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

This study was partially supported by a grant-in-aid from the Scientific Research Japan Society for the Promotion of Science, a JFE grant from the Japanese Foundation for Research and Promotion of Endoscopy, and the Teikyo University Tomoko Fujii memorial research grant for young medical researchers.

Author information

Authors and Affiliations

  1. Department of Surgery, Teikyo University Chiba Medical Center, 3426-3 Anesaki, Ichihara, Chiba, 299-0111, Japan

    Maki Sugimoto, Hideki Yasuda, Keiji Koda, Masato Suzuki, Masato Yamazaki, Tohru Tezuka, Chihiro Kosugi, Ryota Higuchi, Yoshihisa Watayo, Yohsuke Yagawa, Shuichiro Uemura & Hironori Tsuchiya

  2. Department of Gastroenterology, Kobe University Graduate School of Medicine, 7-5-1 Kusunoki-cho, Chuo-ku, Kobe, Hyogo, 650-0017, Japan

    Maki Sugimoto & Takeshi Azuma

Authors
  1. Maki Sugimoto
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hideki Yasuda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Keiji Koda
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Masato Suzuki
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Masato Yamazaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tohru Tezuka
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Chihiro Kosugi
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ryota Higuchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yoshihisa Watayo
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Yohsuke Yagawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Shuichiro Uemura
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Hironori Tsuchiya
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Takeshi Azuma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maki Sugimoto.

About this article

Cite this article

Sugimoto, M., Yasuda, H., Koda, K. et al. Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery. J Hepatobiliary Pancreat Sci 17, 629–636 (2010). https://doi.org/10.1007/s00534-009-0199-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00534-009-0199-y

Keywords

Search

Navigation