Skip to main content
SpringerLink
Log in
Book cover

Intelligent Orthopaedics pp 207–224Cite as

Intelligent HMI in Orthopedic Navigation

  • Chapter
  • First Online:

Part of the book series: Advances in Experimental Medicine and Biology ((AEMB,volume 1093))

Abstract

The human-machine interface (HMI) is an essential part of image-guided orthopedic navigation systems. HMI provides a primary platform to merge surgically relevant pre- and intraoperative images from different modalities and 3D models including anatomical structures and implants to support surgical planning and navigation. With the various input-output techniques of HMI, surgeons can intuitively manipulate anatomical models generated from medical images and/or implant models for surgical planning. Furthermore, HMI recreates sight, sound, and touch feedback for the guidance of surgery operations which helps surgeons to sense more relevant information, e.g., anatomical structures and surrounding tissue, the mechanical axis of limbs, and even the mechanical properties of tissue. Thus, with the help of interactive HMI, precision operations, such as cutting, drilling, and implantation, can be performed more easily and safely.

Classic HMI is based on 2D displays and standard input devices of computers. In contrast, modern visual reality (VR) and augmented reality (AR) techniques allow the showing more information for surgical navigation. Various attempts have been applied to image-guided orthopedic therapy. In order to realize rapid image-based modeling and to create effective interaction and feedback, intelligent algorithms have been developed. Intelligent algorithms can realize fast registration of image to image and image to patients, and the algorithms to compensate the visual offset in AR display have been investigated. In order to accomplish more effective human-computer interaction, various input methods and force sensing/force reflecting methods have been developed. This chapter reviews related human-machine interface techniques for image-guided orthopedic navigation, analyzes several examples of clinical applications, and discusses the trend of intelligent HMI in orthopedic navigation.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Song J, Ding H, Han W et al (2016) A motion compensation method for bi-plane robot-assisted internal fixation surgery of a femur neck fracture. Proc Inst Mech Eng H J Eng Med 230(10):942–948. https://doi.org/10.1177/0954411916663582

    Article  Google Scholar 

  2. Hacihaliloglu I, Wilson DR, Gilbart M et al (2013) Non-iterative partial view 3D ultrasound to CT registration in ultrasound-guided computer-assisted orthopedic surgery. Int J Comput Assist Radiol Surg 8(2):157–168. https://doi.org/10.1007/s11548-012-0747-9

    Article  PubMed  Google Scholar 

  3. Wong KC (2016) Image fusion for computer-assisted bone tumor surgery. In: Zheng G, Li S (eds) Computational radiology for Orthopaedic interventions. Springer, Cham, pp 217–230. https://doi.org/10.1007/978-3-319-23482-3_11

    Chapter  Google Scholar 

  4. Lonner JH (2015) Robotically assisted unicompartmental knee arthroplasty with a handheld image-free sculpting tool. Orthop Clin North Am 25(2):104–113. https://doi.org/10.1016/j.ocl.2015.08.024

    Article  Google Scholar 

  5. Kovler I, Joskowicz L, Weil YA et al (2015) Haptic computer-assisted patient-specific preoperative planning for orthopedic fractures surgery. Int J Comput Assist Radiol Surg 10(10):1535–1546. https://doi.org/10.1007/s11548-015-1162-9

    Article  CAS  PubMed  Google Scholar 

  6. Hettig J, Saalfeld P, Luz M et al (2017) Comparison of gesture and conventional interaction techniques for interventional neuroradiology. Int J Comput Assist Radiol Surg 12(9):1643–1653. https://doi.org/10.1007/s11548-017-1523-7

    Article  PubMed  Google Scholar 

  7. Stefanou MA, Pasparakis D, Mastrokalos D et al (2014) Radiographic assessment of lower limb lengthening in achondroplastic patients, using the ilizarov frame: a 5-19 year follow up study. Int J Orthop 1(4):140–145. https://doi.org/10.6051/j.issn.2311-5106.2014.01.33

    Article  Google Scholar 

  8. Wang GY, Huang WJ, Song Q et al (2016) Computer-assisted virtual preoperative planning in orthopedic surgery for acetabular fractures based on actual computed tomography data. Comput Assist Surg 21(1):160–165. https://doi.org/10.1080/24699322.2016.1240235

    Article  Google Scholar 

  9. Dagnino G, Georgilas I, Morad S et al (2017) Image-guided surgical robotic system for percutaneous reduction of joint fractures. Ann Biomed Eng 45(11):2648–2662. https://doi.org/10.1007/s10439-017-1901-x

    Article  PubMed  PubMed Central  Google Scholar 

  10. Lonner JH, Smith JR, Picard F et al (2015) High degree of accuracy of a novel image-free handheld robot for unicondylar knee arthroplasty in a cadaveric study. Clin Orthop Relat Res 473(1):206–212. https://doi.org/10.1007/s11999-014-3764-x

    Article  PubMed  Google Scholar 

  11. Marien A, Al DCA, Desai M et al (2015) Three-dimensional navigation system integrating position-tracking technology with movable tablet display for percutaneous targeting. BJU Int 115(4):659–665. https://doi.org/10.1111/bju.12948

    Article  PubMed  Google Scholar 

  12. Zeng BW, Meng FL, Ding H et al (2017) A surgical robot with augmented reality visualization for stereoelectroencephalography electrode implantation. Int J Comput Assist Radiol Surg 12(8):1355–1368. https://doi.org/10.1007/s11548-017-1634-1

    Article  PubMed  Google Scholar 

  13. Zhang XR, Chen GW, Liao HG (2017) High-quality see-through surgical guidance system using enhanced 3-d autostereoscopic augmented reality. IEEE Trans Biomed Eng 64(8):1815–1825. https://doi.org/10.1109/tbme.2016.2624632

    Article  PubMed  Google Scholar 

  14. Golab MR, Breedon PJ, Vloeberghs M (2016) A wearable headset for monitoring electromyography responses within spinal surgery. Eur Spine J 25(10):3214–3219. https://doi.org/10.1007/s00586-016-4626-x

    Article  CAS  PubMed  Google Scholar 

  15. Perkins SL, Lin MA, Srinivasan S et al (2017) A mixed-reality system for breast surgical planning. In: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) Proceedings pp 269–274. https://doi.org/10.1109/ISMAR-Adjunct.2017.92

  16. Tepper OM, Rudy HL, Lefkowitz A et al (2017) Mixed reality with HoloLens: where virtual reality meets augmented reality in the operating room. Plast Reconstr Surg 140(5):1066–1070. https://doi.org/10.1097/prs.0000000000003802

    Article  CAS  PubMed  Google Scholar 

  17. Vavra P, Roman J, Zonca P et al (2017) Recent development of augmented reality in surgery: a review. J Healthc Eng 9. https://doi.org/10.1155/2017/4574172

    Article  Google Scholar 

  18. Lopes DS, Parreira PDD, Paulo SF et al (2017) On the utility of 3D hand cursors to explore medical volume datasets with a touchless interface. J Biomed Inform 72(2017):140–149. https://doi.org/10.1016/j.jbi.2017.07.009

    Article  PubMed  Google Scholar 

  19. Mewes A, Saalfeld P, Riabikin O et al (2016) A gesture-controlled projection display for CT-guided interventions. Int J Comput Assist Radiol Surg 11(1):157–164. https://doi.org/10.1007/s11548-015-1215-0

    Article  CAS  PubMed  Google Scholar 

  20. Phil (2014) See the Myo armband in surgery. http://blog.thalmic.com/myo-armband-surgery/. Accessed 5 Feb 2018

  21. Mewes A, Hensen B, Wacker F et al (2017) Touchless interaction with software in interventional radiology and surgery: a systematic literature review. Int J Comput Assist Radiol Surg 12(2):291–305. https://doi.org/10.1007/s11548-016- 1480-6

    Article  PubMed  Google Scholar 

  22. Boiadjiev T, Boiadjiev G, Delchev K et al (2017) Far cortex automatic detection aimed for partial or full bone drilling by a robot system in orthopaedic surgery. Biotechnol Biotechnol Equip 31(1):200–205. https://doi.org/10.1080/13102818.2016.1234947

    Article  Google Scholar 

  23. Boiadjiev T, Boiadjiev G, Delchev K et al (2015) Eliminating of far pedicle cortex perforation by automatic spine drilling. Appl Mech Mater 799-800:505–508. https://doi.org/10.4028/www.scientific.net/AMM. 799-800.505

    Article  Google Scholar 

  24. Boiadjiev T, Zagurski K, Boiadjiev G et al (2011) Identification of bone structure during automatic drilling in orthopedic surgery. Mech Based Des Struc 39(2):285–302. https://doi.org/10.1080/15397734.2011.550863

    Article  Google Scholar 

  25. Boiadjiev G, Kastelov R, Boiadjiev T et al (2013) Design and performance study of an orthopaedic surgery robotized module for automatic bone drilling. Int J Med Robot Comput 9(4):455–463. https://doi.org/10.1002/rcs.1479

    Article  Google Scholar 

  26. Jin HY, Hu Y, Deng Z et al (2014) Model-based state recognition of bone drilling with robotic orthopedic surgery system. In: 2014 IEEE International Conference on Robotics and Automation ICRA, Hong Kong, China, 31 May–7 June 2014. IEEE International Conference on Robotics and Automation ICRA. IEEE, pp 3538–3543. https://doi.org/10.1109/ICRA.2014.6907369

  27. Hu Y, Jin HY, Zhang LW et al (2014) State recognition of pedicle drilling with force sensing in a robotic spinal surgical system. IEEE/ASME Trans Mechatronics 19(1):357–365. https://doi.org/10.1109/tmech.2012.2237179

    Article  Google Scholar 

  28. Tian W, Han XG, Liu B et al (2014) A robot-assisted surgical system using a force-image control method for pedicle screw insertion. PLoS One 9(1):9. https://doi.org/10.1371/journal.pone.0086346

    Article  CAS  Google Scholar 

  29. Dai Y, Xue Y, Zhang JX (2016) Milling state identification based on vibration sense of a robotic surgical system. IEEE Trans Ind Electron 63(10):6184–6193. https://doi.org/10.1109/tie.2016.2574981

    Article  Google Scholar 

  30. Jin HY, Hu Y, Gao P et al (2014) Intraoperative control for robotic spinal surgical system with audio and torque sensing. In: Processing of 2014 International conference on multisensor fusion and information integration for intelligent systems (Mfi), Beijing, China, 28–29 Sept. 2014 2014. IEEE, p 6. https://doi.org/10.1109/MFI.2014.6997711

  31. Sun Y, Jin HY, Hu Y et al (2014) State recognition of bone drilling with audio signal in robotic orthopedics surgery system. In: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA, 14–18 Sept. 2014. IEEE International Conference on Intelligent Robots and Systems. IEEE, pp 3503–3508. https://doi.org/10.1109/IROS.2014.6943051

  32. Cho B, Oka M, Matsumoto N et al (2013) Warning navigation system using real-time safe region monitoring for otologic surgery. Int J Comput Assist Radiol Surg 8(3):395–405. https://doi.org/10.1007/s11548-012-0797-z

    Article  PubMed  Google Scholar 

  33. Cho B, Matsumoto N, Komune S et al (2014) A surgical navigation system for guiding exact cochleostomy using auditory feedback: a clinical feasibility study. Biomed Res Int 2014(3):769659. https://doi.org/10.1155/2014/769659

    Article  PubMed  PubMed Central  Google Scholar 

  34. Dixon BJ, Daly MJ, Chan H et al (2014) Augmented real-time navigation with critical structure proximity alerts for endoscopic skull base surgery. Laryngoscope 124(4):853–859. https://doi.org/10.1002/lary.24385

    Article  PubMed  Google Scholar 

  35. Woerdeman PA, Willems PW, Noordmans HJ et al (2009) Auditory feedback during frameless image-guided surgery in a phantom model and initial clinical experience. J Neurosurg 110(2):257–262. https://doi.org/10.3171/2008.3.17431

    Article  PubMed  Google Scholar 

  36. Vaughan N, Dubey VN, Wainwright TW et al (2016) A review of virtual reality based training simulators for orthopaedic surgery. Med Eng Phys 38(2):59–71. https://doi.org/10.1016/j.medengphy.2015.11.021

    Article  PubMed  Google Scholar 

  37. Vankipuram M, Kahol K, McLaren A et al (2010) A virtual reality simulator for orthopedic basic skills: a design and validation study. J Biomed Inform 43(5):661–668. https://doi.org/10.1016/j.jbi.2010.05.016

    Article  PubMed  Google Scholar 

  38. Cecil J, Gupta A, Pirela-Cruz M (2018) An advanced simulator for orthopedic surgical training. Int J Comput Assist Radiol Surg 13(2):305–319. https://doi.org/10.1007/s11548-017-1688-0

    Article  CAS  PubMed  Google Scholar 

  39. Shen F, Chen B, Guo Q et al (2013) Augmented reality patient-specific reconstruction plate design for pelvic and acetabular fracture surgery. Int J Comput Assist Radiol Surg 8(2):169–179. https://doi.org/10.1007/s11548-012-0775-5

    Article  PubMed  Google Scholar 

  40. MedGadget (2017) CAE Healthcare announces first mixed reality ultrasound simulation solution with Microsoft HoloLens. https://www.medgadget.com/2017/01/cae-healthcare-announces-first-mixed-reality-ultrasound-simulation-solution-with-microsoft-hololens.html. Accessed 10 Feb 2018

  41. Enayati N, De ME, Ferrigno G (2016) Haptics in robot-assisted surgery: challenges and benefits. IEEE Rev Biomed Eng 9:49–65. https://doi.org/10.1109/RBME.2016.2538080

    Article  PubMed  Google Scholar 

  42. Jacobs S, Holzhey D, Strauss G et al (2007) The impact of haptic learning in telemanipulator-assisted surgery. Surg Laparosc Endosc Percutan Tech 17(5):402–406. https://doi.org/10.1097/SLE.0b013e3180f60c23

    Article  PubMed  Google Scholar 

  43. Tholey G, Desai JP, Castellanos AE (2005) Force feedback plays a significant role in minimally invasive surgery: results and analysis. Ann Surg 241(1):102–109. https://doi.org/10.1097/01.sla.0000149301.60553.1e

    Article  PubMed  PubMed Central  Google Scholar 

  44. van der Meijden OAJ, Schijven MP (2009) The value of haptic feedback in conventional and robot-assisted minimal invasive surgery and virtual reality training: a current review. Surg Endosc 23(6):1180–1190. https://doi.org/10.1007/s00464-008-0298-x

    Article  PubMed  PubMed Central  Google Scholar 

  45. Luciano CJ, Banerjee PP, Bellotte B et al (2011) Learning retention of thoracic pedicle screw placement using a high-resolution augmented reality simulator with haptic feedback. Oper Neurosurg 69(suppl_1):14–19. https://doi.org/10.1227/NEU.0b013e31821954ed

    Article  Google Scholar 

  46. Indraccolo C, Paolis LT (2017) Augmented reality and MYO for a touchless interaction with virtual organs. Paper presented at the augmented reality, virtual reality and computer graphics. In: 4th international conference, AVR 2017, Ugento, Italy, 12–15 June 2017

    Google Scholar 

  47. Mentis HM, O’Hara K, Gonzalez G et al (2015) Voice or gesture in the operating room. In: ACM Conference Extended Abstracts on Human Factors in Computing Systems, Seoul, Republic of Korea, 18–23 April 2015. pp 773–780. https://doi.org/10.1145/2702613.2702963

  48. Hotker AM, Pitton MB, Mildenberger P et al (2013) Speech and motion control for interventional radiology: requirements and feasibility. Int J Comput Assist Radiol Surg 8(6):997–1002. https://doi.org/10.1007/s11548-013-0841-7

    Article  PubMed  Google Scholar 

  49. Mylonas GP, Kwok KW, James DRC et al (2012) Gaze-contingent motor Channelling, haptic constraints and associated cognitive demand for robotic MIS. Med Image Anal 16(3):612–631. https://doi.org/10.1016/j.media.2010.07.007

    Article  PubMed  Google Scholar 

  50. Ding H, Yang X, Zheng N et al (2017) Tri-co robot: a Chinese robotic research initiative for enhanced robot interaction capabilities. Natl Sci Rev 0(0):1–3. https://doi.org/10.1093/nsr/nwx148

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, People’s Republic of China

    Guangzhi Wang, Liang Li, Shuwei Xing & Hui Ding

Authors
  1. Guangzhi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shuwei Xing
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hui Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guangzhi Wang .

Editor information

Editors and Affiliations

  1. University of Bern, Bern, Switzerland

    Guoyan Zheng

  2. Beijing Jishuitan Hospital, Peking University, Beijing, Beijing, China

    Wei Tian

  3. Fudan University, Shanghai, China

    Xiahai Zhuang

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Wang, G., Li, L., Xing, S., Ding, H. (2018). Intelligent HMI in Orthopedic Navigation. In: Zheng, G., Tian, W., Zhuang, X. (eds) Intelligent Orthopaedics. Advances in Experimental Medicine and Biology, vol 1093. Springer, Singapore. https://doi.org/10.1007/978-981-13-1396-7_17

Download citation

Publish with us

Policies and ethics

Search

Navigation