Skip to main content

Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration

International Journal of Computer Assisted Radiology and Surgery volume 15, pages 759–769 (2020)Cite this article

Abstract

Purpose

Fluoroscopy is the standard imaging modality used to guide hip surgery and is therefore a natural sensor for computer-assisted navigation. In order to efficiently solve the complex registration problems presented during navigation, human-assisted annotations of the intraoperative image are typically required. This manual initialization interferes with the surgical workflow and diminishes any advantages gained from navigation. In this paper, we propose a method for fully automatic registration using anatomical annotations produced by a neural network.

Methods

Neural networks are trained to simultaneously segment anatomy and identify landmarks in fluoroscopy. Training data are obtained using a computationally intensive, intraoperatively incompatible, 2D/3D registration of the pelvis and each femur. Ground truth 2D segmentation labels and anatomical landmark locations are established using projected 3D annotations. Intraoperative registration couples a traditional intensity-based strategy with annotations inferred by the network and requires no human assistance.

Results

Ground truth segmentation labels and anatomical landmarks were obtained in 366 fluoroscopic images across 6 cadaveric specimens. In a leave-one-subject-out experiment, networks trained on these data obtained mean dice coefficients for left and right hemipelves, left and right femurs of 0.86, 0.87, 0.90, and 0.84, respectively. The mean 2D landmark localization error was 5.0 mm. The pelvis was registered within \(1^{\circ }\) for 86% of the images when using the proposed intraoperative approach with an average runtime of 7 s. In comparison, an intensity-only approach without manual initialization registered the pelvis to \(1^{\circ }\) in 18% of images.

Conclusions

We have created the first accurately annotated, non-synthetic, dataset of hip fluoroscopy. By using these annotations as training data for neural networks, state-of-the-art performance in fluoroscopic segmentation and landmark localization was achieved. Integrating these annotations allows for a robust, fully automatic, and efficient intraoperative registration during fluoroscopic navigation of the hip.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price includes VAT (Canada)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Notes

  1. 1.

    https://youtu.be/5AwGlNkcp9o.

  2. 2.

    https://github.com/rg2/DeepFluoroLabeling-IPCAI2020

References

  1. 1.

    Woerner M, Sendtner E, Springorum R, Craiovan B, Worlicek M, Renkawitz T, Grifka J, Weber M (2016) Visual intraoperative estimation of cup and stem position is not reliable in minimally invasive hip arthroplasty. Acta Orthop 87(3):225–230

    Article  PubMed  PubMed Central  Google Scholar 

  2. 2.

    Slotkin EM, Patel PD, Suarez JC (2015) Accuracy of fluoroscopic guided acetabular component positioning during direct anterior total hip arthroplasty. J Arthroplasty 30(9):102–106

    Article  PubMed  Google Scholar 

  3. 3.

    Troelsen A (2009) Surgical advances in periacetabular osteotomy for treatment of hip dysplasia in adults. Acta Orthop 80(sup332):1–33

    Article  Google Scholar 

  4. 4.

    Kelley TC, Swank ML (2009) Role of navigation in total hip arthroplasty. J Bone Joint Surg Am 91(Supplement):153–158

    Article  PubMed  Google Scholar 

  5. 5.

    Belei P, Skwara A, Fuente MDL, Schkommodau E, Fuchs S, Wirtz DC, Kämper C, Radermacher K (2007) Fluoroscopic navigation system for hip surface replacement. Comput Aided Surg 12(3):160–167

    Article  PubMed  Google Scholar 

  6. 6.

    Malan DF, van der Walt SJ, Raidou RG, van den Berg B, Stoel BC, Botha CP, Nelissen RG, Valstar ER (2016) A fluoroscopy-based planning and guidance software tool for minimally invasive hip refixation by cement injection. Int J Comput Assist Radiol Surg 11(2):281–296

    Article  PubMed  Google Scholar 

  7. 7.

    Grupp RB, Hegeman R, Murphy R, Alexander C, Otake Y, McArthur B, Armand M, Taylor RH (2020) Pose estimation of periacetabular osteotomy fragments with intraoperative X-ray navigation. IEEE Trans Biomed Eng 67(2):441–452

    Article  PubMed  Google Scholar 

  8. 8.

    Gottschling H, Roth M, Schweikard A, Burgkart R (2005) Intraoperative, fluoroscopy-based planning for complex osteotomies of the proximal femur. Int J Med Robot Comput Assist Surg 1(3):67–73

    Article  CAS  Google Scholar 

  9. 9.

    Markelj P, Tomaževič D, Likar B, Pernuš F (2012) A review of 3D/2D registration methods for image-guided interventions. Med Image Anal 16(3):642–661

    Article  CAS  PubMed  Google Scholar 

  10. 10.

    Newell A, Yang K, Deng J (2016) Stacked hourglass networks for human pose estimation. In: Proceedings of European conference on computer vision. Springer, pp 483–499

  11. 11.

    Shelhamer E, Long J, Darrell T (2017) Fully convolutional networks for semantic segmentation. IEEE Trans Pattern Anal Mach Intell 4:640–651

    Article  Google Scholar 

  12. 12.

    Otake Y, Takao M, Yokota F, Fukuda N, Uemura K, Sugano N, Sato Y (2018) Construction and application of large-scale image database in orthopedic surgery. In: Computer assisted orthopaedic surgery for hip and knee. Springer, pp 191–197

  13. 13.

    Miao S, Wang ZJ, Liao R (2016) A CNN regression approach for real-time 2D/3D registration. IEEE Trans Med Imag 35(5):1352–1363

    Article  Google Scholar 

  14. 14.

    Ambrosini P, Ruijters D, Niessen WJ, Moelker A, van Walsum T (2017) Fully automatic and real-time catheter segmentation in X-ray fluoroscopy. In: Proceedings of medical image computing and computer-assisted intervention. Springer, pp 577–585

  15. 15.

    Breininger K, Albarqouni S, Kurzendorfer T, Pfister M, Kowarschik M, Maier A (2018) Intraoperative stent segmentation in X-ray fluoroscopy for endovascular aortic repair. Int J Comput Assist Radiol Surg 13(8):1221–1231

    Article  PubMed  Google Scholar 

  16. 16.

    Gao C, Unberath M, Taylor R, Armand M (2019) Localizing dexterous surgical tools in X-ray for image-based navigation. arXiv preprint arXiv:1901.06672

  17. 17.

    Laina I, Rieke N, Rupprecht C, Vizcaíno JP, Eslami A, Tombari F, Navab N (2017) Concurrent segmentation and localization for tracking of surgical instruments. In: Proceedings of medical image computing and computer-assisted intervention. Springer, pp 664–672

  18. 18.

    Kordon F, Fischer P, Privalov M, Swartman B, Schnetzke M, Franke J, Lasowski R, Maier A, Kunze H (2019) Multi-task localization and segmentation for X-ray guided planning in knee surgery. In: Proceedings of medical image computing and computer-assisted intervention. Springer, pp 622–630

  19. 19.

    Bier B, Goldmann F, Zaech JN, Fotouhi J, Hegeman R, Grupp R, Armand M, Osgood G, Navab N, Maier A, Unberath M (2019) Learning to detect anatomical landmarks of the pelvis in X-rays from arbitrary views. Int J Comput Assist Radiol Surg 14(9):1463–1473

    Article  PubMed  PubMed Central  Google Scholar 

  20. 20.

    Esteban J, Grimm M, Unberath M, Zahnd G, Navab N (2019) Towards fully automatic X-ray to CT registration. In: Proceedings of medical image computing and computer-assisted intervention. Springer, pp 631–639

  21. 21.

    Nikou C, Jaramaz B, DiGioia AM, Levison TJ (2000) Description of anatomic coordinate systems and rationale for use in an image-guided total hip replacement system. In: Proceedings of medical image computing and computer-assisted intervention, pp 1188–1194

    Chapter  Google Scholar 

  22. 22.

    Grupp RB, Armand M, Taylor RH (2018) Patch-based image similarity for intraoperative 2D/3D pelvis registration during periacetabular osteotomy. In: Proceedings of international workshop on clinical image-based procedures. Springer, pp 153–163

  23. 23.

    Storn R, Price K (1997) Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. J Glob Optim 11(4):341–359

    Article  Google Scholar 

  24. 24.

    Shi Y, Eberhart R (1998) A modified particle swarm optimizer. In: Proceedings of IEEE international conference on evolutionary computation. IEEE, pp 69–73

  25. 25.

    Hansen N, Ostermeier A (2001) Completely derandomized self-adaptation in evolution strategies. Evolut Comput 9(2):159–195

    Article  CAS  Google Scholar 

  26. 26.

    Powell MJ (2009) The BOBYQA algorithm for bound constrained optimization without derivatives. Cambridge NA Report NA2009/06. University of Cambridge, Cambridge

  27. 27.

    Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Proceedings of medical image computing and computer-assisted intervention. Springer, pp 234–241

  28. 28.

    Milletari F, Navab N, Ahmadi SA (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: Proceedings of 2016 fourth international conference on 3D vision (3DV). IEEE, pp 565–571

  29. 29.

    Hartley R, Zisserman A (2003) Multiple view geometry in computer vision. Cambridge University Press, Cambridge

    Google Scholar 

  30. 30.

    Unberath M, Zaech JN, Gao C, Bier B, Goldmann F, Lee SC, Fotouhi J, Taylor R, Armand M, Navab N (2019) Enabling machine learning in X-ray-based procedures via realistic simulation of image formation. Int J Comput Assist Radiol Surg 14(9):1517–1528

    Article  PubMed  PubMed Central  Google Scholar 

  31. 31.

    Gao C, Grupp RB, Unberath M, Taylor RH, Armand M (2020) Fiducial-free 2D/3D registration of the proximal femur for robot-assisted femoroplasty. In: Proceedings of SPIE, vol 11315, p 113151C

  32. 32.

    Reyneke CJF, Lüthi M, Burdin V, Douglas TS, Vetter T, Mutsvangwa TE (2018) Review of 2-D/3-D reconstruction using statistical shape and intensity models and X-ray image synthesis: toward a unified framework. IEEE Rev Biomed Eng 12:269–286

    Article  PubMed  Google Scholar 

  33. 33.

    Grupp R, Murphy R, Hegeman R, Alexander C, Unberath M, Otake Y, McArthur B, Armand M, Taylor R (2019) Fast and automatic periacetabular osteotomy fragment pose estimation using intraoperatively implanted fiducials and single-view fluoroscopy. arXiv preprint arXiv:1910.10187

Download references

Acknowledgements

We thank Mr. Demetries Boston for assisting with the cadaveric data acquisition. This research was supported by NIH/NIBIB grants R01EB006839, R21EB020113, Johns Hopkins University Internal Funds, and a Johns Hopkins University Applied Physics Laboratory (Grant No. FNACCX24) Graduate Student Fellowship. Part of this research project was conducted using computational resources at the Maryland Advanced Research Computing Center (MARCC).

Author information

Affiliations

  1. Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA

    Robert B. Grupp, Mathias Unberath, Cong Gao & Russell H. Taylor

  2. Research and Exploratory Development Department, Johns Hopkins University Applied Physics Laboratory, Laurel, MD, USA

    Rachel A. Hegeman & Mehran Armand

  3. Auris Health, Inc., Redwood City, CA, USA

    Ryan J. Murphy

  4. Department of Orthopaedic Surgery, Johns Hopkins Medicine, Baltimore, MD, USA

    Clayton P. Alexander & Mehran Armand

  5. Graduate School of Information Science, Nara Institute of Science and Technology, Ikoma, Nara, Japan

    Yoshito Otake

  6. Department of Surgery and Perioperative Care, Dell Medical School, University of Texas, Austin, TX, USA

    Benjamin A. McArthur

  7. Texas Orthopedics, Austin, TX, USA

    Benjamin A. McArthur

  8. Department of Mechanical Engineering, Johns Hopkins University, Baltimore, MD, USA

    Mehran Armand

Authors
  1. Robert B. Grupp
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mathias Unberath
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cong Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Rachel A. Hegeman
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ryan J. Murphy
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Clayton P. Alexander
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yoshito Otake
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Benjamin A. McArthur
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Mehran Armand
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Russell H. Taylor
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Robert B. Grupp.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

This article does not contain patient data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 2 (mp4 83748 KB)

Supplementary material 1 (pdf 1000 KB)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Grupp, R.B., Unberath, M., Gao, C. et al. Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration. Int J CARS 15, 759–769 (2020). https://doi.org/10.1007/s11548-020-02162-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-020-02162-7

Keywords

  • Landmark detection
  • Semantic segmentation
  • 2D/3D registration
  • X-ray navigation
  • Orthopedics