Skip to main content
SpringerLink
Log in
Book cover

International Conference on Medical Image Computing and Computer-Assisted Intervention

MICCAI 2022: Medical Image Computing and Computer Assisted Intervention – MICCAI 2022 pp 36–45Cite as

On the Dataset Quality Control for Image Registration Evaluation

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13436)

Abstract

Current registration evaluations typically compute target registration error using manually annotated datasets. As a result, the quality of landmark annotations is crucial for unbiased comparisons because registration algorithms are trained and tested using these landmarks. Even though some data providers claim to have mitigated the inter-observer variability by having multiple raters, quality control such as a third-party screening can still be reassuring for intended users. Examining the landmark quality for neurosurgical datasets (RESECT and BITE) poses specific challenges. In this study, we applied the variogram, which is a tool extensively used in geostatistics, to convert 3D landmark distributions into an intuitive 2D representation. This allowed us to identify potential problematic cases efficiently so that they could be examined by experienced radiologists. In both the RESECT and BITE datasets, we identified and confirmed a small number of landmarks with potential localization errors and found that, in some cases, the landmark distribution was not ideal for an unbiased assessment of non-rigid registration errors. Under discussion, we provide some constructive suggestions for improving the utility of publicly available annotated data.

Keywords

  • Image registration
  • Public datasets
  • Quality control

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Bardosi, Z., Freysinger, W.: Estimating FLE distributions of manual fiducial localization in CT images. Int. J. Comput. Assist. Radiol. Surg. 11, 1043–1049 (2016)

    CrossRef  Google Scholar 

  2. Beatriz Garcia, C., Solter, J., Bossa, Matias Husch, A.: On the composition and limitations of publicly availabble covid-19 x-ray imaging datasets. arXiv (2020)

    Google Scholar 

  3. Borovec, J.: BIRL: benchmark on image registration methods with landmark validation. arXiv (2020)

    Google Scholar 

  4. Borovec, J., Munoz-Barrutia, A.: ANHIR: automatic non-rigid histological image registration challenge. IEEE Trans. Med. Imaging 39(10), 3042–3052 (2020)

    CrossRef  Google Scholar 

  5. Christensen, G.E., Geng, X., Kuhl, J.G., Bruss, J., Grabowski, T.J., Pirwani, I.A., Vannier, M.W., Allen, J.S., Damasio, H.: Introduction to the non-rigid image registration evaluation project (NIREP). In: Pluim, J.P.W., Likar, B., Gerritsen, F.A. (eds.) WBIR 2006. LNCS, vol. 4057, pp. 128–135. Springer, Heidelberg (2006). https://doi.org/10.1007/11784012_16

    CrossRef  Google Scholar 

  6. Cressie, N.: Statistics for Spatial Data. Wiley, New York (1991)

    Google Scholar 

  7. Fedorov, A., et al.: 3D slicer as an image computing platform for the quantitative imaging network. Mag. Resonan. Imaging 30(9), 1323–1341 (2012)

    Google Scholar 

  8. Fitzpatrick, J.: The retrospective image registration evaluation project. Insight-Journal (2007)

    Google Scholar 

  9. Fitzpatrick, J.: Fiducial registration error and target registration error are uncorrelated. In: Proceedings of SPIE Medical Imaging. p. 726102G. SPIE (2009)

    Google Scholar 

  10. Hellier, P., et al.: Retrospective evaluation of inter-subject brain registration. IEEE Trans. Med. Imaging 22(9), 1120–1130 (2003)

    CrossRef  Google Scholar 

  11. Hering, A., Murphy, K., van Ginneken, B.: Learn2reg challenge: CT lung registration. In: MICCAI (2021)

    Google Scholar 

  12. Kabus, S., Klinder, T., Murphy, K., van Ginneken, B., Lorenz, C., Pluim, J.P.W.: Evaluation of 4D-CT lung registration. In: Yang, G.-Z., Hawkes, D., Rueckert, D., Noble, A., Taylor, C. (eds.) MICCAI 2009. LNCS, vol. 5761, pp. 747–754. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04268-3_92

    CrossRef  Google Scholar 

  13. Klein, A., et al.: Evaluation of 14 nonlinear deformation algorithms applied to human brain MRI registration. Neuroimage 46(3), 786–802 (2009)

    Google Scholar 

  14. Klein, A., et al.: Evaluation of volume-based and surface-based brain image registration methods. Neuroimage 51(1), 214–220 (2010)

    Google Scholar 

  15. Luo, J., et al.: A Feature-driven active framework for ultrasound-based brain shift compensation. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 30–38. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_4

  16. Luo, J., Frisken, S., Wang, D., Golby, A., Sugiyama, M., Wells III, W.: Are registration uncertainty and error monotonically associated? In: MICCAI 2020 (2020)

    Google Scholar 

  17. Machado, I., et al.: Deformable MRI-ultrasound registration using correlation-based attribute matching for brain shift correction: accuracy and generality in multi-site data. Neuroimage 202(15), (2019)

    Google Scholar 

  18. Mercier, L., Del Maestro, R., Petrecca, K., Araujo, D., Haegelen, C., Collins, D.: Online database of clinical MR and ultrasound images of brain tumors. Med. Phys. 39(6), 3253–3261 (2012)

    Google Scholar 

  19. Murphy, K., et al.: Evaluation of registration methods on thoracic CT: the empire10 challenge. IEEE Trans. Med. Imaging 30(11), 1901–1920 (2011)

    Google Scholar 

  20. Oakden-Rayner, L.: Exploring large scale public medical image datasets. arXiv (2019)

    Google Scholar 

  21. Ou, Y., Akbari, H., Bilello, M., Da, X., Davatzikos, C.: Comparative evaluation of registration algorithms in different brain databases with varying difficulty. IEEE Trans. Med. Imaging 33(10), 2039–2065 (2014)

    Google Scholar 

  22. Reinertsen, I., Collins, D., Drouin, S.: The essential role of open data and software for the future of ultrasound-based neuronavigation. Front. Oncol. 10 (2020)

    Google Scholar 

  23. Rohlfing, T.: Image similarity and tissue overlaps as surrogates for image registration accuracy: widely used but unreliable. IEEE Trans. Med. Imaging 31(2), 153–16 (2012)

    Google Scholar 

  24. dos Santos, T., et al.: Pose-independent surface matching for intra-operative soft-tissue marker-less registration. Med. Image Anal. 18(7), 1101–1114 (2014)

    Google Scholar 

  25. Shamir, R.R., Joskowicz, L., Shoshan, Y.: Fiducial optimization for minimal target registration error in image-guided neurosurgery. IEEE Trans. Med. Imaging 31(3), 725–737 (2012)

    Google Scholar 

  26. Song, J.: Methods for evaluating image registration. Dissertation, University of Iowa (2017)

    Google Scholar 

  27. Sonka, M.: Handbook of medical imaging: medical image processing and analysis. In: SPIE (2000)

    Google Scholar 

  28. Sotiras, A., Davatzikos, C., Paragios, N.: Deformable medical image registration: A survey. IEEE Trans. Med. Imaging 32(7), 1153–1190 (2013)

    Google Scholar 

  29. Thompson, S., Penney, G., Dasgupta, P., Hawkes, D.: Improved modelling of tool tracking errors by modelling dependent marker errors. IEEE Trans. Med. Imaging 32(2), 165–177 (2013)

    Google Scholar 

  30. Wang, J., Zhang, M.: Deepflash: An efficient network for learning-based medical image registration. In: CVPR 2020, pp. 4444–4452. IEEE (2020)

    Google Scholar 

  31. West, J., Fitzpatrick, J., Wang, M., Woods, R.: Comparison and evaluation of retrospective intermodality brain image registration techniques. J. Comput. Assist. Tomogr. 21(4), 554–566 (1997)

    Google Scholar 

  32. Xiao, Y., Fortin, M., Unsgard, G., Rivaz, H., Reinertsen, I.: Retrospective evaluation of cerebral tumors (resect): a clinical database of pre-operative MRI and intra-operative ultrasound in low-grade glioma surgeries. Med. Phys. 44(7), 3875–3882 (2017)

    Google Scholar 

  33. Xiao, Y., Rivaz, H., Chabanas, M.: Evaluation of MRI to ultrasound registration methods for brain shift correction: the curious2018 challenge. IEEE Trans. Med. Imaging 39(3), 777–786 (2020)

    Google Scholar 

  34. Xu, Z., et al.: Evaluation of six registration methods for the human abdomen on clinically acquired CT. IEEE Trans. Biomed. Eng. 63(8), 1563–1572 (2016)

    Google Scholar 

  35. Yassa, M., Stark, C.E.: A quantitative evaluation of cross-participant registration techniques for MRI studies of the medial temporal lobe. Neuroimage 442, 319–327 (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Brigham and Women’s Hospital, Harvard Medical School, Boston, USA

    Jie Luo, Nazim Haouchine, Tina Kapur, Lipeng Ning, William M. Wells III & Sarah Frisken

  2. Department of Mechanical Engineering and Material Science, Duke University, Durham, USA

    Guangshen Ma

  3. Department of Biomedical Engineering, Chinese University of Hong Kong, Hong Kong, Hong Kong

    Zhe Xu

  4. Department of Bioengineering, Stanford University, Stanford, USA

    Yixin Wang

Authors
  1. Jie Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guangshen Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nazim Haouchine
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhe Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yixin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tina Kapur
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Lipeng Ning
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. William M. Wells III
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Sarah Frisken
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jie Luo .

Editor information

Editors and Affiliations

  1. Rochester Institute of Technology, Rochester, NY, USA

    Linwei Wang

  2. Chinese University of Hong Kong, Hong Kong, Hong Kong

    Qi Dou

  3. University of Virginia, Charlottesville, VA, USA

    P. Thomas Fletcher

  4. National Center for Tumor Diseases (NCT/UCC), Dresden, Germany

    Stefanie Speidel

  5. Case Western Reserve University, Cleveland, OH, USA

    Shuo Li

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Luo, J. et al. (2022). On the Dataset Quality Control for Image Registration Evaluation. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13436. Springer, Cham. https://doi.org/10.1007/978-3-031-16446-0_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16446-0_4

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16445-3

  • Online ISBN: 978-3-031-16446-0

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation