Advertisement

Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration

Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12263)

Abstract

Differentiable rendering is a technique to connect 3D scenes with corresponding 2D images. Since it is differentiable, processes during image formation can be learned. Previous approaches to differentiable rendering focus on mesh-based representations of 3D scenes, which is inappropriate for medical applications where volumetric, voxelized models are used to represent anatomy. We propose a novel Projective Spatial Transformer module that generalizes spatial transformers to projective geometry, thus enabling differentiable volume rendering. We demonstrate the usefulness of this architecture on the example of 2D/3D registration between radiographs and CT scans. Specifically, we show that our transformer enables end-to-end learning of an image processing and projection model that approximates an image similarity function that is convex with respect to the pose parameters, and can thus be optimized effectively using conventional gradient descent. To the best of our knowledge, we are the first to describe the spatial transformers in the context of projective transmission imaging, including rendering and pose estimation. We hope that our developments will benefit related 3D research applications. The source code is available at https://github.com/gaocong13/Projective-Spatial-Transformers.

This is a preview of subscription content, log in to check access.

References

  1. 1.
    Ferrante, E., Oktay, O., Glocker, B., Milone, D.H.: On the adaptability of unsupervised CNN-based deformable image registration to unseen image domains. In: Shi, Y., Suk, H.-I., Liu, M. (eds.) MLMI 2018. LNCS, vol. 11046, pp. 294–302. Springer, Cham (2018).  https://doi.org/10.1007/978-3-030-00919-9_34CrossRefGoogle Scholar
  2. 2.
    Gibson, E., et al.: Automatic multi-organ segmentation on abdominal CT with dense V-networks. IEEE Trans. Med. Imaging 37(8), 1822–1834 (2018)CrossRefGoogle Scholar
  3. 3.
    Grupp, R., et al.: Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration. arXiv preprint arXiv:1911.07042 (2019)
  4. 4.
    Hansen, N., Müller, S.D., Koumoutsakos, P.: Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES). Evol. Comput. 11(1), 1–18 (2003)CrossRefGoogle Scholar
  5. 5.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  6. 6.
    Haskins, G., Kruger, U., Yan, P.: Deep learning in medical image registration: a survey. Mach. Vis. Appl. 1–18 (2020).  https://doi.org/10.1007/s00138-020-01060-x
  7. 7.
    Henderson, P., Ferrari, V.: Learning single-image 3D reconstruction by generative modelling of shape, pose and shading. Int. J. Comput. Vis. 128, 835–854 (2020).  https://doi.org/10.1007/s11263-019-01219-8CrossRefGoogle Scholar
  8. 8.
    Hou, B., et al.: Predicting slice-to-volume transformation in presence of arbitrary subject motion. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 296–304. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-66185-8_34CrossRefGoogle Scholar
  9. 9.
    Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in Neural Information Processing Systems, pp. 2017–2025 (2015)Google Scholar
  10. 10.
    Krčah, M., Székely, G., Blanc, R.: Fully automatic and fast segmentation of the femur bone from 3D-CT images with no shape prior. In: 2011 IEEE International Symposium on Biomedical Imaging: from Nano to Macro, pp. 2087–2090. IEEE (2011)Google Scholar
  11. 11.
    Krebs, J., et al.: Robust non-rigid registration through agent-based action learning. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 344–352. Springer, Cham (2017).  https://doi.org/10.1007/978-3-319-66182-7_40CrossRefGoogle Scholar
  12. 12.
    Kuang, D., Schmah, T.: FAIM – a ConvNet method for unsupervised 3D medical image registration. In: Suk, H.-I., Liu, M., Yan, P., Lian, C. (eds.) MLMI 2019. LNCS, vol. 11861, pp. 646–654. Springer, Cham (2019).  https://doi.org/10.1007/978-3-030-32692-0_74CrossRefGoogle Scholar
  13. 13.
    Liao, R., et al.: An artificial agent for robust image registration. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)Google Scholar
  14. 14.
    Liu, S., Li, T., Chen, W., Li, H.: Soft rasterizer: a differentiable renderer for image-based 3D reasoning. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7708–7717 (2019)Google Scholar
  15. 15.
    Loper, M.M., Black, M.J.: OpenDR: an approximate differentiable renderer. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8695, pp. 154–169. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10584-0_11CrossRefGoogle Scholar
  16. 16.
    Maes, F., Collignon, A., Vandermeulen, D., Marchal, G., Suetens, P.: Multimodality image registration by maximization of mutual information. IEEE Trans. Med. Imaging 16(2), 187–198 (1997)CrossRefGoogle Scholar
  17. 17.
    Mahendran, S., Ali, H., Vidal, R.: 3D pose regression using convolutional neural networks. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2174–2182 (2017)Google Scholar
  18. 18.
    Miao, S., et al.: Dilated FCN for multi-agent 2D/3D medical image registration. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)Google Scholar
  19. 19.
    Miao, S., Wang, Z.J., Liao, R.: A CNN regression approach for real-time 2D/3D registration. IEEE Trans. Med. Imaging 35(5), 1352–1363 (2016)CrossRefGoogle Scholar
  20. 20.
    Miolane, N., Mathe, J., Donnat, C., Jorda, M., Pennec, X.: Geomstats: a python package for riemannian geometry in machine learning. arXiv preprint arXiv:1805.08308 (2018)
  21. 21.
    Penney, G.P., Weese, J., Little, J.A., Desmedt, P., Hill, D.L., et al.: A comparison of similarity measures for use in 2-D-3-D medical image registration. IEEE Trans. Med. Imaging 17(4), 586–595 (1998)CrossRefGoogle Scholar
  22. 22.
    Roth, H.R., et al.: A new 2.5D representation for lymph node detection using random sets of deep convolutional neural network observations. In: Golland, P., Hata, N., Barillot, C., Hornegger, J., Howe, R. (eds.) MICCAI 2014. LNCS, vol. 8673, pp. 520–527. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10404-1_65CrossRefGoogle Scholar
  23. 23.
    Salehi, S.S.M., Khan, S., Erdogmus, D., Gholipour, A.: Real-time deep registration with geodesic loss. arXiv preprint arXiv:1803.05982 (2018)
  24. 24.
    Smith, L.N.: Cyclical learning rates for training neural networks. In: 2017 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 464–472. IEEE (2017)Google Scholar
  25. 25.
    Würfl, T., et al.: Deep learning computed tomography: learning projection-domain weights from image domain in limited angle problems. IEEE Trans. Med. Imaging 37(6), 1454–1463 (2018)CrossRefGoogle Scholar
  26. 26.
    Yan, X., Yang, J., Yumer, E., Guo, Y., Lee, H.: Perspective transformer nets: learning single-view 3D object reconstruction without 3D supervision. In: Advances in Neural Information Processing Systems, pp. 1696–1704 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Johns Hopkins UniversityBaltimoreUSA

Personalised recommendations

Cite paper

Buy options