Skip to main content
SpringerLink
Log in
Book cover

International Workshop on Biomedical Image Registration

WBIR 2022: Biomedical Image Registration pp 103–115Cite as

SuperWarp: Supervised Learning and Warping on U-Net for Invariant Subvoxel-Precise Registration

  • 583 Accesses

Part of the Lecture Notes in Computer Science book series (LNCS,volume 13386)

Abstract

In recent years, learning-based image registration methods have gradually moved away from direct supervision with target warps to self-supervision using segmentations, producing promising results across several benchmarks. In this paper, we argue that the relative failure of supervised registration approaches can in part be blamed on the use of regular U-Nets, which are jointly tasked with feature extraction, feature matching, and estimation of deformation. We introduce one simple but crucial modification to the U-Net that disentangles feature extraction and matching from deformation prediction, allowing the U-Net to warp the features, across levels, as the deformation field is evolved. With this modification, direct supervision using target warps begins to outperform self-supervision approaches that require segmentations, presenting new directions for registration when images do not have segmentations. We hope that our findings in this preliminary workshop paper will re-ignite research interest in supervised image registration techniques. Our code is publicly available from https://github.com/balbasty/superwarp.

Keywords

  • Image registration
  • Optical flow
  • Supervised learning

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   54.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   69.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Zhang, K., Zuo, W., Chen, Y., Meng, D., Zhang, L.: Beyond a Gaussian denoiser: residual learning of deep CNN for image denoising. IEEE Trans. Image Process. 26, 3142–3155 (2017)

    CrossRef  MathSciNet  Google Scholar 

  2. Dong, C., Loy, C.C., He, K., Tang, X.: Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38, 295–307 (2016)

    CrossRef  Google Scholar 

  3. Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    CrossRef  Google Scholar 

  4. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of CVPR, pp. 3431–3440 (2015)

    Google Scholar 

  5. Noh, H., Hong, S., Han, B.: Learning deconvolution network for semantic segmentation. In: Proceedings of CVPR (2015)

    Google Scholar 

  6. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of CVPR, pp. 2414–2423 (2016)

    Google Scholar 

  7. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43

    CrossRef  Google Scholar 

  8. de Vos, B.D., Berendsen, F.F., Viergever, M.A., Staring, M., Išgum, I.: End-to-end unsupervised deformable image registration with a convolutional neural network. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS -2017. LNCS, vol. 10553, pp. 204–212. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67558-9_24

    CrossRef  Google Scholar 

  9. Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: An unsupervised learning model for deformable medical image registration. In: Proceedings of CVPR (2018)

    Google Scholar 

  10. Dalca, A.V., Balakrishnan, G., Guttag, J., Sabuncu, M.R.: Unsupervised learning for fast probabilistic diffeomorphic registration. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 729–738. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_82

    CrossRef  Google Scholar 

  11. Mok, T.C.W., Chung, A.C.S.: Fast symmetric diffeomorphic image registration with convolutional neural networks. In: Proceedings of CVPR (2020)

    Google Scholar 

  12. Cao, X., et al.: Deformable image registration based on similarity-steered CNN regression. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 300–308. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_35

    CrossRef  Google Scholar 

  13. Rohé, M.-M., Datar, M., Heimann, T., Sermesant, M., Pennec, X.: SVF-Net: learning deformable image registration using shape matching. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 266–274. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_31

    CrossRef  Google Scholar 

  14. Yang, X., Kwitt, R., Styner, M., Niethammer, M.: Quicksilver: fast predictive image registration – a deep learning approach. Neuroimage 158, 378–396 (2017)

    CrossRef  Google Scholar 

  15. Horn, B.K.P., Schunck, B.G.: Determining optical flow. Artif. Intell. 17, 185–203 (1981)

    CrossRef  Google Scholar 

  16. Sun, D., Roth, S., Black, M.J.: A quantitative analysis of current practices in optical flow estimation and the principles behind them. Int. J. Comput. Vis. 106, 115–137 (2014)

    CrossRef  Google Scholar 

  17. Black, M.J., Anandan, P.: The robust estimation of multiple motions: parametric and piecewise-smooth flow fields. Comput. Vis. Image Underst. 63, 75–104 (1996)

    CrossRef  Google Scholar 

  18. Papenberg, N., Bruhn, A., Brox, T., Didas, S., Weickert, J.: Highly accurate optic flow computation with theoretically justified warping. Int. J. Comput. Vis. 67, 141–158 (2006)

    CrossRef  Google Scholar 

  19. Roth, S., Lempitsky, V., Rother, C.: Discrete-continuous optimization for optical flow estimation. In: Cremers, D., Rosenhahn, B., Yuille, A.L., Schmidt, F.R. (eds.) Statistical and Geometrical Approaches to Visual Motion Analysis. LNCS, vol. 5604, pp. 1–22. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03061-1_1

    CrossRef  Google Scholar 

  20. Zach, C., Pock, T., Bischof, H.: A duality based approach for realtime TV-L 1 optical flow. In: Hamprecht, F.A., Schnörr, C., Jähne, B. (eds.) Pattern Recognition, pp. 214–223. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74936-3_22

    CrossRef  Google Scholar 

  21. Sun, D., Roth, S., Lewis, J.P., Black, M.J.: Learning Optical Flow. In: Forsyth, D., Torr, P., Zisserman, A. (eds.) ECCV 2008. LNCS, vol. 5304, pp. 83–97. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88690-7_7

    CrossRef  Google Scholar 

  22. Nagel, H., Enkelmann, W.: An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8, 565–593 (1986)

    CrossRef  Google Scholar 

  23. Wedel, A., Cremers, D., Pock, T., Bischof, H.: Structure- and motion-adaptive regularization for high accuracy optic flow. In: Proceedings of ICCV, pp. 1663–1668 (2009)

    Google Scholar 

  24. Zimmer, H., Bruhn, A., Weickert, J.: Optic flow in harmony. Int. J. Comput. Vis. 93, 368–388 (2011)

    CrossRef  MathSciNet  Google Scholar 

  25. Zimmer, H., et al.: Complementary optic flow. In: Cremers, D., Boykov, Y., Blake, A., Schmidt, F.R. (eds.) EMMCVPR 2009. LNCS, vol. 5681, pp. 207–220. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-03641-5_16

    CrossRef  Google Scholar 

  26. Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: Proceedings of CVPR, pp. 2432–2439 (2010)

    Google Scholar 

  27. Werlberger, M., Pock, T., Bischof, H.: Motion estimation with non-local total variation regularization. In: Proceedings of CVPR, pp. 2464–2471 (2010)

    Google Scholar 

  28. Ranftl, R., Bredies, K., Pock, T.: Non-local total generalized variation for optical flow estimation. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8689, pp. 439–454. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10590-1_29

    CrossRef  Google Scholar 

  29. Krähenbühl, P., Koltun, V.: Efficient nonlocal regularization for optical flow. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7572, pp. 356–369. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33718-5_26

    CrossRef  Google Scholar 

  30. Bredies, K., Kunisch, K., Pock, T.: Total generalized variation. SIAM J. Imaging Sci. 3, 492–526 (2010)

    CrossRef  MathSciNet  Google Scholar 

  31. Liu, C., Yuen, J., Torralba, A.: Sift flow: dense correspondence across scenes and its applications. IEEE Trans. Pattern Anal. Mach. Intell. 33, 978–994 (2011)

    CrossRef  Google Scholar 

  32. Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33, 500–513 (2011)

    CrossRef  Google Scholar 

  33. Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: DeepFlow: large displacement optical flow with deep matching. In: ICCV, pp. 1385–1392 (2013)

    Google Scholar 

  34. Hu, Y., Song, R., Li, Y.: Efficient coarse-to-fine patchmatch for large displacement optical flow. In: Proceedings of CVPR, pp. 5704–5712 (2016)

    Google Scholar 

  35. Lempitsky, V., Rother, C., Roth, S., Blake, A.: Fusion moves for markov random field optimization. IEEE Trans. Pattern Anal. Mach. Intell. 32, 1392–1405 (2010)

    CrossRef  Google Scholar 

  36. Wedel, A., Pock, T., Zach, C., Bischof, H., Cremers, D.: An improved algorithm for TV-L1 optical flow. In: Proceedings of Statistical and Geometrical Approaches to Visual Motion Analysis, pp. 23–45 (2009)

    Google Scholar 

  37. Dosovitskiy, A., et al.: FlowNet: learning optical flow with convolutional networks. In: Proceedings of CVPR (2015)

    Google Scholar 

  38. Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: evolution of optical flow estimation with deep networks. In: Proceedings of CVPR (2017)

    Google Scholar 

  39. Sun, D., Yang, X., Liu, M.-Y., Kautz, J.: PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In: Proceedings of CVPR (2018)

    Google Scholar 

  40. Ranjan, A., Black, M.J.: Optical flow estimation using a spatial pyramid network. In: Proceedings of CVPR (2017)

    Google Scholar 

  41. Yu, J.J., Harley, A.W., Derpanis, K.G.: Back to basics: unsupervised learning of optical flow via brightness constancy and motion smoothness. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 3–10. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-49409-8_1

    CrossRef  Google Scholar 

  42. Liu, P., Lyu, M., King, I., Xu, J.: SelFlow: self-supervised learning of optical flow. In: Proceedings of CVPR (2019)

    Google Scholar 

  43. Wang, Y., Yang, Y., Yang, Z., Zhao, L., Wang, P., Xu, W.: Occlusion aware unsupervised learning of optical flow. In: Proceedings of CVPR (2018)

    Google Scholar 

  44. Hur, J., Roth, S.: Iterative residual refinement for joint optical flow and occlusion estimation. In: Proceedings of CVPR (2019)

    Google Scholar 

  45. Liu, P., King, I., Lyu, M.R., Xu, J.: DDFlow: learning optical flow with unlabeled data distillation. In: Proceedings of AAAI Conference on Artificial Intelligence, vol. 33, pp. 8770–8777 (2019)

    Google Scholar 

  46. Ashburner, J.: A fast diffeomorphic image registration algorithm. Neuroimage 38, 95–113 (2007)

    CrossRef  Google Scholar 

  47. Blake, A., Zisserman, A.: Visual Reconstruction. MIT Press, Cambridge (1987)

    CrossRef  Google Scholar 

  48. Pratt, W.K., Kane, J., Andrews, H.C.: Hadamard transform image coding. Proc. IEEE. 57, 58–68 (1969)

    CrossRef  Google Scholar 

  49. Lee, C.-Y., Xie, S., Gallagher, P., Zhang, Z., Tu, Z.: Deeply-supervised nets. In: Proceedings of MLR, pp. 562–570 (2015)

    Google Scholar 

  50. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of ICLR (2017)

    Google Scholar 

  51. Fischl, B.: FreeSurfer. NeuroImage. 62, 774–781 (2012)

    CrossRef  Google Scholar 

  52. Hoffmann, M., Billot, B., Greve, D.N., Iglesias, J.E., Fischl, B., Dalca, A.V.: SynthMorph: learning contrast-invariant registration without acquired images. IEEE Trans. Med. Imaging 41, 543–558 (2021)

    CrossRef  Google Scholar 

Download references

Acknowledgments

Support for this research provided in part by the BRAIN Initiative Cell Census Network grant U01MH117023, NIBIB (P41EB015896, 1R01EB023281, R01EB-006758, R21EB018907, R01EB019956, P41EB030006, P41EB028741), NIA (1R-56AG064027, 1R01AG064027, 5R01AG008122, R01AG016495, 1R01AG070988), NIMH (R01MH123195, R01MH121885, 1RF1MH123195), NINDS (R01NS05–25851, R21-NS072652, R01NS070963, R01NS083534, 5U01NS086625, 5U24NS-10059103, R01NS105820), ARUK (IRG2019A-003), and was made possible by resources from Shared Instrumentation Grants 1S10RR023401, 1S10RR019307, and 1S10-RR023043. Additional support was provided by the NIH Blueprint for Neuroscience Research (5U01-MH093765), part of the multi-institutional Human Connectome Project.

Author information

Authors and Affiliations

  1. MGH/HST Martinos Center for Biomedical Imaging, Boston, USA

    Sean I. Young, Yaël Balbastre, Adrian V. Dalca, William M. Wells, Juan Eugenio Iglesias & Bruce Fischl

  2. Massachusetts Institute of Technology, Cambridge, USA

    Sean I. Young, Yaël Balbastre, Adrian V. Dalca, William M. Wells, Juan Eugenio Iglesias & Bruce Fischl

Authors
  1. Sean I. Young
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yaël Balbastre
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Adrian V. Dalca
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. William M. Wells
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Juan Eugenio Iglesias
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bruce Fischl
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sean I. Young .

Editor information

Editors and Affiliations

  1. Radboud University Nijmegen Medical Center, Nijmegen, The Netherlands

    Alessa Hering

  2. Helmholtz Center Munich, Neuherberg, Germany

    Julia Schnabel

  3. University of Virginia, Charlottesville, VA, USA

    Miaomiao Zhang

  4. CONICET, Universidad Nacional del Litoral, Santa Fe, Argentina

    Dr. Enzo Ferrante

  5. Universität zu Lübeck, Lübeck, Germany

    Mattias Heinrich

  6. GALILEO, Technical University of Munich, Munich, Germany

    Daniel Rueckert

Rights and permissions

Reprints and Permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Young, S.I., Balbastre, Y., Dalca, A.V., Wells, W.M., Iglesias, J.E., Fischl, B. (2022). SuperWarp: Supervised Learning and Warping on U-Net for Invariant Subvoxel-Precise Registration. In: Hering, A., Schnabel, J., Zhang, M., Ferrante, E., Heinrich, M., Rueckert, D. (eds) Biomedical Image Registration. WBIR 2022. Lecture Notes in Computer Science, vol 13386. Springer, Cham. https://doi.org/10.1007/978-3-031-11203-4_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-11203-4_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-11202-7

  • Online ISBN: 978-3-031-11203-4

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation