Content-Aware Unsupervised Deep Homography Estimation

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 12346)


Homography estimation is a basic image alignment method in many applications. It is usually conducted by extracting and matching sparse feature points, which are error-prone in low-light and low-texture images. On the other hand, previous deep homography approaches use either synthetic images for supervised learning or aerial images for unsupervised learning, both ignoring the importance of handling depth disparities and moving objects in real world applications. To overcome these problems, in this work we propose an unsupervised deep homography method with a new architecture design. In the spirit of the RANSAC procedure in traditional methods, we specifically learn an outlier mask to only select reliable regions for homography estimation. We calculate loss with respect to our learned deep features instead of directly comparing image content as did previously. To achieve the unsupervised training, we also formulate a novel triplet loss customized for our network. We verify our method by conducting comprehensive comparisons on a new dataset that covers a wide range of scenes with varying degrees of difficulties for the task. Experimental results reveal that our method outperforms the state-of-the-art including deep solutions and feature-based solutions.


Homography Deep homography Image alignment RANSAC 



This research was supported in part by National Key Research and Development Program of China under Grant 2017YFA0700800, in part by National Natural Science Foundation of China under Grants (NSFC, No. 61872067 and No. 61720106004) and in part by Research Programs of Science and Technology in Sichuan Province under Grant 2019YFH0016.

Supplementary material (83.4 mb)
Supplementary material 1 (zip 85432 KB)


  1. 1.
    Altwaijry, H., Veit, A., Belongie, S.J., Tech, C.: Learning to detect and match keypoints with deep architectures. In: Proceedings of BMVC (2016)Google Scholar
  2. 2.
    Baker, S., Matthews, I.: Lucas-kanade 20 years on: a unifying framework. Int. J. Comput. Vis. 56(3), 221–255 (2004)CrossRefGoogle Scholar
  3. 3.
    Barath, D., Matas, J., Noskova, J.: MAGSAC: marginalizing sample consensus. In: Proceedings of CVPR, pp. 10197–10205 (2019)Google Scholar
  4. 4.
    Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). Scholar
  5. 5.
    Bian, J., Lin, W.Y., Matsushita, Y., Yeung, S.K., Nguyen, T.D., Cheng, M.M.: GMS: grid-based motion statistics for fast, ultra-robust feature correspondence. In: Proceedings of CVPR, pp. 4181–4190 (2017)Google Scholar
  6. 6.
    Brown, M., Lowe, D.: Recognising panoramas. In: Proceedings of ICCV, p. 1218 (2003)Google Scholar
  7. 7.
    DeTone, D., Malisiewicz, T., Rabinovich, A.: Deep image homography estimation. arXiv preprint arXiv:1606.03798 (2016)
  8. 8.
    Evangelidis, G.D., Psarakis, E.Z.: Parametric image alignment using enhanced correlation coefficient maximization. IEEE Trans. Pattern Anal. Mach. Intell. 30(10), 1858–1865 (2008)CrossRefGoogle Scholar
  9. 9.
    Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Gelfand, N., Adams, A., Park, S.H., Pulli, K.: Multi-exposure imaging on mobile devices. In: Proceedings of ACM Multimedia, pp. 823–826 (2010)Google Scholar
  11. 11.
    Godard, C., Mac, O., Firman, M., Brostow, G.J.: Digging into self-supervised monocular depth estimation. In: Proceedings of ICCV, pp. 3828–3838 (2019)Google Scholar
  12. 12.
    Guo, H., Liu, S., He, T., Zhu, S., Zeng, B., Gabbouj, M.: Joint video stitching and stabilization from moving cameras. IEEE Trans. Image Process. 25(11), 5491–5503 (2016)MathSciNetCrossRefGoogle Scholar
  13. 13.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2003)zbMATHGoogle Scholar
  14. 14.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of CVPR, pp. 770–778 (2016)Google Scholar
  15. 15.
    Holland, P.W., Welsch, R.E.: Robust regression using iteratively reweighted least-squares. Commun. Stat. Theo. Methods 6(9), 813–827 (1977)CrossRefGoogle Scholar
  16. 16.
    Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: FlowNet 2.0: evolution of optical flow estimation with deep networks. In: Proceedings of CVPR, pp. 2462–2470 (2017)Google Scholar
  17. 17.
    Jaderberg, M., Simonyan, K., Zisserman, A., et al.: Spatial transformer networks. In: Advances in Neural Information Processing Systems, pp. 2017–2025 (2015)Google Scholar
  18. 18.
    Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  19. 19.
    Lin, K., Jiang, N., Liu, S., Cheong, L.F., Do, M., Lu, J.: Direct photometric alignment by mesh deformation. In: Proceedings of CVPR, pp. 2405–2413 (2017)Google Scholar
  20. 20.
    Liu, S., Tan, P., Yuan, L., Sun, J., Zeng, B.: MeshFlow: minimum latency online video stabilization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 800–815. Springer, Cham (2016). Scholar
  21. 21.
    Liu, S., Yuan, L., Tan, P., Sun, J.: Bundled camera paths for video stabilization. ACM Trans. Graph. 32(4), 78 (2013)Google Scholar
  22. 22.
    Liu, Z., Yuan, L., Tang, X., Uyttendaele, M., Sun, J.: Fast burst images denoising. ACM Trans. Graph. 33(6), 1–9 (2014)Google Scholar
  23. 23.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vis. 60(2), 91–110 (2004)CrossRefGoogle Scholar
  24. 24.
    Lucas, B.D., Kanade, T., et al.: An iterative image registration technique with an application to stereo vision. In: Proceedings of IJCAI (1981)Google Scholar
  25. 25.
    Ma, J., Zhao, J., Jiang, J., Zhou, H., Guo, X.: Locality preserving matching. Int. J. Comput. Vis. 127(5), 512–531 (2019)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Mur-Artal, R., Montiel, J.M.M., Tardos, J.D.: ORB-SLAM: a versatile and accurate monocular slam system. IEEE Trans. Robot. 31(5), 1147–1163 (2015)CrossRefGoogle Scholar
  27. 27.
    Nguyen, T., Chen, S.W., Shivakumar, S.S., Taylor, C.J., Kumar, V.: Unsupervised deep homography: a fast and robust homography estimation model. IEEE Robot. Autom. Lett. 3(3), 2346–2353 (2018)CrossRefGoogle Scholar
  28. 28.
    Revaud, J., Weinzaepfel, P., Harchaoui, Z., Schmid, C.: Deepmatching: hierarchical deformable dense matching. Int. J. Comput. Vis. 120(3), 300–323 (2016)MathSciNetCrossRefGoogle Scholar
  29. 29.
    Rublee, E., Rabaud, V., Konolige, K., Bradski, G.R.: ORB: an efficient alternative to SIFT or SURF. In: Proceedings of ICCV, vol. 11, pp. 2564–2571 (2011)Google Scholar
  30. 30.
    Simon, G., Fitzgibbon, A.W., Zisserman, A.: Markerless tracking using planar structures in the scene. In: Proceedings of International Symposium on Augmented Reality, pp. 120–128 (2000)Google Scholar
  31. 31.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  32. 32.
    Tian, Y., Yu, X., Fan, B., Wu, F., Heijnen, H., Balntas, V.: SOSNet: second order similarity regularization for local descriptor learning. In: Proceedings of CVPR, pp. 11016–11025 (2019)Google Scholar
  33. 33.
    Weinzaepfel, P., Revaud, J., Harchaoui, Z., Schmid, C.: Deepflow: large displacement optical flow with deep matching. In: Proceedings of CVPR, pp. 1385–1392 (2013)Google Scholar
  34. 34.
    Wronski, B., et al.: Handheld multi-frame super-resolution. ACM Trans. Graph. 38(4), 1–18 (2019)CrossRefGoogle Scholar
  35. 35.
    Yi, K.M., Trulls, E., Lepetit, V., Fua, P.: LIFT: learned invariant feature transform. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 467–483. Springer, Cham (2016). Scholar
  36. 36.
    Zaragoza, J., Chin, T.J., Brown, M.S., Suter, D.: As-projective-as-possible image stitching with moving DLT. In: Proceedings of CVPR, pp. 2339–2346 (2013)Google Scholar
  37. 37.
    Zhang, F., Liu, F.: Parallax-tolerant image stitching. In: Proceedings of CVPR, pp. 3262–3269 (2014)Google Scholar
  38. 38.
    Zhang, J., et al.: Learning two-view correspondences and geometry using order-aware network. In: Proceedings of ICCV, pp. 5845–5854 (2019)Google Scholar
  39. 39.
    Zhang, X., Zhou, X., Lin, M., Sun, J.: ShuffleNet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of CVPR, pp. 6848–6856 (2018)Google Scholar
  40. 40.
    Zhang, Z.: A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell. 22(11), 1330–1334 (2000)CrossRefGoogle Scholar
  41. 41.
    Zhou, T., Brown, M., Snavely, N., Lowe, D.G.: Unsupervised learning of depth and ego-motion from video. In: Proceedings of CVPR, pp. 1851–1858 (2017)Google Scholar
  42. 42.
    Zou, D., Tan, P.: CoSLAM: collaborative visual slam in dynamic environments. IEEE Trans. Pattern Anal. Mach. Intell. 35(2), 354–366 (2012)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.University of Electronic Science and Technology of ChinaChengduChina
  2. 2.Megvii TechnologyBeijingChina

Personalised recommendations