An Improved Learning Framework for Covariant Local Feature Detection

  • Nehal Doiphode
  • Rahul MitraEmail author
  • Shuaib Ahmed
  • Arjun Jain
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11366)


Learning feature detection has been largely an unexplored area when compared to handcrafted feature detection. Recent learning formulations use the covariant constraint in their loss function to learn covariant detectors. However, just learning from covariant constraint can lead to detection of unstable features. To impart further, stability detectors are trained to extract pre-determined features obtained by hand-crafted detectors. However, in the process they lose the ability to detect novel features. In an attempt to overcome the above limitations, we propose an improved scheme by incorporating covariant constraints in form of triplets with addition to an affine covariant constraint. We show that using these additional constraints one can learn to detect novel and stable features without using pre-determined features for training. Extensive experiments show our model achieves state-of-the-art performance in repeatability score on the well known datasets such as Vgg-Affine, EF, and Webcam.


Local features Covariant detection Deep learning 


  1. 1.
    Balntas, V., Riba, E., Ponsa, D., Mikolajczyk, K.: Learning local feature descriptors with triplets and shallow convolutional neural networks. In: BMVC 2016, pp. 119:1–119:11 (2016)Google Scholar
  2. 2.
    Bay, H., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 404–417. Springer, Heidelberg (2006). Scholar
  3. 3.
    Brown, M., Lowe, D.G.: Automatic panoramic image stitching using invariant features. IJCV 74, 59–73 (2007)CrossRefGoogle Scholar
  4. 4.
    Brown, M., Szeliski, R., Winder, S.: Multi-image matching using multi-scale oriented patches. In: CVPR 2005, pp. 510–517 (2005)Google Scholar
  5. 5.
    Harris, C., Stephens, M.: A combined corner and edge detector. In: Alvey Vision Conference 1988, pp. 147–151 (1988)Google Scholar
  6. 6.
    Lenc, K., Vedaldi, A.: Learning covariant feature detectors. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 100–117. Springer, Cham (2016). Scholar
  7. 7.
    Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60, 91–110 (2004)CrossRefGoogle Scholar
  8. 8.
    Mikolajczyk, K., Schmid, C.: Indexing based on scale invariant interest points. In: ICCV 2001, pp. 525–531 (2001)Google Scholar
  9. 9.
    Mikolajczyk, K., Schmid, C.: Scale & affine invariant interest point detectors. IJCV 60, 63–86 (2004)CrossRefGoogle Scholar
  10. 10.
    Mikolajczyk, K., et al.: A comparison of affine region detectors. IJCV 65, 43–72 (2005)CrossRefGoogle Scholar
  11. 11.
    Mishchuk, A., Mishkin, D., Radenovic, F., Matas, J.: Working hard to know your neighbor’s margins: local descriptor learning loss. In: NIPS 2017 (2017)Google Scholar
  12. 12.
    Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). Scholar
  13. 13.
    Rosten, E., Porter, R., Drummond, T.: Faster and better: a machine learning approach to corner detection. PAMI 32, 105–119 (2010)CrossRefGoogle Scholar
  14. 14.
    Savinov, N., Seki, A., Ladicky, L., Sattler, T., Pollefeys, M.: Quad-networks: unsupervised learning to rank for interest point detection. In: CVPR 2017, pp. 3929–3937 (2017)Google Scholar
  15. 15.
    Snavely, N., Seitz, S.M., Szeliski, R.: Photo tourism: exploring photo collections in 3D. In: ACM SIGGRAPH 2006, pp. 835–846 (2006)Google Scholar
  16. 16.
    Strecha, C., Lindner, A., Ali, K., Fua, P.: Training for task specific keypoint detection. In: Denzler, J., Notni, G., Süße, H. (eds.) DAGM 2009. LNCS, vol. 5748, pp. 151–160. Springer, Heidelberg (2009). Scholar
  17. 17.
    Tian, Y., Fan, B., Wu, F.: L2-Net: deep learning of discriminative patch descriptor in euclidean space. In: CVPR 2017, pp. 6128–6136 (2017)Google Scholar
  18. 18.
    Verdie, Y., Yi, K.M., Fua, P., Lepetit, V.: TILDE: a temporally invariant learned detector. In: CVPR 2015, pp. 5279–5288 (2015)Google Scholar
  19. 19.
    Yi, K.M., Trulls, E., Lepetit, V., Fua, P.: LIFT: learned invariant feature transform. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 467–483. Springer, Cham (2016). Scholar
  20. 20.
    Zhang, H., Zha, Z.J., Yang, Y., Yan, S., Gao, Y., Chua, T.S.: Attribute-augmented semantic hierarchy: towards bridging semantic gap and intention gap in image retrieval. In: ACM MM 2013, pp. 33–42 (2013)Google Scholar
  21. 21.
    Zhang, X., Yu, F.X., Karaman, S.: Learning discriminative and transformation covariant local feature detectors. In: CVPR 2017 (2017)Google Scholar
  22. 22.
    Zitnick, C.L., Ramnath, K.: Edge foci interest points. In: ICCV 2011, pp. 359–366 (2011)Google Scholar
  23. 23.
    Zitova, B., Flusser, J.: Image registration methods: a survey. Image Vis. Comput. 21, 977–1000 (2003)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Nehal Doiphode
    • 1
  • Rahul Mitra
    • 2
    Email author
  • Shuaib Ahmed
    • 3
  • Arjun Jain
    • 2
  1. 1.University of PennsylvaniaPhiladelphiaUSA
  2. 2.Indian Institute of Technology BombayMumbaiIndia
  3. 3.Mercedes-Benz Research and Development India Private LimitedBengaluruIndia

Personalised recommendations