LIFT: Learned Invariant Feature Transform

  • Kwang Moo YiEmail author
  • Eduard Trulls
  • Vincent Lepetit
  • Pascal Fua
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9910)


We introduce a novel Deep Network architecture that implements the full feature point handling pipeline, that is, detection, orientation estimation, and feature description. While previous works have successfully tackled each one of these problems individually, we show how to learn to do all three in a unified manner while preserving end-to-end differentiability. We then demonstrate that our Deep pipeline outperforms state-of-the-art methods on a number of benchmark datasets, without the need of retraining.


Local features Feature descriptors Deep Learning 

Supplementary material (21.8 mb)
Supplementary material 1 (zip 22293 KB)


  1. 1.
    Lowe, D.: Distinctive image features from scale-invariant keypoints. IJCV 20(2), 91–110 (2004)CrossRefGoogle Scholar
  2. 2.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: SURF: speeded up robust features. CVIU 10(3), 346–359 (2008)Google Scholar
  3. 3.
    Tola, E., Lepetit, V., Fua, P.: A fast local descriptor for dense matching. In: CVPR (2008)Google Scholar
  4. 4.
    Rublee, E., Rabaud, V., Konolidge, K., Bradski, G.: ORB: an efficient alternative to SIFT or SURF. In: ICCV (2011)Google Scholar
  5. 5.
    Mainali, P., Lafruit, G., Tack, K., Van Gool, L., Lauwereins, R.: Derivative-based scale invariant image feature detector with error resilience. TIP 23(5), 2380–2391 (2014)MathSciNetGoogle Scholar
  6. 6.
    Verdie, Y., Yi, K.M., Fua, P., Lepetit, V.: TILDE: a temporally invariant learned DEtector. In: CVPR (2015)Google Scholar
  7. 7.
    Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C.: MatchNet: unifying feature and metric learning for patch-based matching. In: CVPR (2015)Google Scholar
  8. 8.
    Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: CVPR (2015)Google Scholar
  9. 9.
    Yi, K., Verdie, Y., Lepetit, V., Fua, P.: Learning to assign orientations to feature points. In: CVPR (2016)Google Scholar
  10. 10.
    Simo-Serra, E., Trulls, E., Ferraz, L., Kokkinos, I., Fua, P., Moreno-Noguer, F.: Discriminative learning of deep convolutional feature point descriptors. In: ICCV (2015)Google Scholar
  11. 11.
    Jaderberg, M., Simonyan, K., Zisserman, A., Kavukcuoglu, K.: Spatial transformer networks. In: NIPS (2015)Google Scholar
  12. 12.
    Chapelle, O., Wu, M.: Gradient descent optimization of smoothed information retrieval metrics. Inf. Retrieval 13(3), 216–235 (2009)CrossRefGoogle Scholar
  13. 13.
    Harris, C., Stephens, M.: A combined corner and edge detector. In: Fourth Alvey Vision Conference (1988)Google Scholar
  14. 14.
    Moravec, H.: Obstacle avoidance and navigation in the real world by a seeing robot rover. Technical report CMU-RI-TR-80-03, Robotics Institute, Carnegie Mellon University, Stanford University, September 1980Google Scholar
  15. 15.
    Rosten, E., Drummond, T.: Machine learning for high-speed corner detection. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3951, pp. 430–443. Springer, Heidelberg (2006). doi: 10.1007/11744023_34 CrossRefGoogle Scholar
  16. 16.
    Matas, J., Chum, O., Martin, U., Pajdla, T.: Robust wide baseline stereo from maximally stable extremal regions. In: BMVC, pp. 384–393, September 2002Google Scholar
  17. 17.
    Mikolajczyk, K., Schmid, C.: An affine invariant interest point detector. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002. LNCS, vol. 2350, pp. 128–142. Springer, Heidelberg (2002). doi: 10.1007/3-540-47969-4_9 CrossRefGoogle Scholar
  18. 18.
    Förstner, W., Dickscheid, T., Schindler, F.: Detecting interpretable and accurate scale-invariant keypoints. In: ICCV, September 2009Google Scholar
  19. 19.
    Zitnick, C., Ramnath, K.: Edge foci interest points. In: ICCV (2011)Google Scholar
  20. 20.
    Mainali, P., Lafruit, G., Yang, Q., Geelen, B., Van Gool, L., Lauwereins, R.: SIFER: scale-invariant feature detector with error resilience. IJCV 104(2), 172–197 (2013)CrossRefzbMATHGoogle Scholar
  21. 21.
    Šochman, J., Matas, J.: Learning a fast emulator of a binary decision process. In: Yagi, Y., Kang, S.B., Kweon, I.S., Zha, H. (eds.) ACCV 2007. LNCS, vol. 4844, pp. 236–245. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-76390-1_24 CrossRefGoogle Scholar
  22. 22.
    Trujillo, L., Olague, G.: Using evolution to learn how to perform interest point detection. In: ICPR, pp. 211–214 (2006)Google Scholar
  23. 23.
    Strecha, C., Bronstein, A., Bronstein, M., Fua, P.: LDAHash: improved matching with smaller descriptors. PAMI 34(1), 66–78 (2012)CrossRefGoogle Scholar
  24. 24.
    Winder, S., Brown, M.: Learning local image descriptors. In: CVPR, June 2007Google Scholar
  25. 25.
    Perez, C., Olague, G.: Genetic programming as strategy for learning image descriptor operators. Intell. Data Anal. 17, 561–583 (2013)Google Scholar
  26. 26.
    Simonyan, K., Vedaldi, A., Zisserman, A.: Learning local feature descriptors using convex optimisation. PAMI 36, 1573–1585 (2014)CrossRefGoogle Scholar
  27. 27.
    Zbontar, J., LeCun, Y.: Computing the stereo matching cost with a convolutional neural network. In: CVPR (2015)Google Scholar
  28. 28.
    Balntas, V., Johns, E., Tang, L., Mikolajczyk, K.: PN-Net: conjoined triple deep network for learning local image descriptors. In: arXiv Preprint (2016)Google Scholar
  29. 29.
    Wilson, K., Snavely, N.: Robust global translations with 1DSfM. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8691, pp. 61–75. Springer, Heidelberg (2014). doi: 10.1007/978-3-319-10578-9_5 Google Scholar
  30. 30.
    Wu, C.: Towards linear-time incremental structure from motion. In: 3DV (2013)Google Scholar
  31. 31.
    Paulin, M., Douze, M., Harchaoui, Z., Mairal, J., Perronnin, F., Schmid, C.: Local convolutional features with unsupervised training for image retrieval. In: ICCV (2015)Google Scholar
  32. 32.
    Sermanet, P., Chintala, S., LeCun, Y.: Convolutional neural networks applied to house numbers digit classification. In: ICPR (2012)Google Scholar
  33. 33.
    Strecha, C., Hansen, W., Van Gool, L., Fua, P., Thoennessen, U.: On benchmarking camera calibration and multi-view stereo for high resolution imagery. In: CVPR (2008)Google Scholar
  34. 34.
    Aanaes, H., Dahl, A.L., Pedersen, K.S.: Interesting interest points. IJCV 97, 18–35 (2012)CrossRefGoogle Scholar
  35. 35.
    Mikolajczyk, K., Schmid, C.: A performance evaluation of local descriptors. In: CVPR, pp. 257–263, June 2003Google Scholar
  36. 36.
    Alcantarilla, P.F., Bartoli, A., Davison, A.J.: KAZE features. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7577, pp. 214–227. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-33783-3_16 Google Scholar
  37. 37.
    Tola, E., Lepetit, V., Fua, P.: Daisy: an efficient dense descriptor applied to wide baseline stereo. PAMI 32(5), 815–830 (2010)CrossRefGoogle Scholar
  38. 38.
    Bellavia, F., Tegolo, D.: Improving sift-based descriptors stability to rotations. In: ICPR (2010)Google Scholar
  39. 39.
    Mikolajczyk, K., Schmid, C.: Scale and affine invariant interest point detectors. IJCV 60, 63–86 (2004)CrossRefGoogle Scholar
  40. 40.
    Fan, B., Wu, F., Hu, Z.: Aggregating gradient distributions into intensity orders: a novel local image descriptor. In: CVPR (2011)Google Scholar
  41. 41.
    Wang, Z., Fan, B., Wu, F.: Local intensity order pattern for feature description. In: ICCV (2011)Google Scholar
  42. 42.
    Zitnick, C.L.: Binary coherent edge descriptors. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 170–182. Springer, Heidelberg (2010). doi: 10.1007/978-3-642-15552-9_13 CrossRefGoogle Scholar
  43. 43.
    Leutenegger, S., Chli, M., Siegwart, R.: BRISK: binary robust invariant scalable keypoints. In: ICCV (2011)Google Scholar
  44. 44.
    Alahi, A., Ortiz, R., Vandergheynst, P.: FREAK: fast retina keypoint. In: CVPR (2012)Google Scholar
  45. 45.
    Felzenszwalb, P., Girshick, R., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part based models. PAMI 32(9), 1627–1645 (2010)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  • Kwang Moo Yi
    • 1
    Email author
  • Eduard Trulls
    • 1
  • Vincent Lepetit
    • 2
  • Pascal Fua
    • 1
  1. 1.Computer Vision LaboratoryEcole Polytechnique Fédérale de Lausanne (EPFL)LausanneSwitzerland
  2. 2.Institute for Computer Graphics and VisionGraz University of TechnologyGrazAustria

Personalised recommendations