Deep Local Descriptors with Domain Adaptation

  • Shuwen Qiu
  • Weihong Deng
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11257)


Due to the different distributions of training and testing datasets, the performance of the trained model based on the training set can rarely achieve the most optimal. Inspired by the successful application of domain adaptation in the object recognition area, we apply domain adaptation methods to CNN based local feature descriptors based on their own traits. Different from previous domain adaptation methods that focus only on the fully connected layer, we apply maximum mean discrepancy (MMD) criterion to both the fully connected layer and the convolutional layer, which makes the primary local filters of CNN adaptive to the target dataset in an unsupervised manner. Extensive experiments on Photo Tour and HPatches dataset show that domain adaption is effective to local feature descriptors, and, more importantly, the convolutional layer adaption can further improve the performance of traditional domain adaptation.


Local feature descriptor Domain adaptation Maximum Mean Discrepancy 


  1. 1.
    Gretton, A., Sejdinovic, D., Strathmann, H., et al.: Optimal kernel choice for large-scale two-sample tests. In: Advances in Neural Information Processing Systems, pp. 1205–1213 (2012)Google Scholar
  2. 2.
    Ramdas, A., Reddi, S.J., Poczos, B., et al.: On the high-dimensional power of linear-time kernel two-sample testing under mean-difference alternatives. arXiv preprint arXiv:1411.6314 (2014)
  3. 3.
    Gong, B., Grauman, K., Sha, F.: Connecting the dots with landmarks: discriminatively learning domain-invariant features for unsupervised domain adaptation. In: International Conference on Machine Learning, pp. 222–230 (2013)Google Scholar
  4. 4.
    Simo-Serra, E., Trulls, E., Ferraz, L., et al.: Discriminative learning of deep convolutional feature point descriptors. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 118–126. IEEE (2015)Google Scholar
  5. 5.
    Tzeng, E., Hoffman, J., Zhang, N., et al.: Deep domain confusion: maximizing for domain invariance. arXiv preprint arXiv:1412.3474 (2014)
  6. 6.
    Tzeng, E., Hoffman, J., Darrell, T., et al.: Simultaneous deep transfer across domains and tasks. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 4068–4076. IEEE (2015)Google Scholar
  7. 7.
    Tzeng, E., Hoffman, J., Saenko, K., Darrell, T.: Adversarial discriminative domain adaptation. arXiv preprint arXiv:1702.05464 (2017)
  8. 8.
    Zhang, K., ScholKopf, B., Muandet, K., Wang, Z.: Domain adaptation under target and conditional shift. In: International Conference on Machine Learning, pp. 819–827 (2013)Google Scholar
  9. 9.
    Duan, L., Tsang, I.W., Xu, D.: Domain transfer multiple kernel learning. IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 34(3), 465–479 (2012)CrossRefGoogle Scholar
  10. 10.
    Ghifary, M., Kleijn, W.B., Zhang, M.: Domain adaptive neural networks for object recognition. In: Pham, D.-N., Park, S.-B. (eds.) PRICAI 2014. LNCS (LNAI), vol. 8862, pp. 898–904. Springer, Cham (2014). Scholar
  11. 11.
    Borgwardt, K.M., Gretton, A., Rasch, M.J., Kriegel, H.P., Smola, A.J.: Integrating structured biological data by kernel maximum mean discrepancy. Bioinformatics 22(14), e49–e57 (2006)CrossRefGoogle Scholar
  12. 12.
    Long, M., Zhu, H., Wang, J., et al.: Deep transfer learning with joint adaptation networks. arXiv preprint arXiv:1605.06636 (2016)
  13. 13.
    Long, M., Cao, Y., Wang, J., et al.: Learning transferable features with deep adaptation networks. arXiv preprint arXiv:1502.02791 (2015)
  14. 14.
    Long, M., Zhu, H., Wang, J., Jordan, M.I.: Unsupervised domain adaptation with residual transfer networks. In: Advances in Neural Information Processing Systems (NIPS), pp. 136–144 (2016)Google Scholar
  15. 15.
    van der Maaten, L.J.P., Hinton, G.E.: Visualizing high-dimensional data using t-SNE. Mach. Learn. Res. 9, 2579–2605 (2008)zbMATHGoogle Scholar
  16. 16.
    Sugiyama, M., Nakajima, S., Kashima, H., et al.: Direct importance estimation with model selection and its application to covariate shift adaptation. In: Advances in Neural Information Processing Systems, pp. 1433–1440 (2008)Google Scholar
  17. 17.
    Russakovsky, O., et al.: ImageNet Large Scale Visual Recognition Challenge. Technical report. arXiv:1409.0575 (2014)
  18. 18.
    Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., Vaughan, J.W.: A theory of learning from different domains. Mach. Learn. 79(1–2), 151–175 (2010)MathSciNetCrossRefGoogle Scholar
  19. 19.
    Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 13451359 (2010)CrossRefGoogle Scholar
  20. 20.
    Winder, S., Hua, G., Brown, M.: Picking the best daisy. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2009, pp. 178–185. IEEE (2009)Google Scholar
  21. 21.
    Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4353–4361. IEEE (2015)Google Scholar
  22. 22.
    Balntas, V., Lenc, K., Vedaldi, A., et al.: HPatches: a benchmark and evaluation of handcrafted and learned local descriptors. In: Computer Vision and Pattern Recognition (CVPR), vol. 4, no. 5, p. 6 (2017)Google Scholar
  23. 23.
    Balntas, V., Riba, E., Ponsa, D.: Learning local feature descriptors with triplets and shallow convolutional neural networks. BMVC 1(2), 3 (2016)Google Scholar
  24. 24.
    Han, X., Leung, T., Jia, Y., Sukthankar, R., Berg, A.C.: MatchNet: unifying feature and metric learning for patch-based matching. In: Computer Vision and Pattern Recognition, pp. 3279–3286. IEEE (2015)Google Scholar
  25. 25.
    Ganin, Y., Lempitsky, V.: Unsupervised domain adaptation by backpropagation. arXiv preprint arXiv:1409.7495 (2014)
  26. 26.
    Tian, Y., Fan, B., Wu, F.: L2-Net: deep learning of discriminative patch descriptor in euclidean space. In: Conference on Computer Vision and Pattern Recognition, pp. 6128–6136. IEEE Computer Society (2017)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  1. 1.School of Information and Communication EngineeringBeijing University of Posts and TelecommunicationsBeijingChina

Personalised recommendations