DeepJDOT: Deep Joint Distribution Optimal Transport for Unsupervised Domain Adaptation

  • Bharath Bhushan DamodaranEmail author
  • Benjamin Kellenberger
  • Rémi Flamary
  • Devis Tuia
  • Nicolas Courty
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11208)


In computer vision, one is often confronted with problems of domain shifts, which occur when one applies a classifier trained on a source dataset to target data sharing similar characteristics (e.g. same classes), but also different latent data structures (e.g. different acquisition conditions). In such a situation, the model will perform poorly on the new data, since the classifier is specialized to recognize visual cues specific to the source domain. In this work we explore a solution, named DeepJDOT, to tackle this problem: through a measure of discrepancy on joint deep representations/labels based on optimal transport, we not only learn new data representations aligned between the source and target domain, but also simultaneously preserve the discriminative information used by the classifier. We applied DeepJDOT to a series of visual recognition tasks, where it compares favorably against state-of-the-art deep domain adaptation methods.


Deep domain adaptation Optimal transport 



This work benefited from the support of Region Bretagne grant and OATMIL ANR-17-CE23-0012 project of the French National Research Agency (ANR). The constructive comments and suggestions of anonymous reviewers are gratefully acknowledged.


  1. 1.
    Patel, V.M., Gopalan, R., Li, R., Chellappa, R.: Visual domain adaptation: a survey of recent advances. IEEE SPM 32(3), 53–69 (2015)CrossRefGoogle Scholar
  2. 2.
    Saenko, K., Kulis, B., Fritz, M., Darrell, T.: Adapting visual category models to new domains. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6314, pp. 213–226. Springer, Heidelberg (2010). Scholar
  3. 3.
    Gopalan, R., Li, R., Chellappa, R.: Domain adaptation for object recognition: an unsupervised approach. In: ICCV, pp. 999–1006 (2011)Google Scholar
  4. 4.
    Courty, N., Flamary, R., Tuia, D., Rakotomamonjy, A.: Optimal transport for domain adaptation. IEEE TPAMI 39(9), 1853–1865 (2017)CrossRefGoogle Scholar
  5. 5.
    Courty, N., Flamary, R., Habrard, A., Rakotomamonjy, A.: Joint distribution optimal transportation for domain adaptation. In: NIPS (2017)Google Scholar
  6. 6.
    Sun, B., Saenko, K.: Deep CORAL: correlation alignment for deep domain adaptation. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 443–450. Springer, Cham (2016). Scholar
  7. 7.
    Luo, Z., Zou, Y., Hoffman, J., Fei-Fei, L.: Label efficient learning of transferable representations across domains and tasks. In: NIPS (2017)Google Scholar
  8. 8.
    Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17(1), 2030–2096 (2016)MathSciNetzbMATHGoogle Scholar
  9. 9.
    Liu, M.Y., Tuzel, O.: Coupled generative adversarial networks. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) NIPS, pp. 469–477 (2016)Google Scholar
  10. 10.
    Ben-David, S., Blitzer, J., Crammer, K., Pereira, F.: Analysis of representations for domain adaptation. In: NIPS, pp. 137–144 (2007)Google Scholar
  11. 11.
    Jhuo, I.H., Liu, D., Lee, D.T., Chang, S.F.: Robust visual domain adaptation with low-rank reconstruction. In: CVPR, pp. 2168–2175 (2012)Google Scholar
  12. 12.
    Hoffman, J., Rodner, E., Donahue, J., Saenko, K., Darrell, T.: Efficient learning of domain-invariant image representations. In: ICLR (2013)Google Scholar
  13. 13.
    Aljundi, R., Tuytelaars, T.: Lightweight unsupervised domain adaptation by convolutional filter reconstruction. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9915, pp. 508–515. Springer, Cham (2016). Scholar
  14. 14.
    Long, M., Cao, Y., Wang, J., Jordan, M.I.: Learning transferable features with deep adaptation networks. In: ICML, pp. 97–105 (2015)Google Scholar
  15. 15.
    Long, M., Wang, J., Jordan, M.I.: Unsupervised domain adaptation with residual transfer networks. In: NIPS (2016)Google Scholar
  16. 16.
    Haeusser, P., Frerix, T., Mordvintsev, A., Cremers, D.: Associative domain adaptation. In: ICCV (2017)Google Scholar
  17. 17.
    Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Simultaneous deep transfer across domains and tasks. In: ICCV (2015)Google Scholar
  18. 18.
    Liu, M.Y., Breuel, T., Kautz, J.: Unsupervised image-to-image translation networks. In: Guyon, I., et al. (eds.) NIPS, pp. 700–708 (2017)Google Scholar
  19. 19.
    Sankaranarayanan, S., Balaji, Y., Castillo, C.D., Chellappa, R.: Generate to adapt: Aligning domains using generative adversarial networks. CoRR abs/1704.01705 (2017)Google Scholar
  20. 20.
    Murez, Z., Kolouri, S., Kriegman, D., Ramamoorthi, R., Kim, K.: Image to Image Translation for Domain Adaptation. ArXiv e-prints, December 2017Google Scholar
  21. 21.
    Tzeng, E., Hoffman, J., Darrell, T., Saenko, K.: Adversarial discriminative domain adaptation. In: CVPR (2017)Google Scholar
  22. 22.
    Monge, G.: Mémoire sur la théorie des déblais et des remblais. De l’Imprimerie Royale (1781)Google Scholar
  23. 23.
    Kantorovich, L.: On the translocation of masses. C.R. (Doklady) Acad. Sci. URSS (N.S.) 37, 199–201 (1942)Google Scholar
  24. 24.
    Villani, C.: Optimal Transport: Old and New, Grundlehren der mathematischen Wissenschaften. Springer, Heidelberg (2009). Scholar
  25. 25.
    Courty, N., Flamary, R., Tuia, D.: Domain adaptation with regularized optimal transport. In: ECML (2014)Google Scholar
  26. 26.
    Perrot, M., Courty, N., Flamary, R., Habrard, A.: Mapping estimation for discrete optimal transport. In: NIPS, pp. 4197–4205 (2016)Google Scholar
  27. 27.
    Redko, I., Habrard, A., Sebban, M.: Theoretical analysis of domain adaptation with optimal transport. In: ECML/PKDD, pp. 737–753 (2017)Google Scholar
  28. 28.
    Shen., J., Qu, Y., Zhang, W., Yu, Y.: Wasserstein distance guided representation learning for domain adaptation. In: AAAI (2018)Google Scholar
  29. 29.
    Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: ICML, pp. 214–223 (2017)Google Scholar
  30. 30.
    Cuturi, M.: Sinkhorn distances: lightspeed computation of optimal transportation. In: NIPS, pp. 2292–2300 (2013)Google Scholar
  31. 31.
    Genevay, A., Cuturi, M., Peyré, G., Bach, F.: Stochastic optimization for large-scale optimal transport. In: NIPS, pp. 3432–3440 (2016)Google Scholar
  32. 32.
    Seguy, V., Damodaran, B.B., Flamary, R., Courty, N., Rolet, A., Blondel, M.: Large-scale optimal transport and mapping estimation. In: ICLR (2018)Google Scholar
  33. 33.
    Shmelkov, K., Schmid, C., Alahari, K.: Incremental learning of object detectors without catastrophic forgetting. In: ICCV, Venice, Italy (2017)Google Scholar
  34. 34.
    Li, Z., Hoiem, D.: Learning without forgetting. IEEE TPAMI (in press)Google Scholar
  35. 35.
    Genevay, A., Peyré, G., Cuturi, M.: Sinkhorn-autodiff: tractable wasserstein learning of generative models. arXiv preprint arXiv:1706.00292 (2017)
  36. 36.
    Lecun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)CrossRefGoogle Scholar
  37. 37.
    Hull, J.J.: A database for handwritten text recognition research. IEEE TPAMI 16(5), 550–554 (1994)CrossRefGoogle Scholar
  38. 38.
    Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning. In: NIPS worksophs (2011)Google Scholar
  39. 39.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE TPAMI 33(5), 898–916 (2011)CrossRefGoogle Scholar
  40. 40.
    Ghifary, M., Kleijn, W.B., Zhang, M., Balduzzi, D., Li, W.: Deep reconstruction-classification networks for unsupervised domain adaptation. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 597–613. Springer, Cham (2016). Scholar
  41. 41.
    Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., Erhan, D.: Domain separation networks. In: NIPS (2016) 343–351Google Scholar
  42. 42.
    French, G., Mackiewicz, M., Fisher, M.: Self-ensembling for visual domain adaptation. In: International Conference on Learning Representations (2018)Google Scholar
  43. 43.
    Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)Google Scholar
  44. 44.
    Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  45. 45.
    Sun, B., Feng, J., Saenko, K.: Return of frustratingly easy domain adaptation. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. AAAI 2016, AAAI Press, pp. 2058–2065 (2016)Google Scholar
  46. 46.
    Long, M., Wang, J., Ding, G., Sun, J., Yu, P.S.: Transfer feature learning with joint distribution adaptation. In: 2013 IEEE International Conference on Computer Vision, pp. 2200–2207, December 2013Google Scholar
  47. 47.
    Long, M., Cao, Y., Wang, J., Jordan, M.: Learning transferable features with deep adaptation networks. In: Bach, F., Blei, D., (eds.) Proceedings of the 32nd International Conference on Machine Learning, Volume 37 of Proceedings of Machine Learning Research., Lille, France, PMLR, pp. 97–105, 07–09 July 2015Google Scholar
  48. 48.
    Peng, X., Usman, B., Kaushik, N., Hoffman, J., Wang, D., Saenko, K.: Visda: the visual domain adaptation challenge (2017)Google Scholar
  49. 49.
    Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). Scholar
  50. 50.
    Real, E., Shlens, J., Mazzocchi, S., Pan, X., Vanhoucke, V.: Youtube-boundingboxes: a large high-precision human-annotated data set for object detection in video. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7464–7473. IEEE (2017)Google Scholar
  51. 51.
    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Bharath Bhushan Damodaran
    • 1
    Email author
  • Benjamin Kellenberger
    • 2
  • Rémi Flamary
    • 3
  • Devis Tuia
    • 2
  • Nicolas Courty
    • 1
  1. 1.Université de Bretagne Sud, IRISA, UMR 6074, CNRSLorientFrance
  2. 2.Wageningen UniversityWageningenThe Netherlands
  3. 3.Université Côte d’Azur, OCA, UMR 7293, CNRS, Laboratoire LagrangeNiceFrance

Personalised recommendations