International Journal of Computer Vision

, Volume 119, Issue 1, pp 46–59

Large Scale Retrieval and Generation of Image Descriptions

  • Vicente Ordonez
  • Xufeng Han
  • Polina Kuznetsova
  • Girish Kulkarni
  • Margaret Mitchell
  • Kota Yamaguchi
  • Karl Stratos
  • Amit Goyal
  • Jesse Dodge
  • Alyssa Mensch
  • Hal DauméIII
  • Alexander C. Berg
  • Yejin Choi
  • Tamara L. Berg
Article

DOI: 10.1007/s11263-015-0840-y

Cite this article as:
Ordonez, V., Han, X., Kuznetsova, P. et al. Int J Comput Vis (2016) 119: 46. doi:10.1007/s11263-015-0840-y
  • 936 Downloads

Abstract

What is the story of an image? What is the relationship between pictures, language, and information we can extract using state of the art computational recognition systems? In an attempt to address both of these questions, we explore methods for retrieving and generating natural language descriptions for images. Ideally, we would like our generated textual descriptions (captions) to both sound like a person wrote them, and also remain true to the image content. To do this we develop data-driven approaches for image description generation, using retrieval-based techniques to gather either: (a) whole captions associated with a visually similar image, or (b) relevant bits of text (phrases) from a large collection of image + description pairs. In the case of (b), we develop optimization algorithms to merge the retrieved phrases into valid natural language sentences. The end result is two simple, but effective, methods for harnessing the power of big data to produce image captions that are altogether more general, relevant, and human-like than previous attempts.

Keywords

Retrieval Image description Data driven Big data Natural language processing 

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Vicente Ordonez
    • 1
  • Xufeng Han
    • 1
  • Polina Kuznetsova
    • 2
  • Girish Kulkarni
    • 2
  • Margaret Mitchell
    • 3
  • Kota Yamaguchi
    • 4
  • Karl Stratos
    • 5
  • Amit Goyal
    • 6
  • Jesse Dodge
    • 7
  • Alyssa Mensch
    • 8
  • Hal DauméIII
    • 9
  • Alexander C. Berg
    • 1
  • Yejin Choi
    • 10
  • Tamara L. Berg
    • 1
  1. 1.University of North CarolinaChapel HillUSA
  2. 2.Stony Brook UniversityStony BrookUSA
  3. 3.Microsoft ResearchRedmondUSA
  4. 4.Tohoku UniversitySendaiJapan
  5. 5.Columbia UniversityNew YorkUSA
  6. 6.Yahoo! LabsSunnyvaleUSA
  7. 7.Carnegie Mellon UniversityPittsburghUSA
  8. 8.University of PennsylvaniaPhiladelphiaUSA
  9. 9.University of MarylandCollege ParkUSA
  10. 10.University of WashingtonSeattleUSA

Personalised recommendations