Show Me How to Tie a Tie: Evaluation of Cross-Lingual Video Retrieval

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9822)


In this study we investigate the potential of cross-lingual video retrieval for how-to questions. How-to questions are the most frequent among wh-questions and constitute almost 1 % of the entire query stream. At the same time, how-to videos are popular on video sharing services. We analyzed a dataset of 500M+ Russian how-to questions. First, we carried out manual labelling of 1,000 queries that shows that about two thirds of all how-to question queries are potentially suitable for answers in the form of video in a language other than the language of the query. Then, we evaluated video retrieval quality for original and machine translated queries on a crowdsourcing platform. The evaluation reveals that machine translated questions yield video search quality comparable to the quality for original questions. Cross-lingual video search for how-to queries can improve recall and diversity of search results, as well as compensate the shortage of original content in emerging markets.


How-to questions Video retrieval Question answering Cross-lingual information retrieval Machine translation Query translation Evaluation 


  1. 1.
    Bojar, O., et al.: Findings of the 2015 workshop on statistical machine translation. In: WMT (2015)Google Scholar
  2. 2.
    Cao, J., Nunamaker, J.F.: Question answering on lecture videos: a multifaceted approach. In: JCDL (2004)Google Scholar
  3. 3.
    Chua, T.S., Hong, R., Li, G., Tang, J.: From text question-answering to multimedia QA on web-scale media resources. In: LS-MMRM Workshop (2009)Google Scholar
  4. 4.
    Filippova, K., Hall, K.B.: Improved video categorization from text metadata and user comments. In: SIGIR (2011)Google Scholar
  5. 5.
    Giampiccolo, D., et al.: Overview of the CLEF 2007 multilingual question answering track. In: Peters, C., et al. (eds.) CLEF 2007. LNCS, vol. 5152, pp. 200–236. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  6. 6.
    Hong, R., Wang, M., Li, G., Nie, L., Zha, Z.J., Chua, T.S.: Multimedia question answering. IEEE Trans. Multimed. 19(4), 72–78 (2012)CrossRefGoogle Scholar
  7. 7.
    Li, G., et al.: Video reference: question answering on youtube. In: MM (2009)Google Scholar
  8. 8.
    Li, G., Li, H., Ming, Z., Hong, R., Tang, S., Chua, T.S.: Question answering over community-contributed web videos. IEEE Trans. Multimed. 17(4), 46–57 (2010)CrossRefGoogle Scholar
  9. 9.
    Mitamura, T., et al.: Overview of the NTCIR-7 ACLIA tasks: advanced cross-lingual information access. In: NTCIR-7 Workshop (2008)Google Scholar
  10. 10.
    Nie, L., Wang, M., Gao, Y., et al.: Beyond text QA: multimedia answer generation by harvesting web information. IEEE Trans. Multimed. 15(2), 426–441 (2013)CrossRefGoogle Scholar
  11. 11.
    Pang, B., Kumar, R.: Search in the lost sense of query: question formulation in web search queries and its temporal changes. In: ACL, vol. 2 (2011)Google Scholar
  12. 12.
    Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: ACL (2002)Google Scholar
  13. 13.
    Parton, K., McKeown, K.R., Allan, J., Henestroza, E.: Simultaneous multilingual search for translingual information retrieval. In: CIKM (2008)Google Scholar
  14. 14.
    Pavlick, E., Post, M., Irvine, A., Kachaev, D., Callison-Burch, C.: The language demographics of amazon mechanical turk. TACL 2, 79–92 (2014)Google Scholar
  15. 15.
    Snover, M., Dorr, B., Schwartz, R., Micciulla, L., Makhoul, J.: A study of translation edit rate with targeted human annotation. In: AMTA (2006)Google Scholar
  16. 16.
    Spink, A., Ozmultu, H.C.: Characteristics of question format web queries: an exploratory study. Inf. Process. Manage. 38(4), 453–471 (2002)CrossRefMATHGoogle Scholar
  17. 17.
    Surdeanu, M., Ciaramita, M., Zaragoza, H.: Learning to rank answers to non-factoid questions from web collections. Comput. Linguist. 37(2), 351–383 (2011)CrossRefGoogle Scholar
  18. 18.
    Torrey, C., Churchill, E.F., McDonald, D.W.: Learning how: the search for craft knowledge on the internet. In: CHI (2009)Google Scholar
  19. 19.
    Völske, M., Braslavski, P., Hagen, M., Lezina, G., Stein, B.: What users ask a search engine: analyzing one billion russian question queries. In: CIKM (2015)Google Scholar
  20. 20.
    Weber, I., Ukkonen, A., Gionis, A.: Answers, not links: Extracting tips from yahoo! answers to address how-to web queries. In: WSDM (2012)Google Scholar
  21. 21.
    Wu, Y.C., Chang, C.H., Lee, Y.S.: CLVQ: cross-language video question/answering system. In: IEEE MMSE (2004)Google Scholar
  22. 22.
    Yang, H., Chaisorn, L., Zhao, Y., Neo, S.Y., Chua, T.S.: VideoQA: question answering on news video. In: MM (2003)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Pavel Braslavski
    • 1
  • Suzan Verberne
    • 2
  • Ruslan Talipov
    • 1
  1. 1.Ural Federal UniversityYekaterinburgRussia
  2. 2.Radboud UniversityNijmegenThe Netherlands

Personalised recommendations