Advertisement

Final Report of the NTCIR-14 OpenLiveQ-2 Task

Conference paper
First Online:
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11966)

Abstract

This is the final report of the OpenLiveQ-2 task at NTCIR-14. This task aimed to provide an open live test environment of Yahoo Japan Corporation’s community question-answering service (Yahoo! Chiebukuro) for question retrieval systems. The task was simply defined as follows: given a query and a set of questions with their answers, return a ranked list of questions. Submitted runs were evaluated both offline and online. In the online evaluation, we employed pairwise preference multileaving, a multileaving method that showed high efficiency over the other methods in a recent study. We describe the details of the task, data, and evaluation methods, and then report official results at NTCIR-14 OpenLiveQ-2. Furthermore, we demonstrate the effectiveness and efficiency of the proposed evaluation methodology.

Keywords

Online evaluation Interleaving Community question answering 
This is a preview of subscription content, log in to check access.

Notes

Acknowledgments

We would like to thank the OpenLiveQ-2 participants for their contributions to the OpenLiveQ-2 task.

References

  1. 1.
    Arora, P., Jones, G.: DCU at the NTCIR-14 OpenLiveQ-2 task. In: NTCIR-14 Conference (2019)Google Scholar
  2. 2.
    Cao, X., Cong, G., Cui, B., Jensen, C.S.: A generalized framework of exploring category information for question retrieval in community question answer archives. In: WWW, pp. 201–210 (2010)Google Scholar
  3. 3.
    Chen, M., Li, L., Sun, Y., Zhang, J.: Erler at the NTCIR-13 OpenLiveQ task. In: NTCIR-13 Conference (2017)Google Scholar
  4. 4.
    Kato, M.P., Manabe, T., Fujita, S., Nishida, A., Yamamoto, T.: Challenges of multileaved comparison in practice: lessons from NTCIR-13 OpenLiveQ task. In: CIKM, pp. 1515–1518 (2018)Google Scholar
  5. 5.
    Kato, M.P., Nishida, A., Manabe, T., Fujita, S., Yamamoto, T.: Overview of the NTCIR-14 OpenLiveQ-2 task. In: NTCIR-14 Conference (2019)Google Scholar
  6. 6.
    Kato, M.P., Yamamoto, T., Manabe, T., Nishida, A., Fujita, S.: Overview of the NTCIR-13 OpenLiveQ task. In: NTCIR-13 Conference (2017)Google Scholar
  7. 7.
    Manabe, T., Fujita, S., Nishida, A.: YJRS at the NTCIR-14 OpenLiveQ-2 task. In: NTCIR-14 Conference (2019)Google Scholar
  8. 8.
    Oosterhuis, H., de Rijke, M.: Sensitive and scalable online evaluation with theoretical guarantees. In: CIKM, pp. 77–86 (2017)Google Scholar
  9. 9.
    Robertson, S., Zaragoza, H., Taylor, M.: Simple BM25 extension to multiple weighted fields. In: CIKM, pp. 42–49 (2004)Google Scholar
  10. 10.
    Sakai, T.: Alternatives to Bpref. In: SIGIR, pp. 71–78 (2007)Google Scholar
  11. 11.
    Sakai, T., Ferro, N., Soboroff, I., Zeng, Z., Xiao, P., Maistro, M.: Overview of the NTCIR-14 centre task. In: NTCIR-14 Conference (2019)Google Scholar
  12. 12.
    Sakai, T., Song, R.: Evaluating diversified search results using per-intent graded relevance. In: SIGIR, pp. 1043–1052 (2011)Google Scholar
  13. 13.
    Sato, T., Nagase, Y., Uraji, M.: OKSAT at NTCIR-14 OpenLiveQ-2 task -reorder questions by using white and black words. In: NTCIR-14 Conference (2019)Google Scholar
  14. 14.
    Tanioka, H.: AITOK at the NTCIR-14 OpenLiveQ-2 task. In: NTCIR-14 Conference (2019)Google Scholar
  15. 15.
    Wang, K., Ming, Z., Chua, T.S.: A syntactic tree matching approach to finding similar questions in community-based QA services. In: SIGIR, pp. 187–194 (2009)Google Scholar
  16. 16.
    Zhou, G., Liu, Y., Liu, F., Zeng, D., Zhao, J.: Improving question retrieval in community question answering using world knowledge. In: IJCAI, pp. 2239–2245 (2013)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.University of TsukubaTsukubaJapan
  2. 2.Yahoo Japan CorporationTokyoJapan
  3. 3.University of HyogoKobeJapan

Personalised recommendations

Cite paper

Buy options