Abstract
This is the final report of the OpenLiveQ-2 task at NTCIR-14. This task aimed to provide an open live test environment of Yahoo Japan Corporation’s community question-answering service (Yahoo! Chiebukuro) for question retrieval systems. The task was simply defined as follows: given a query and a set of questions with their answers, return a ranked list of questions. Submitted runs were evaluated both offline and online. In the online evaluation, we employed pairwise preference multileaving, a multileaving method that showed high efficiency over the other methods in a recent study. We describe the details of the task, data, and evaluation methods, and then report official results at NTCIR-14 OpenLiveQ-2. Furthermore, we demonstrate the effectiveness and efficiency of the proposed evaluation methodology.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Arora, P., Jones, G.: DCU at the NTCIR-14 OpenLiveQ-2 task. In: NTCIR-14 Conference (2019)
Cao, X., Cong, G., Cui, B., Jensen, C.S.: A generalized framework of exploring category information for question retrieval in community question answer archives. In: WWW, pp. 201–210 (2010)
Chen, M., Li, L., Sun, Y., Zhang, J.: Erler at the NTCIR-13 OpenLiveQ task. In: NTCIR-13 Conference (2017)
Kato, M.P., Manabe, T., Fujita, S., Nishida, A., Yamamoto, T.: Challenges of multileaved comparison in practice: lessons from NTCIR-13 OpenLiveQ task. In: CIKM, pp. 1515–1518 (2018)
Kato, M.P., Nishida, A., Manabe, T., Fujita, S., Yamamoto, T.: Overview of the NTCIR-14 OpenLiveQ-2 task. In: NTCIR-14 Conference (2019)
Kato, M.P., Yamamoto, T., Manabe, T., Nishida, A., Fujita, S.: Overview of the NTCIR-13 OpenLiveQ task. In: NTCIR-13 Conference (2017)
Manabe, T., Fujita, S., Nishida, A.: YJRS at the NTCIR-14 OpenLiveQ-2 task. In: NTCIR-14 Conference (2019)
Oosterhuis, H., de Rijke, M.: Sensitive and scalable online evaluation with theoretical guarantees. In: CIKM, pp. 77–86 (2017)
Robertson, S., Zaragoza, H., Taylor, M.: Simple BM25 extension to multiple weighted fields. In: CIKM, pp. 42–49 (2004)
Sakai, T.: Alternatives to Bpref. In: SIGIR, pp. 71–78 (2007)
Sakai, T., Ferro, N., Soboroff, I., Zeng, Z., Xiao, P., Maistro, M.: Overview of the NTCIR-14 centre task. In: NTCIR-14 Conference (2019)
Sakai, T., Song, R.: Evaluating diversified search results using per-intent graded relevance. In: SIGIR, pp. 1043–1052 (2011)
Sato, T., Nagase, Y., Uraji, M.: OKSAT at NTCIR-14 OpenLiveQ-2 task -reorder questions by using white and black words. In: NTCIR-14 Conference (2019)
Tanioka, H.: AITOK at the NTCIR-14 OpenLiveQ-2 task. In: NTCIR-14 Conference (2019)
Wang, K., Ming, Z., Chua, T.S.: A syntactic tree matching approach to finding similar questions in community-based QA services. In: SIGIR, pp. 187–194 (2009)
Zhou, G., Liu, Y., Liu, F., Zeng, D., Zhao, J.: Improving question retrieval in community question answering using world knowledge. In: IJCAI, pp. 2239–2245 (2013)
Acknowledgments
We would like to thank the OpenLiveQ-2 participants for their contributions to the OpenLiveQ-2 task.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Kato, M.P., Nishida, A., Manabe, T., Fujita, S., Yamamoto, T. (2019). Final Report of the NTCIR-14 OpenLiveQ-2 Task. In: Kato, M., Liu, Y., Kando, N., Clarke, C. (eds) NII Testbeds and Community for Information Access Research. NTCIR 2019. Lecture Notes in Computer Science(), vol 11966. Springer, Cham. https://doi.org/10.1007/978-3-030-36805-0_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-36805-0_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-36804-3
Online ISBN: 978-3-030-36805-0
eBook Packages: Computer ScienceComputer Science (R0)