Abstract
Search tasks provide a medium for the evaluation of system performance and the underlying analytical aspects of IR systems. Researchers have recently developed new interfaces or mechanisms to support vague information needs and struggling search. However, little attention has been paid to the generation of a unified task set for evaluation and comparison of search engine improvements for struggling search. Generation of such tasks is inherently difficult, as each task is supposed to trigger struggling and exploring user behavior rather than simple search behavior. Moreover, the everchanging landscape of information needs would render old task sets less ideal if not unusable for system evaluation. In this paper, we propose a task generation method and develop a crowd-powered platform called TaskGenie to generate struggling search tasks online. Our experiments and analysis show that the generated tasks are qualified to emulate struggling search behaviors consisting of ‘repeated similar queries’ and ‘quick-back clicks’, etc. – tasks of diverse topics, high quality and difficulty can be created using this framework. For the benefit of the community, we publicly released the platform, a task set containing 80 topically diverse struggling search tasks generated and examined in this work, and the corresponding anonymized user behavior logs.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
- 2.
Wikinews: https://en.wikinews.org/; wikivoyage: https://www.wikivoyage.org/.
- 3.
Anonymized URL– https://github.com/sst20190816/WISE2020.
- 4.
- 5.
- 6.
- 7.
References
Ageev, M., Guo, Q., Lagun, D., Agichtein, E.: Find it if you can: a game for modeling different types of web search success using interaction data. In: SIGIR, pp. 345–354 (2011)
Aula, A., Khan, R.M., Guan, Z.: How does search behavior change as search becomes more difficult? In: SIGCHI, pp. 35–44 (2010)
Baykan, E., Henzinger, M., Marian, L., Weber, I.: Purely URL-based topic classification. In: WWW, pp. 1109–1110 (2009)
Bhagat, R., Hovy, E.: What is a paraphrase? Comput. Linguist. 39(3), 463–472 (2013)
Braarud, P.Ø., Kirwan, B.: Task complexity: what challenges the crew and how do they cope. In: Skjerve, A., Bye, A. (eds.) Simulator-based Human Factors Studies Across 25 Years, pp. 233–251. Springer, London (2010). https://doi.org/10.1007/978-0-85729-003-8_15
Capra, R., Arguello, J., O’Brien, H., Li, Y., Choi, B.: The effects of manipulating task determinability on search behaviors and outcomes. In: SIGIR, pp. 445–454 (2018)
De Beaugrande, R.A., Dressler, W.U.: Introduction to Text Linguistics, vol. 1. Longman, London (1981)
Gadiraju, U., et al.: Crowdsourcing versus the laboratory: towards human-centered experiments using the crowd. In: Archambault, D., Purchase, H., Hoßfeld, T. (eds.) Evaluation in the Crowd. Crowdsourcing and Human-Centered Experiments. LNCS, vol. 10264, pp. 6–26. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66435-4_2
Gadiraju, U., Yu, R., Dietze, S., Holtz, P.: Analyzing knowledge gain of users in informational search sessions on the web. In: CHIIR 2018 (2018)
Hassan, A.: A semi-supervised approach to modeling web search satisfaction. In: SIGIR, pp. 275–284 (2012)
Hassan, A., Jones, R., Klinkner, K.L.: Beyond DCG: user behavior as a predictor of a successful search. In: WSDM, pp. 221–230 (2010)
Hassan, A., White, R.W., Dumais, S.T., Wang, Y.M.: Struggling or exploring?: disambiguating long search sessions. In: WSDM, pp. 53–62 (2014)
Hassan Awadallah, A., White, R.W., Pantel, P., Dumais, S.T., Wang, Y.M.: Supporting complex search tasks. In: CIKM, pp. 829–838 (2014)
Kellar, M., Watters, C., Shepherd, M.: A goal-based classification of web information tasks. ASIST 43(1), 1–22 (2006)
Kenter, T., De Rijke, M.: Short text similarity with word embeddings. In: CIKM, pp. 1411–1420 (2015)
Kim, Y., Hassan, A., White, R.W., Zitouni, I.: Modeling dwell time to predict click-level satisfaction. In: WSDM, pp. 193–202 (2014)
Kolln, M., Funk, R.: Understanding English Grammar. Longman, London (1982)
Liu, C., Liu, J., Cole, M., Belkin, N.J., Zhang, X.: Task difficulty and domain knowledge effects on information search behaviors. ASIS&T 49(1), 1–10 (2012)
Liu, J., Kim, C.S., Creel, C.: Why do users feel search task difficult? In: The 76th ASIS&T. American Society for Information Science (2013)
Mai, J.E.: Looking for Information: A Survey of Research on Information Seeking, Needs, and Behavior. Emerald Group Publishing, Bingley (2016)
Mitra, B.: Exploring session context using distributed representations of queries and reformulations. In: SIGIR, pp. 3–12 (2015)
Odijk, D., White, R.W., Hassan Awadallah, A., Dumais, S.T.: Struggling and success in web search. In: CIKM, pp. 1551–1560 (2015)
Pirolli, P., Card, S.: The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In: IA, vol. 5, pp. 2–4 (2005)
Robinson, P.: Task complexity, task difficulty, and task production: exploring interactions in a componential framework. Appl. Linguist. 22(1), 27–57 (2001)
Singer, G., Norbisrath, U., Lewandowski, D.: Ordinary search engine users carrying out complex search tasks. J. Inf. Sci. 39(3), 346–358 (2013)
Singer, P., et al.: Why we read Wikipedia. In: WWW, pp. 1591–1600 (2017)
White, R.W.: Interactions with Search Systems. Cambridge University Press, Cambridge (2016)
White, R.W., Roth, R.A.: Exploratory search: beyond the query-response paradigm. In: Synthesis Lectures on Information Concepts, Retrieval, and Services, vol. 1, no. 1, pp. 1–98 (2009)
Wilson, M.L., Kules, B., Shneiderman, B., et al.: From keyword search to exploration: designing future search interfaces for the web. Found. Trends® Web Sci. 2(1), 1–97 (2010)
Xu, L., Zhou, X.: Generating tasks for study of struggling search. In: Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, pp. 267–270 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Xu, L., Zhou, X., Gadiraju, U. (2020). TaskGenie: Crowd-Powered Task Generation for Struggling Search. In: Huang, Z., Beek, W., Wang, H., Zhou, R., Zhang, Y. (eds) Web Information Systems Engineering – WISE 2020. WISE 2020. Lecture Notes in Computer Science(), vol 12343. Springer, Cham. https://doi.org/10.1007/978-3-030-62008-0_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-62008-0_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-62007-3
Online ISBN: 978-3-030-62008-0
eBook Packages: Computer ScienceComputer Science (R0)