Predicting Search Task Difficulty
Search task difficulty refers to a user’s assessment about the amount of effort required to complete a search task. Our goal in this work is to learn predictive models of search task difficulty. We evaluate features derived from the user’s interaction with the search engine as well as features derived from the user’s level of interest in the task and level of prior knowledge in the task domain. In addition to user-interaction features used in prior work, we evaluate features generated from scroll and mouse-movement events on the SERP. In some situations, we may prefer a system that can predict search task difficulty early in the search session. To this end, we evaluate features in terms of whole-session evidence and first-round evidence, which excludes all interactions starting with the second query. Our results found that the most predictive features were different for whole-session vs. first-round prediction, that mouseover features were effective for first-round prediction, and that level of interest and prior knowledge features did not improve performance.
KeywordsSearch Task Average Precision Search Behavior Mouse Movement Search Session
Unable to display preview. Download preview PDF.
- 1.Anderson, L.W., Krathwohl, D.R.: A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives (2001)Google Scholar
- 2.Aula, A., Khan, R.M., Guan, Z.: How does search behavior change as search becomes more difficult? In: CHI, pp. 35–44 (2010)Google Scholar
- 5.Campbell, D.J.: Task complexity: A review and analysis. The Academy of Management Review 13(1), 40–52 (1988)Google Scholar
- 7.Feild, H.A., Allan, J., Jones, R.: Predicting searcher frustration. In: SIGIR, pp. 34–41 (2010)Google Scholar
- 9.Kim, J.: Task difficulty as a predictor and indicator of web searching interaction. In: CHI, pp. 959–964 (2006)Google Scholar
- 11.Liu, J., Cole, M.J., Liu, C., Bierig, R., Gwizdka, J., Belkin, N.J., Zhang, J., Zhang, X.: Search behaviors in different task types. In: JCDL, pp. 69–78 (2010)Google Scholar
- 12.Liu, J., Gwizdka, J., Liu, C., Belkin, N.J.: Predicting task difficulty for different task types. ASIS&T, 16:1–16:10 (2010)Google Scholar
- 13.Liu, J., Liu, C., Cole, M., Belkin, N.J., Zhang, X.: Exploring and predicting search task difficulty. In: CIKM, pp. 1313–1322 (2012)Google Scholar
- 14.Liu, J., Liu, C., Gwizdka, J., Belkin, N.J.: Can search systems detect users‘ task difficulty?: some behavioral signals. In: SIGIR, pp. 845–846 (2010)Google Scholar
- 15.Smucker, M.D., Allan, J., Carterette, B.: A comparison of statistical significance tests for information retrieval evaluation. In: CIKM, pp. 623–632 (2007)Google Scholar
- 16.White, R.W., Dumais, S.T.: Characterizing and predicting search engine switching behavior. In: CIKM, pp. 87–96 (2009)Google Scholar
- 18.Wu, W.-C., Kelly, D., Edwards, A., Arguello, J.: Grannies, tanning beds, tattoos and nascar: evaluation of search tasks with varying levels of cognitive complexity. In: IIIX, pp. 254–257 (2012)Google Scholar