In interactive information retrieval, researchers consider the user behaviour towards systems and search tasks in order to adapt search results and to improve the search experience of users. Analysing the users’ past interactions with the system is one typical approach. In this paper, we analyse the user behaviour in retrieval sessions towards Marcia Bates’ search stratagems such as “Footnote Chasing”, “Citation Searching”, “Keyword Searching”, “Author Searching” and “Journal Run” in a real-life academic search engine. In fact, search stratagems represent high-level search behaviour as the users go beyond simple execution of queries and investigate more of the system functionalities. We performed analyses of these five search stratagems using two datasets extracted from the social sciences search engine sowiport. A specific focus was the detection of the search phase and frequency of the usage of these stratagems. In addition, we explored the impact of these stratagems on the whole search process performance. We addressed mainly the usage patterns’ observation of the stratagems, their impact on the conduct of retrieval sessions and explored whether they are used similarly in both datasets. From the observation and metrics proposed, we can conclude that the utilisation of search stratagems in real retrieval sessions leads to an improvement of the precision in terms of positive interactions. For both datasets (SUSS 14–15 and SUSS 16–17), the user behaviour was similar as all stratagems appear most frequently in the middle of a session. However, the difference is that “Footnote Chasing”, “Citation Searching” and “Journal Run” appear mostly at the end of a session while Keyword and Author Searching appear typically at the beginning. Thus, we can conclude from the log analysis that the improvement of search functionalities including personalisation and/or recommendation could be achieved by considering references, citations, and journals in the ranking process.
KeywordsWhole-session evaluation Information behaviour Retrieval session log Cited reference searching Stratagem search Academic search
This work was funded by Deutsche Forschungsgemeinschaft (DFG), Grant No. MA 3964/5-1; the AMUR project at GESIS together with the working group of Norbert Fuhr. The AMUR project aims at improving the support of interactive retrieval sessions following two major goals: improving user guidance and system tuning. We thank Julia Achenbach for her proof reading of the final version of this paper.
- Akbar, M., Shaffer, C. A., & Fox, E. A. (2012). Deduced social networks for an educational digital library. In Proceedings of JCDL’12 (pp. 43–46). ACM. https://doi.org/10.1145/2232817.2232828.
- Belkin, N., Cool, C., Kelly, D., Lin, S. J., Park, S., Perez-Carballo, J., et al. (2001). Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval. Information Processing and Management, 37(3), 403–434. https://doi.org/10.1016/S0306-4573(00)00055-8.CrossRefzbMATHGoogle Scholar
- Borlund, P. (2016). Interactive information retrieval: An evaluation perspective. In Proceedings of the 2016 ACM on conference on human information interaction and retrieval, CHIIR’16 (pp. 151–151). New York, NY: ACM. https://doi.org/10.1145/2854946.2870648.
- Carevic, Z., & Mayr, P. (2015). Extending search facilities via bibliometric-enhanced stratagems. In Proceedings of the BIR workshop at ECIR 2015, Vienna, Austria (pp. 40–46). http://ceur-ws.org/Vol-1344/paper5.pdf.
- Carevic, Z., & Mayr, P. (2016). Survey on high-level search activities based on the stratagem level in digital libraries (pp. 54–66). Cham: Springer. https://doi.org/10.1007/978-3-319-43997-6_5.
- Carevic, Z., Schüller, S., Mayr, P., & Fuhr, N. (2018). Contextualised browsing in a digital library’s living lab. In Proceedings of the 18th ACM/IEEE on joint conference on digital libraries, Fort Worth, Texas, USA (pp. 89–98). New York, NY: ACM. https://doi.org/10.1145/3197026.3197054. https://arxiv.org/abs/1804.06426.
- Cohen, J. (1988). Statistical power analyses for the social sciences. NJ, Lawrence Erlbauni Associates: Hillsdale.Google Scholar
- Dungs, S., & Fuhr, N. (2017). Advanced hidden Markov models for recognizing search phases. In Proceedings of the ACM SIGIR international conference on theory of information retrieval, ICTIR’17 (pp. 257–260). New York, NY: ACM. https://doi.org/10.1145/3121050.3121090.
- Hienert, D., & Mutschke, P. (2016). A usefulness-based approach for measuring the local and global effect of IIR services. In Proceedings of CHIIR’16 (pp. 153–162). ACM . https://doi.org/10.1145/2854946.2854962.
- Kacem, A., & Mayr, P. (2017). Analysis of footnote chasing and citation searching in an academic search engine. In Proceedings of the 2nd joint workshop on bibliometric-enhanced information retrieval and natural language processing for digital libraries (BIRNDL 2017) co-located with the 40th international ACM SIGIR conference on research and development in information, Tokyo, Japan (pp. 91–100). http://ceur-ws.org/Vol-1888/paper8.pdf.
- Kacem, A., & Mayr, P. (2018). Users are not influenced by high impact and core journals while searching. In Proceedings of the 7th international workshop on bibliometric-enhanced information retrieval (BIR 2018) co-located with the 40th European conference on information retrieval (ECIR 2018), Grenoble, France, March 26, 2018 (pp. 63–75).Google Scholar
- Kelly, D., et al. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval®, 3(1–2), 1–224.Google Scholar
- Kriewel, S., & Fuhr, N. (2010). An evaluation of an adaptive search suggestion system. In 32nd European conference on information retrieval research (ECIR 2010) (pp. 544–555). Springer.Google Scholar
- Liu, C., Belkin, N. J., & Cole, M. J. (2012). Personalization of search results using interaction behaviors in search sessions. In Proceedings of SIGIR’12 (pp. 205–214). ACM. https://doi.org/10.1145/2348283.2348314.
- Mahoui, M., & Cunningham, S. J. (2001). Search behavior in a research-oriented digital library. In Proceedings of ECDL 2001 (pp. 13–24). Springer.Google Scholar
- Mayr, P. (2016). How do practitioners, PhD students and postdocs in the social sciences assess topic-specific recommendations? In Proceedings of the joint workshop on bibliometric-enhanced information retrieval and natural language processing for digital libraries (BIRNDL2016) (pp. 84–92). Newark, NJ: CEUR-WS.org. http://ceur-ws.org/Vol-1610/paper10.pdf.
- Mayr, P. (2016). Sowiport user search sessions data set (SUSS). https://doi.org/10.7802/1380.
- Rohini, U., & Ambati, V. (2005). A collaborative filtering based re-ranking strategy for search in digital libraries (pp. 194–203). Berlin: Springer.Google Scholar
- Tran, V., Maxwell, D., Fuhr, N., & Azzopardi, L. (2017). Personalised search time prediction using Markov chains. In Proceedings of the ACM SIGIR international conference on theory of information retrieval, ICTIR’17 (pp. 237–240). New York, NY: ACM. https://doi.org/10.1145/3121050.3121085.
- Zhuang, M., Toms, E. G., & Demartini, G. (2016). The relationship between user perception and user behaviour in interactive information retrieval evaluation. In N. Ferro, F. Crestani, M. F. Moens, J. Mothe, F. Silvestri, G. M. Di Nunzio, C. Hauff, & G. Silvello (Eds.), Advances in information retrieval (pp. 293–305). Cham: Springer.CrossRefGoogle Scholar