Abstract
We conducted a lab-based experiment to investigate relationship between multiple criteria used in information relevance judgments and eye fixation behavior on search engine results. We collected eye-tracking data and conducted gaze-cued retrospective think-aloud (RTA). Data from RTA was coded with criteria used by participants in judging search results as relevant. The criteria were analyzed in relation to search engine result page (SERP) sequence and result rank on SERPs. The results of our study aligned with previous research, showing the effect of result rank on SERPs. Our results newly showed that specific source and topicality were the two most often used criteria for relevance judgments. Specific source was the most often used criteria initially but was then surpassed by topicality on subsequent SERPs and on lower result ranks. On first SERPs, fixation duration was significantly longer on results judged on topicality than on specific source. Pupils dilated significantly on the top ranked result on most SERP pages.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Balatsoukas, P., & Ruthven, I. (2012). An eye-tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine. Journal of the American Society for Information Science and Technology, 63(9), 1728–1746. https://doi.org/10.1002/asi.22707.
Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for information Science and Technology, 54(10), 913–925. https://doi.org/10.1002/asi.10286.
da Costa Pereira, C., Dragoni, M., & Pasi, G. (2012). Multidimensional relevance: Prioritized aggregation in a personalized Information Retrieval setting. Information processing & management, 48(2), 340–357. https://doi.org/10.1016/j.ipm.2011.07.001.
Fogg, B. J., & Tseng, H. (1999, May). The elements of computer credibility. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems (pp. 80–87). New York, NY, USA: ACM.https://doi.org/10.1145/302979.303001.
Gerjets, P., et al. (2011). Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking-aloud protocols and eye-tracking data. Learning and Instruction, 21(2), 220–231. https://doi.org/10.1016/j.learninstruc.2010.02.005.
Greisdorf, H. (2003). Relevance thresholds: A multi-stage predictive model of how users evaluate information. Information Processing & Management, 39(3), 403–423. https://doi.org/10.1016/S0306-4573(02)00032-8.
Gwizdka, J. (2018). Inferring web page relevance using pupillometry and single channel EEG. In Information Systems and Neuroscience (pp. 175–183). Springer, Cham.
Janes, J. W. (1994). Other people’s judgments: A comparison of users’ and others’ judgments of document relevance, topicality, and utility. Journal of the American Society for Information Science, 45(3), 160–171. https://doi.org/10.1002/(SICI)1097-4571(199404)45:3%3c160:AID-ASI6%3e3.0.CO;2-4.
Kahneman, D., & Beatty, J. (1966). Pupil diameter and load on memory. Science, 154(3756), 1583–1585. https://doi.org/10.1126/science.154.3756.1583.
Krugman, H. E. (1964). Some applications of pupil measurement. Journal of Marketing Research, 1(4), 15–19.
Maglaughlin, K. L., & Sonnenwald, D. H. (2002). User perspectives on relevance criteria: A comparison among relevant, partially relevant, and not-relevant judgments. Journal of the American Society for Information Science and Technology, 53(5), 327–342. https://doi.org/10.1002/asi.10049.
Saracevic, T. (1996, October). Relevance reconsidered. In Proceedings of the second conference on conceptions of library and information science (CoLIS 2) (pp. 201–218). New York: ACM.
Schamber, L., et al. (1990). A re-examination of relevance: Toward a dynamic, situational definition. Information Processing & Management, 26(6), 755–776. https://doi.org/10.1016/0306-4573(90)90050-C.
Zhang, Y., Zhang, J., Lease, M., & Gwizdka, J. (2014, July). Multidimensional relevance modeling via psychometrics and crowdsourcing. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval (pp. 435–444). ACM. https://doi.org/10.1145/2600428.2609577.
Acknowledgements
This research has been funded, in part, by Portuguese Foundation for Science and Technology and the Digital Media Program at UT-Austin. We thank master’s student Han Han for her help with manual creation of AOIs.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Gwizdka, J., Chang, YS. (2020). Search Results Viewing Behavior vis-à-vis Relevance Criteria. In: Davis, F., Riedl, R., vom Brocke, J., Léger, PM., Randolph, A., Fischer, T. (eds) Information Systems and Neuroscience. Lecture Notes in Information Systems and Organisation, vol 32. Springer, Cham. https://doi.org/10.1007/978-3-030-28144-1_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-28144-1_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-28143-4
Online ISBN: 978-3-030-28144-1
eBook Packages: Business and ManagementBusiness and Management (R0)