Skip to main content
SpringerLink
Log in

Choice overload and recommendation effectiveness in related-article recommendations

Analyzing the Sowiport digital library

International Journal on Digital Libraries Aims and scope Submit manuscript

Cite this article

Abstract

Choice overload describes a situation in which a person has difficulty in making decisions due to too many options. We examine choice overload when displaying related-article recommendations in digital libraries, and examine the effectiveness of recommendation algorithms in this domain. We first analyzed existing digital libraries, and found that only 30% of digital libraries show related-article recommendations to their users. Of these libraries, the majority (74%) displays 3–5 related articles; 28% of them display 6–10 related articles; and no digital library displayed more than ten related-article recommendations. We then conducted our experimental evaluation through GESIS’ digital library Sowiport, with recommendations delivered by recommendations-as-a-service provider Mr. DLib. We use four metrics to analyze 41.3 million delivered recommendations: click-through rate (CTR), percentage of clicked recommendation sets (clicked set rate, CSR), average clicks per clicked recommendation set (ACCS), and time to first click (TTFC), which is the time between delivery of a set of recommendations to the first click. These metrics help us to analyze choice overload and can yield evidence for finding the ideal number of recommendations to display. We found that with increasing recommendation set size, i.e., the numbers of displayed recommendations, CTR decreases from 0.41% for one recommendation to 0.09% for 15 recommendations. Most recommendation sets only receive one click. ACCS increases with set size, but increases more slowly for six recommendations and more. When displaying 15 recommendations, the average clicks per set is at a maximum (1.15). Similarly, TTFC increases with larger recommendation set size, but increases more slowly for sets of more than five recommendations. While CTR and CSR do not indicate choice overload, ACCS and TTFC point toward 5–6 recommendations as being optimal for Sowiport. Content-based filtering yields the highest CTR with 0.118%, while stereotype recommendations yield the highest ACCS (1.28). Stereotype recommendations also yield the highest TTFC. This means that users take more time before clicking stereotype recommendations when compared to recommendations based on other algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. http://mr-dlib.org.

  2. http://sowiport.gesis.org.

  3. Most of them listed on https://en.wikipedia.org/wiki/List_of_digital_library_projects.

  4. Some explanations about Sowiport and Mr. DLib are copied from [19].

  5. http://www.gesis.org.

  6. Here, as a data cleaning process, we considered recommendations as clicked if they were clicked within 30 min of delivery of the recommendation set. This is true for 90.77% of all first clicks in the dataset (39,709 of 43,748).

References

  1. Arunachalam, B., Henneberry, S.R., Lusk, J.L., Norwood, F.B.: An empirical investigation into the excessive-choice effect. Am. J. Agric. Econ. 91(3), 810–825 (2009). https://doi.org/10.1111/j.1467-8276.2009.01260.x

    Article  Google Scholar 

  2. Azzopardi, L., Zuccon, G.: Two scrolls or one click: a cost model for browsing search results. In: Ferro, N., et al. (eds.) Advances in Information Retrieval, pp. 696–702. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30671-1_55

    Chapter  Google Scholar 

  3. Bawden, D., Robinson, L.: The dark side of information: overload, anxiety and other paradoxes and pathologies. J. Inf. Sci. 35(2), 180–191 (2009)

    Article  Google Scholar 

  4. Beel, J., Dinesh, S.: Real-world recommender systems for academia: the pain and gain in building, operating, and researching them. In: BIR ECIR, pp. 6–17 (2017)

  5. Beel, J., Langer, S.: A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems. In: Kapidakis, S., Mazurek, C., Werla, M. (eds.) Research and Advanced Technology for Digital Libraries, pp. 153–168. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24592-8_12

    Chapter  Google Scholar 

  6. Beel, J., Gipp, B., Langer, S., Genzmehr, M., Wilde, E., Nürnberger, A., Pitman, J.: Introducing Mr. DLib, a machine-readable digital library. In: Proceedings of the 11th ACM/IEEE Joint Conference on Digital Libraries (JCDL‘11), ACM, pp. 463–464. https://doi.org/10.1145/1998076.1998187 (2011)

  7. Beel, J., Gipp, B., Langer, S., Breitinger, C.: Research paper recommender systems: a literature survey. Int. J. Digit. Lib. (2016) https://doi.org/10.1007/s00799-015-0156-0

  8. Beel, J., Aizawa, A., Breitinger, C., Gipp, B.: Mr. DLib: Recommendations-as-a-Service (RaaS) for academia. In: 2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pp. 1–2 (2017) https://doi.org/10.1109/JCDL.2017.7991606

  9. Beel, J., Smyth, B., Collins, A.: RARD II: The 2nd related-article recommendation dataset. arXiv:1807.06918 [cs.IR] (2018)

  10. Beierle, F., Aizawa, A., Beel, J.: Exploring choice overload in related-article recommendations in digital libraries. In: Proceedings of the 5th International Workshop on Bibliometric-enhanced Information Retrieval (BIR2017), co-located with the 39th European Conference on Information Retrieval (ECIR), CEUR-WS, pp. 51–61 (2017). http://ceur-ws.org/Vol-1823/paper5.pdf

  11. Besedeš, T., Deck, C., Sarangi, S., Shor, M.: Reducing choice overload without reducing choices. Rev. Econ. Stat. 97(4), 793–802 (2015)

    Article  Google Scholar 

  12. Bollen, D., Knijnenburg, B.P., Willemsen, M.C., Graus, M.: Understanding choice overload in recommender systems. In: Proceedings of the Fourth ACM Conference on Recommender Systems, ACM, RecSys ’10, pp. 63–70 (2010). https://doi.org/10.1145/1864708.1864724

  13. Bornmann, L., Mutz, R.: Growth rates of modern science: a bibliometric analysis based on the number of publications and cited references. J. Assoc. Inf. Sci. Technol. 66(11), 2215–2222 (2015)

    Article  Google Scholar 

  14. Buturak, G., Evren, Ö.: Choice overload and asymmetric regret. Theor. Econ. 12(3), 1029–1056 (2017)

    Article  MathSciNet  Google Scholar 

  15. Chernev, A., Böckenholt, U., Goodman, J.: Choice overload: a conceptual review and meta-analysis. J. Consum. Psychol. 25(2), 333–358 (2015)

    Article  Google Scholar 

  16. Chiravirakul, P., Payne, S.J.: Choice overload in search engine use? In: Proceedings of the 32nd Annual ACM Conference on Human Factors in Computing Systems, ACM, CHI ’14, pp. 1285–1294 (2014). https://doi.org/10.1145/2556288.2557149

  17. Collins, A., Tkaczyk, D., Aizawa, A., Beel, J.: Position bias in recommender systems for digital libraries. In: Chowdhury, G., McLeod, J., Gillet, V., Willett, P. (eds.) Transforming Digital Worlds, pp. 335–344. Springer International Publishing, Cham (2018)

    Chapter  Google Scholar 

  18. Dellaert, B., Baker, T., Johnson, E.: Partitioning sorted sets: overcoming choice overload while maintaining decision quality. Columbia Business School Research Paper 18 (2017)

  19. Dinesh, S., Beel, J., Mayr, P., Carevic, Z., Raghvendra, J.: Stereotype and most-popular recommendations in the digital library Sowiport. In: Proceedings of the 15th International Symposium of Information Science (ISI), Werner Hülsbusch, Glückstadt, Germany, pp. 96–108 (2017)

  20. Fasolo, B., McClelland, G.H., Todd, P.M.: Escaping the tyranny of choice: when fewer attributes make choice easier. Market. Theory 7(1), 13–26 (2007). https://doi.org/10.1177/1470593107073842

    Article  Google Scholar 

  21. Feyer, S., Siebert, S., Gipp, B., Aizawa, A., Beel, J.: Integration of the scientific recommender system Mr. DLib into the reference manager JabRef. In: Proceedings of the 39th European Conference on Information Retrieval (ECIR) (2017)

  22. Haynes, G.A.: Testing the boundaries of the choice overload phenomenon: the effect of number of options and time pressure on decision difficulty and satisfaction. Psychol. Market. 26(3), 204–212 (2009)

    Article  Google Scholar 

  23. Hienert, D., Sawitzki, F., Mayr, P.: Digital library research in action: supporting information retrieval in Sowiport. D-Lib Magazine 21(3/4) (2015)

  24. Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, SIGIR ’05, pp. 154–161 (2005)

  25. Jones, M., Marsden, G., Mohd-Nasir, N., Boone, K., Buchanan, G.: Improving Web interaction on small displays. Comput. Netw. 31(11–16), 1129–1137 (1999). https://doi.org/10.1016/S1389-1286(99)00013-4

    Article  Google Scholar 

  26. Jugovac, M., Jannach, D.: Interacting with recommenders—overview and research directions. ACM Trans. Interact. Intell. Syst. (TiiS) 7(3), 10 (2017)

    Google Scholar 

  27. Kelly, D., Azzopardi, L.: How many results per page? A study of SERP size, search behavior and user experience. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, SIGIR ’15, pp. 183–192 (2015)

  28. Kim, J., Thomas, P., Sankaranarayana, R., Gedeon, T., Yoon, H.J.: Eye-tracking analysis of user behavior and performance in web search on large and small screens. J. Assoc. Inf. Sci. Technol. 66(3), 526–544 (2015). https://doi.org/10.1002/asi.23187

    Article  Google Scholar 

  29. Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H., Newell, C.: Explaining the user experience of recommender systems. User Model. User Adapt. Interact. 22(4–5), 441–504 (2012). https://doi.org/10.1007/s11257-011-9118-4

    Article  Google Scholar 

  30. Kopp, O., Breitenbücher, U., Müller, T.: Cloudref-to-wards collaborative reference management in the cloud. In: Proceedings of the 10th Central European Workshop on Services and their Composition (2018)

  31. Langer, S., Beel, J.: Apache Lucene as content-based-filtering recommender system: 3 lessons learned. In: Proceedings of the 5th International Workshop on Bibliometric-Enhanced Information Retrieval (BIR2017), co-located with the 39th European Conference on Information Retrieval (ECIR), CEUR-WS, pp. 85–92 (2017). http://ceur-ws.org/Vol-1823/paper8.pdf

  32. Langer, S., Genzmehr, M., Beel, J., Nürnberger, A.: Introducing Docear’s research paper recommender system. In: Proceedings of the 13th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL’13), ACM, pp. 459–460 (2013). https://doi.org/10.1145/2467696.2467786

  33. Linden, G.: Marissa Mayer at Web 2.0. (2006). http://glinden.blogspot.com/2006/11/marissa-mayer-at-web-20.html

  34. Moser, C., Phelan, C., Resnick, P., Schoenebeck, S.Y., Reinecke, K.: No such thing as too much chocolate: evidence against choice overload in e-commerce. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, ACM, New York, NY, USA, CHI ’17, pp. 4358–4369 (2017). https://doi.org/10.1145/3025453.3025778

  35. Motajcsek, T., Le Moine, J.Y., Larson, M., Kohlsdorf, D., Lommatzsch, A., Tikk, D., Alonso, O., Cremonesi, P., Demetriou, A., Dobrajs, K., Garzotto, F., Göker, A., Hopfgartner, F., Malagoli, D., Nguyen, T.N., Novak, J., Ricci, F., Scriminaci, M., Tkalcic, M., Zacchi, A.: Algorithms aside: Recommendation as the lens of life. In: Proceedings of the 10th ACM Conference on Recommender Systems, ACM, New York, NY, USA, RecSys ’16, pp. 215–219 (2016). https://doi.org/10.1145/2959100.2959164

  36. Muggeo, V.M.R.: Estimating regression models with unknown break-points. Stat. Med. 22, 3055–3071 (2003). https://doi.org/10.1002/sim.1545

    Article  Google Scholar 

  37. Muggeo, V.M.R.: Segmented: an R package to fit regression models with broken-line relationships. R News 8(1), 20–25 (2008)

    Google Scholar 

  38. Oulasvirta, A., Hukkinen, J.P., Schwartz, B.: When more is less: the paradox of choice in search engine use. In: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, SIGIR ’09, pp. 516–523 (2009)

  39. Park, J.Y., Jang, S.S.: Confused by too many choices? Choice overload in tourism. Tour. Manag. 35, 1–12 (2013)

    Article  Google Scholar 

  40. Radlinski, F., Joachims, T.: Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In: Proceedings of the 21st National Conference on Artificial Intelligence, Vol. 2. AAAI Press, AAAI’06, pp. 1406–1412 (2006)

  41. Reed, D.D., Reed, F.D.D., Chok, J., Brozyna, G.A.: The “tyranny of choice”: choice overload as a possible instance of effort discounting. Psycholog. Rec. 61(4), 547–560 (2011)

    Article  Google Scholar 

  42. Scheibehenne, B., Greifeneder, R., Todd, P.M.: Can there ever be too many options? A meta-analytic review of choice overload. J. Consum. Res. 37(3), 409–425 (2010)

    Article  Google Scholar 

  43. Schwartz, B.: The tyranny of choice. Sci. Am. 290(4), 70–75 (2004)

    Article  Google Scholar 

  44. Schwarzer, M., Schubotz, M., Meuschke, N., Breitinger, C., Markl, V., Gipp, B.: Evaluating link-based recommendations for wikipedia. In: 2016 IEEE/ACM Joint Conference on Digital Libraries (JCDL), pp. 191–200 (2016)

  45. Spillane, B., Lawless, S., Wade, V.: Perception of bias: the impact of user characteristics, website design and technical features. In: Proceedings of the International Conference on Web Intelligence, ACM, pp. 227–236 (2017)

  46. Stempfhuber, M., Schaer, P., Shen, W.: Enhancing visibility: integrating grey literature in the sowiport information cycle. Conf. Pap. Int. Conf. Grey Lit. 9, 23–29 (2008)

    Google Scholar 

  47. Van Noorden, R.: Global scientific output doubles every nine years. Nat. Blog. (2014) http://blogs.nature.com/news/2014/05/global-scientific-output-doubles-every-nine-years.html

  48. Willemsen, M.C., Graus, M.P., Knijnenburg, B.P.: Understanding the role of latent feature diversification on choice difficulty and satisfaction. User Model. User Adapt. Interact. 26(4), 347–389 (2016). https://doi.org/10.1007/s11257-016-9178-6

    Article  Google Scholar 

  49. Zheng, Y., Mobasher, B., Burke, R.: Emotions in context-aware recommender systems. In: Tkalčič, M., De Carolis, B., de Gemmis, M., Odić, A., Košir, A. (eds.) Emotions and Personality in Personalized Services, pp. 311–326. Springer, Cham (2016). 10.1007/978-3-319-31413-6\_15

    Google Scholar 

Download references

Acknowledgements

We are grateful for the support provided by Jan Blechschmidt, Boris Lorbeer, Sophie Siebert, Stefan Feyer, Siddharth Dinesh, Zeljko Carevic, and Mr. DLib’s partners. We furthermore thank the reviewers for their valuable comments.

Author information

Authors and Affiliations

  1. Service-centric Networking, Telekom Innovation Laboratories, Technische Universität Berlin, Berlin, Germany

    Felix Beierle

  2. Digital Content and Media Sciences Research Division, National Institute of Informatics (NII), Tokyo, Japan

    Akiko Aizawa & Joeran Beel

  3. School of Computer Science and Statistics, ADAPT Centre, Trinity College Dublin, Dublin, Ireland

    Andrew Collins & Joeran Beel

Authors
  1. Felix Beierle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Akiko Aizawa
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrew Collins
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Joeran Beel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Felix Beierle.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work has received funding from project DYNAMIC (http://www.dynamic-project.de) (Grant No. 01IS12056), which is funded as part of the Software Campus initiative by the German Federal Ministry of Education and Research (BMBF). This work was also supported by a fellowship within the FITweltweit programme of the German Academic Exchange Service (DAAD). This publication also has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under Grant Number 13/RC/2106.

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Beierle, F., Aizawa, A., Collins, A. et al. Choice overload and recommendation effectiveness in related-article recommendations. Int J Digit Libr 21, 231–246 (2020). https://doi.org/10.1007/s00799-019-00270-7

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00799-019-00270-7

Keywords

search

Navigation