Skip to main content
SpringerLink
Log in

Evaluating product search and recommender systems for E-commerce environments

  • Published:
Electronic Commerce Research Aims and scope Submit manuscript

Abstract

Online systems that help users select the most preferential item from a large electronic catalog are known as product search and recommender systems. Evaluation of various proposed technologies is essential for further development in this area. This paper describes the design and implementation of two user studies in which a particular product search tool, known as example critiquing, was evaluated against a chosen baseline model. The results confirm that example critiquing significantly reduces users’ task time and error rate while increasing decision accuracy. Additionally, the results of the second user study show that a particular implementation of example critiquing also made users more confident about their choices. The main contribution is that through these two user studies, an evaluation framework of three criteria was successfully identified, which can be used for evaluating general product search and recommender systems in E-commerce environments. These two experiments and the actual procedures also shed light on some of the most important issues which need to be considered for evaluating such tools, such as the preparation of materials for evaluation, user task design, the context of evaluation, the criteria, the measures and the methodology of result analyses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bederson, B. B. (2000). Fisheye Menus. In Proceedings of ACM conference on user interface software and technology (UIST 2000) (pp. 217–226). New York: Assoc. Comput. Mach.

    Chapter  Google Scholar 

  2. Bederson, B. B., et al. (2004). DateLens: a fisheye calendar interface for PDAs. ACM Transactions on Computer-Human Interaction, 11(1), 90–119.

    Article  Google Scholar 

  3. Burke, R. (2002). Hybrid recommender systems: survey and experiments. User Modeling and User-Adapted Interaction, 12(4), 331–370.

    Article  Google Scholar 

  4. Burke, R., Hammond, K., & Young, B. (1997). The findme approach to assisted browsing. IEEE Expert: Intelligent Systems and Their Applications, 12(4), 32–40.

    Google Scholar 

  5. Callahan, E., & Koenemann, J. (2000). A comparative usability evaluation of user interfaces for online product catalogs. In Proceedings of ACM electronic commerce conference (pp. 197–206). New York: Assoc. Comput. Mach.

    Chapter  Google Scholar 

  6. Carenini, G., & Poole, D. (2002). Constructed preferences and value-focused thinking: implications for AI research on preference elicitation. In AAAI-02 workshop on preferences in AI and CP: symbolic approaches, Edmonton, Canada.

  7. Faltings, B., Torrens, M., & Pu, P. (2004). Solution generation with qualitative models of preferences. International Journal of Computational Intelligence and Applications, 20(2), 246–263.

    Google Scholar 

  8. Fischer, G. W., & Hawkins, S. A. (1993). Strategy compatibility, scale compatibility, and the prominence effect. Journal of Experimental Psychology: Human Perception and Performance, 19, 580–597.

    Article  Google Scholar 

  9. Furnas, G. W. (1986). Generalized fisheye views. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 16–23). New York: Assoc. Comput. Mach.

    Google Scholar 

  10. Janecek, P., & Pu, P. (2002). A framework for designing fisheye views to support multiple semantic contexts. In Proceedings of the international conference on advanced visual interfaces (AVI’02). New York: Assoc. Comput. Mach.

    Google Scholar 

  11. Jedetski, J., Adelman, L., & Yeo, C. (2002). How web site decision technology affects consumers. Internet Computing, 6(2), 72–79.

    Article  Google Scholar 

  12. Ha, V., & Haddawy, P. (2003). Similarity of personal preferences: theoretical foundations and empirical analysis. Artificial Intelligence, 46(2), 9–173.

    Google Scholar 

  13. Haubl, G., & Trifts, V. (2000). Consumer decision making in online shopping environments: the effects of interactive decision aids. Marketing Science, 19(1), 4–21.

    Article  Google Scholar 

  14. Herlocker, J., Konstan, J., Terveen, L. G., & Riedl, J. T. (2004). Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1), 5–53.

    Article  Google Scholar 

  15. Hogarth, R. (1987). Judgment and choice. New York: Wiley.

    Google Scholar 

  16. Keeney, R. L., & Raiffa, H. (1976). Decisions with multiple objectives: preferences and value tradeoffs. New York: Wiley.

    Google Scholar 

  17. Lamping, J., Rao, R., & Pirolli, P. (1995). A focus + context technique based on hyperbolic geometry for visualizing large hierarchies. In CHI’95, ACM conference on human factors in computing systems (pp. 401–408).

  18. Linden, G., Hanks, S., & Lesh, N. (1997). Interactive assessment of user preference models: the automated travel assistant. In Proceedings of user modeling’97 (pp. 67–78).

  19. Maes, P., Guttman, R., & Moukas, A. (1999). Agents that buy and sell: transforming commerce as we know it. Communications of the ACM, 42(3), 81–91.

    Article  Google Scholar 

  20. McCarthy, K. et al. (2004). On the dynamic generation of compound critiques in conversational recommender systems. In Proceedings of the third international conference on adaptive hypermedia and adaptive web-based systems.

  21. McCarthy, K., et al. (2005). Experiments in dynamic critiquing. In Proceedings of the international conference on intelligent user interfaces (IUI05) (pp. 175–182).

  22. Payne, J. W., Bettman, J. R., & Johnson, E. J. (1993). The adaptive decision maker. Cambridge: Cambridge University Press.

    Google Scholar 

  23. Pirolli, P., Card, S. K., & Wege, M. M. (2003). The effects of information scent on visual search in the hyperbolic tree browser. ACM Transactions on Computer-Human Interaction, 10(1), 20–53.

    Article  Google Scholar 

  24. Pu, P., & Faltings, B. (2000). Enriching buyers’ experiences: the SmartClient approach. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 289–296). New York: Assoc. Comput. Mach.

    Chapter  Google Scholar 

  25. Pu, P., & Faltings, B. (2004). Interface technologies for decision tradeoff problems using CSP. International Journal of Constraints, 9(4), 289–310.

    Article  Google Scholar 

  26. Pu, P., & Kumar, P. (2004). Evaluating example-based search tools. In Proceedings of the ACM conference on electronic commerce (EC’04) (pp. 208–217). New York: Assoc. Comput. Mach.

    Chapter  Google Scholar 

  27. Rao, R., & Card, S. K. (1994). The table lens: merging graphical and symbolic representations in an interactive focus + context visualization for tabular information. In Proceeding of Human factors in computing systems (CHI’94) (pp. 318–322). New York: Assoc. Comput. Mach.

    Google Scholar 

  28. Reilly, J., et al. (2004). Dynamic critiquing. In Proceedings of the seventh European conference on case-based reasoning (ECCBR-04).

  29. Reilly, J., et al. (2004). Incremental critiquing. In Proceedings of the 24th SGAI international conference on innovative techniques and applications of artificial intelligence (AI-04).

  30. Resnick, P. et al. (1994). Grouplens: an open architecture for collaborative filtering of netnews. In Proceedings of the ACM conference on computer supported cooperative work (pp. 130–137). New York: Assoc. Comput. Mach.

    Google Scholar 

  31. Sarkar, M., & Brown, M. H. (1992). Graphical fisheye views of graphs. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 83–91). New York: Assoc. Comput. Mach.

    Google Scholar 

  32. Smyth, B., et al. (2004). Compound critiques feedback for conversational recommender systems. In Proceedings of the IEEE/WIC/ACM international conference on web intelligence (WI-04).

  33. Spence, R., & Apperly, M. (1982). Data base navigation: an office environment for the professional. Behaviour and Information Technology, 1(1), 43–54.

    Article  Google Scholar 

  34. Spiekermann, S., & Paraschiv, C. (2002). Motivating human-agent interaction: transferring insights from behavioral marketing to interface design. Journal of Electronic Commerce Research, 2, 255–285.

    Article  Google Scholar 

  35. Stolze, M. (2000). Soft navigation in electronic product catalogs. International Journal on Digital Libraries, 3(1), 60–66.

    Article  Google Scholar 

  36. Torrens, M., & Faltings, B. (1999). SmartClients: constraint satisfaction as a paradigm for scaleable intelligent information systems. In Workshop notes, artificial intelligence for electronic commerce, the sixteenth national conference on artificial intelligence (AAAI’99) (pp. 10–15). Merlo Park: AAAI Press.

    Google Scholar 

  37. Torrens, M., Faltings, B., & Pu, P. (2002). SmartClients: constraint satisfaction as a paradigm for scaleable intelligent information systems. International Journal of Constraints, 7(1), 49–69.

    Article  Google Scholar 

  38. Torrens, M., Weigel, R., & Faltings, B. (1997). Java constraint library: bringing constraints technology on the internet using the Java language. In Workshop notes, constraints and agents, the fourteenth national conference on artificial intelligence (AAAI’97). Merlo Park: AAAI Press.

    Google Scholar 

  39. Tversky, A., Sattath, S., & Slovic, P. (1988). Contigent weighting in judgement and choice. Psychology Review, 95, 371–384.

    Article  Google Scholar 

  40. Zhang, J., & Pu, P. (2004). Survey of solving multi-attribute decision problems (Technical Report No. IC/200454). Swiss Federal Institute of Technology in Lausanne (EPFL), Lausanne, Switzerland, June, 2004.

Download references

Author information

Authors and Affiliations

  1. Human Computer Interaction Group, School of Computer and Communication Sciences, Swiss Federal Institute of Technology in Lausanne (EPFL), 1015, Lausanne, Switzerland

    Pearl Pu & Li Chen

  2. Business Administration, Darden Graduate School of Business, University of Virginia, Charlottesville, USA

    Pratyush Kumar

Authors
  1. Pearl Pu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Pratyush Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pearl Pu.

Additional information

This research was supported by the Swiss National Science Foundation.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Pu, P., Chen, L. & Kumar, P. Evaluating product search and recommender systems for E-commerce environments. Electron Commerce Res 8, 1–27 (2008). https://doi.org/10.1007/s10660-008-9015-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10660-008-9015-z

Keywords

Search

Navigation