Skip to main content

Beyond Theory and Data in Preference Modeling: Bringing Humans into the Loop

Part of the Lecture Notes in Computer Science book series (LNAI,volume 9346)

Abstract

Many mathematical frameworks aim at modeling human preferences, employing a number of methods including utility functions, qualitative preference statements, constraint optimization, and logic formalisms. The choice of one model over another is usually based on the assumption that it can accurately describe the preferences of humans or other subjects/processes in the considered setting and is computationally tractable. Verification of these preference models often leverages some form of real life or domain specific data; demonstrating the models can predict the series of choices observed in the past. We argue that this is not enough: to evaluate a preference model, humans must be brought into the loop. Human experiments in controlled environments are needed to avoid common pitfalls associated with exclusively using prior data including introducing bias in the attempt to clean the data, mistaking correlation for causality, or testing data in a context that is different from the one where the data were produced. Human experiments need to be done carefully and we advocate a multi-disciplinary research environment that includes experimental psychologists and AI researchers. We argue that experiments should be used to validate models. We detail the design of an experiment in order to highlight some of the significant computational, conceptual, ethical, mathematical, psychological, and statistical hurdles to testing whether decision makers’ preferences are consistent with a particular mathematical model of preferences.

Keywords

  • Preference Model
  • Human Subject Research
  • Pairwise Preference
  • Artificial Intelligence Researcher
  • Video Rental

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-319-23114-3_1
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-319-23114-3
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Fig. 1.

Notes

  1. 1.

    http://maxsat.ia.udl.cat/introduction/.

  2. 2.

    Human-subjects standards vary from country to country and are also sometimes imposed by international academic societies.

  3. 3.

    For an example of the complexities involved in testing transitivity of preferences, including a critical review of the prior literature, see, e.g. [11, 57, 58, 60].

  4. 4.

    See, e.g., http://psychfiledrawer.org/.

References

  1. Allen, T.E., Goldsmith, J., Mattei, N.: Counting, ranking, and randomly generating CP-nets. In: 8th Workshop on Advances in Preference Handling (MPREF 2014), AAAI-14 Workshop Series (2014)

    Google Scholar 

  2. Allen, T.E.: CP-nets with indifference. In: 2013 51st Annual Allerton Conference on Communication, Control, and Computing (Allerton), pp. 1488–1495. IEEE (2013)

    Google Scholar 

  3. Bache, K., Lichman, M.: UCI machine learning repository, School of Information and Computer Sciences, University of California, Irvine (2013). http://archive.ics.uci.edu/ml

  4. Becker, G., DeGroot, M., Marschak, J.: Stochastic models of choice behavior. Behav. Sci. 8, 41–55 (1963)

    Google Scholar 

  5. Bennett, J., Lanning, S.: The Netflix prize. In: Proceedings of the KDD Cup and Workshop (2007)

    Google Scholar 

  6. Boutilier, C., Brafman, R., Domshlak, C., Hoos, H., Poole, D.: CP-nets: A tool for representing and reasoning with conditional ceteris paribus preference statements. J. Artif. Intell. Res. 21, 135–191 (2004)

    MathSciNet  MATH  Google Scholar 

  7. Boutilier, C., Patrascu, R., Poupart, P., Schuurmans, D.: Constraint-based optimization and utility elicitation using the minimax decision criterion. Artif. Intell. 170(8), 686–713 (2006)

    MathSciNet  MATH  Google Scholar 

  8. Brafman, R.I., Domshlak, C.: Preference handling–an introductory tutorial. AI Mag. 30(1), 58 (2009)

    Google Scholar 

  9. Braziunas, D., Boutilier, C.: Assessing regret-based preference elicitation with the utpref recommendation system. In: Proceedings of the 11th ACM Conference on Electronic Commerce (EC), pp. 219–228. ACM (2010)

    Google Scholar 

  10. Carbone, E., Hey, J.: Which error story is best? J. Risk Uncertain. 20, 161–176 (2000)

    MATH  Google Scholar 

  11. Cavagnaro, D., Davis-Stober, C.: Transitive in our preferences, but transitive in different ways: an analysis of choice variability. Decision 1(1), 102–122 (2014)

    Google Scholar 

  12. Chevaleyre, Y., Endriss, U., Lang, J., Maudet, N.: Preference handling in combinatorial domains: from AI to social choice. AI Mag. 29(4), 37–46 (2008)

    Google Scholar 

  13. Cohen, J.: The earth is round (\(p<.05\)). Am. Psychol. 49(12), 997 (1994)

    Google Scholar 

  14. Crump, M.J.C., McDonnell, J.V., Gureckis, T.M.: Evaluating Amazon’s mechanical Turk as a tool for experimental behavioral research. PloS one 8(3), e57410 (2013)

    Google Scholar 

  15. Davis-Stober, C.: Analysis of multinomial models under inequality constraints: applications to measurement theory. J. Math. Psychol. 53, 1–13 (2009)

    MathSciNet  MATH  Google Scholar 

  16. De Groot, A.D.: Thought and Choice in Chess (Psychological Studies), 2nd edn. Mouton de Gruyter, The Hague (1978)

    Google Scholar 

  17. De Saint-Cyr, F.D., Lang, J., Schiex, T.: Penalty logic and its link with Dempster-Shafer theory. In: Proceedings of the UAI, pp. 204–211 (1994)

    Google Scholar 

  18. Dodson, T., Mattei, N., Guerin, J., Goldsmith, J.: An English-language argumentation interface for explanation generation with Markov decision processes in the domain of academic advising. ACM Trans. Interact. Intell. Syst. (TiiS) 3(3), 18 (2013)

    Google Scholar 

  19. Domshlak, C., Hüllermeier, E., Kaci, S., Prade, H.: Preferences in AI: an overview. Artif. Intell. 175(7), 1037–1052 (2011)

    MathSciNet  Google Scholar 

  20. Dubois, D., Lang, J., Prade, H.: A brief overview of possibilistic logic. In: Kruse, R., Siegel, P. (eds.) Symbolic and Quantitative Approaches to Uncertainty. LNCS, vol. 548, pp. 53–57. Springer, Heidelberg (1991)

    Google Scholar 

  21. Fiorini, S.: Determining the automorphism group of the linear ordering polytope. Discret. Appl. Math. 112, 121–128 (2001)

    MathSciNet  MATH  Google Scholar 

  22. Fürnkranz, J., Hüllermeier, E.: Preference Learning. Springer, New York (2010)

    MATH  Google Scholar 

  23. Goldsmith, J., Junker, U.: Preference handling for artificial intelligence. AI Mag. 29(4), 9 (2009)

    Google Scholar 

  24. Grötschel, M., Jünger, M., Reinelt, G.: Facets of the linear ordering polytope. Math. Program. 33, 43–60 (1985)

    MathSciNet  MATH  Google Scholar 

  25. Guo, S., Sanner, S., Bonilla, E.V.: Gaussian process preference elicitation. In: Advances in Neural Information Processing Systems, pp. 262–270 (2010)

    Google Scholar 

  26. Ioannidis, J.: Why most published research findings are false. PLoS Med. 2(8), e124 (2005)

    Google Scholar 

  27. Jawaheer, G., Weller, P., Kostkova, P.: Modeling user preferences in recommender systems: a classification framework for explicit and implicit user feedback. ACM Trans. Interact. Intell. Syst. (TiiS) 4(2), 8 (2014). http://doi.acm.org/10.1145/2512208

    Google Scholar 

  28. Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlinski, F., Gay, G.: Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans. Inf. Syst. (TOIS) 25(2), 7 (2007)

    Google Scholar 

  29. Kagel, J., Roth, A.: The Handbook of Experimental Economics. Princeton University Press, Princeton (1995)

    Google Scholar 

  30. Kamishima, T.: Nantonac collaborative filtering: recommendation based on order responses. In: The 9th International Conference on Knowledge Discovery and Data Mining (KDD), pp. 583–588 (2003)

    Google Scholar 

  31. Kass, R., Raftery, A.: Bayes factors. J. Am. Stat. Assoc. 90(430), 773–795 (1995)

    MathSciNet  MATH  Google Scholar 

  32. Klugkist, I., Hoijtink, H.: The Bayes factor for inequality and about equality constrained models. Comput. Stat. Data Anal. 51, 6367–6379 (2007)

    MathSciNet  MATH  Google Scholar 

  33. Koppen, M.: Random utility representation of binary choice probabilities: critical graphs yielding critical necessary conditions. J. Math. Psychol. 39, 21–39 (1995)

    MathSciNet  MATH  Google Scholar 

  34. Li, M., Vo, Q.B., Kowalczyk, R.: Efficient heuristic approach to dominance testing in CP-nets. In: Proceedings of the AAMAS, pp. 353–360, Richland, SC, USA (2011)

    Google Scholar 

  35. Linden, G., Smith, B., York, J.: Amazon.com recommendations: item-to-item collaborative filtering. IEEE Internet Comput. 7(1), 76–80 (2003)

    Google Scholar 

  36. Loomes, G., Sugden, R.: Testing different stochastic specifications of risky choice. Economica 65, 581–598 (1998)

    Google Scholar 

  37. Luce, R.: Individual Choice Behavior: A Theoretical Analysis. Wiley, New York (1959)

    MATH  Google Scholar 

  38. Luce, R.: Four tensions concerning mathematical modeling in psychology. Annu. Rev. Psychol. 46, 1–26 (1995)

    Google Scholar 

  39. Luce, R.: Joint receipt and certainty equivalents of gambles. J. Math. Psychol. 39, 73–81 (1995)

    MathSciNet  MATH  Google Scholar 

  40. MacKenzie, I.S., Castellucci, S.J.: Empirical research methods for human-computer interaction. In: Proceedings of the CHI, pp. 1013–1014 (2014)

    Google Scholar 

  41. Mao, A., Procaccia, A.D., Chen, Y.: Better human computation through principled voting. In: Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI) (2013)

    Google Scholar 

  42. Martí, R., Reinelt, G.: The Linear Ordering Problem: Exact and Heuristic Methods in Combinatorial Optimization. Applied Mathematical Science, vol. 175. Springer, Heidelberg (2011)

    MATH  Google Scholar 

  43. Mason, W., Suri, S.: Conducting behavioral research on Amazon’s Mechanical Turk. Behav. Res. Methods 44(1), 1–23 (2012)

    Google Scholar 

  44. Mattei, N., Walsh, T.: Preflib: a library for preferences http://www.preflib.org. In: Perny, P., Pirlot, M., Tsoukiàs, A. (eds.) ADT 2013. LNCS, vol. 8176, pp. 259–270. Springer, Heidelberg (2013)

    Google Scholar 

  45. Milgram, S.: Behavioral study of obedience. J. Abnorm. Soc. Psychol. 67(4), 371 (1963)

    Google Scholar 

  46. Morey, R., Rouder, J., Verhagen, J., Wagenmakers, E.: Why hypothesis tests are essential for psychological science: a comment on Cumming. Psychol. Sci. 25(6), 1289–1290 (2014)

    Google Scholar 

  47. Müller, H., Sedley, A., Ferrall-Nunge, E.: Designing unbiased surveys for HCI research. In: Proceedings of the CHI, pp. 1027–1028 (2014)

    Google Scholar 

  48. Myung, J., Karabatsos, G., Iverson, G.: A Bayesian approach to testing decision making axioms. J. Math. Psychol. 49, 205–225 (2005)

    MathSciNet  MATH  Google Scholar 

  49. Narayanan, A., Shmatikov, V.: Robust de-anonymization of large sparse datasets. In: IEEE Symposium on Security and Privacy, pp. 111–125 (2008)

    Google Scholar 

  50. Nisbett, R.E., Wilson, T.D.: Telling more than we can know: verbal reports on mental processes. Psychol. Rev. 84(3), 231 (1977)

    Google Scholar 

  51. Nordgren, L.F., Dijksterhuis, A.: The devil is in the deliberation: thinking too much reduces preference consistency. J. Consum. Res. 36(1), 39–46 (2009)

    Google Scholar 

  52. Pommeranz, A., Broekens, J., Wiggers, P., Brinkman, W.P., Jonker, C.M.: Designing interfaces for explicit preference elicitation: a user-centered investigation of preference representation and elicitation process. User Model. User-Adap. Inter. 22(4–5), 357–397 (2012)

    Google Scholar 

  53. Popova, A., Regenwetter, M., Mattei, N.: A behavioral perspective on social choice. Ann. Math. Artif. Intell. 68(1–3), 5–30 (2013)

    MathSciNet  Google Scholar 

  54. Popper, K.: The Logic of Scientific Discovery. Hutchinson, London (1959)

    MATH  Google Scholar 

  55. Pu, P., Chen, L.: Trust building with explanation interfaces. In: Proceedings of the 11th International Conference on Intelligent User Interfaces (IUI), pp. 93–100 (2006)

    Google Scholar 

  56. Pu, P., Chen, L., Hu, R.: Evaluating recommender systems from the user’s perspective: survey of the state of the art. User Model. User-Adap. Inter. 22(4–5), 317–355 (2012)

    Google Scholar 

  57. Regenwetter, M., Dana, J., Davis-Stober, C.P.: Transitivity of preferences. Psychol. Rev. 118, 42–56 (2011)

    Google Scholar 

  58. Regenwetter, M., Dana, J., Davis-Stober, C.P.: Testing transitivity of preferences on two-alternative forced choice data. Front. Psychology 1, 148 (2010). doi:10.3389/fpsyg.2010.00148

    CrossRef  Google Scholar 

  59. Regenwetter, M., Davis-Stober, C.P., Lim, S.H., Guo, Y., Popova, A., Zwilling, C., Cha, Y.C., Messner, W.: QTest: quantitative testing of theories of binary choice. Decision 1(1), 2–34 (2014)

    Google Scholar 

  60. Regenwetter, M., Davis-Stober, C.: Behavioral variability of choices versus structural inconsistency of preferences. Psychol. Rev. 119(2), 408–416 (2012)

    Google Scholar 

  61. Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.): Recommender Systems Handbook. Springer, New York (2011)

    MATH  Google Scholar 

  62. Rossi, F., Beek, P.V., Walsh, T.: Handbook of Constraint Programming. Elsevier, Amsterdam (2006)

    MATH  Google Scholar 

  63. Rossi, F., Venable, K.B., Walsh, T.: A short introduction to preferences: between artificial intelligence and social choice. Synth. Lect. Artif. Intell. Mach. Learn. 5(4), 1–102 (2011)

    Google Scholar 

  64. Schooler, J.: Unpublished results hide the decline effect. Nature 470(7335), 437 (2011)

    Google Scholar 

  65. Simmons, J., Nelson, L., Simonsohn, U.: False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22(11), 1359–1366 (2011)

    Google Scholar 

  66. Tintarev, N., Masthoff, J.: Designing and evaluating explanations for recommender systems. In: Ricci, F., Rokach, L., Shapira, B., Kantor, P.B. (eds.) Recommender Systems Handbook, pp. 479–510. Springer, London (2011)

    Google Scholar 

  67. Tversky, A.: Intransitivity of preferences. Psychol. Rev. 76, 31–48 (1969)

    Google Scholar 

  68. Waldman, K.: Facebook’s unethical experiment. Slate (2014). http://www.slate.com/articles/health_and_science/science/2014/06/facebook_unethical_experiment_it_made_news_feeds_happier_or_sadder_to_manipulate.html

  69. Wetzels, R., Matzke, D., Lee, M., Rouder, J., Iverson, G., Wagenmakers, E.: Statistical evidence in experimental psychology an empirical comparison using 855 t tests. Perspect. Psychol. Sci. 6(3), 291–298 (2011)

    Google Scholar 

  70. Wilson, T.D., Schooler, J.W.: Thinking too much: introspection can reduce the quality of preferences and decisions. J. Pers. Soc. Psychol. 60(2), 181 (1991)

    Google Scholar 

  71. von Winterfeldt, D., Chung, N.K., Luce, R., Cho, Y.: Tests of consequence monotonicity in decision making under uncertainty. J. Exper. Psychol. Learn. Mem. Cogn. 23, 406–426 (1997)

    Google Scholar 

  72. Zhu, Y., Truszczynski, M.: On optimal solutions of answer set optimization problems. In: Cabalar, P., Son, T.C. (eds.) LPNMR 2013. LNCS, vol. 8148, pp. 556–568. Springer, Heidelberg (2013)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nicholas Mattei .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2015 Springer International Publishing Switzerland

About this paper

Cite this paper

Allen, T.E. et al. (2015). Beyond Theory and Data in Preference Modeling: Bringing Humans into the Loop. In: Walsh, T. (eds) Algorithmic Decision Theory. ADT 2015. Lecture Notes in Computer Science(), vol 9346. Springer, Cham. https://doi.org/10.1007/978-3-319-23114-3_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-23114-3_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-23113-6

  • Online ISBN: 978-3-319-23114-3

  • eBook Packages: Computer ScienceComputer Science (R0)