Autonomous Agents and Multi-Agent Systems

, Volume 32, Issue 1, pp 163–187 | Cite as

Social decisions and fairness change when people’s interests are represented by autonomous agents

  • Celso M. de Melo
  • Stacy Marsella
  • Jonathan Gratch


There has been growing interest on agents that represent people’s interests or act on their behalf such as automated negotiators, self-driving cars, or drones. Even though people will interact often with others via these agent representatives, little is known about whether people’s behavior changes when acting through these agents, when compared to direct interaction with others. Here we show that people’s decisions will change in important ways because of these agents; specifically, we showed that interacting via agents is likely to lead people to behave more fairly, when compared to direct interaction with others. We argue this occurs because programming an agent leads people to adopt a broader perspective, consider the other side’s position, and rely on social norms—such as fairness—to guide their decision making. To support this argument, we present four experiments: in Experiment 1 we show that people made fairer offers in the ultimatum and impunity games when interacting via agent representatives, when compared to direct interaction; in Experiment 2, participants were less likely to accept unfair offers in these games when agent representatives were involved; in Experiment 3, we show that the act of thinking about the decisions ahead of time—i.e., under the so-called “strategy method”—can also lead to increased fairness, even when no agents are involved; and, finally, in Experiment 4 we show that participants were less likely to reach an agreement with unfair counterparts in a negotiation setting. We discuss theoretical implications for our understanding of the nature of people’s social behavior with agent representatives, as well as practical implications for the design of agents that have the potential to increase fairness in society.


Agent representatives Decision making Fairness Strategy method 



This work is supported by the National Science Foundation, under Grant BCS-1419621, and the Air Force Office of Scientific Research, under Grant FA9550-14-1-0364. The content does not necessarily reflect the position or the policy of any Government, and no official endorsement should be inferred.


  1. 1.
    Chugh, D., Bazerman, M., & Banaji, M. (2005). Bounded ethicality as a psychological barrier to recognizing conflicts of interest. In D. Moore, G. Loewenstein, D. Cain, & M. Bazerman (Eds.), Conflicts of interest: Challenges and solutions in business, law, medicine, and public policy (pp. 74–95). Cambridge: Cambridge University Press.CrossRefGoogle Scholar
  2. 2.
    Haslam, N. (2006). Dehumanization: An integrative review. Personality and Social Psychology Review, 10, 252–264.CrossRefGoogle Scholar
  3. 3.
    Trope, Y., & Nira, L. (2010). Construal-level theory of psychological distance. Psychological Review, 117, 440–463.CrossRefGoogle Scholar
  4. 4.
    Pruitt, D., & Kimmel, M. (1977). Twenty years of experimental gaming: Critique, synthesis, and suggestions for the future. Annual Review of Psychology, 28, 363–392.CrossRefGoogle Scholar
  5. 5.
    Kollock, P. (1998). Social dilemmas: The anatomy of cooperation. Annual Review of Sociology, 24, 183–214.CrossRefGoogle Scholar
  6. 6.
    Lin, R., & Kraus, S. (2010). Can automated agents proficiently negotiate with humans? Communications of the ACM, 53, 78–88.CrossRefGoogle Scholar
  7. 7.
    Jennings, N., Faratin, P., Lomuscio, A., Parsons, S., Wooldridge, M., et al. (2001). Automated negotiation: Prospects, methods and challenges. Group Decision and Negotiation, 10, 199–215.CrossRefGoogle Scholar
  8. 8.
    Davenport, T., & Harris, J. (2005). Automated decision making comes of age. MIT Sloan Management Review, 46, 83–89.Google Scholar
  9. 9.
    Fehr, E., & Schmidt, K. (1999). A theory of fairness, competition, and cooperation. The Quarterly Journal of Economics, 114, 817–868.CrossRefMATHGoogle Scholar
  10. 10.
    Royzman, E., & Baron, J. (2002). The preference for indirect harm. Social Justice Research, 15, 165–184.CrossRefGoogle Scholar
  11. 11.
    Hoffman, E., McCabe, K., & Smith, V. (1996). Social distance and other-regarding behavior in dictator games. American Economic Review, 86, 653–660.Google Scholar
  12. 12.
    Goeree, J., McConnell, M., Mitchell, T., Tromp, T., & Yariv, L. (2010). The 1/D law of giving. American Economic Journal: Microeconomics, 2, 183–203.Google Scholar
  13. 13.
    Yu, R., Hu, P., & Zhang, P. (2015). Social distance and anonymity modulate fairness consideration: An ERP study. Science Reports, 5, 1–12.Google Scholar
  14. 14.
    Pronin, E., Olivola, C., & Kennedy, K. (2008). Doing unto future selves as you would do unto others: Psychological distance and decision making. Personality and Social Psychology Bulletin, 34, 224–236.CrossRefGoogle Scholar
  15. 15.
    Nowak, M., & May, R. (1992). Evolutionary games and spatial chaos. Nature, 359, 826–829.CrossRefGoogle Scholar
  16. 16.
    Reeves, B., & Nass, C. (1996). The media equation: How people treat computers, television, and new media like real people and places. New York: Cambridge University Press.Google Scholar
  17. 17.
    Lin, R., Kraus, S., Oshrat, Y., & Gal, Y. (2010). Facilitating the evaluation of automated negotiators using peer designed agents. In Proceedings of the 24th AAAI conference on artificial intelligence (AAAI’10).Google Scholar
  18. 18.
    Chalamish, M., Sarne, D., & Lin, R. (2013). Enhancing parking simulations using peer-designed agents. IEEE Transactions on Intelligent Transportation Systems, 14(1), 492–498.CrossRefGoogle Scholar
  19. 19.
    Grosz, B., Kraus, S., & Talman, S. (2004). The influence of social dependencies on decision-making: Initial investigations with a new game. In Proceedings of the international conference on autonomous agents and multi-agent systems (AAMAS’04).Google Scholar
  20. 20.
    Elmalech, A., & Sarne, D. (2013). Evaluating the applicability of peer-designed agents for mechanism evaluation. Web Intelligence and Agent Systems: An International Journal, 12(2), 171–191.Google Scholar
  21. 21.
    Lin, R, Oshrat, Y., & Kraus, S. (2009). Investigating the benefits of automated negotiations in enhancing negotiation skills of people. In Proceedings of the international conference on autonomous agents and multi-agent systems (AAMAS’09).Google Scholar
  22. 22.
    Lin, R., Gal, Y., Kraus, S., & Mazliah, Y. (2014). Training with automated agents improves people’s behavior in negotiation and coordination tasks. Decision Support Systems, 60, 1–9.CrossRefGoogle Scholar
  23. 23.
    Elmalech, A., Sarne, D., & Agmon, N. (2014). Can agent development affect developer’s strategy? In Proceedings of the 24th AAAI conference on artificial intelligence (AAAI’10).Google Scholar
  24. 24.
    Nass, C., & Moon, Y. (2000). Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56, 81–103.CrossRefGoogle Scholar
  25. 25.
    Nass, C., Moon, Y., & Carney, P. (1999). Are people polite to computers? Responses to computer-based interviewing systems. Journal of Applied Psychology, 29, 1093–1110.Google Scholar
  26. 26.
    Nass, C., Fogg, B., & Moon, Y. (1996). Can computers be teammates? International Journal of Human–Computer Studies, 45, 669–678.CrossRefGoogle Scholar
  27. 27.
    Nass, C., Moon, Y., & Green, N. (1997). Are computers gender-neutral? Gender stereotypic responses to computers. Journal of Applied Social Psychology, 27, 864–876.CrossRefGoogle Scholar
  28. 28.
    Nass, C., Isbister, K., & Lee, E.-J. (2000). Truth is beauty: Researching conversational agents. In J. Cassell (Ed.), Embodied conversational agents (pp. 374–402). Cambridge, MA: MIT Press.Google Scholar
  29. 29.
    Gratch, J., Wang, N., Gerten, J., Fast, E., & Duffy, R. (2007). Creating rapport with virtual agents. In C. Pelachaud, et al. (Eds.), Intelligent virtual agents (pp. 125–138). Berlin: Springer.CrossRefGoogle Scholar
  30. 30.
    Riek, L., Paul, P., & Robinson, P. (2010). When my robot smiles at me: Enabling human–robot rapport via real-time head gesture mimicry. Journal on Multimodal User Interfaces, 3, 99–108.CrossRefGoogle Scholar
  31. 31.
    de Melo, C., Carnevale, P., Read, S., & Gratch, J. (2014). Reading people’s minds from emotion expressions in interdependent decision making. Journal of Personality and Social Psychology, 106, 73–88.CrossRefGoogle Scholar
  32. 32.
    Salem, M., Ziadee, M., & Sakr, M. (2014). Marhaba, how may I help you? Effects of politeness and culture on robot acceptance and anthropomorphization. In Proceedings of the 2014 ACM/IEEE international conference on human–robot interaction.Google Scholar
  33. 33.
    Eyssel, F., & Kuchenbrandt, D. (2012). Social categorization of social robots: Anthropomorphism as a function of robot group membership. British Journal of Social Psychology, 51(4), 724–731.CrossRefGoogle Scholar
  34. 34.
    Gray, H., Gray, K., & Wegner, D. (2007). Dimensions of mind perception. Science, 315, 619.CrossRefGoogle Scholar
  35. 35.
    Blascovich, J., Loomis, J., Beall, A., Swinth, K., Hoyt, C., et al. (2002). Immersive virtual environment technology as a methodological tool for social psychology. Psychological Inquiry, 13, 103–124.CrossRefGoogle Scholar
  36. 36.
    Epley, N., Waytz, A., & Cacioppo, J. (2007). On seeing human: A three-factor theory of anthropomorphism. Psychological Review, 114, 864–886.CrossRefGoogle Scholar
  37. 37.
    Waytz, A., Gray, K., Epley, N., & Wegner, D. (2010). Causes and consequences of mind perception. Trends Cognitive Science, 14, 383–388.CrossRefGoogle Scholar
  38. 38.
    Gallagher, H., Anthony, J., Roepstorff, A., & Frith, C. (2002). Imaging the intentional stance in a competitive game. NeuroImage, 16, 814–821.CrossRefGoogle Scholar
  39. 39.
    McCabe, K., Houser, D., Ryan, L., Smith, V., & Trouard, T. (2001). A functional imaging study of cooperation in two-person reciprocal exchange. Proceedings of the National Academy of Sciences USA, 98, 11832–11835.CrossRefGoogle Scholar
  40. 40.
    Rilling, J., Gutman, D., Zeh, T., Pagnoni, G., Berns, G., et al. (2002). A neural basis for social cooperation. Neuron, 35, 395–405.CrossRefGoogle Scholar
  41. 41.
    Kircher, T., Blümel, I., Marjoram, D., Lataster, T., Krabbendam, L., et al. (2009). Online mentalising investigated with functional MRI. Neuroscience Letters, 454, 176–181.CrossRefGoogle Scholar
  42. 42.
    de Melo, C., Carnevale, P., & Gratch, J. (2014). Humans vs. computers: Impact of emotion expressions on people’s decision making. IEEE Transactions on Affective Computing, 6, 127–136.CrossRefGoogle Scholar
  43. 43.
    de Melo, C., Marsella, S., & Gratch, J. (2016). People don’t feel guilty about exploiting machines. ACM Transactions on Computer–Human Interaction, 23(2), 8.CrossRefGoogle Scholar
  44. 44.
    Sanfey, A., Rilling, J., Aronson, J., Nystrom, L., & Cohen, J. (2003). The neural basis of economic decision-making in the ultimatum game. Science, 300, 1755–1758.CrossRefGoogle Scholar
  45. 45.
    Cormier, D., Newman, G., Nakane, M., Young, J., & Durocher, S. (2013). Would you do as a robot commands? An obedience study for human–robot interaction. In Proceedings of the 1st international conference on human–agent interaction (iHAI).Google Scholar
  46. 46.
    Lucas, G., Gratch, J., King, A., & Morency, L.-P. (2014). It’s only a computer: Virtual humans increase willingness to disclose. Computers in Human Behavior, 37, 94–100.CrossRefGoogle Scholar
  47. 47.
    Güth, W., & Tietz, R. (1990). Ultimatum bargaining behavior: A survey and comparison of experimental results. Journal of Economic Psychology, 11, 417–449.CrossRefGoogle Scholar
  48. 48.
    Oosterbeek, H., Sloof, R., & Van de Kuilen, G. (2004). Cultural differences in ultimatum game experiments: Evidence from a meta-analysis. Experimental Economics, 7, 171–188.CrossRefMATHGoogle Scholar
  49. 49.
    Blount, S., & Bazerman, M. (1996). The inconsistent evaluation of absolute versus comparative payoffs in labor supply and bargaining. Journal of Economic Behavior and Organization, 30, 227–240.CrossRefGoogle Scholar
  50. 50.
    Brandts, J., & Charness, G. (2000). Hot vs. cold: Sequential responses and preference stability in experimental games. Experimental Economics, 2, 227–238.CrossRefMATHGoogle Scholar
  51. 51.
    Rauhut, H., & Winter, F. (2010). A sociological perspective on measuring social norms by means of strategy method experiments. Social Science Research, 39, 1181–1194.CrossRefGoogle Scholar
  52. 52.
    Güth, W., Schmittberger, R., & Schwarze, B. (1982). An experimental analysis of ultimatum bargaining. Journal of Economic Behavior and Organization, 3, 367–388.CrossRefGoogle Scholar
  53. 53.
    Henrich, J., Boyd, R., Bowles, S., Camerer, C., Fehr, E., et al. (2001). In search of homo economicus: Behavioral experiments in 15 small-scale societies. American Economic Review, 91, 73–78.CrossRefGoogle Scholar
  54. 54.
    Camerer, C., & Thaler, R. (1995). Ultimatums, dictators, and manners. Journal of Economic Perspectives, 9, 209–219.CrossRefGoogle Scholar
  55. 55.
    Yamagishi, T., Horita, H., Shinada, M., Tanida, S., & Cook, K. (2009). The private rejection of unfair offers and emotional commitment. Proceedings of the National Academy of Science USA, 106, 11520–11523.CrossRefGoogle Scholar
  56. 56.
    Blount, S. (1995). When social outcomes aren’t fair: The effect of causal attributions on preferences. Organizational Behavior and Human Decision Processes, 63, 131–144.CrossRefGoogle Scholar
  57. 57.
    Rabin, M. (1993). Incorporating fairness into game theory and economics. American Economic Review, 83, 1281–1302.Google Scholar
  58. 58.
    Falk, A., & Fischbacher, U. (2006). A theory of reciprocity. Game and Economic Behavior, 54, 293–315.MathSciNetCrossRefMATHGoogle Scholar
  59. 59.
    Starmer, C., & Sugden, R. (1991). Does the random-lottery incentive system elicit true preferences? An experimental investigation. The American Economic Review, 81, 971–978.Google Scholar
  60. 60.
    Camerer, C., & Hogarth, R. (1999). The effects of financial incentives in experiments: A review and capital-labor-production framework. Journal of Risk and Uncertainty, 19, 7–42.CrossRefMATHGoogle Scholar
  61. 61.
    Adler, R., Iacobelli, F., & Gutstein, Y. (2016). Are you convinced? A Wizard of Oz study to test emotional vs. rational persuasion strategies in dialogues. Computers in Human Behavior, 57, 75–81.CrossRefGoogle Scholar
  62. 62.
    DeVault, D., Artstein, R., Benn, G., Dey, T., Fast, E., et al. (2014). Simsensei Kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the international conference on autonomous agents and multi-agent systems (AAMAS’14).Google Scholar
  63. 63.
    Hersch, G. (2015). Experimental economics’ inconsistent ban on deception. Studies in History and Philosophy of Science, 52, 13–19.CrossRefGoogle Scholar
  64. 64.
    Paolacci, G., Chandler, J., & Ipeirotis, P. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.Google Scholar
  65. 65.
    Charness, G., & Gneezy, U. (2008). What’s in a name? Anonymity and social distance in dictator and ultimatum games. Journal of Economic Behavior and Organization, 68, 29–35.CrossRefGoogle Scholar
  66. 66.
    Bogardus, E. (1926). Social distance in the city. Proceedings Public American Society Society, 20, 40–46.Google Scholar
  67. 67.
    Van Kleef, G., De Dreu, C., & Manstead, A. (2004). The interpersonal effects of anger and happiness in negotiations. Journal of Personality and Social Psychology, 86, 57–76.CrossRefGoogle Scholar
  68. 68.
    de Melo, C., Carnevale, P., & Gratch, J. (2011). The effect of expression of anger and happiness in computer agents on negotiations with humans. In Proceedings of the international conference on autonomous agents and multi-agent systems (AAMAS’11).Google Scholar
  69. 69.
    Trope, Y., & Liberman, N. (2003). Temporal construal. Psychological Review, 110, 403–421.CrossRefGoogle Scholar
  70. 70.
    Liberman, N., & Trope, Y. (2010). Construal-level theory of psychological distance. Psychological Review, 117, 440–463.CrossRefGoogle Scholar
  71. 71.
    Agerström, J., & Björklund, F. (2009). Temporal distance and moral concerns: Future morally questionable behavior is perceived as more wrong and evokes stronger prosocial intentions. Basic Applied Social Psychology, 31, 49–59.CrossRefGoogle Scholar
  72. 72.
    Agerström, J., & Björklund, F. (2009). Moral concerns are greater for temporally distant events and are moderated by value strength. Social Cognition, 27, 261–282.CrossRefGoogle Scholar
  73. 73.
    Gong, H., & Medin, D. (2012). Construal levels and moral judgment: Some complications. Judgment and Decision Making, 7, 628–638.Google Scholar
  74. 74.
    Kortenkamp, K., & Moore, C. (2006). Time, uncertainty, and individual differences in decisions to cooperate in resource dilemmas. Personality and Social Psychology Bulletin, 32, 603–615.CrossRefGoogle Scholar
  75. 75.
    Henderson, M., Trope, Y., & Carnevale, P. (2006). Negotiation from a near and distant time perspective. Journal of Personality and Social Psychology, 91, 712–729.CrossRefGoogle Scholar
  76. 76.
    De Dreu, C., Giacomantonio, M., Shalvi, S., & Sligte, D. (2009). Getting stuck or stepping back: Effects of obstacles in the negotiation of creative solutions. Journal of Experimental Social Psychology, 45, 542–548.CrossRefGoogle Scholar
  77. 77.
    Rand, D., & Nowak, M. (2013). Human cooperation. Trends in Cognitive Science, 17, 413–425.CrossRefGoogle Scholar
  78. 78.
    Nowak, M., & Sigmund, K. (2005). Evolution of indirect reciprocity. Nature, 437, 1291–1298.CrossRefGoogle Scholar
  79. 79.
    Andreoni, J., & Bernheim, B. (2009). Social image and the 50–50 norm. A theoretical and experimental analysis of audience effects. Econometrica, 77, 1607–1636.MathSciNetCrossRefMATHGoogle Scholar
  80. 80.
    Axelrod, R., & Hamilton, W. (1981). The evolution of cooperation. Science, 211, 1390–1396.MathSciNetCrossRefMATHGoogle Scholar
  81. 81.
    Carnevale, P., Puitt, D., & Britton, S. (1979). Looking tough: The negotiator under constituent surveillance. Personality and Social Psychology Bulletin, 5(1), 118–121.CrossRefGoogle Scholar
  82. 82.
    Azaria, A., Richardson, A., & Rosenfeld, A. (2016). Autonomous agents and human cultures in the trust–revenge game. Journal of Autonomous Agents and Multi-agent Systems, 30, 486–505.CrossRefGoogle Scholar
  83. 83.
    Fulmer, C., & Gelfand, M. (2015). Trust after violations: Are collectivists more or less forgiving? Journal of Trust Research, 5, 109–131.CrossRefGoogle Scholar
  84. 84.
    Nouri, E., & Traum, D. (2014). Cultural differences in playing repeated ultimatum game online with virtual humans. In Proceedings of system sciences conference (HICSS).Google Scholar
  85. 85.
    Breazeal, C. (2003). Toward sociable robots. Robotics and Autonomous Systems, 42, 167–175.CrossRefMATHGoogle Scholar
  86. 86.
    Dresner, K., & Stone, P. (2007). Sharing the road: Autonomous vehicles meet human drivers. In Proceedings of the international joint conference on artificial intelligence (IJCAI’07) (pp. 1263–1268).Google Scholar
  87. 87.
    Gupte, S. (2012) A survey of quadrotor Unmanned Aerial Vehicles. In Proceedings of IEEE Southeastcon (pp. 1–6).Google Scholar
  88. 88.
    Arkin, R. (2009). Ethical robots in warfare. IEEE Technology and Society Magazine, 28, 30–33.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2017

Authors and Affiliations

  • Celso M. de Melo
    • 1
  • Stacy Marsella
    • 2
  • Jonathan Gratch
    • 1
  1. 1.Institute for Creative TechnologiesUniversity of Southern CaliforniaPlaya VistaUSA
  2. 2.College of Computer and Information ScienceNortheastern UniversityBostonUSA

Personalised recommendations