De-instrumentalizing HCI: Social Psychology, Rapport Formation, and Interactions with Artificial Social Agents

  • Ritwik BanerjiEmail author
Part of the Human–Computer Interaction Series book series (HCIS)


Decisions in designing artificial social interactants to reproduce culturally-specific forms of human sociality evince a range of conceptions of the norms and cognitive processes involved in the human social interactions themselves. Regarding the use of machine learning (ML) in such systems, decisions whether or not to use this approach implicitly presents questions on the nature of the interpersonal adaptation that takes place and indicate a range of conceptions of the values which structure these interactions. In the design of virtual performers of musical free improvisation, several designers assume that the experience of equal partnership between improvisers can only be afforded through deployment of ML in such systems. By contrast, tests of agents not based in ML reveal that human beings experience illusions of “adaptation” in interactions with systems which lack any adaptive capacity. Such results suggest that HCI research with artificial social interactants may be used to raise new questions about the nature of human interaction and interpersonal adaptation in the formation of relationships over time.



Earlier versions of this chapter benefitted from the insightful critiques of Nick Seaver, Zachary Chase Lipton, as well as editors and reviewers of this volume. Financial support for this project came from the Fulbright U.S. Young Journalist’s Fellowship in Germany, the Berlin Program for Advanced German and European Studies, and the Berkeley-Mellon Fellowship.


  1. Amershi S, Cakmak M, Knox WB, Kulesza T (2014) Power to the people: the role of humans in interactive machine learning. AI Mag 35(4):105–120CrossRefGoogle Scholar
  2. Andersen K, Brooks R (2003) Special guest: Rodney brooks. Studio 360: WNYC. National Public Radio, Washington, DCGoogle Scholar
  3. Assayag G, Dubnov S (2004) Using factor oracles for machine improvisation. Soft Comput 8(9):604–610zbMATHGoogle Scholar
  4. Assayag G, Bloch G, Chemillier M, Cont A, Dubnov S (2006) Omax brothers: a dynamic topology of agents for improvization learning. In: Proceedings of the 1st ACM workshop on audio and music computing multimedia, pp 125–132Google Scholar
  5. Assayag G, Bloch G, Cont A, Dubnov S (2010) Interaction with machine improvisation. In: Argamon S, Burns K, Dubnov S (eds) The structure of style. Springer, Berlin and Heidelberg, pp 219–245CrossRefGoogle Scholar
  6. Backstrom MJ (2013). The field of cultural production and the limits of freedom in improvisation. Crit Stud Improv/Études critiques en improvisation 9(1)Google Scholar
  7. Bailey D (1980/1993) Improvisation: its nature and practice in music. Da Capo Press, New YorkGoogle Scholar
  8. Banerji R (2010) Maxine Banerji: the mutually beneficial practices of youth development and interactive systems development. eContact! J Can Electroacoust Commun 12(3).
  9. Banerji R (2012) Maxine’s Turing test: a player-program as co-ethnographer of socio-aesthetic interaction in improvised music. In: Proceedings of the Artificial Intelligence and Interactive Digital Entertainment Conference (AIIDE’12)Google Scholar
  10. Banerji R (2016) Balancing defiance and cooperation: the design and human critique of a virtual free improviser. In: Proceedings of the international computer music conference, pp 49–54Google Scholar
  11. Banerji R, Maxine, the Astromusicologist (2014) Palmer square. pfMentum, San DiegoGoogle Scholar
  12. Bardzell J, Bardzell S (2011) Pleasure is your birthright: digitally enabled designer sex toys as a case of third-wave HCI. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 257–266Google Scholar
  13. Bardzell J, Bardzell S (2016) Humanistic HCI. Interactions 23(2):20–29CrossRefGoogle Scholar
  14. Baumer EP, Silberman M (2011) When the implication is not to design (technology). In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, New York, pp 2271–2274Google Scholar
  15. Beins B (2011) Entwurf und Ereignis (Scheme and Event). In: Beins B, Kesten C, Nauck G, Neumann A (eds) echtzeitmusik berlin: selbstbestimmung einer szene (self-defining a scene). Wolke Verlag, HofheimGoogle Scholar
  16. Bernieri F, Gillis JS (1995) The judgment of rapport: a cross-cultural comparison between Americans and Greeks. J Nonverbal Behav 19(2):115–130CrossRefGoogle Scholar
  17. Bernieri FJ, Gillis JS, Davis JM, Grahe JE (1996) Dyad rapport and the accuracy of its judgment across situations: a lens model analysis. J Pers Soc Psychol 71(1):110CrossRefGoogle Scholar
  18. Blackwell T, Bentley P (2002) Improvised music with swarms. In: Proceedings of the World conference on computational intelligence, pp 1462–1467Google Scholar
  19. Blackwell T, Young M (2004) Self-organised music. Organised Sound 9(2):123–136CrossRefGoogle Scholar
  20. Bødker S (2006) When second wave HCI meets third wave challenges. In: Proceedings of the 4th Nordic conference on Human-computer interaction: changing roles. ACM, New York, pp 1–8Google Scholar
  21. Boehm C (1993) Egalitarian behavior and reverse dominance hierarchy. Curr Anthropol 34(3):227–254CrossRefGoogle Scholar
  22. Boehner K, DePaula R, Dourish P, Sengers P (2007) How emotion is made and measured. Int J Hum Comput Stud 65(4):275–291CrossRefGoogle Scholar
  23. Borgo D (2002) Synergy and Surrealestate: the orderly disorder of free improvisation. Pac Rev Ethnomusicol 10:1–24Google Scholar
  24. Bornstein RF (1989) Exposure and affect: overview and meta-analysis of research, 1968–1987. Psychol Bull 106(2):265–289CrossRefGoogle Scholar
  25. Bown O (2011) Experiments in modular design for the creative composition of live algorithms. Comput Music J 35(3):73–85CrossRefGoogle Scholar
  26. Bown O (2015) Player responses to a live algorithm: conceptualising computational creativity without recourse to human comparisons? In: Proceedings of the sixth international conference on computational creativityGoogle Scholar
  27. Brinner B (1995) Knowing music, making music: Javanese gamelan and the theory of musical competence and interaction. University of Chicago Press, ChicagoGoogle Scholar
  28. Burrell J (2016) How the machine ‘thinks’: understanding opacity in machine learning algorithms. Big Data Soc 3(1):1–12CrossRefGoogle Scholar
  29. Canonne C (2013) Focal points in collective free improvisation. Perspect New Music 51(1):40–55CrossRefGoogle Scholar
  30. Canonne C, Garnier N (2015) Individual decisions and perceived form in collective free improvisation. J New Music Res 44(2):145–167CrossRefGoogle Scholar
  31. Cappella JN (1990) On defining conversational coordination and rapport. Psychol Inq 1(4):303–305CrossRefGoogle Scholar
  32. Carey B (2012) Designing for cumulative interactivity: the _derivations system. In: Proceedings of the international conference on new interfaces for musical expressionGoogle Scholar
  33. Carles P, Comolli J-L (2015 [1971]) Free jazz/Black power. University Press of Mississippi, MississippiGoogle Scholar
  34. Casal DP (2008) Time after time: short-circuiting the emotional distance between algorithm and human improvisors. In: Proceedings of the international computer music conferenceGoogle Scholar
  35. Casal DP, Morelli D (2007). Remembering the future: applications of genetic co-evolution in music improvisation. In: Proceedings of the European conference on artificial lifeGoogle Scholar
  36. Cassell J, Gill AJ, Tepper PA (2007) Coordination in conversation and rapport. In: Proceedings of the workshop on embodied language processing. Association for Computational Linguistics, pp 41–50Google Scholar
  37. Chadabe J (1997) Interaction. In: Electric sound: the past and promise of electronic music. Prentice-Hall, Upper Saddle River, pp 286–323Google Scholar
  38. Chaudron C (1984) The effects of feedback on students’ composition revisions. RELC J 15(2):1–14MathSciNetCrossRefGoogle Scholar
  39. Collins N (2008) Reinforcement learning for live musical agents. In: Proceedings of the International Computer Music Conference (ICMC), BelfastGoogle Scholar
  40. Cont A, Dubnov S, Assayag G (2006) A framework for anticipatory machine improvisation and style imitation. In: Proceedings of the conference on Anticipatory Behavior in Adaptive Learning Systems (ABiALS)Google Scholar
  41. Corbett J (1994) Evan Parker: saxophone botany. In: Extended play: sounding off from John Cage to Dr. Funkenstein. Duke University Press, Duke, pp 201–208Google Scholar
  42. Corbett J (2016) A Listener’s guide to free improvisation. University of Chicago Press, ChicagoCrossRefGoogle Scholar
  43. Crabtree A, Rodden T, Tolmie P, Button G (2009) Ethnography considered harmful. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 879–888Google Scholar
  44. Dourish P (2006) Implications for design. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 541–550Google Scholar
  45. Edwards PN (1997) The closed world: computers and the politics of discourse in Cold War America. MIT Press, Cambridge, MAGoogle Scholar
  46. Eiben AE, Smith JE (2003) Introduction to evolutionary computing. Springer, Berlin/HeidelbergCrossRefGoogle Scholar
  47. Eklund R (2002) Ingressive speech as an indication that humans are talking to humans (and not to machines). In: Proceedings of the 7th International Conference on Spoken Language Processing (ICSLP)Google Scholar
  48. Fails JA, Olsen DR Jr (2003) Interactive machine learning. In: Proceedings of the 8th international conference on Intelligent user interfaces. ACM, New York, pp 39–45Google Scholar
  49. Fiebrink R, Cook PR, Trueman D (2011) Human model evaluation in interactive supervised learning. In: Proceedings of the SIGCHI conference on human factors in computing systems, pp 147–156Google Scholar
  50. Fischlin D, Heble A, Lipsitz G (2013) Take two: sounds of surprise; rights, risks and responsibilities in improvised music. In: The fierce urgency of now: improvisation, rights, and the ethics of co-creation. Duke University Press, Duke, pp 203–219CrossRefGoogle Scholar
  51. Forsythe D (2002) Studying those who study us: an anthropologist in the world of artificial intelligence. Stanford University Press, StanfordGoogle Scholar
  52. Friedman B, Nissenbaum H (1996) Bias in computer systems. ACM Trans Inf Syst (TOIS) 14(3):330–347CrossRefGoogle Scholar
  53. Fuchsberger V, Moser C, Tscheligi M (2012) Values in action (ViA): combining usability, user experience and user acceptance. In: CHI’12 extended abstracts on human factors in computing systems. ACM, New York, pp 1793–1798Google Scholar
  54. Giles H, Coupland N, Coupland I (eds) (1991) Contexts of accommodation: developments in applied sociolinguistics. Cambridge University Press, CambridgeGoogle Scholar
  55. Goffman E (1955) On face-work: an analysis of ritual elements in social interaction. Psychiatry Interpersonal Biol Process 18(3):213–231CrossRefGoogle Scholar
  56. Goffman E (1967) Interaction ritual: essays in face-to-face behavior. Anchor Books, New YorkGoogle Scholar
  57. Grahe JE, Bernieri FJ (1999) The importance of nonverbal cues in judging rapport. J Nonverbal Behav 23(4):253–269CrossRefGoogle Scholar
  58. Gratch J et al (2006) Virtual Rapport. In: Gratch J, Young M, Aylett R, Ballin D, Olivier P (eds) Intelligent Virtual Agents. IVA. Springer, Berlin/Heidelberg, pp 14–27CrossRefGoogle Scholar
  59. Gratch J, Wang N, Gerten, J., Fast, E., & Duffy, R. (2007). Creating rapport with virtual agents. In: Pelachaud C, Martin J-C, André E, Chollet G, Karpouzis K, Pelé D (eds) International Workshop on Intelligent Virtual Agents. Springer, Berlin, pp 125–138Google Scholar
  60. Harrison S, Tatar D, Sengers P (2007) The three paradigms of HCI. alt. chiGoogle Scholar
  61. Helmreich S (2001) After culture: reflections on the apparition of anthropology in artificial life, a science of simulation. Cult Anthropol 16(4):612–627CrossRefGoogle Scholar
  62. Hofstadter DR (1995) The ineradicable Eliza effect and its dangers. In: Hofstadter DR (ed) Fluid concepts and creative analogies: computer models of the fundamental mechanisms of thought. Basic books, New York, pp 155–168Google Scholar
  63. Hsu W (2005) Using timbre in a computer-based improvisation system. In: Proceedings of the international computer music conferenceGoogle Scholar
  64. Hsu W (2010) Strategies for managing timbre and interaction in automatic improvisation systems. Leonardo Music J 20:33–39CrossRefGoogle Scholar
  65. Hsu W, Sosnick M (2009) Evaluating interactive music systems: an HCI approach. In: Proceedings of the international conference on New Interfaces in Musical Expression (NIME)Google Scholar
  66. Huang L, Morency L-P, Gratch J (2011) Virtual rapport 2.0. In: Vilhjálmsson HH, Kopp S, Marsella S, Thórisson KR (eds) Proceedings of the 10th International Conference on Intelligent Virtual Agents. Springer, Berlin/Heidelberg, pp 68–79Google Scholar
  67. Huron DB (2006) Sweet anticipation: music and the psychology of expectation. MIT Press, Cambridge, MAGoogle Scholar
  68. Husserl E (1913/2012) Ideas: general introduction to pure phenomenology (WRB Gibson, Trans.). RoutledgeGoogle Scholar
  69. Kendon A (1990) Conducting interaction: patterns of behavior in focused encounters. Cambridge University Press, CambridgeGoogle Scholar
  70. Kennedy J, Eberhart R (1995) Particle swarm optimization. In: Proceedings of the IEEE conference on neural networks, pp 1942–1948Google Scholar
  71. Kiousis S (2002) Interactivity: a concept explication. New Media Soc 4(3):355–383. Scholar
  72. Kofsky F (1970) Black nationalism and the revolution in music. Pathfinder Press, New YorkGoogle Scholar
  73. Lakin JL, Chartrand TL (2003) Using nonconscious behavioral mimicry to create affiliation and rapport. Psychol Sci 14(4):334–339CrossRefGoogle Scholar
  74. Lange BR (2011) Teaching the ethics of free improvisation. Crit Stud Improvis/Études critiques en improvisation 7(2)Google Scholar
  75. Langton CG (1997) Artificial life: an overview. MIT Press, Cambridge, MAGoogle Scholar
  76. Lewis GE (1993). Voyager (G Lewis, R Mitchell, V. I. V. I. System, Trans.). Avant, JapanGoogle Scholar
  77. Lewis GE (1999) Interacting with latter-day musical automata. Contemp Music Rev 18(3):99–112CrossRefGoogle Scholar
  78. Lewis GE (2000) Too many notes: computers, complexity and culture inVoyager. Leonardo Music J 10:33–39CrossRefGoogle Scholar
  79. Lewis GE (2007) Mobilitas animi: improvising technologies, intending chance. Parallax 13(4):108–122CrossRefGoogle Scholar
  80. Linson A (2014) Investigating the cognitive foundations of collaborative musical free improvisation: experimental case studies using a novel application of the subsumption architecture. Doctoral thesis, The Open UniversityGoogle Scholar
  81. Linson A, Dobbyn C, Lewis GE, Laney R (2015) A Subsumption agent for collaborative free improvisation. Comput Music J 39(4):96–115CrossRefGoogle Scholar
  82. Maatman R, Gratch J, Marsella S (2005) Natural behavior of a listening agent. In: Intelligent virtual agents. Springer, Heidelberg, pp 25–36CrossRefGoogle Scholar
  83. Monson I (1996) Saying something: jazz improvisation and interaction. University of Chicago Press, ChicagoGoogle Scholar
  84. Moreland RL, Zajonc RB (1982) Exposure effects in person perception: familiarity, similarity, and attraction. J Exp Soc Psychol 18(5):395–415CrossRefGoogle Scholar
  85. Morrissey K, Kirakowski J (2013) ‘Realness’ in Chatbots: establishing quantifiable criteria. In: Kurosu M (ed) Proceedings of the 15th international conference on human-computer interaction. Springer, Berlin/Heidelberg, pp 87–96Google Scholar
  86. Nissenbaum H (2001) How computer systems embody values. Computer 34(3):120–119CrossRefGoogle Scholar
  87. Pachet F (2003) The continuator: musical interaction with style. J New Music Res 32(3):333–341. Scholar
  88. Pras A, Schober MF, Spiro N (2017) What about their performance do free jazz improvisers agree upon? A case study [Original Research]. Front Psychol 8(966).
  89. Reis HT, Maniaci MR, Caprariello PA, Eastwick PW, Finkel EJ (2011) Familiarity does indeed promote attraction in live interaction. J Pers Soc Psychol 101(3):557CrossRefGoogle Scholar
  90. Rodriguez AW (2016) Harmolodic pedagogy and the challenge of omni-musicality. Jazz Perspect 9(2):173–192CrossRefGoogle Scholar
  91. Rowe R (1992) Interactive music systems: machine listening and composing. MIT Press, Cambridge, MAGoogle Scholar
  92. Seaver N (2012) Algorithmic recommendations and synaptic functions. Limn 1(2)Google Scholar
  93. Sengers P, Kaye J, Boehner K, Fairbank J, Gay G, Medynskiy Y et al (2004) Culturally embedded computing. IEEE Pervasive Comput 3(1):14–21CrossRefGoogle Scholar
  94. Serenko A, Bontis N (2011) What’s familiar is excellent: the impact of exposure effect on perceived journal quality. J Informet 5(1):219–223CrossRefGoogle Scholar
  95. Seyfert R, Roberge J (2016) Algorithmic cultures: essays on meaning, performance and new technologies. Routledge, New YorkGoogle Scholar
  96. Snyder JK, Fessler DM (2014) Narcotics anonymous: anonymity, admiration, and prestige in an egalitarian community. Ethos 42(4):440–459CrossRefGoogle Scholar
  97. Spellman AB (1966) Four lives in the bebop business. Pantheon Books, New YorkGoogle Scholar
  98. Stanyek J (1999) Articulating intercultural free improvisation: Evan Parker’s synergetics project. Resonance 7(2):44–47Google Scholar
  99. Steinbeck P (2010) “Patience, sincerity, and consistency”: Fred Anderson’s musical and social practices. Crit Stud Improvis/Études critiques en improvisation 6(2)Google Scholar
  100. Suchman L (2006) Human-machine reconfigurations: plans and situated actions. Cambridge University Press, CambridgeCrossRefGoogle Scholar
  101. Sunardi C (2011) Negotiating authority and articulating gender: performer interaction in Malang, East Java. Ethnomusicology 55(1):32–54CrossRefGoogle Scholar
  102. Tickle-Degnen L, Rosenthal R (1990) The nature of rapport and its nonverbal correlates. Psychol Inq 1(4):285–293CrossRefGoogle Scholar
  103. Waterman E (2008) Naked intimacy: eroticism, improvisation, and gender. Crit Stud Improvis/Études critiques en improvisation 4(2)Google Scholar
  104. Weizenbaum J (1976) Computer power and human reason. W. H. Freeman and Company, San FranciscoGoogle Scholar
  105. Wilf E (2013a) Sociable robots, jazz music, and divination: contingency as a cultural resource for negotiating problems of intentionality. Am Ethnol 40(4):605–618CrossRefGoogle Scholar
  106. Wilf E (2013b) Toward an anthropology of computer-mediated, algorithmic forms of sociality. Curr Anthropol 54(6):716–739CrossRefGoogle Scholar
  107. Wilson GB, MacDonald RA (2015) Musical choices during group free improvisation: a qualitative psychological investigation. Psychol Music:1–15Google Scholar
  108. Woodburn J (1982) Egalitarian societies. Man 17(3):431–451CrossRefGoogle Scholar
  109. Yee-King MJ (2007) An automated music improviser using a genetic algorithm driven synthesis engine. In: Giacobini M (ed) Applications of evolutionary computing (EvoWorkshops 2007) proceedings. Springer, Berlin/Heidelberg, pp 567–576Google Scholar
  110. Young M (2008) NN music: improvising with a ‘living’ computer. In: Kronland-Martinet R, Ystad S, Jensen K (eds) Computer music modeling and retrieval: sense of sounds. Springer, Berlin, pp 337–350CrossRefGoogle Scholar
  111. Young M (2010) Identity and intimacy in human-computer improvisation. Leonardo Music J 20(online supplement)Google Scholar
  112. Zajonc RB (1968) Attitudinal effects of mere exposure. J Pers Soc Psychol 9(2, part 2):1–27CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of MusicUniversity of CaliforniaBerkeleyUSA

Personalised recommendations