Skip to main content
SpringerLink
Log in

Should Conversational Agents Care About Our Gender Identity?

  • Conference paper
  • First Online:
Chatbot Research and Design (CONVERSATIONS 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14524))

Included in the following conference series:

  • 345 Accesses

Abstract

Chatbots are increasing their relevance in the global market. Nonetheless, companies are still struggling to develop chatbots that provide their clients with an optimal experience and, so far, few insights have been obtained to improve their related User Experience (UX). This study investigates whether chatbots that consider users’ gender identity result in an improved UX, and whether sensitivity towards this social matter moderates this relationship. Therefore, a one-factor within-subjects experiment was conducted, involving participants interacting with two versions of a buying-assistant chatbot. In one condition, the chatbot provided a more personalised interaction by presuming the user’s gender identity by their sex assigned at birth and conversing with them using a gender-specific language (e.g., ‘women’s clothing’, ‘men’s clothing’). The second condition, instead, used a gender-neutral approach, using gender-neutral language (e.g., ‘clothing’). We hypothesised that the chatbot presuming a cisgender identity of the user would result in a higher UX than the gender-neutral chatbot, and that this effect would be more substantial for users who score low on gender sensitivity. UX was measured by evaluating the chatbot’s Usability, Empathy and Supportiveness. Results indicate that the chatbot presuming a cisgender identity was considered significantly more usable and supportive, but less empathetic. A moderation effect of gender sensitivity on the evaluation of the chatbots was not found.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Følstad, A., Brandtzæg, P.B.: Chatbots and the new world of HCI. Interactions. 24, 38–42 (2017). https://doi.org/10.1145/3085558

    Article  Google Scholar 

  2. Van Pinxteren, M.M.E., Pluymaekers, M., Lemmink, J.G.A.M.: Human-like communication in conversational agents: a literature review and research agenda. JOSM. 31, 203–225 (2020). https://doi.org/10.1108/JOSM-06-2019-0175

    Article  Google Scholar 

  3. Grudin, J., Jacques, R.: Chatbots, Humbots, and the Quest for Artificial General Intelligence. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–11. ACM, Glasgow, Scotland, UK (2019)

    Google Scholar 

  4. Juniper Research: Chatbots to facilitate $142 billion of retail spend by 2024, driven by omnichannel strategies. https://www.juniperresearch.com/press/chatbots-to-facilitate-$142-billion-of-retail

    Google Scholar 

  5. Gnewuch, U., Morana, S., Maedche, A.: Towards designing cooperative and social conversational agents for customer service. Presented at the 38th International Conference on Information Systems, Seoul (2017)

    Google Scholar 

  6. Ait Baha, T., El Hajji, M., Es-Saady, Y., Fadili, H.: The power of personalization: a systematic review of personality-adaptive chatbots. Sn Comput. Sci. 4, 661 (2023). https://doi.org/10.1007/s42979-023-02092-6

  7. Daniel, F., Matera, M., Zaccaria, V., Dell’Orto, A.: Toward truly personal chatbots: on the development of custom conversational assistants. In: Proceedings of the 1st International Workshop on Software Engineering for Cognitive Services. pp. 31–36. ACM, Gothenburg Sweden (2018)

    Google Scholar 

  8. Følstad, A., Skjuve, M.: Chatbots for customer service: user experience and motivation. In: Proceedings of the 1st International Conference on Conversational User Interfaces. pp. 1–9. ACM, Dublin Ireland (2019)

    Google Scholar 

  9. The Deloitte Consumer Review: Made-to-order: The rise of mass personalisation. Deloitte (2015)

    Google Scholar 

  10. Chakrabarti, C., Luger, G.F.: Artificial conversations for customer service chatter bots: Architecture, algorithms, and evaluation metrics. Expert Syst. Appl. 42, 6878–6897 (2015). https://doi.org/10.1016/j.eswa.2015.04.067

    Article  Google Scholar 

  11. Ling, E.C., Tussyadiah, I., Tuomi, A., Stienmetz, J., Ioannou, A.: Factors influencing users’ adoption and use of conversational agents: A systematic review. Psychol. Mark. 38, 1031–1051 (2021). https://doi.org/10.1002/mar.21491

    Article  Google Scholar 

  12. Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., Weston, J.: Personalizing Dialogue Agents: I have a dog, do you have pets too? In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2204–2213. Association for Computational Linguistics, Melbourne, Australia (2018)

    Google Scholar 

  13. Hu, T., et al.: Touch Your Heart: A Tone-aware Chatbot for Customer Care on Social Media User Experience - Interactions (2018). http://arxiv.org/abs/1803.02952

    Google Scholar 

  14. Stokoe, E.H., Smithson, J.: Making gender relevant: conversation analysis and gender categories in interaction. Discourse Soc. 12, 217–244 (2001). https://doi.org/10.1177/0957926501012002005

    Article  Google Scholar 

  15. Kitzinger, C.: Doing gender: a conversation analytic perspective. Gend. Soc. 23, 94–98 (2009). https://doi.org/10.1177/0891243208326730

    Article  Google Scholar 

  16. Thüring, M., Mahlke, S.: Usability, aesthetics and emotions in human–technology interaction. Int. J. Psychol. 42, 253–264 (2007). https://doi.org/10.1080/00207590701396674

    Article  Google Scholar 

  17. Bevan, N., Kirakowski, J., Maissel, J.: What is Usability? In: Human Aspects in Computing: Proceedings of the 4th International Conference on HCI, Stuttgart, Germany (1991)

    Google Scholar 

  18. Broadbent, E., et al.: Robots with display screens: a robot with a more humanlike face display is perceived to have more mind and a better personality. PLoS ONE 8, e72589 (2013). https://doi.org/10.1371/journal.pone.0072589

    Article  Google Scholar 

  19. Nowak, K.L., Rauh, C.: The influence of the avatar on online perceptions of anthropomorphism, androgyny, credibility, homophily, and attraction. J Comp Mediated Comm. 11, 153–178 (2005). https://doi.org/10.1111/j.1083-6101.2006.tb00308.x

    Article  Google Scholar 

  20. Araujo, T.: Living up to the chatbot hype: the influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions. Comput. Hum. Behav. 85, 183–189 (2018). https://doi.org/10.1016/j.chb.2018.03.051

    Article  Google Scholar 

  21. Reeves, B., Nass, C.: How People Treat Computers, Television, and New Media Like Real People and Places. Cambridge University Press, Center for the Study of Language and Information (1996)

    Google Scholar 

  22. Bainbridge, W.A., Hart, J., Kim, E.S., Scassellati, B.: The effect of presence on human-robot interaction. In: RO-MAN 2008 - The 17th IEEE International Symposium on Robot and Human Interactive Communication, pp. 701–706. IEEE, Munich, Germany (2008)

    Google Scholar 

  23. Lee, S., Choi, J.: Enhancing user experience with conversational agent for movie recommendation: effects of self-disclosure and reciprocity. Int. J. Hum. Comput. Stud. 103, 95–105 (2017). https://doi.org/10.1016/j.ijhcs.2017.02.005

    Article  Google Scholar 

  24. Schuetzler, R., Grimes, M., Giboney, J., Buckman, J.: Facilitating natural conversational agent interactions: lessons from a deception experiment. In: Human Computer Interaction, Auckland (2014)

    Google Scholar 

  25. Mahlke, S.: User Experience of Interaction with Technical Systems (2008)

    Google Scholar 

  26. Kaptein, M., Markopoulos, P., De Ruyter, B., Aarts, E.: Two acts of social intelligence: the effects of mimicry and social praise on the evaluation of an artificial agent. AI & Soc. 26, 261–273 (2011). https://doi.org/10.1007/s00146-010-0304-4

    Article  Google Scholar 

  27. Laban, G., Araujo, T.: The effect of personalization techniques in users’ perceptions of conversational recommender systems. In: Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, pp. 1–3. ACM, Virtual Event Scotland UK (2020)

    Google Scholar 

  28. Laban, G., Araujo, T.: Don’t Take it Personally: resistance to individually targeted recommendations from conversational recommender agents. In: Proceedings of the 10th International Conference on Human-Agent Interaction, pp. 57–66. ACM, Christchurch New Zealand (2022)

    Google Scholar 

  29. Trivedi, J.: Examining the customer experience of using banking chatbots and its impact on brand love: the moderating role of perceived risk. J. Internet Commerce 18, 91–111 (2019). https://doi.org/10.1080/15332861.2019.1567188

    Article  Google Scholar 

  30. Przegalinska, A., Ciechanowski, L., Stroz, A., Gloor, P., Mazurek, G.: In bot we trust: a new methodology of chatbot performance measures. Bus. Horiz. 62, 785–797 (2019). https://doi.org/10.1016/j.bushor.2019.08.005

    Article  Google Scholar 

  31. Hassenzahl, M., Tractinsky, N.: User experience - a research agenda. Behav. Inform. Technol. 25, 91–97 (2006). https://doi.org/10.1080/01449290500330331

    Article  Google Scholar 

  32. Plocher, T.A., Garg, C., Chestnut, J.: Connecting culture, user characteristics and user interface design. In: On Human-Computer Interaction: Ergonomics and User Interfaces, pp. 803–807. Lawrence Erlbaum, Munich, Germany (1999)

    Google Scholar 

  33. Wood, W., Eagly, A.H.: Gender identity. In: Handbook of Individual Differences in Social Behavior, pp. 109–125. The Guilford Press (2009)

    Google Scholar 

  34. Jenkins, K.: Amelioration and inclusion: gender identity and the concept of woman. Ethics 126, 394–421 (2016). https://doi.org/10.1086/683535

    Article  Google Scholar 

  35. Jenkins, K.: Toward an Account of Gender Identity. Ergo Open Access J. Philos. 5 (2018). https://doi.org/10.3998/ergo.12405314.0005.027

  36. Yen, C., Chiang, M.-C.: Trust me, if you can: a study on the factors that influence consumers’ purchase intention triggered by chatbots based on brain image evidence and self-reported assessments. Behav. Inform. Technol. 40, 1177–1194 (2021). https://doi.org/10.1080/0144929X.2020.1743362

    Article  Google Scholar 

  37. Tsai, W.-H.S., Liu, Y., Chuan, C.-H.: How chatbots’ social presence communication enhances consumer engagement: the mediating role of parasocial interaction and dialogue. JRIM. 15, 460–482 (2021). https://doi.org/10.1108/JRIM-12-2019-0200

    Article  Google Scholar 

  38. Müller, L., Mattke, J., Maier, C., Weitzel, T., Graser, H.: Chatbot acceptance: a latent profile analysis on individuals’ trust in conversational agents. In: Proceedings of the 2019 on Computers and People Research Conference, pp. 35–42. ACM, Nashville TN USA (2019)

    Google Scholar 

  39. Bartneck, C., Kulić, D., Croft, E., Zoghbi, S.: Measurement instruments for the anthropomorphism, animacy, likeability, perceived Intelligence, and Perceived Safety of Robots. Int. J. Soc. Robot. 1, 71–81 (2009). https://doi.org/10.1007/s12369-008-0001-3

    Article  Google Scholar 

  40. Ren, R., Castro, J.W., Acuña, S.T., De Lara, J.: Evaluation techniques for chatbot usability: a systematic mapping study. Int. J. Soft. Eng. Knowl. Eng. 29, 1673–1702 (2019). https://doi.org/10.1142/S0218194019400163

    Article  Google Scholar 

  41. Laranjo, L., et al.: Conversational agents in healthcare: a systematic review. J. Am. Med. Inform. Assoc. 25, 1248–1258 (2018). https://doi.org/10.1093/jamia/ocy072

    Article  Google Scholar 

  42. Kocaballi, A.B., et al.: The personalization of conversational agents in health care: systematic review. J. Med. Internet Res. 21, e15360 (2019). https://doi.org/10.2196/15360

    Article  Google Scholar 

  43. United Nations: Guidelines for gender-inclusive language in English. https://www.un.org/en/gender-inclusive-language/guidelines.shtml

  44. Kocaballi, A.B., Laranjo, L., Coiera, E.: Understanding and measuring user experience in conversational interfaces. Interact. Comput. 31, 192–207 (2019). https://doi.org/10.1093/iwc/iwz015

    Article  Google Scholar 

  45. Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., Mctear, M.: Usability testing of a healthcare chatbot: Can we use conventional methods to assess conversational user interfaces? In: Proceedings of the 31st European Conference on Cognitive Ergonomics, pp. 207–214. ACM, BELFAST United Kingdom (2019)

    Google Scholar 

  46. Rubin, R.B., Martin, M.M.: Development of a measure of interpersonal communication competence. Commun. Res. Rep. 11, 33–44 (1994). https://doi.org/10.1080/08824099409359938

    Article  Google Scholar 

  47. Habarth, J.M.: Development of the heteronormative attitudes and beliefs scale. Psychol. Sexuality 6, 166–188 (2015). https://doi.org/10.1080/19419899.2013.876444

    Article  Google Scholar 

  48. R Core Team R: A language and environment for statistical computing (2022). https://www.R-project.org/

  49. Figner, B., et al.: Standard Operating Procedures For Using Mixed-Effects Models (2020). http://decision-lab.org/wp-content/uploads/2020/07/SOP_Mixed_Models_D2P2_v1_0_0.pdf

  50. Lüdecke, D., Ben-Shachar, M., Patil, I., Waggoner, P., Makowski, D.: Performance: an R package for assessment, comparison and testing of statistical models. JOSS 6, 3139 (2021). https://doi.org/10.21105/joss.03139

  51. Dixon, M., Freeman, K., Toman, N.: Stop Trying to Delight Your Customers (2010). https://hbr.org/2010/07/stop-trying-to-delight-your-customers

  52. Haugeland, I.K.F., Følstad, A., Taylor, C., Bjørkli, C.A.: Understanding the user experience of customer service chatbots: an experimental study of chatbot interaction design. Int. J. Hum. Comput. Stud. 161, 102788 (2022)

    Google Scholar 

  53. Brandtzæg, P.B., Skjuve, M., Kristoffer Dysthe, K.K., Følstad, A.: When the social becomes non-human: young people’s perception of social support in chatbots. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–13. ACM, Yokohama Japan (2021)

    Google Scholar 

  54. Litman, D.J., Pan, S.: Designing and evaluating an adaptive spoken dialogue system. User Model. User-Adap. Inter.Adap. Inter. 12, 111–137 (2002). https://doi.org/10.1023/A:1015036910358

    Article  Google Scholar 

  55. Go, E., Sundar, S.S.: Humanizing chatbots: the effects of visual, identity and conversational cues on humanness perceptions. Comput. Hum. Behav.. Hum. Behav. 97, 304–316 (2019). https://doi.org/10.1016/j.chb.2019.01.020

    Article  Google Scholar 

  56. Følstad, A., Nordheim, C.B., Bjørkli, C.A.: What makes users trust a chatbot for customer service? an exploratory interview study. In: Tiropanis, T., Vakali, A., Sartori, L., Burnap, P. (eds.): INSCI 2015. LNCS, vol. 9089. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18609-2

  57. Liu, X., Wang, Q.: Study on Application of a quantitative evaluation approach for software architecture adaptability. In: Fifth International Conference on Quality Software (QSIC 2005), pp. 265–272. IEEE, Melbourne, Australia (2005)

    Google Scholar 

  58. Singmann, H., Bolker, B., Westfall, J., Aust, F., Ben-Shachar, M.S.: afex: Analysis of Factorial Experiments (2022). https://CRAN.R-project.org/package=afex

Download references

Author information

Authors and Affiliations

  1. Faculty of Political and Social Sciences, Gent University, Gent, Belgium

    Arturo Cocchi

  2. Communication and Media Group, Radboud University, Nijmegen, The Netherlands

    Tibor Bosse & Michelle van Pinxteren

Authors
  1. Arturo Cocchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tibor Bosse
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michelle van Pinxteren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arturo Cocchi .

Editor information

Editors and Affiliations

  1. SINTEF, Oslo, Norway

    Asbjørn Følstad

  2. University of Amsterdam, Amsterdam, The Netherlands

    Theo Araujo

  3. CERTH-ITI, Thessaloniki, Greece

    Symeon Papadopoulos

  4. Department of Computer Science, Durham University, Durham, UK

    Effie L.-C. Law

  5. Design Informatics, University of Edinburgh, Edinburgh, UK

    Ewa Luger

  6. Centre for AI Research, University of Agder, Grimstad, Norway

    Morten Goodwin

  7. TH Lübeck – University of Applied Sciences, Lübeck, Germany

    Sebastian Hobert

  8. SINTEF & University of Oslo, Oslo, Norway

    Petter Bae Brandtzaeg

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cocchi, A., Bosse, T., van Pinxteren, M. (2024). Should Conversational Agents Care About Our Gender Identity?. In: Følstad, A., et al. Chatbot Research and Design. CONVERSATIONS 2023. Lecture Notes in Computer Science, vol 14524. Springer, Cham. https://doi.org/10.1007/978-3-031-54975-5_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-54975-5_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-54974-8

  • Online ISBN: 978-3-031-54975-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation