This section introduces the main question we wish to explore by discussing the literature on bias in social robotics and its implications on design ethics. First (2.1), we clarify the role biases play with reference to social robots – and, more specifically, to ECAs (2.2) – by focusing the attention on the design strategy of bias alignment. Secondly, we underline its ethical significance by introducing the feedback hypothesis, both in general (2.3) and as it applies to ECAs (2.4). In light of this, we state our research question and set the stage for critically discussing four possible answers to it.
Bias alignment
Biases play a massive role in structuring human relations and the social life. Implicit assumptions based on approximate generalizations, rules of thumb and long-lasting habits influence the interpretation of our experience. In a sense, biases function as social scripts (De Angeli & Brahnam 2006) that assist us in coping with information incompleteness and complexity in everyday situations. As predetermined and rather fix schemes of information management, social biases – that is, biases that co-structure relations between humans – concern different aspects of the social sphere. However, the function they execute is similar. Basically, biases suggest associations between easily perceivable data or cues (such as ethnicity, age, gender, physical appearance, apparent economic status, social roles, specific tasks, and so on) and other, less immediately accessible but highly valuable pieces of information (such as competence, trustworthiness, intelligence, kindness, and so on). In doing so, biases concur to forming expectations and subsequent social behaviour (Fig. 1).
The fact that social roles and specific tasks are cues that trigger biased expectations is of particular importance. Indeed, it is reasonable to suppose that similar biased expectations will be triggered when artificial agents substitute human agents in given practical contexts. The automation of social roles and tasks might cause the same biased expectations influencing human social relations to be extended on to technological entities that are put in the place of humans. In many cases, this leads users to partially anthropomorphize artificial agents, projecting onto them typically human features such as gender, ethnicity, or social status. These projections, in turn, trigger biased information associations concerning competence, authority, trustworthiness, and other socially relevant features (Tay et al., 2014) (Fig. 2).
This intuition, that was firstly introduced and empirically confirmed in the human-computer interaction Computers Are Social Actors [CASA] studies (Nass et al., 1994; Nass et al., 1996), in recent years has been found to generally apply to human-robot interactions as well and has grown to become a widely accepted notion in social robotics (McDonnell & Baxter, 2019; Weßel et al., 2021). When social roles or tasks are automated, the way in which users respond to computer programs appear to be very similar to how they respond to social robotsFootnote 3.
The fact that social biases are relevant even in human-machine interactions is dense of design implications for social robotics. First of all, social biases have appeared to many in the social robotics community as conditions for designing successful interactions – which is arguably the central design goal in this field (Carpenter et al., 2009; Nomura, 2017). In light of their tremendous impact on user mental models, it seems reasonable to consider social biases as factors that massively influence the extent to which interactions are perceived as easy, pleasant, engaging, and effective. Ignoring their power would probably lead to interaction failures, suboptimal interaction quality and eventually rejections of the technology (Jung et al., 2016; Bryant et al., 2016).
The power of biases, however, is not just there to be coped with. It can be harnessed as well. Intelligently stimulating biases through the insertion of the right design cues might turn out to be the most successful strategy for optimizing interactions (Siegel et al., 2009). In other words, the cues that in human-human relations trigger biases that are conducive to effective interactions can be reproduced in the design of the artificial agent to which we want the corresponding human task to be delegated. Perceivable traits commonly associated with ethnicity, gender, age, physical beauty, and so on become tools in the designers’ hands. Through the adoption of various design cues, designers can influence the formation of user mental models of the technology and steer them towards desired outcomes, such as maximizing the feeling of trustworthiness suggested by a healthcare robot or the impression of competence produced by a smart assistant.
The practice of leveraging the power of biases in order to maximize the quality of interactions seems intimately connected to the idea that a seamless, almost “natural” introduction of artificial agents in the fabric of society must be promoted (Breazeal, 2003; Isaac & Bridewell, 2017; Eyssel & Kuchenbrandt, 2012). This implies a design strategy we call bias alignment. According to this strategy, artificial agents should be specifically designed to trigger the same social biases that are triggered in the corresponding human-human interaction that is being automated. In so doing, the unavoidable strangeness and difference of artificial agents will be countered, or covered, by their apparent social similarity to good old humans, so they will “fit in” just fine in the social context where they are deployed, without requiring extra cognitive efforts on the user part.
Bias Alignment and ECAs
To further clarify these points, let us see how bias alignment works as applied to a specific class of social robots, i.e., Embodied Conversational Agents (ECAs).
ECAs are systems designed to mimic human-human interaction using natural language via text or voice. Moreover, ECAs are endowed with an embodied avatar representing an anthropomorphic body or part of it (Silvervarg et al., 2012). Both natural language processing and embodied avatars serve as platforms to incorporate different cues aimed at easing the interaction with human users. As explained, these elements facilitate the triggering of social biases that decrease the discomfort users experience when interacting with inanimate beings (Powers et al., 2005; Robertson, 2010; Jung et al., 2016; Kraus et al., 2018).
Gender biases offer themselves as a useful example of bias alignment with reference to ECAs. Gender attribution can be encouraged by-design through different cues that, as effectively described by Robertson (2010), serve as “cultural genitals”. In the absence of physical genitals, gender cues are proxies that trigger the identification of an ECA as a ‘malebot’ or a ‘fembot’Footnote 4.
Gender attribution is rather easy to trigger, so multifarious cues based on traits that men and women are thought to have (descriptive stereotypes) or ‘should’ have (prescriptive stereotypes; cf. Brahnam & De Angeli, 2012) can be used. Some cues are physiognomic in nature: the hairstyle, the size of the eyes and of the head, the shoulder width, and the colour of the lips (De Angeli & Brahnam, 2006; Robertson, 2010; Eyssel & Hegel, 2012; Bernotat et al., 2017, 2021; Trovato et al., 2018). Even simple fashion accessories could suffice. In a work conducted by Jung et al., (2016), a male hat rather than a pair of pink earmuffs was enough for users to classify the ECA as respectively “male” and “female”.
Beyond the bodily domain, gender attribution can be triggered also through the bot’s tone of voice: e.g., fembots are usually endowed with high-pitch voices, which are commonly attributed to women (Robertson, 2010). Cues can be based on even more subtle proxies, such as specific skills or personal traits. For instance, the ‘masculinity’ or ‘femininity’ associated to a given task – e.g., changing a flat tyre vs. babysitting – will encourage users to project a specific gender to the system to which the task is delegated even if its avatar and voice are built to resist gender attribution. Indeed, gender identification is complex and involves multiple levels (Ladwig & Ferstl, 2018; Sutton, 2020).
Aligning the design of ECAs to users’ expectations through gender cues could be essential for acceptability. Gender biases triggered by design cues importantly influence user mental models and expectations. Different studies (Nass et al., 1997; De Angeli & Brahnam, 2006; Kraus et al., 2018) show that systems with male cues are seen as more dominant and assertive, while fembots are thought to be kinder, more communicative and helpful (Nass et al., 1994; Kuchenbrandt et al., 2014; Reich-Stiebert & Eyssel, 2017). Depending on the contexts of application, the power of gender biases could lead to adoption or disuse (Nass & Moon, 2000).
Consider this hypothetical scenario. Imagine that a company decides to develop a smart assistant to help people carrying out basic car maintenance tasks like changing the oil, a flat tire, or a consumed wiper. Let us also suppose that data collected in interviews show that the target customer group heavily associates the job as a mechanical with the male gender. Suppose further that previous psychological research showed that the male gender is commonly associated with higher degrees of authority, competence, and trustworthiness than the female gender. In this scenario, the bias alignment design strategy would suggest inserting some design cues that nudge users into projecting the male gender onto the technology.Footnote 5 First, this will meet the users’ biased expectations concerning the ‘masculinity’ of the mechanical domain, ensuring a seamless introduction of the technology in its social context. Furthermore, this will leverage a double bias concerning competence and trustworthiness: (a) the bias according to which men are more competent than women when it comes to car maintenance; and (b) the bias according to which men are generally more authoritative and trustworthy than women. Adding a simple design cue to the technology like a low-pitched voice – which is commonly associated with the male gender – will do the trick.
By aligning design to social biases, their power will have been leashed and channelled towards the design goal of effective, pleasant, and smooth interaction. Stated in this teleological sense – i.e., according to a means-end approach where efficiency is at stake –bias alignment is a widely accepted design strategy in the social robotics community (Kraus et al., 2018; McDonnel & Baxter, 2019; Tay et al., 2014). As a recent UNESCO report (West et al., 2019) shows, for instance, feminine attributes are often used for the characterization of Personal Assistant (such as Alexa or Siri) so to nudge users into perceiving the system as more sympathetic and, at the same time, easier to control and dominate. This does not mean, however, that there is full agreement on the actual effectiveness of bias alignment (Sandry, 2015; Brahnam & Weaver, 2015; Reich-Stieber & Eyssel, 2017). Nevertheless, the dissident voices are considerably fewer than the agreeing ones.
The Feedback Hypothesis
Through bias alignment, social biases can be leveraged to design better human-machine interactions. The benefits in terms of user acceptability are widely acknowledged. Are there any risks that should be considered?
As seen, implementing cues by design allows for social biases to be projected onto robots, so that users perceive them as familiar interlocutors. Interestingly, this implies that interactions with robots are intended to belong to the same practical category of interactions with humans. Accordingly, bias projection must be conceived as bidirectional as well. If biases triggered by humans transfer to corresponding interactions with ECAs, it is reasonable to assume that biases triggered by ECAs will feedback onto corresponding human relations as well. Let us call this claim the feedback hypothesis.
Even though, at least to our knowledge, there is no hard evidence supporting it, the feedback hypothesis has already been posited by some authors (Carpenter, 2009; Robertson, 2010; Sparrow, 2017; Bisconti, 2021) and appears to be conceptually sound enough for its potential impacts to be taken seriously. Impacts are relevant because, if the hypothesis holds, the social biases harnessed through design would be reinforced during interactions with robots and transferred back to humans. This would generate a lock-in situation where such biases, already deeply rooted in cultural views as they are, would become even further institutionalized and normalized, significantly increasing the amount of effort needed to eradicate them.
At first sight, it seems reasonable to claim that not every instance of bias leveraging is necessarily problematic from this perspective. For example, suppose that psychological research suggests that users tend to behave more calmly and collaboratively when they interact with artificial agents coloured in blue instead of red. As a consequence, a smart assistant is given a blue suit, and not a red one, to wear on its digital avatar body. No risk seems to arise.
Things are different when the leveraged biases have ethical significance, which is rather common when social relationships are concerned. For the purposes of our paper, suffice it to say that biases are unethical if they convey epistemically problematic beliefs which promote unacceptable discrimination, thus harming people by reducing their individuality to inconsistent and often spiteful generalizations - which is a clear offence to human dignityFootnote 6.
As already said, many social biases are extensively based on highly sensitive perceptual data like ethnicity, skin colour, age, and gender. It is thus sensible to fear that leveraging discriminatory biases might enforce and solidify the connected forms of social discrimination, making it even harder to eradicate them and promote equal and fair social treatment. If this were the case, technology would become a booster of socially discriminatory biases and would engender a lock-in effect where pre-existing social biases would be stabilized and eventually normalized.
A lock-in effect involving discriminatory biases would be highly detrimental. Indeed, it would worsen the condition of both victims and perpetrators. Discriminated groups would have to face a silent, invisible but systemic diffusion of harmful biases. Users would be put in a position where the discriminatory biases they unintentionally share are opaquely strengthened and amplified through interactions with social technologies, working against their own moral aspirations.
In light of this, the best solution could be to separate risky biases from safe ones. Yet, it seems unlikely that biases can be absolutely categorized as discriminatory and non-discriminatory. Potentially discriminatory biases might be triggered through cues without engendering evident discriminatory outcomes. Gender biases are a good example here. Suppose, for instance, that conclusive evidence was found that gendered ECAs work better than non-gendered ones. Suppose further that a company is tasked to develop an ECA to offer customer service inside a women’s sauna. After some debate, the design team opts for a fembot, since they believe that, given the context of use, customers may react negatively to a malebot. In this case, we surmise, no unethical discrimination seems involved. Therefore, the way in which cues are implemented and the contexts of use affect the risks posed by design choices that exploit biases. It follows that the risks involved in bias alignment requires careful consideration so that the legitimate purpose of maximising interaction quality is pursued within the boundaries of what is ethically permissible.
Let us introduce another fictional example to better grasp the risks that the hypothesis wishes to stress. Suppose now that the team is appointed to design an ECA to carry out secretarial tasks assisting executives in a company where 80% of senior positions are held by white men aged between 55 and 65 years. Suppose further that psychological surveys showed that male subjects in the same age span tend to associate the secretary role with the female gender, while the male gender is associated with managerial roles. The team thus infers that a malebot would be perceived as strange if deployed to carry out secretary tasks, while a fembot would fit in just fine. Consequently, they implement gender cues studied to trigger the desired biases in the specific user group.
Once the fembot is deployed, users’ biased expectations will be reinforced through day-to-day interaction with the technology. Since no hard line can be traced between human-machine and human-human practical dimensions, the reinforced bias will affect the chances of women getting executive roles and the chances of those (unconsciously) biased to become aware of their discriminative beliefs and correct the aim. Moreover, since fembots seem to be often verbally addressed in inappropriate ways (see infra), the same habits may transfer to human workers filling the same role, increasing the chance of abuse to come to pass (Fig. 3).
As the example shows, in this case bias alignment would cause the normalization of the biased expectations held by the user group. This would in turn make it even harder for moral awareness to arise and discriminatory biases to be eradicated. Here, the mere technical dimension is not sufficient to provide a satisfactory justification to the adoption of the bias alignment strategy. An ethical justification is needed.
The Feedback Hypothesis and ECAs
As the next pages show, research on ECAs and gender bias seems to corroborate the conclusion presented in the previous section. Gender cues are commonly acknowledged as powerful means to improve the quality of human-machine interactions and, as such, are largely inserted in the design of ECAs. However, gender bias alignment can also lead both to the transfer of discriminatory behaviours and to the solidification of pre-existing unethical biases (Sucameli, 2021).
Analysing the language users tend to adopt when interacting with ECAs highlights how strongly word choices are influenced by gender cues. Utterances characterized by the presence of explicit sexual references are common in interactions with fembotsFootnote 7. As women are often the object of sexual harassment, equally often fembots trigger abusive behaviours and the adoption of a discriminatory vocabulary. Brahnam and De Angeli (2012) show that a fembot named Kathy receives more inputs containing words which refers to its physical appearance, more swear words and more explicit sexual demands or offensive comments compared to a conversational agent with a masculine avatar. Figure 4 reports an explicit conversation between Kathy and a user that shows how foul and repulsive conversations might become.
Building on these results, De Angeli and Brahnam (2006) underline how such a disinhibited behaviour has a clear impact on the evolution of sex stereotypes. These aspects of human conversations with ECAs not only constitute a clear and dramatic representation of how women are often considered and approached in our society, but they can also lead to an increase of abuses to women.
Interestingly enough, the way ECAs respond to these disinhibited utterances is equally problematic. Curry and Rieser (2018) identify three classes of possible responses: (1) Nonsensical Responses: non-grammatical, non-coherent, no-answer, search result, and ‘don’t know’ responses; (2) Negative Responses: humorous refusal, polite refusal, deflection, retaliation; (3) Positive Responses: play-along, joke, and flirtation.
Every type of response presents its own risks. On the one hand, the inability to (or the choice not to) reply to the users’ utterances (e.g. nonsensical responses), probably represents the less risky type of response and, as such, it is often preferred as a design solution. For instance, West et al. (2019) highlights how most used Personal Assistants (Siri, Alexa, Google Assistant, Cortana) almost never provide negative answers or signal the inappropriateness of the received verbal harassment to the user. Similarly, if fembots respond with a joke to a sexualised comment or request, the interaction may convey the idea that the type of language used by the human interlocutor – and, therefore, the related attitude – is acceptable after all, almost humorous, and appropriate for human interactions as well. As such, these responses might contribute to reinforcing sexist and abusive attitudes.
On the other hand, the use of an aggressive response – as in the case of retaliation, in which the system insults users back – raises the ethical dilemma linked to the potential immorality of allowing ECAs to insult human beings. However, some developers choose to design ECAs this way in order to provoke the listener and bring to light the problem of discrimination against women. It is the case of AYA, a virtual assistant which replies to verbal abuse with very aggressive utterances (Søndergaard & Hansen, 2018; Lee et al., 2021). In other cases (Winkle, 2021) the ECA uses a standard, argumentative or aggressive behaviour to express feministic sentiments and discourage user verbal abuse.
To sum up, the feedback hypothesis provides grounds to reasonably suspect that the bias alignment design strategy might pose severe risks of an ethical nature. It follows that such strategy requires to be ethically assessed. We are finally ready to pose our main research question:
Q
Is it ethically permissible to align the design of ECAs to gender biases in order to improve interactions and maximize user satisfaction?