1 Introduction

Affective computing—the study and development of artificial intelligence (AI) and machines that detect and respond to affective states—has recently received increased interest outside its original field owing to the emergence of commercial applications. For example, in 2016, Affectiva’s emotion-sensing technology was integrated with the videogame Nevermind, allowing it to dynamically adjust to the players’ emotions. Similarly, in 2017, France Loisirs, a French bookstore, tested Angus’s emotion-sensing technology to detect the emotions of browsing customers and alert employees in real-time, enabling them to offer customer support when needed.

Major international firms, such as Apple and Google, have already invested in affective computing technology to enhance customer experience (Research And Markets, 2020), paving the way for its large-scale marketing applications. In fact, the industry of affective computing technologies is expected to be worth US$123.3 billion by 2026, with an estimated compound annual growth rate of 36.4% in 2020–2026 (Research And Markets, 2020). Nevertheless, affective computing has received little attention in marketing literature. Therefore, the need for research on the impact of affective computing on marketing is urgent.

Affective computing relies on the bodily changes (e.g., smile, heart rate acceleration, sweating) characterizing affective states or emotions. Machines sense these changes—biometrics, brain activity, facial expressions, body movement, or voice/speech can be used as input—and algorithms are trained to identify the corresponding emotion (D’Mello et al., 2018). Machines can also be trained to respond to the identified emotion (e.g., a robot smiling in response to a human’s smile).

Affective computing is expected to transform marketing practice because: (1) firms could collect new types of insights into consumer emotions to personalize their experience in an unprecedented manner; (2) human employees would no longer be needed to understand and manage consumer emotions during their interactions; and (3) bots (robots, chatbots, voice assistants) that inform and serve consumers could be augmented with emotional intelligence.

Although affective computing and AI are related, the critical differences between them call for marketing research on affective computing in its own right. Whereas AI mimics human IQ type of intelligence when performing cognitive tasks, affective computing simulates emotional intelligence. The latter may be less easily embraced than the former because emotions are considered to be “our human advantage,” distinguishing us from machines (Forbes, 2021). Therefore, studying how emotionally intelligent machines, beyond pure cognitive AI, can be accepted by consumers and employees is essential.

To this end, we advance a framework that distinguishes between three uses of affective computing in marketing: empathic (to detect emotions), collaborative (to detect emotions and assist humans in responding to them), and interactive (to detect and respond to emotions). This framework draws upon the work by Picard (1997), which specifies that affective computing can serve to detect emotions or to detect and express emotions.

2 Research agenda

We identify research directions for all three uses of affective computing (Table 1 presents an expanded list of the research questions).

Table 1 Opportunities for future research
Full size table

3 Empathic affective computing

Empathic affective computing (for affective computing systems that detect emotions) is relevant to marketing managers who want to understand consumers’ emotions. Emotion detection is central to market research, as many consumer decisions are driven by emotions. As such, empathic affective computing will likely advance market research by offering a new range of methods for studying which emotions are triggered by a product, an advertisement, or another element of the marketing mix.

Academic research should examine the predictive abilities of affective computing methods relative to those of traditional methods. Affective computing methods keep increasing in accuracy and may even outperform survey methods, as anecdotal evidence suggests. For example, the food company Mars used Affectiva’s Affdex technology, which identifies emotions based on facial expressions, to determine consumers’ emotional response to their ads and found that it predicted short-term sales more accurately than self-reports (Affectiva, n.d.). Academic research is needed to corroborate this finding.

Moreover, empathic affective computing offers opportunities for studying unconscious emotions. These are emotions that affective computing methods can sense but that study participants would not necessarily be cognitively aware of (Winkielman & Berridge, 2004). When and how do unconscious emotions affect consumer attitudes and behaviors? We encourage academics to extend existing knowledge about the role of emotions in the consumer decision-making process by exploring the respective roles of conscious and unconscious emotions in this process.

4 Collaborative affective computing

Collaborative affective computing (to detect emotions and assist humans in responding to them) can cooperate with employees during consumer interactions by helping them understand the consumer’s emotions and prompting them to respond accordingly.

However, a prerequisite is that frontline employees are willing to use affective computing systems. Accordingly, research is needed to identify the factors influencing the acceptance of and trust in these systems. Prior research has shown that in decision-making situations, humans are unwilling to trust forecasts made by algorithms—a phenomenon called algorithm aversion (Dietvorst et al., 2015). Does a similar aversion apply to affective computing systems? If so, how can it be alleviated?

Research should explore if consumers consent to their emotions being tracked by affective computing systems during interactions with the firm. Humana, an insurance company, experimented with Cogito’s affective computing method to assist its call center agents (these agents are alerted when an emotion that needs to be addressed is detected in a consumer’s voice) and saw the net promotor score and issue resolution increase by 28% and 6%, respectively (Inc, 2017). Although consumers may benefit from collaborative affective computing by being better served, they may feel uncomfortable by having their emotions systematically analyzed. How do consumers evaluate this trade-off? Moreover, we encourage academics to investigate how consumers expect firms to inform them about their use of affective computing systems during service interactions.

Finally, collaborative affective computing raises questions regarding the psychological outcomes experienced by employees. Although the case of Humana suggests that collaborative affective computing improves the task performance of frontline employees, its impact on employees’ well-being remains unknown. Frontline employees, who perform emotional labor, are particularly likely to experience stress and burnout. Does collaborative affective computing alleviate the burden of emotional labor? Or does it make employees worry that machines/AI could take over their jobs, thereby reducing their motivation? Do employees feel dehumanized, as they are partly deprived from using their emotional intelligence?

5 Interactive affective computing

Interactive affective computing (for affective computing systems that detect and respond to emotions) can enrich consumer–bot interactions by augmenting the (ro)bot with emotional intelligence. This technology can be applied to service robots (Mende et al., 2019) or bots built in physical objects (e.g., headphones) or on digital platforms (e.g., Spotify). We review each application.

Interactive affective computing has potential for transforming organizational frontlines by enabling service robots to adapt their emotional responses based on the emotions of the consumers who interact with them. This raises questions regarding the impact of such robots on service climate and on consumer responses. Past research (e.g., Mende et al., 2019) has focused on how the physical appearance of robots influences consumers’ propensity to anthropomorphize them; however, how robots’ behaviors influence consumer responses remains unknown. Robots augmented with emotional intelligence will behave in a more human-like manner. Would this increase their acceptance and likeability? Or would consumers perceive such robots as excessively anthropomorphized, thereby triggering discomfort?

Another research direction relates to the effectiveness of robots’ emotional displays. The positive emotions displayed by (human) employees generate positive emotions in consumers, a phenomenon called emotional contagion (Hennig-Thurau et al., 2006). Does emotional contagion occur when consumers interact with emotionally augmented robots? If so, does it enable building rapport between robots and consumers?

Further, questions arise regarding the authenticity of the emotions displayed by robots augmented with emotional intelligence, which is a critical factor in (human) employee–consumer interactions (Hennig-Thurau et al., 2006). However, what is authentic emotional display in robots? This construct needs to be re-conceptualized.

Interactive affective computing also offers the potential for real-time emotion-based personalization of bot-enabled products, creating opportunities for product innovation. For instance, Neurowear’s Mico headphones sense the user’s brain activity to identify their affective state and play songs that match it. Similarly, Spotify—which offers voice-controlled access to its streaming platform—is working on a voice recognition system that can identify users’ emotions. Research is needed to determine when and how consumers derive value from using products and services that adapt in real-time to their emotions. We encourage academics to study how emotion-based personalization benefits hedonic versus utilitarian products.

Examining the circumstances under which consumers are willing to trade their emotional data for an emotionally personalized experience is also important. The amount and type of data collected by affective computing systems raise privacy concerns, and its implications need to be examined.

6 Conclusion

Understanding and responding to consumer emotions is fundamental to marketing practice, and affective computing will likely transform this process. As marketing applications of affective computing rise, new research is needed to understand how these applications affect consumers, employees, and companies.