Abstract
More and more companies have started to use nonhuman agents for employment interviews, making the selection process easier, faster, and unbiased. To assess the effectiveness of the above, in this paper, we systematically analyzed, reviewed, and compared human interaction with a social robot, a digital human, and another human under the same scenario simulating the first phase of a job interview. Our purpose is to allow the understanding of human reactions, concluding to a disclosure of the human needs towards human – nonhuman interaction. We also explored how the appearance and the physical presence of an agent can affect human perception, expectations, and emotions. To support our research, we used time-related and acoustic features of audio data, as well as psychometric data. Statistically significant differences were found for almost all extracted features and especially for intensity, speech rate, frequency, and response time. We also developed a Machine Learning model that can recognize the nature of the interlocutor a human interacts with. Although human was generally preferred, the interest level was higher and the shyness level was lower during human-robot interaction. Thus, we believe that, following some improvements, social robots, compared to digital humans, have the potential to act effectively as job interviewers.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Woods, S.A., Ahmed, S., Nikolaou, I., Costa, A.C., Anderson, N.R.: Personnel selection in the digital age: a review of validity and applicant reactions, and future research challenges. Europ. J. Work Organ. Psychol. 29, 64–77 (2020)
Homan, A.C., Van Knippenberg, D., Van Kleef, G.A., De Dreu, C.K.W.: Bridging faultlines by valuing diversity: diversity beliefs, information elaboration, and performance in diverse work groups. J. Appl. Psychol. 92, 1189 (2007)
Seibt, J., Vestergaard, C.: Fair proxy communication: using social robots to modify the mechanisms of implicit social cognition. Research Ideas and Outcomes 4, e31827 (2018)
Li, J.: The benefit of being physically present: a survey of experimental works comparing copresent robots, telepresent robots and virtual agents. Int. J. Hum.-Comput. Stud. 77, 23–37 (2015)
Mollahosseini, A., Abdollahi, H., Sweeny, T.D., Cole, R., Mahoor, M.H.: Role of embodiment and presence in human perception of robots’ facial cues. Int. J. Hum.-Comput. Stud. 116, 25–39 (2018)
Nørskov, S., Damholdt, M.F., Ulhøi, J.P., Jensen, M.B., Ess, C., Seibt, J.: Applicant fairness perceptions of a robot-mediated job interview: a video vignette-based experimental survey. Frontiers in Robotics and AI. Frontiers, 163 (2020)
Savage, M.: Meet Tengai, the job interview robot who won’t judge you. BBC Oline 12 (2019)
Ahmad, M.I., Mubin, O., Patel, H.: Exploring the potential of NAO robot as an interviewer. In: Proceedings of the 6th International Conference on Human-Agent Interaction, pp. 324–326 (2018)
Edwards, A., Omilion-Hodges, L., Edwards, C.: How do patients in a medical interview perceive a robot versus human physician? In: Proceedings of the companion of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, pp. 109–110 (2017)
Elkins, A.C., Gupte, A., Cameron, L.: Humanoid robots as interviewers for automated credibility assessment. In: Nah, F.F.-H., Siau, K. (eds.) HCII 2019, vol. 11589, pp. 316–325. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22338-0_26
HireVue Full Platform. hirevue.com (2022). https://www.hirevue.com/demo/full-platform-em. Accessed 8 Feb
Chapman, D.S., Rowe, P.M.: The impact of videoconference technology, interview structure, and interviewer gender on interviewer evaluations in the employment interview: a field experiment. J. Occup. Organ. Psychol. 74, 279–298 (2001)
Stritzke, W.G.K., Nguyen, A., Durkin, K.: Shyness and computer-mediated communication: a self-presentational theory perspective. Media Psychol. 6, 1–22 (2004)
Joinson, A.N.: Self‐disclosure in computer‐mediated communication: the role of self‐awareness and visual anonymity. European journal of social psychology 31, 177–192 (2001)
Langer, M., König, C.J., Fitili, A.: Information as a double-edged sword: the role of computer experience and information on applicant reactions towards novel technologies for personnel selection. Comput. Hum. Behav. 81, 19–30 (2018)
Gebhard, P., Baur, T., Damian, I., Mehlmann, G., Wagner, J., André, E.: Exploring interaction strategies for virtual characters to induce stress in simulated job interviews. In: Proceedings of the 2014 International Conference on Autonomous Agents and Multi-agent Systems, pp. 661–668 (2014)
Anderson, K.: The TARDIS framework: intelligent virtual agents for social coaching in job interviews. In: Reidsma, D., Katayose, H., Nijholt, A. (eds.) ACE 2013. LNCS, vol. 8253, pp. 476–491. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-03161-3_35
Gebhard, P., et al.: Serious games for training social skills in job interviews. IEEE Trans. Games 11, 340–351 (2018)
Georgiou, K., Gouras, A., Nikolaou, I.: Gamification in employee selection: the development of a gamified assessment. Int. J. Selection Assessment 27, 91–103 (2019)
Youssef, B., Atef, M.C., Jones, H., Sabouret, N., Pelachaud, C., Ochs, M.: Towards a socially adaptive virtual agent. In: International Conference on Intelligent Virtual Agents, pp. 3–16. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21996-7_1
Baur, T., Damian, I., Gebhard, P., Porayska-Pomsta, K., André, E.: A job interview simulation: Social cue-based interaction with a virtual character. In: 2013 International Conference on Social Computing, pp. 220–227. IEEE (2013)
Kawahara, T.: Spoken dialogue system for a human-like conversational robot ERICA. In: D’Haro, L.F., Banchs, R.E., Li, H. (eds.) 9th International Workshop on Spoken Dialogue System Technology. LNEE, vol. 579, pp. 65–75. Springer, Cham (2019). https://doi.org/10.1007/978-981-13-9443-0_6
Inoue, K., Hara, K., Lala, D., Nakamura, S., Takanashi, K., Kawahara, T.: A job interview dialogue system with autonomous android ERICA. In: Increasing Naturalness and Flexibility in Spoken Dialogue Interaction: 10th International Workshop on Spoken Dialogue Systems, pp. 291–297. Springer Singapore. https://doi.org/10.1007/978-981-15-9323-9_25
Kumazaki, H., et al.: Job interview training targeting nonverbal communication using an android robot for individuals with autism spectrum disorder. Autism 23, 1586–1595 (2019)
Wood, L.J., Dautenhahn, K., Lehmann, H., Robins, B., Rainer, A., Syrdal, D.S.: Robot-mediated interviews: Do robots possess advantages over human interviewers when talking to children with special needs? In: Herrmann, G., Pearson, M.J., Lenz, A., Bremner, P., Spiers, A., Leonards, U. (eds.) ICSR 2013, vol. 8239, pp. 54–63. Springer, Cham (2013). https://doi.org/10.1007/978-3-319-02675-6_6
Wilson, I.: Using Praat and Moodle for teaching segmental and suprasegmental pronunciation. In: Proceedings of the 3rd international WorldCALL Conference: Using Technologies for Language Learning (WorldCALL 2008) (2008)
Crumpton, J., Bethel, C.L.: A survey of using vocal prosody to convey emotion in robot speech. Int. J. Soc. Robot. 8, 271–285 (2016)
Poria, S., Cambria, E., Bajpai, R., Hussain, A.: A review of affective computing: From unimodal analysis to multimodal fusion. Inf. Fusion 37, 98–125 (2017)
Dasgupta, P.B.: Detection and analysis of human emotions through voice and speech pattern processing. arXiv preprint arXiv:1710.10198 (2017)
Johnstone, T.: The effect of emotion on voice production and speech acoustics. Thesis Commons (2017)
Giannakopoulos, T.: Pyaudioanalysis: An open-source python library for audio signal analysis. PloS One 10, e0144610 (2015)
Watson, D., Clark, L.A.: The PANAS-X: Manual for the positive and negative affect schedule-expanded form (1999)
Abhang, P.A., Gawali, B.W., Mehrotra, S.C.: Technological basics of EEG recording and operation of apparatus. Introduction to EEG-and Speech-Based Emotion Recognition, pp. 19–50 . Academic Press (2016)
Teixeira, J.P., Oliveira, C., Lopes, C.: Vocal acoustic analysis–jitter, shimmer and hnr parameters. Procedia Technol. 9, 1112–1122 (2013)
de Felippe, A.C.N., Helena, M., Grillo, M.M., Grechi, T.H.: Standardization of acoustic measures for normal voice patterns. Brazilian J. Otorhinolaryngology 72, 659–664 (2006)
Baka, E., Vishwanath, A., Mishra, N., Vleioras, G., Thalmann, N.M.: “Am I talking to a human or a robot?”: a preliminary study of human’s perception in human-humanoid interaction and its effects in cognitive and emotional states. In: Computer Graphics International Conference, pp. 240–252. Springer (2019)
Yumoto, E., Gould, W.J., Baer, T.: Harmonics‐to‐noise ratio as an index of the degree of hoarseness. The journal of the Acoustical Society of America 71. Acoustical Society of America: 1544–1550 (1982)
Acknowledgment
This research is partly supported by the National Research Foundation, Singapore under its International Research Centers in Singapore Funding Initiative, and Institute for Media Innovation, Nanyang Technological University (IMI-NTU). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Baka, E., Mishra, N., Sylligardos, E., Magnenat-Thalmann, N. (2022). Social Robots and Digital Humans as Job Interviewers: A Study of Human Reactions Towards a More Naturalistic Interaction. In: Kurosu, M. (eds) Human-Computer Interaction. Technological Innovation. HCII 2022. Lecture Notes in Computer Science, vol 13303. Springer, Cham. https://doi.org/10.1007/978-3-031-05409-9_34
Download citation
DOI: https://doi.org/10.1007/978-3-031-05409-9_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-05408-2
Online ISBN: 978-3-031-05409-9
eBook Packages: Computer ScienceComputer Science (R0)