Skip to main content
SpringerLink
Log in

Using Online Customer Reviews to Classify, Predict, and Learn About Domestic Robot Failures

  • Published:
International Journal of Social Robotics Aims and scope Submit manuscript

Abstract

There is a knowledge gap regarding which types of failures robots undergo in domestic settings and how these failures influence customer experience. We classified 10,072 customer reviews of small utilitarian domestic robots on Amazon by the robotic failures described in them, grouping failures into twelve types and three categories (Technical, Interaction, and Service). We identified sources and types of failures previously overlooked in the literature, combining them into an updated failure taxonomy. We analyzed their frequencies and relations to customer star ratings. Results indicate that for utilitarian domestic robots, Technical failures were more detrimental to customer experience than Interaction or Service failures. Issues with Task Completion and Robustness and Resilience were commonly reported and had the most significant negative impact. Future failure-prevention and response strategies for utilitarian domestic robots should address the robot’s technical ability to meet functional goals, operate and maintain structural integrity over time. Usability and interaction design were less detrimental to customer experience, indicating that customers may be more forgiving of failures that impact these aspects for the robots and practical uses examined. Further, we developed a Natural Language Processing model capable of predicting whether a customer review contains content that describes a failure and the type of failure it describes. With this knowledge, designers and researchers of robotic systems can prioritize design and development efforts towards essential issues.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Data Availability

The datasets generated during the current study are available from the corresponding author on reasonable request.

References

  1. Müller C (2019) Market for professional and domestic service robots booms in 2018. IFR Secretariat Blog

  2. Honig S, Oron-Gilad T (2018) Understanding and resolving failures in human–robot interaction: literature review and model development. Front Psychol 9:861. https://doi.org/10.3389/fpsyg.2018.00861

    Article  Google Scholar 

  3. Carlson J, Murphy RR (2005) How UGVs physically fail in the field. IEEE Trans Rob 21:423–437. https://doi.org/10.1109/TRO.2004.838027

    Article  Google Scholar 

  4. Cuevas CAG, Rodriguez RAC (2018) Robot failure. In: Sotelo R, Arriaga J, Aron M (eds) Complications in robotic urologic surgery. Springer International Publishing, Cham, pp 23–28

    Chapter  Google Scholar 

  5. Rajih E, Tholomier C, Cormier B et al (2017) Error reporting from the da Vinci surgical system in robotic surgery: a Canadian multispecialty experience at a single academic centre. Canadian Urol Assoc J 11:E197–E202. https://doi.org/10.5489/cuaj.4116

    Article  Google Scholar 

  6. Zorn KC, Gofrit ON, Orvieto MA et al (2007) Da Vinci robot error and failure rates: single institution experience on a single three-arm robot unit of more than 700 consecutive robot-assisted laparoscopic radical prostatectomies. J Endourol 21:1341–1344. https://doi.org/10.1089/end.2006.0455

    Article  Google Scholar 

  7. Steinbauer G (2013) A survey about faults of robots used in RoboCup. In: Chen X, Stone P, Sucar LE van der ZT (eds) Lecture notes in computer science. Springer, Berlin, pp 344–355

  8. Stiber M, Huang CM (2020) Not all errors are created equal: exploring human responses to robot errors with varying severity. In: ACM international conference on multimodal interaction late-breaking report

  9. Short E, Hart J, Vu M, Scassellati B (2010) No fair!! an interaction with a cheating robot. In: 2010 5th ACM/IEEE international conference on human–robot interaction (HRI), IEEE. pp 219–226

  10. Tolmeijer S, Weiss A, Hanheide M, et al (2020) Taxonomy of trust-relevant failures and mitigation strategies. In: ACM/IEEE international conference on human–robot interaction

  11. Correia F, Guerra C, Mascarenhas S, et al (2018) Exploring the impact of fault justification in human–robot trust. In: Proceedings of the 17th international conference on autonomous agents and multiagent systems (AAMAS 2018)

  12. Waveren S van, Carter EJ, Leite I (2019) Take one for the team: the effects of error severity in collaborative tasks with social robots. In: IVA 2019—proceedings of the 19th ACM international conference on intelligent virtual agents

  13. Rossi A, Dautenhahn K, Koay KL, Walters ML (2018) The impact of peoples’ personal dispositions and personalities on their trust of robots in an emergency scenario. Paladyn J Behav Robot 9:137–154

    Article  Google Scholar 

  14. Chung W, Tseng T-LB (2012) Discovering business intelligence from online product reviews: a rule-induction framework. Expert Syst Appl 39:11870–11879

    Article  Google Scholar 

  15. Herzog C, Handke C, Hitters E (2019) Analyzing talk and text II: thematic analysis. In: den Bulck H, Puppis M, Donders K, Van Audenhove L (eds) The Palgrave Handbook of methods for media policy research. Springer International Publishing, Cham, pp 385–401

    Chapter  Google Scholar 

  16. Flook R, Shrinah A, Wijnen L et al (2019) On the impact of different types of errors on trust in human–robot interaction. Interact Stud Soc Behav Commun Biol Artif Syst 20:455–486. https://doi.org/10.1075/is.18067.flo

    Article  Google Scholar 

  17. Jin J, Ji P, Liu Y (2014) Prioritising engineering characteristics based on customer online reviews for quality function deployment. J Eng Des 25:303–324

    Article  Google Scholar 

  18. Randall N, Šabanović S, Milojević S, Gupta A (2021) Top of the class: mining product characteristics associated with crowdfunding success and failure of home robots. Int J Soc Robot. https://doi.org/10.1007/s12369-021-00776-8

    Article  Google Scholar 

  19. Friedman B, Kahn Jr PH, Hagman J (2003) Hardware companions? What online AIBO discussion forums reveal about the human–robotic relationship. In: Proceedings of the SIGCHI conference on human factors in computing systems. pp 273–280

  20. Mubin O, Khan A, Obaid M (2016) # Naorobot: exploring Nao discourse on Twitter. In: Proceedings of the 28th Australian conference on computer-human interaction. pp 155–159

  21. Carter EJ, Reig S, Tan XZ, et al (2020) Death of a robot: social media reactions and language usage when a robot stops operating. In: Proceedings of the 2020 ACM/IEEE international conference on human–robot interaction. pp 589–597

  22. Tian L, Oviatt S (2021) A taxonomy of social errors in human–robot interaction. ACM Trans Hum Robot Interact 10:1–32. https://doi.org/10.1145/3439720

    Article  Google Scholar 

  23. Smith A, Anderson M (2016) Online shopping and E-Commerce. In: Pew research center. https://www.pewresearch.org/internet/2016/12/19/online-reviews/

  24. Valant J (2015) Online consumer reviews: the case of misleading or fake reviews

  25. Yang B, Liu Y, Liang Y, Tang M (2019) Exploiting user experience from online customer reviews for product design. Int J Inf Manage 46:173–186

    Article  Google Scholar 

  26. Jin J, Liu Y, Ji P, Kwong CK (2019) Review on recent advances in information mining from big consumer opinion data for product design. J Comput Inf Sci Eng. https://doi.org/10.1115/1.4041087

    Article  Google Scholar 

  27. See-To EWK, Ngai EWT (2018) Customer reviews for demand distribution and sales nowcasting: a big data approach. Ann Oper Res 270:415–431. https://doi.org/10.1007/s10479-016-2296-z

    Article  MATH  Google Scholar 

  28. Li H, Bhowmick SS, Sun A (2010) Affinity-driven prediction and ranking of products in online product review sites. In: Proceedings of the 19th ACM international conference on Information and knowledge management. pp 1745–1748

  29. Palese B, Usai A (2018) The relative importance of service quality dimensions in E-commerce experiences. Int J Inf Manage 40:132–140

    Article  Google Scholar 

  30. Decker R, Trusov M (2010) Estimating aggregate consumer preferences from online product reviews. Int J Res Mark 27:293–307

    Article  Google Scholar 

  31. Li Y-M, Chen H-M, Liou J-H, Lin L-F (2014) Creating social intelligence for product portfolio design. Decis Support Syst 66:123–134

    Article  Google Scholar 

  32. Xiao S, Wei C-P, Dong M (2016) Crowd intelligence: analyzing online product reviews for preference measurement. Inf Manag 53:169–182

    Article  Google Scholar 

  33. Law D, Gruss R, Abrahams AS (2017) Automated defect discovery for dishwasher appliances from online consumer reviews. Expert Syst Appl 67:84–94

    Article  Google Scholar 

  34. Trenz M, Berger B (2013) Analyzing online customer reviews-an interdisciplinary literature review and research agenda

  35. Hu N, Zhang J, Pavlou PA (2009) Overcoming the J-shaped distribution of product reviews. Commun ACM 52:144–147

    Article  Google Scholar 

  36. Koh NS, Hu N, Clemons EK (2010) Do online reviews reflect a product’s true perceived quality? An investigation of online movie reviews across cultures. Electron Commer Res Appl 9:374–385

    Article  Google Scholar 

  37. Li X, Hitt LM (2008) Self-selection and information role of online product reviews. Inf Syst Res 19:456–474

    Article  Google Scholar 

  38. Hu Y, Li X (2011) Context-dependent product evaluations: an empirical analysis of internet book reviews. J Interact Mark 25:123–133

    Article  Google Scholar 

  39. Singh JP, Irani S, Rana NP et al (2017) Predicting the “helpfulness” of online consumer reviews. J Bus Res 70:346–355

    Article  Google Scholar 

  40. Wu Y, Ngai EWT, Wu P, Wu C (2020) Fake online reviews: literature review, synthesis, and directions for future research. Decis Support Syst. https://doi.org/10.1016/j.dss.2020.113280

    Article  Google Scholar 

  41. Timoshenko A, Hauser JR (2019) Identifying customer needs from user-generated content. Mark Sci 38:1–20

    Article  Google Scholar 

  42. Li H, Ye Q, Law R (2013) Determinants of customer satisfaction in the hotel industry: an application of online review analysis. Asia Pac J Tour Res 18:784–802

    Article  Google Scholar 

  43. Haque TU, Saber NN, Shah FM (2018) Sentiment analysis on large scale Amazon product reviews. In: 2018 IEEE international conference on innovative research and development (ICIRD), IEEE. pp 1–6

  44. Lee J, Kim H, Choi J (2019) A study on user experience factors of display-type artificial intelligence speakers through semantic network analysis: focusing on online review analysis of the Amazon Echo. J Converg Cult Technol 5:9–23

    Google Scholar 

  45. Liu P, Xu Z, Ai J, Wang F (2017) Identifying indicators of fake reviews based on spammer’s behavior features. In: 2017 IEEE international conference on software quality, reliability and security companion (QRS-C). pp 396–403

  46. Lu Y, Zhang L, Xiao Y, Li Y (2013) Simultaneously detecting fake reviews and review spammers using factor graph model. In: Proceedings of the 5th annual ACM web science conference. pp 225–233

  47. Lin Y, Zhu T, Wu H, et al (2014) Towards online anti-opinion spam: spotting fake reviews from the review sequence. In: 2014 IEEE/ACM international conference on advances in social networks analysis and mining (ASONAM 2014), IEEE. pp 261–264

  48. Costa A, Guerreiro J, Moro S, Henriques R (2019) Unfolding the characteristics of incentivized online reviews. J Retail Consum Serv 47:272–281

    Article  Google Scholar 

  49. Luo JM, Vu HQ, Li G, Law R (2021) Understanding service attributes of robot hotels: a sentiment analysis of customer online reviews. Int J Hosp Manag 98:103032. https://doi.org/10.1016/j.ijhm.2021.103032

    Article  Google Scholar 

  50. Carames K, Mui K, Azad A, Giang* WCW (2021) Studying robot vacuums using online retailer reviews to understand human-automation interaction. In: Proceedings of the human factors and ergonomics society annual meeting. SAGE Publications Sage CA, Los Angeles, CA. pp 1029–1033

  51. Brooks DJ (2017) A human-centric approach to autonomous robot failures. Ph.D. dissertation, Department of Computer Science, University. Lowell, MA

  52. Benlian A, Koufaris M, Hess T (2011) Service quality in software-as-a-service: developing the SaaS-Qual measure and examining its role in usage continuance. J Manag Inf Syst 28:85–126

    Article  Google Scholar 

  53. Auer J (2018) Essays on price and usage effects. Universitätsbibliothek Tübingen, Tübingen

    Google Scholar 

  54. He W, Tian X, Tao R et al (2017) Application of social media analytics: a case of analyzing online hotel reviews. Online Inf Rev. https://doi.org/10.1108/OIR-07-2016-0201

    Article  Google Scholar 

  55. Devlin J, Chang MW, Lee K, Toutanova K (2018) Bert: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181004805

  56. Sutcliffe A, Rugg G (1998) A taxonomy of error types for failure analysis and risk assessment. Int J Hum Comput Interact 10:381–405. https://doi.org/10.1207/s15327590ijhc1004

    Article  Google Scholar 

  57. Adubor O, St. John R, Steinfeld A (2017) Personal safety is more important than cost of damage during robot failure. In: Proceedings of the companion of the 2017 ACM/IEEE International conference on human–robot interaction—HRI ’17. ACM Press, New York, USA, pp 403–403

  58. Salem M, Lakatos G, Amirabdollahian F, Dautenhahn K (2015) Towards safe and trustworthy social robots: ethical challenges and practical issues. In: International conference on social robotics. Springer, pp 584–593

  59. Akalin N, Kristoffersson A, Loutfi A (2021) Investigating the multidisciplinary perspective of perceived safety in human–robot interaction. arXiv preprint arXiv:210605854

  60. Washburn A, Adeleye A, An T, Riek LD (2020) Robot errors in proximate HRI: how functionality framing affects perceived reliability and trust. ACM Trans Hum Robot Interact 9:1–21. https://doi.org/10.1145/3380783

    Article  Google Scholar 

  61. Rosenthal S, Veloso M, Dey AK (2012) Is someone in this office available to help me? J Intell Rob Syst 66:205–221. https://doi.org/10.1007/s10846-011-9610-4

    Article  Google Scholar 

  62. Cha E, Matarić M (2016) Using nonverbal signals to request help during human–robot collaboration. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS). pp 5070–5076

  63. Honig S, Oron-Gilad T (2021) How user presence impacts perceptions and operation routines of robotic vacuum cleaners–a ‘Stay at home’experiment. In: Congress of the international ergonomics association. Springer, pp 282–290

  64. Kaur K, Singh T (2021) What motivates consumers to write online reviews? Qualitative research in the Indian cultural context. J Global Mark 34(3):170–188. https://doi.org/10.1080/08911762.2021.1882022

    Article  Google Scholar 

  65. Yoo KH, Gretzel U (2008) What motivates consumers to write online travel reviews? Inf Technol Tour 10:283–295

    Article  Google Scholar 

  66. Dixit S, Badgaiyan AJ, Khare A (2019) An integrated model for predicting consumer’s intention to write online reviews. J Retail Consum Serv 46:112–120

    Article  Google Scholar 

  67. Beldad A, Voutsas C (2018) Understanding the motivation to write reviews for mobile apps among German users: testing the expanded theory of planned behavior using a structural equation modeling approach. J Technol Behav Sci 3:301–311

    Article  Google Scholar 

  68. Cheung CMK, Lee MKO (2012) What drives consumers to spread electronic word of mouth in online consumer-opinion platforms. Decis Support Syst 53:218–225

    Article  Google Scholar 

  69. Ma WWK, Chan A (2014) Knowledge sharing and social media: altruism, perceived online attachment motivation, and perceived online relationship commitment. Comput Hum Behav 39:51–58

    Article  Google Scholar 

  70. Sun T, Youn S, Wu G, Kuntaraporn M (2006) Online word-of-mouth (or mouse): an exploration of its antecedents and consequences. J Comput Mediat Commun 11:1104–1127

    Article  Google Scholar 

  71. Mangold WG, Smith KT (2012) Selling to millennials with online reviews. Bus Horiz 55:141–153

    Article  Google Scholar 

  72. Archak N, Ghose A, Ipeirotis PG (2011) Deriving the pricing power of product features by mining consumer reviews. Manage Sci 57:1485–1509

    Article  MATH  Google Scholar 

  73. Schlosser AE (2011) Can including pros and cons increase the helpfulness and persuasiveness of online reviews? The interactive effects of ratings and arguments. J Consum Psychol 21:226–239

    Article  Google Scholar 

  74. Latikka R, Turja T, Oksanen A (2019) Self-efficacy and acceptance of robots. Comput Hum Behav 93:157–163

    Article  Google Scholar 

Download references

Acknowledgements

The first author is supported by the Helmsley Charitable Trust through the Agricultural, Biological and Cognitive Robotics Initiative and by the Marcus Endowment Fund both at Ben-Gurion University of the Negev.

Funding

No research funding was received for conducting this study. The first author was supported by scholarships as noted in the acknowledgments.

Author information

Authors and Affiliations

  1. Ben-Gurion University of the Negev, Be’er Sheva, Israel

    Shanee Honig, Yisrael Parmet & Tal Oron-Gilad

  2. Bar-Ilan University, Ramat Gan, Israel

    Alon Bartal

Authors
  1. Shanee Honig
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alon Bartal
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yisrael Parmet
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tal Oron-Gilad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shanee Honig.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

See Tables

Table 8 All failures classified by type, category and their applicability to the different robot types included in the analysis. 1 = failure was found in the reviews of the robot, R = Failure is relevant to this type of robot but no evidence of it was found in the reviews for the robot, N/A = this failure is not likely applicable for this type of robot
Full size table

8 and

Table 9 Statistical comparisons of the number of failures per review across different robots (* = p < .01, df = 1)
Full size table

9.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Honig, S., Bartal, A., Parmet, Y. et al. Using Online Customer Reviews to Classify, Predict, and Learn About Domestic Robot Failures. Int J of Soc Robotics (2022). https://doi.org/10.1007/s12369-022-00929-3

Download citation

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s12369-022-00929-3

Keywords

Search

Navigation