Skip to main content
SpringerLink
Log in

Building a speech recognition system with privacy identification information based on Google Voice for social robots

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Currently, many smart speakers, even social robots, appear on the market to help people's lives become more convenient. Usually, people use smart speakers to check their daily schedule or control home appliances in their house. Many social robots also include smart speakers. They have the common property of being used in voice control machines. Regardless of where the smart speaker is installed and used, when people start a conversation with voice equipment, a security or privacy risk is exposed. Hence, we want to build a speech recognition (SR) that contains the privacy identification information (PII) system in this paper. We call this the SR-PII system. We used a Google Artificial-Intelligence-Yourself (AIY) Voice Kit released from Google to build a simple, smart dialog speaker and included our SR-PII system. In our experiments, we test SR accuracy and the reliability of privacy settings in three environments (quiet, noise, and playing music). We also examine the cloud response and speaker response times during our experiments. The results show that the speaker response is approximately 3.74 s in the cloud environment and approximately 9.04 s from the speaker. We also showed the response accuracy of the speaker, which successfully prevented personal information with the SR-PII system in three environments. The speaker has a response mean time of approximately 8.86 s with 93% mean accuracy in a quiet room, approximately 9.18 s with 89% mean accuracy in a noisy environment, and approximately 9.62 s with 90% mean accuracy in an environment that plays music. We conclude that the SR-PII system can secure private information and that the most important factor affecting the response speed of the speaker is the network connection status. We hope that people can, through our experiments, have some guidelines in building social robots and installing the SR-PII system to protect users’ personal identification information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Graves A, Jaitly N (2014) Towards end-to-end speech recognition with recurrent neural networks. In: Proceedings of the International Conference on Machine Learning, pp 1764–1772

  2. Dong L, Xu S, Xu B (2018) Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp 5884–5888

  3. Hinton G (2012) Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Process Mag 29(6):82–97

    Article  Google Scholar 

  4. Xiong W, Wu L, Alleva F, Droppo J, Huang X, Stolcke A (2018) The Microsoft 2017 conversational speech recognition system. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp 5934–5938

  5. Salazar J, Kirchhoff K, Huang Z (2019) Self-attention networks for connectionist temporal classification in speech recognition. In: Proceedings of the ICASSP - IEEE International Conference on Acoustics, Speech and Signal Processing, pp 7115–7119

  6. Haeb-Umbach R, Watanabe R, Nakatani T, Bacchiani M (2019) Speech processing for digital home assistants. IEEE Signal Process Mag 36(6):111–124

    Article  Google Scholar 

  7. Li B, Sainath T, Narayanan A, Caroselli J (2017) Acoustic modeling for Google Home. In: Interspeech, pp 399–403

  8. Barker J, Watanabe S, Vincent E, Tr-mal J (2018) The fifth CHiME speech separation and recognition challenge: Dataset, task, and baselines. In: Interspeech, pp 1561–1565

  9. Yu D, Deng L (2015) Automatic speech recognition a deep learning approach. Springer, p 4

  10. Li B (2017) Acoustic modeling for google home. In: Proceedings of the Interspeech, pp 399–403

  11. Tian Z, Yi J, Tao J, Bai Y, Wen Z (2019) Self-attention transducers for end-to-end speech recognition. In: Proceedings of the Interspeech, pp 4395–4399

  12. Gannot S, Vincent E, Markovich-Golan S, Ozerov A (2017) A consolidated perspective on multimicrophone speech enhancement and source separation. In: IEEE/ACM transactions on ASLP, vol 25, no 4, pp 692–730

  13. Gangamohan P, Mittal VK, Yegnanarayana B (2012) A flexible analysis synthesis tool (fast) for studying the characteristic features of emotion in speech. In: Consumer Communications and Networking Conference (CCNC), pp 250–254

  14. Yegnanarayana B, Murty K (2009) Event-based instantaneous fundamental frequency estimation from speech signals. In: IEEE transactions on audio speech and language processing, vol 17, no 4, pp 614–624

  15. Krothapalli SR, Yadav J, Sarkar S, Koolagudi SG, Vuppala AK (2012) Neural network-based feature transformation for emotion independent speaker identification. Int J Speech Technol 15(3):335–349

    Article  Google Scholar 

  16. Desai S, Black AW, Yegnanarayana B, Prahallad K (2010) Spectral mapping using artificial neural networks for voice conversion. IEEE Trans Audio Speech Lang Process 18(5):954–964

    Article  Google Scholar 

  17. Saon G (2017) English conversational telephone speech recognition by humans and machines. In: Proceedings of the Interspeech, pp 132–136

  18. Karita S, Soplin NEY, Watanabe S, Delcroix M, Ogawa A, Nakatani T (2019) Improving transformer-based end-to-end speech recognition with connectionist temporal classification and language model integration. In: Proceedings of the Interspeech, pp 1408–1412

  19. Rahulamathavan Y, Sutharsini KR, Ray IG, Lu R, Rajarajan M (2019) Privacy-preserving vector-based speaker verification. In: IEEE/ACM Transactions on the audio speech language process, vol 27, no 3, pp 496–506

  20. European Parliament and Council (2016) Regulation (EU) 2016/679 of the European Parliament and the council of 27 Apr. 2016 on the Protection of Natural Persons With Regard to the Processing of Personal Data and on the Free Movement of Such Data and Repealing Directive 95/46/EC (General Data Protection Regulation

  21. Nautsch A, Jasserand C, Kindt E, Todisco M, Trancoso L, Evans N (2019) The GDPR & speech data: Reflections of legal and technology communities first steps towards a common understanding. In: Proceedings of the Interspeech, pp 3695–3699

  22. Papernot N, McDaniel P, Sinha A, Wellman MP (2018) SoK: security and privacy in machine learning. In: Proceedings of the IEEE European symposium security, privacy, pp 399–414

  23. Povey D, et al (2011) The Kaldi speech recognition toolkit. In: Proceedings of the IEEE workshop automatic speech recognition understanding

  24. Lin P-C, Yankson B, Hung PCK (2020) A prototype of privacy identification system for smart toy dialogue design. In: 17th IEEE International Conference on Networking, Sensing and Control (IEEE ICNSC 2020), Nanjing, China, pp 1–6

  25. Lin P-C, Yankson B, Lu ZH, Hung PCK (2019) Children privacy identification system in LINE Chatbot for smart toys. In: IEEE International Conference on Cloud Computing (IEEE CLOUD 2019), Milan, Italy

  26. Lin P-C, Lin YH, Lai C-A, Chauhan V, Arbai N (2019) Movie recommender system with perceptual engineering in Chatbot. In: The 2nd International Conference on Innovative & Advanced Multidisciplinary Research (ICIAMR 2019), Park Avenue Convention Centre, Singapore, Oct. 12–13

  27. Thomas L, Lothar F, Kilian P, Kai R (2004) Exploitation of public and private WiFi coverage for new business models. In: Lamersdorf W, Tschammer V, Amarger S (eds)

  28. Bahl P, Padmanabhan VN (2000) RADAR: an in-building RF-based user location and tracking system. In: Proceedings of 19th Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM’00), vol 2, pp 775–784

  29. Manikanta K, Kiran J, Dinesh B, Sachin K (2015) Decimeter Level Localization Using Wi-Fi. In: Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. SIGCOMM '15. New York, NY, USA. ACM. pp 269–282

  30. Franzago M (2018) Collaborative model-driven software engineering: a classification framework and a research map. IEEE Trans Softw Eng. https://doi.org/10.1109/TSE.2017.2755039

    Article  Google Scholar 

  31. Storey MA (2017) How social and communication channels shape and challenge a participatory culture in software development. In: IEEE Transactions on the Software Engineering, vol 43, no 2, pp 185–204

  32. Lin B (2016) Why developers are slacking off: understanding how software teams use slack. In: Proceedings of the 19th ACM Conference on Computer-Supported Cooperative Work & Social Computing (CSCW 16), pp 333–336

  33. Tian Y (2017) APIBot: question answering bot for API documentation. In: Proceedings of the 32nd IEEE/ACM Conference on the Automated Software Engineering (ASE 17), pp 153–158

  34. Nenkov N, Dimitrov G, Dyachenko Y, Koeva K (2016) Artificial intelligence technologies for personnel learning management systems. In: 2016 IEEE 8th International Conference on Intelligent Systems (IS), pp 189–195. https://doi.org/10.1109/IS.2016.7737420.

  35. Nguyen MH (2017) The latest market research trends and landscape in the growing AI Chatbot Industry. https://www.businessinsider.de/chatbot-market-stats-trends-size-ecosystem-research-2017-10?r=US&IR=T.

  36. Fawthrop A (2019) Voice payments set to drive a new era of commerce for Amazon Pay

  37. Payments pioneer launches Voice Pay transactions (2017) Biometric Technology Today, vol 15, pp 2–3, 05

  38. The Rise of Voice Payment Technology in Banking (2019)

  39. Hattersley L (2018) AIY Projects: create a voice kita with your raspberry Pi. In: The MagPi 2017.

  40. Lee H (2018) Voice user interface projects: build voice-enabled applications using dialogflow for google home and Alexa skills kit for Amazon Echo. Packt Publishing Ltd

  41. Gunawan TS, Mokhtar MN, Kartiwi M, Ismail N, Effendi MR, Qodim H (2020) Development of voice-based smart home security system using google voice kit. In: 2020 6th International Conference on Wireless and Telematics (ICWT), 2020, pp 1–4. https://doi.org/10.1109/ICWT50448.2020.9243633.

  42. Vimalkumar M, Sharma SK, Singh JB, Yogesh K (2021) Okay google, what about my privacy? User’s privacy perceptions and acceptance of voice-based digital assistants. Comput Hum Behav 120:106763. https://doi.org/10.1016/j.chb.2021.106763

    Article  Google Scholar 

  43. Krazit T (2010) Google finding its voice In: CNet. Retrieved from https://www.cnet.com/news/google-finding-its-voice/.

  44. Epstein J, Klinkenberg WD (2001) From eliza to internet: a brief history of computerized assessment. Comput Hum Behav 17(3):295–314. https://doi.org/10.1016/S0747-5632(01)00004-8

    Article  Google Scholar 

  45. Marketer E (2019) US voice assistant users 2019. In: Insider intelligence trends, forecasts and statistics. Retrieved February 1, 2021, from eMarketer website: https://www.emarketer.com/content/us-voice-assistant-users-2019.

  46. Foehr J, Germelmann CC (2020) Alexa, can I trust you? Exploring consumer paths to trust in smart voice-interaction technologies. J Assoc Consum Res 5(2):181–205. https://doi.org/10.1086/707731

    Article  Google Scholar 

  47. Bellman S, Johnson EJ, Kobrin SJ, Lohse GL (2004) International differences in information privacy concerns: a global survey of consumers. Inf Soc 20(5):313–324. https://doi.org/10.1080/01972240490507956

    Article  Google Scholar 

  48. Miltgen CL, Henseler J, Gelhard C, Popovi A (2016) Introducing new products that affect consumer privacy: a mediation model. J Bus Res 69(10):4659–4666. https://doi.org/10.1016/j.jbusres.2016.04.015

    Article  Google Scholar 

  49. Zeng F, Ye Q, Li J, Yang Z (2020) Does self-disclosure matter? A dynamic two-stage perspective for the personalization-privacy paradox. J Bus Res. https://doi.org/10.1016/j.jbusres.2020.02.006

    Article  Google Scholar 

  50. Herrero A, San MH, Garcia-De MLS (2017) Explaining the adoption of social networks sites for sharing user-generated content: a revision of the UTAUT2. Comput Hum Behav 71:209–217. https://doi.org/10.1016/j.chb.2017.02.007

    Article  Google Scholar 

  51. Merhi M, Hone K, Tarhini A (2019) A cross-cultural study of the intention to use mobile banking between Lebanese and British consumers: extending UTAUT2 with security, privacy, and trust”. Technol Soc 59:101. https://doi.org/10.1016/j.techsoc.2019.101151

    Article  Google Scholar 

  52. Kowalczuk P (2018) Consumer acceptance of smart speakers: a mixed-methods approach. J Res Indian Med 12(4):418–431. https://doi.org/10.1108/JRIM-01-2018-0022

    Article  Google Scholar 

  53. Pitardi V, Marriott RH (2021) Alexa, she’s not human, but Unveiling the drivers of consumers’ trust in voice-based artificial intelligence. Psychol Mark. https://doi.org/10.1002/mar.21457

    Article  Google Scholar 

  54. Smith H, Dinev T, Xu H (2011) Information privacy research: an interdisciplinary review. MIS Q 35(4):989–1016

    Article  Google Scholar 

  55. Weisbaum H (2018) Hey Alexa, How Secure Are Voice-Activated Assistants Like You? In: NBC News, https://www.nbcnews.com/tech/security/hey-alexa-how-secure-arevoice-activated-assistants-you-n824566

  56. Song L, Mittal P (2017) Inaudible Voice Commands. In Princeton University, New Jersey, USA

    Google Scholar 

  57. Russakovskii A (2017) Google Is Permanently Nerfing All Home Minis Because Mine Spied on Everything I Said 24/7 [Update x2]. In: Android Police https://www.androidpolice.com/2017/10/10/google-nerfing-home-minis-mine-spied-everything-said-247/

  58. Greatrex C, White M (2017) Parrot Manages to Fool Amazon’s Alexa and Orders His Own GiftBox Without His Owners Knowing. In: Mirror, https://www.mirror.co.uk/news/uk-news/parrot-manages-fool-amazons-alexa-11207953.

  59. Capital F (2016) Voice Recognition: Risks to Our Privacy In: Forbes, https://www.forbes.com/sites/realspin/2016/10/06/voice-recognition-every-single-day-every-word-you-say/#19368ea6786d

  60. Finn RL, Wright D, Friedewald M (2013) Seven types of privacy. In: Gutwirth S, Leenes R, De Hert P, Poullet Y (eds) European Data Protection: Coming of Age. Springer, New York, pp 3–32. https://doi.org/10.1007/978-94-007-5170-5_1

    Chapter  Google Scholar 

  61. Warren SD, Brandeis LD (1890) The right to privacy. Harward Law Rev 4:193–220. https://doi.org/10.2307/1321160

    Article  Google Scholar 

  62. Westin A (1967) Privacy and freedom. Atheneum Press, Cambridge

    Google Scholar 

  63. Altman I (1975) The environment and social behavior: privacy, personal space, territory, and crowding. Wadsworth Publishing Company, Monterey

    Google Scholar 

  64. Bygrave LA (2002) Data protection law: approaching its rationale, logic and limits. Wolters Kluwer, Alphen aan den Rijn

    Google Scholar 

  65. Solove DJ (2008) Understanding privacy. Harvard University Press, Cambridge

    Google Scholar 

  66. Kaminski ME (2015) Robots in the home: What will we have agreed to? Idaho Law Rev 51:661–677

    Google Scholar 

  67. Yankson B (2021) Continuous improvement process (CIP)-based privacy-preserving framework for smart connected toys. Int J Inf Secur 20:849–869. https://doi.org/10.1007/s10207-020-00535-2

    Article  Google Scholar 

  68. Kaminski ME, Rueben M, Grimm C, Smart WD (2017) Averting robot eyes. Maryland Law Rev. 76:983–1023

    Google Scholar 

  69. Krasnova H, Veltri NF, Günther O (2012) Self-disclosure and privacy calculus on social networking sites: the role of culture. Bus Inform Syst Eng 4:127–135. https://doi.org/10.1007/s12599-012-0216-6

    Article  Google Scholar 

  70. Trepte S, Reinecke L, Ellison NB, Quiring O, Yao MZ, Ziegele M (2017) A cross-cultural perspective on the privacy calculus. Soc Med Soc 3:1–13. https://doi.org/10.1177/2056305116688035

    Article  Google Scholar 

  71. Baxter P, Kennedy J, Senft E, Lemaignan S, Belpaeme T (2016) From characterising three years of HRI to methodology and reporting recommendations. In: 11th ACM/IEEE International Conference on Human–Robot Interaction (HRI) (Christchurch), pp 391–398. https://doi.org/10.1109/HRI.2016.7451777

  72. Lee MK, Tang KP, Forlizzi J, Kiesler S (2011) Understanding users' perception of privacy in human–robot interaction. In: Proceedings of the 6th International Conference on Human–Robot Interaction. ACM, Lausanne, pp 181–182. https://doi.org/10.1145/1957656.1957721

  73. Syrdal DS, Walters ML, Otero N, Koay KL, Dautenhahn K (2007) He knows when you are sleeping – Privacy and the personal robot companion. In: Proceedings of the 2007 AAAI workshop human implications of human–robot interaction, AAAI, Washington DC, pp 28–33

  74. Butler DJ, Huang J, Roesner F, Cakmak M (2015) The privacy-utility tradeoff for remotely teleoperated robots. In: Proceedings of the tenth annual ACM/IEEE International Conference on Human–Robot Interaction. ACM, Portland OR, pp. 27–34. https://doi.org/10.1145/2696454.2696484

  75. Krupp MM, Rueben M, Grimm CM, Smart WD (2017) Privacy and telepresence robotics: What do non-scientists think? In: Proceedings of the Companion of the 2017 ACM/IEEE International Conference on Human–Robot Interaction. ACM, Vienna, pp 175–176. https://doi.org/10.1145/3029798.3038384

  76. Rueben M, Bernieri FJ, Grimm CM, Smart WD (2017) Framing effects on privacy concerns about a home telepresence robot. In: Proceedings of the 2017 ACM/IEEE International Conference on Human–Robot Interaction. ACM, Vienna, pp 435–444. https://doi.org/10.1145/2909824.3020218

  77. Lutz C, Schöttler M, Hoffmann CP (2019) The privacy implications of social robots: scoping review and expert interviews. Mobile Media Commun 7:412–434. https://doi.org/10.1177/2050157919843961

    Article  Google Scholar 

  78. Eurobarometer (2015) Special Eurobarometer 427: Autonomous systems

  79. Madden M, Rainie L (2015) Americans' attitudes about privacy, security and surveillance. Pew Internet, Science and Tech Report

  80. Lutz C, Hoffmann CP, Ranzini G (2020) Data capitalism and the user: an exploration of privacy cynicism in Germany. New Media Soc 22:1168–1187. https://doi.org/10.1177/1461444820912544

    Article  Google Scholar 

Download references

Acknowledgements

This research was supported by the Ministry of Education, R.O.C., under the grant TEEP@AsiaPlus. The work in this paper was also supported by the Ministry of Science and Technology under Grant No. MOST 109-2221-E-035-063-MY2. The authors would like to express their appreciation to Feng Chia University and the anonymous reviewers for their feedback.

Author information

Authors and Affiliations

  1. Faculty of Information Engineering and Computer Science, Feng Chia University, Taichung City, Taiwan

    Pei-Chun Lin

  2. CEHC, University at Albany, State University of New York, Albany, USA

    Benjamin Yankson

  3. Graduate School of Information Science and Technology, The University of Tokyo, Tokyo, Japan

    Vishal Chauhan & Manabu Tsukada

Authors
  1. Pei-Chun Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Benjamin Yankson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vishal Chauhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Manabu Tsukada
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pei-Chun Lin.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, PC., Yankson, B., Chauhan, V. et al. Building a speech recognition system with privacy identification information based on Google Voice for social robots. J Supercomput 78, 15060–15088 (2022). https://doi.org/10.1007/s11227-022-04487-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-022-04487-3

Keywords

Search

Navigation