Skip to main content
SpringerLink
Log in

Evaluation of AI-Based Digital Assistants in Smart Manufacturing

  • Conference paper
  • First Online:
Advances in Production Management Systems. Smart Manufacturing and Logistics Systems: Turning Ideas into Action (APMS 2022)

Part of the book series: IFIP Advances in Information and Communication Technology ((IFIPAICT,volume 664))

Abstract

Industry 5.0 complements the Industry 4.0 paradigm by highlighting research and innovation as drivers for a transition to a sustainable, human-centric and resilient industry. In this context, new types of interactions between operators and machines are facilitated, that can be realized through artificial intelligence (AI) based and voice-enabled Digital Intelligent Assistants (DIA). Apart from the existing technological challenges, this direction requires new methodologies for the evaluation of such technological solutions that will be able to treat AI in manufacturing as a socio-technical system. In this paper, we propose a framework for the evaluation of voice-enabled AI solutions in Industry 5.0, which consists of four dimensions: the trustworthiness of the AI system; the usability of the DIA; the cognitive workload of individual users; and the overall business benefits for the corporation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 119.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Maddikunta, P.K.R., et al.: Industry 5.0: a survey on enabling technologies and potential applications. J. Ind. Inf. Integr. 26, 100257 (2021)

    Google Scholar 

  2. Wellsandt, S., et al.: Hybrid-augmented intelligence in predictive maintenance with digital intelligent assistants. In: Annual Reviews in Control (In Press, Corrected Proof) (2022)

    Google Scholar 

  3. Dhiman, H., Wächter, C., Fellmann, M., Röcker, C.: Intelligent assistants. Bus. Inf. Syst. Eng. 1–21 (2022)

    Google Scholar 

  4. Rabelo, R.J., Romero, D., Zambiasi, S.P.: Softbots supporting the operator 4.0 at smart factory environments. In: Moon, I., Lee, G., Park, J., Kiritsis, D., Von Cieminski, G. (eds.) Advances in Production Management Systems. Smart Manufacturing for Industry 4.0. APMS 2018. IFIP Advances in Information and Communication Technology, vol. 536, pp. 456–464. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99707-0_57

  5. Bousdekis, A., et al.: Human-AI collaboration in quality control with augmented manufacturing analytics. In: Dolgui, A., Bernard, A., Lemoine, D., von Cieminski, G., Romero, D. (eds.) Advances in Production Management Systems. Artificial Intelligence for Sustainable and Resilient Production Systems. APMS 2021. IFIP Advances in Information and Communication Technology, vol. 633, pp.303–310. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85910-7_32

  6. Wellsandt, S., Hribernik, K., Thoben, K.D.: Anatomy of a digital assistant. In: Dolgui, A., Bernard, A., Lemoine, D., von Cieminski, G., Romero, D. (eds.) Advances in Production Management Systems. Artificial Intelligence for Sustainable and Resilient Production Systems. APMS 2021. IFIP Advances in Information and Communication Technology, vol. 633, pp. 321–330. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-85910-7_34

  7. High-Level Independent Group on Artificial Intelligence (AI HLEG). Ethics Guidelines for Trustworthy AI. https://ec.europa.eu/digital

  8. Floridi, L.: Establishing the rules for building trustworthy AI. Nat. Mach. Intell. 1(6), 261–262 (2019)

    Article  Google Scholar 

  9. Baneres, D., Guerrero-Roldán, A.E., Rodríguez-González, M.E., Karadeniz, A.: A predictive analytics infrastructure to support a trustworthy early warning system. Appl. Sci. 11(13), 5781 (2021)

    Article  Google Scholar 

  10. Thiebes, S., Lins, S., Sunyaev, A.: Trustworthy artificial intelligence. Electron. Mark. 31(2), 447–464 (2020). https://doi.org/10.1007/s12525-020-00441-4

    Article  Google Scholar 

  11. Georgieva, I., Lazo, C., Timan, T., Van Veenstra, A.F.: From AI ethics principles to data science practice: a reflection and a gap analysis based on recent frameworks and practical experience. AI Ethics 1–15 (2022)

    Google Scholar 

  12. Kocaballi, A.B., Laranjo, L., Coiera, E.: Understanding and measuring user experience in conversational interfaces. Interact. Comput. 31(2), 192–207 (2019)

    Article  Google Scholar 

  13. Finstad, K.: The usability metric for user experience. Interact. Comput. 22(5), 323–327 (2010)

    Article  Google Scholar 

  14. Lewis, J.R.: IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int. J. Hum. Comput. Interact. 7(1), 57–78 (1995)

    Article  Google Scholar 

  15. Kirakowski, J.: Software usability measurement inventory SUMI. SUMI (2011). http://sumi.uxp.ie/en/index.php

  16. Hassenzahl, M., Burmester, M., Koller, F.: AttrakDiff: a questionnaire to measure perceived hedonic and pragmatic quality. Mensch Comput. 57, 187–196 (2003)

    Google Scholar 

  17. Brooke, J.: SUS-A quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996)

    Google Scholar 

  18. Zwakman, D.S., Pal, D., Arpnikanondt, C.: Usability evaluation of artificial intelligence-based voice assistants: the case of Amazon Alexa. SN Comput. Sci. 2(1), 1–16 (2021). https://doi.org/10.1007/s42979-020-00424-4

    Article  Google Scholar 

  19. Murad, C., Munteanu, C., Cowan, B.R., Clark, L.: Revolution or evolution? Speech interaction and HCI design guidelines. IEEE Pervasive Comput. 18(2), 33–45 (2019)

    Article  Google Scholar 

  20. Holmes, S., Moorhead, A., Bond, R., Zheng, H., Coates, V., McTear, M.: Usability testing of a healthcare chatbot: can we use conventional methods to assess conversational user interfaces? In: Proceedings of the 31st European Conference on Cognitive Ergonomics, pp. 207–214 (2019)

    Google Scholar 

  21. Cowan, B.R., et al.: What can i help you with? Infrequent users’ experiences of intelligent personal assistants. In: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, pp. 1–12 (2017)

    Google Scholar 

  22. Babel, M., McGuire, G., King, J.: Towards a more nuanced view of vocal attractiveness. PLoS ONE 9(2), e88616 (2014)

    Article  Google Scholar 

  23. Hone, K.S., Graham, R.: Towards a tool for the subjective assessment of speech system interfaces (SASSI). Nat. Lang. Eng. 6(3–4), 287–303 (2000)

    Article  Google Scholar 

  24. Polkosky, M.D.: Machines as mediators: the challenge of technology for interpersonal communication theory and research, pp. 48–71. Routledge (2008)

    Google Scholar 

  25. Turunen, M., Hakulinen, J., Melto, A., Heimonen, T., Laivo, T., Hella, J.: SUXES-user experience evaluation method for spoken and multimodal interaction. In: Tenth Annual Conference of the International Speech Communication Association (2009)

    Google Scholar 

  26. Sweller, J.: Cognitive load during problem solving: effects on learning. Cognit. Sci. 12(2), 257–285 (1988)

    Article  Google Scholar 

  27. Brachten, F., Brünker, F., Frick, N.R., Ross, B., Stieglitz, S.: On the ability of virtual agents to decrease cognitive load: an experimental study. Inf. Syst. e-Bus. Manag. 18(2), 187–207 (2020)

    Google Scholar 

  28. Rubio, S., Díaz, E., Martín, J., Puente, J.M.: Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol. 53(1), 61–86 (2004)

    Article  Google Scholar 

  29. Cao, A., Chintamani, K.K., Pandya, A.K., Ellis, R.D.: NASA TLX: software for assessing subjective mental workload. Behav. Res. Methods 41(1), 113–117 (2009). https://doi.org/10.3758/BRM.41.1.113

    Article  Google Scholar 

  30. Meshkati, N., Hancock, P.A., Rahimi, M., Dawes, S.M.: Techniques in mental workload assessment (1995)

    Google Scholar 

  31. Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, no. 9, pp. 904–908. Sage publications, Sage CA: Los Angeles, CA (2006)

    Google Scholar 

  32. Castro, S.C., Quinan, P.S., Hosseinpour, H., Padilla, L.: Examining effort in 1d uncertainty communication using individual differences in working memory and NASA-TLX. IEEE Trans. Vis. Comput. Graph. 28(1), 411–421 (2021)

    Article  Google Scholar 

  33. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): results of empirical and theoretical research. In: Advances in psychology, vol. 52, pp. 139–183. North-Holland (1988)

    Google Scholar 

  34. Zhu, L., Johnsson, C., Varisco, M., Schiraldi, M.M.: Key performance indicators for manufacturing operations management–gap analysis between process industrial needs and ISO 22400 standard. Procedia Manuf. 25, 82–88 (2018)

    Article  Google Scholar 

  35. Galy, E., Cariou, M., Mélan, C.: What is the relationship between mental workload factors and cognitive load types? Int. J. Psychophysiol. 83(3), 269–275 (2012)

    Article  Google Scholar 

  36. Matt, C., Hess, T., Benlian, A.: Digital transformation strategies. Bus. Inf. Syst. Eng. 57(5), 339–343 (2015)

    Article  Google Scholar 

  37. Dellermann, D., Ebel, P., Söllner, M., Leimeister, J.M.: Hybrid intelligence. Bus. Inf. Syst. Eng. 61(5), 637–643 (2019)

    Google Scholar 

  38. Mirbabaie, M., Stieglitz, S., Brünker, F., Hofeditz, L., Ross, B., Frick, N.R.: Understanding collaboration with virtual assistants–the role of social identity and the extended self. Bus. Inf. Syst. Eng. 63(1), 21–37 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

This work is partly funded by the European Union’s Horizon 2020 project COALA “COgnitive Assisted agile manufacturing for a LAbor force supported by trustworthy Artificial Intelligence” (Grant agreement No 957296). The work presented here reflects only the authors’ view and the European Commission is not responsible for any use that may be made of the information it contains.

Author information

Authors and Affiliations

  1. Information Management Unit (IMU), Institute of Communication and Computer Systems (ICCS), National Technical University of Athens (NTUA), Athens, Greece

    Alexandros Bousdekis, Gregoris Mentzas & Dimitris Apostolou

  2. Department of Informatics, University of Piraeus, Piraeus, Greece

    Dimitris Apostolou

  3. BIBA - Bremer Institut für Produktion und Logistik GmbH at the University of Bremen, Bremen, Germany

    Stefan Wellsandt

Authors
  1. Alexandros Bousdekis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gregoris Mentzas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dimitris Apostolou
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefan Wellsandt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexandros Bousdekis .

Editor information

Editors and Affiliations

  1. Pohang University of Science and Technology, Pohang, Korea (Republic of)

    Duck Young Kim

  2. ZF Friedrichshafen AG, Friedrichshafen, Germany

    Gregor von Cieminski

  3. Tecnológico de Monterrey, Mexico City, Mexico

    David Romero

Rights and permissions

Reprints and permissions

Copyright information

© 2022 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bousdekis, A., Mentzas, G., Apostolou, D., Wellsandt, S. (2022). Evaluation of AI-Based Digital Assistants in Smart Manufacturing. In: Kim, D.Y., von Cieminski, G., Romero, D. (eds) Advances in Production Management Systems. Smart Manufacturing and Logistics Systems: Turning Ideas into Action. APMS 2022. IFIP Advances in Information and Communication Technology, vol 664. Springer, Cham. https://doi.org/10.1007/978-3-031-16411-8_58

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16411-8_58

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16410-1

  • Online ISBN: 978-3-031-16411-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation