Developing and Operating Artificial Intelligence Models in Trustworthy Autonomous Systems

Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 415)


Companies dealing with Artificial Intelligence (AI) models in Autonomous Systems (AS) face several problems, such as users’ lack of trust in adverse or unknown conditions, gaps between software engineering and AI model development, and operation in a continuously changing operational environment. This work-in-progress paper aims to close the gap between the development and operation of trustworthy AI-based AS by defining an approach that coordinates both activities. We synthesize the main challenges of AI-based AS in industrial settings. We reflect on the research efforts required to overcome these challenges and propose a novel, holistic DevOps approach to put it into practice. We elaborate on four research directions: (a) increased users’ trust by monitoring operational AI-based AS and identifying self-adaptation needs in critical situations; (b) integrated agile process for the development and evolution of AI models and AS; (c) continuous deployment of different context-specific instances of AI models in a distributed setting of AS; and (d) holistic DevOps-based lifecycle for AI-based AS.


DevOps Autonomous Systems AI Trustworthiness 



This work has been partially supported by the Beatriz Galindo programme (BGP18/00075) and the Catalan Research Agency (AGAUR, contract 2017 SGR 1694).


  1. 1.
    Arnold, M., Bellamy, R.K., et al.: FactSheets: increasing trust in AI services through supplier’s declarations of conformity. IBM J. Res. Dev. 63(4/5), 6-1 (2019)Google Scholar
  2. 2.
    Gharib, M., Lollini, P., et al.: On the safety of automotive systems incorporating machine learning based components. In: DSN-W, pp. 271–274 (2018)Google Scholar
  3. 3.
    Borg, M., Englund, C., et al.: Safely entering the deep: a review of verification and validation for machine learning and a challenge elicitation in the automotive industry. J. Autom. Softw. Eng. 1(1), 1 (2019)CrossRefGoogle Scholar
  4. 4.
    Ozkaya, I.: What is really different in engineering AI-enabled systems? IEEE Softw. 37(4), 3–6 (2020)CrossRefGoogle Scholar
  5. 5.
    Serban, A., van der Blom, K., Hoos, H., Visser, J.: Adoption and effects of software engineering best practices in machine learning. arXiv:2007.14130 (2020)
  6. 6.
    Aniculaesei, A., Grieser, J., et al.: Toward a holistic software systems engineering approach for dependable autonomous systems. In: SEFAIAS, pp. 23–30 (2018)Google Scholar
  7. 7.
    European Commission: On Artificial Intelligence - a European approach to excellence and trust.
  8. 8.
    Ribeiro, M.T., Singh, S., Guestrin, C.: Why should I trust you? Explaining the predictions of any classifier. In: ACM SIGKDD, pp. 1135–1144 (2016)Google Scholar
  9. 9.
    Akkiraju, R., et al.: Characterizing machine learning processes: a maturity framework. In: Fahland, D., Ghidini, C., Becker, J., Dumas, M. (eds.) BPM 2020. LNCS, vol. 12168, pp. 17–31. Springer, Cham (2020). Scholar
  10. 10.
    Kim, M., Zimmermann, T., DeLine, R., Begel, A.: Data scientists in software teams: state of the art and challenges. TSE 44(11), 1024–1038 (2017)Google Scholar
  11. 11.
    Heck, P.: Software Engineering for Machine Learning Applications.
  12. 12.
    Santhanam, P., Farchi, E., Pankratius, V.: Engineering reliable deep learning systems. arXiv:1910.12582 (2019)
  13. 13.
    Saltz, J., Crowston, K., et al.: Comparing data science project management methodologies via a controlled experiment. In: HICSS (2017)Google Scholar
  14. 14.
    Lwakatare, L.E., Raj, A., Crnkovic, I., Bosch, J., Olsson, H.H.: Large-scale machine learning systems in real-world industrial settings: a review of challenges and solutions. Inf. Softw. Technol. 127, 106368 (2020)CrossRefGoogle Scholar
  15. 15.
    Garcia, R., Sreekanti, V., et al.: Context: the missing piece in the machine learning lifecycle. In: KDD CMI Workshop, vol. 114 (2018)Google Scholar
  16. 16.
    Nguyen-Duc, A., Sundbø, I., et al.: A multiple case study of artificial intelligent system development in industry. In: EASE, pp. 1–10 (2020)Google Scholar
  17. 17.
    Kästner, C., Kang, E.: Teaching software engineering for AI-enabled systems. arXiv preprint arXiv:2001.06691 (2020)
  18. 18.
    Banijamali, A., Jamshidi, P., Kuvaja, P., Oivo, M.: Kuksa: a cloud-native architecture for enabling continuous delivery in the automotive domain. In: Franch, X., Männistö, T., Martínez-Fernández, S. (eds.) PROFES 2019. LNCS, vol. 11915, pp. 455–472. Springer, Cham (2019). Scholar
  19. 19.
    IBM: An architectural blueprint for autonomic computing. IBM White Paper, vol. 31, no. 2003, pp. 1–6 (2006)Google Scholar
  20. 20.
    Siebert, J., et al.: Towards guidelines for assessing qualities of machine learning systems. In: Shepperd, M., Brito e Abreu, F., Rodrigues da Silva, A., Pérez-Castillo, R. (eds.) QUATIC 2020. CCIS, vol. 1266, pp. 17–31. Springer, Cham (2020). Scholar
  21. 21.
    Trustworthy AI: explainability, safety and verifiability.
  22. 22.
  23. 23.
    Breck, E., Cai, S., Nielsen, E., Salib, M., Sculley, D.: The ML test score: a rubric for ML production readiness and technical debt reduction. In: Big Data, pp. 1123–1132 (2017)Google Scholar
  24. 24.
  25. 25.
    Gustavsson, T.: Benefits of agile project management in a non-software development context: a literature review. PM World J. 114–124 (2016)Google Scholar
  26. 26.
    Shearer, C.: The CRISP-DM model: the new blueprint for data mining. J. Data Warehous. 5(4), 13–22 (2000)Google Scholar
  27. 27.
    Studer, S., Bui, T.B., et al.: Towards CRISP-ML (Q): a machine learning process model with quality assurance methodology. arXiv:2003.05155 (2020)
  28. 28.
  29. 29.
  30. 30.
    Capilla, R., Fuentes, L., Lochau, M.: Software variability in dynamic environments. J. Syst. Softw. 156, 62–64 (2019)CrossRefGoogle Scholar
  31. 31.
    Torrey, L., Shavlik, J.: Transfer learning. In: Handbook of Research on Machine Learning Applications and Trends, pp. 242–264. IGI Global (2010)Google Scholar
  32. 32.
    Settles, B., Brachman, R.J.: Active learning. In: Synthesis Lectures on AI and ML. Morgan Claypool (2012)Google Scholar
  33. 33.
  34. 34.
  35. 35.
    Ebert, C., Gallardo, G., Hernantes, J., Serrano, N.: DevOps. IEEE Softw. 33(3), 94–100 (2016)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2021

Authors and Affiliations

  1. 1.Universitat Politècnica de Catalunya - BarcelonaTechBarcelonaSpain
  2. 2.Fraunhofer IESEKaiserslauternGermany

Personalised recommendations