Abstract
Companies across domains are rapidly engaged in shifting computational power and intelligence from centralized cloud to fully decentralized edges to maximize value delivery, strengthen security and reduce latency. However, most companies have only recently started pursuing this opportunity and are therefore at the early stage of the cloud-to-edge transition. To provide an overview of AI deployment in the context of edge/cloud/hybrid architectures, we conduct a systematic literature review and a grey literature review. To advance understanding of how to integrate, deploy, operationalize and evolve AI models, we derive a framework from existing literature to accelerate the end-to-end deployment process. The framework is organized into five phases: Design, Integration, Deployment, Operation and Evolution. We make an attempt to analyze the extracted results by comparing and contrasting them to derive insights. The contribution of the paper is threefold. First, we conduct a systematic literature review in which we review the contemporary scientific literature and provide a detailed overview of the state-of-the-art of AI deployment. Second, we review the grey literature and present the state-of-practice and experience of practitioners while deploying AI models. Third, we present a framework derived from existing literature for the end-to-end deployment process and attempt to compare and contrast SLR and GLR results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Amershi, S., et al.: Software engineering for machine learning: a case study. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pp. 291–300. IEEE, May 2019
Bernardi, L., Mavridis, T., Estevez, P.: 150 successful machine learning models: 6 lessons learned at booking. com. In: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1743–1751, July 2019
Sculley, D., et al.: Hidden technical debt in machine learning systems. In: Advances in Neural Information Processing Systems, pp. 2503–2511 (2015)
Dahlmeier, D.: On the challenges of translating NLP research into commercial products. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 92–96, July 2017
Crankshaw, D., Gonzalez, J., Bailis, P.: Research for practice: prediction-serving systems. Commun. ACM 61(8), 45–49 (2018)
Hill, C., Bellamy, R., Erickson, T., Burnett, M.: Trials and tribulations of developers of intelligent systems: a field study. In: 2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC), pp. 162–170. IEEE, September 2016
Provost, F., Kohavi, R.: Guest editors’ introduction: on applied research in machine learning. Mach. Learn. 30(2–3), 127–132 (1998)
Keele, S.: Guidelines for performing systematic literature reviews in software engineering, vol. 5. Technical report, Version 2.3 EBSE Technical Report, EBSE (2007)
Kitchenham, B.A., Budgen, D., Brereton, P.: Evidence-Based Software Engineering and Systematic Reviews, vol. 4. CRC Press, Boca Raton (2015)
Garousi, V., Felderer, M., Mäntylä, M.V.: The need for multivocal literature reviews in software engineering: complementing systematic literature reviews with grey literature. In: Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering, pp. 1–6, June 2016
Williams, A.: Using reasoning markers to select the more rigorous software practitioners’ online content when searching for grey literature. In: Proceedings of the 22nd International Conference on Evaluation and Assessment in Software Engineering 2018, pp. 46–56, June 2018
Xiaojing, X.I.E., Govardhan, S.S.: A service mesh-based load balancing and task scheduling system for deep learning applications. In: 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID), pp. 843–849. IEEE, May 2020
Serebryakov, S., Milojicic, D., Vassilieva, N., Fleischman, S., Clark, R.D.: Deep learning cookbook: recipes for your AI infrastructure and applications. In 2019 IEEE International Conference on Rebooting Computing (ICRC), pp. 1–9. IEEE, November 2019
Li, J., Li, J. and Zhang, H., 2018, August. Deep learning based parking prediction on cloud platform. In: 2018 4th International Conference on Big Data Computing and Communications (BIGCOM), pp. 132–137. IEEE
Brayford, D., Vallecorsa, S., Atanasov, A., Baruffa, F., Riviera, W.: Deploying AI frameworks on secure HPC systems with containers. In: 2019 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1–6. IEEE, September 2019
Tuli, S., Basumatary, N., Buyya, R.: EdgeLens: deep learning based object detection in integrated IoT, fog and cloud computing environments. In: 2019 4th International Conference on Information Systems and Computer Networks (ISCON), pp. 496–502. IEEE, November 2019
Brumbaugh, E., et al.: Bighead: a framework-agnostic, end-to-end machine learning platform. In: 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pp. 551–560. IEEE, October 2019
Salza, P., Hemberg, E., Ferrucci, F., O’Reilly, U.M.: Towards evolutionary machine learning comparison, competition, and collaboration with a multi-cloud platform. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1263–1270, July 2017
Gupta, N., Anantharaj, K., Subramani, K.: Containerized architecture for edge computing in smart home: a consistent architecture for model deployment. In: 2020 International Conference on Computer Communication and Informatics (ICCCI), pp. 1–8. IEEE, January 2020
Cartas, A., et al.: A reality check on inference at mobile networks edge. In: Proceedings of the 2nd International Workshop on Edge Systems, Analytics and Networking, pp. 54–59, March 2019
Zhang, X., Wang, Y., Lu, S., Liu, L., Shi, W.: OpenEI: an open framework for edge intelligence. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pp. 1840–1851. IEEE, July 2019
Rovnyagin, M.M., Timofeev, K., Elenkin, A.A., Shipugin, V.A.: Cloud computing architecture for high-volume ML-based solutions. In: 2019 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (EIConRus), pp. 315–318. IEEE, January 2019
Pääkkönen, P., Pakkala, D.: Extending reference architecture of big data systems towards machine learning in edge computing environments. J. Big Data 7, 1–29 (2020). https://doi.org/10.1186/s40537-020-00303-y
Li, Y., Han, Z., Zhang, Q., Li, Z., Tan, H.: Automating cloud deployment for deep learning inference of real-time online services. In: IEEE INFOCOM 2020-IEEE Conference on Computer Communications, pp. 1668–1677. IEEE, July 2020
Data blueprint - IOA Knowledge Base. https://ioakb.com/fileattachment?file=b3O7X9NGQKO2OrG8ohmRvw%3D%3D&v=3
Machine Learning Based Advanced Analytics using Intel Technology. https://media.bitpipe.com/io_14x/io_147787/item_1954603/machine-learning-based-advanced-analytics-using-intel-ForDistribution-2.pdf
Next Generation Hybrid Cloud Data Analytics Solution. https://builders.intel.com/docs/datacenterbuilders/next-gen-hybrid-cloud-data-analytics-solution.pdf
Getting Started with Machine Learning in the Cloud. https://www.oracle.com/a/ocom/docs/machine-learning-goes-to-the-cloud-ebook.pdf
Machine Learning Lens-AWS Well-Architected Framework. https://d1.awsstatic.com/whitepapers/architecture/wellarchitected-Machine-Learning-Lens.pdf.Move the Algorithms, Not the Data
Easterbrook, S., Singer, J., Storey, M.-A., Damian, D.: Selecting empirical methods for software engineering research. In: Shull, F., Singer, J., Sjoberg D.I.K. (eds.) Guide to Advanced Empirical Software Engineering, pp. 285–311. Springer, London (2008). https://doi.org/10.1007/978-1-84800-044-5_11
Crankshaw, D., et al.: The missing piece in complex analytics: low latency, scalable model management and serving with velox. arXiv preprint arXiv:1409.3809, September 2014
Bosch, J., Olsson, H.H., Crnkovic, I.: Engineering AI systems: a research agenda. In: Artificial Intelligence Paradigms for Smart Cyber-Physical Systems, pp. 1–19. IGI Global (2020)
Arpteg, A., Brinne, B., Crnkovic-Friis, L., Bosch, J.: Software engineering challenges of deep learning. In: 2018 44th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), pp. 50–59. IEEE, August 2018
Baylor, D., et al.: TFX: a tensorflow-based production-scale machine learning platform. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1387–1395 (2017)
Crankshaw, D., et al.: The missing piece in complex analytics: low latency, scalable (2014)
Olston, C., et al.: Tensor flow-serving: flexible, high-performance ml serving. In: Workshop on ML Systems at NIPS (2017)
Breck, E., Cai, S., Nielsen, E., Salib, M., Sculley, D.: What’s your ML test score? A rubric for ML production systems (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
John, M.M., Holmström Olsson, H., Bosch, J. (2021). Architecting AI Deployment: A Systematic Review of State-of-the-Art and State-of-Practice Literature. In: Klotins, E., Wnuk, K. (eds) Software Business. ICSOB 2020. Lecture Notes in Business Information Processing, vol 407. Springer, Cham. https://doi.org/10.1007/978-3-030-67292-8_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-67292-8_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-67291-1
Online ISBN: 978-3-030-67292-8
eBook Packages: Computer ScienceComputer Science (R0)