Abstract
The process of building a complex Artificial Intelligence (AI) system must address two issues to guarantee its correct functioning: 1) the software engineering errors that are inadvertently introduced during the building of the system; and 2) the inherent uncertainty in finding a solution that generates actionable insights due to the application of AI techniques in building complex and interdependent modules. The first issue is addressed with the help of both a declarative approach to programming and the modularization of many complex algorithms, all in the form of libraries. However, the second issue calls for an incremental, agile-system engineering approach without fully committing to its development path, which is uncertain.
In this chapter, we detail a scrum-based agile approach that we pursue at AdventHealth to build intelligent data products in the consumer analytics space to better serve our patients. We make use of natural language processing (NLP) in conjunction with cutting-edge machine learning (ML) and deep learning (DL) techniques to build various data products, including an understanding of consumer needs, their grievances, and how well we serve our consumers. Each sprint of our agile approach typically incorporates formulation of the problem or the revision of a previous formulation, a time-consuming data preparation involving large, noisy, and incomplete data, and analysts’ evaluation of the system to provide feedback. Our overall experience with the agile approach is its quick turn-around time for incremental deliverables, where meeting the requirements of our stakeholders.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Blei, D., Ng, A., Jordan, M.: Latent dirichlet allocation. J. Mach. Learn. Res. 3(5), 993–1022 (2003)
Cho, K., van Merrienboer, B., Gulcehre, C., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of EMNLP (2014)
Cockburn, A., Highsmith, J.: Agile software development: the people factor. Computer 34(11), 131–133 (2001)
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuglu, K., Kuksa, P.: Natural Language Processing (Almost) from Scratch. J. Mach. Learn. Res. 12, 2493–2537 (2011)
Das, S.: High-Level Data Fusion. Artech House, Norwood (2008)
Das, S.: Computational Business Analytics. Chapman and Hall/CRC Press, Boca Raton (2014)
Dumais, S.T., Furnas, G.W., Landauer, T.K., Deerwester, S.: Using latent semantic analysis to improve information retrieval. In: Proceedings of CHI 1988: Conference on Human Factors in Computing, Washington, DC (1988)
Fowler, M., Highsmith, J.: The agile manifesto. Softw. Dev. 9(8), 28–35 (2001)
Fox, J., Das, S.: Safe and Sound: Artificial Intelligence in Hazardous Applications. AAAI-MIT Press, Cambridge (2000)
Graves, A., Mohamed, A.-R., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of ICASSP (2013)
Hofmann, T.: Probabilistic latent semantic analysis. In: Proceedings of the Conference on Uncertainty in Artificial Intelligence (UAI), Stockholm (1999)
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (2019)
Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences (2014). https://arxiv.org/abs/1404.2188
Karpathy, A., Fei-Fei, L.: Deep visual-semantic alignments for generating image descriptions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Kroll, P., MacIsaac, B.: Agility and Discipline Made Easy. Addison-Wesley Professional, Boston (2006)
LeadingAgile: Cheat Sheet for Product Backlog Refinement (Grooming). LeadingAgile (2021)
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)
Li, S.: Multi-Class Text Classification Model Comparison and Selection. Towards Data Science (2018). https://towardsdatascience.com/multi-class-text-classification-model-comparison-and-selection-5eb066197568
Mikolov, T., Chen, K., Corrado, K., Dean, J.: Efficient estimation of word representations in vector space (2013). https://arxiv.org/abs/1301.3781
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Proceedings of NIPS (2013)
Planning Poker (2021). https://www.planningpoker.com/
Product Plan (2021). https://www.productplan.com/glossary/fibonacci-agile-estimation/
Royce, W.: Managing the development of large software systems. In: Proceedings of IEEE WESCON, vol. 26, pp. 328–388 (1970)
Sarawagi, S., Bhamidipaty, A.: Interactive deduplication using active learning. In: International Conference on Very Large Data Bases (VLDB), pp. 269–278 (2002)
Schmidhuber, J., Hochreiter, S.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Scrum Institute (2021). https://www.scrum-institute.org/Burndown_Chart.php
Scrum.org (2021). https://www.scrum.org/resources/what-is-scrum
Software Testing Help: Agile Methodology: A Beginner’s Guide to Agile Method and Scrum (2021). http://softwaretestinghelp.com/
Sutskever, I., Martens, J., Hinton, G.: Generating text with recurrent neural networks. In: Proceedings of the ICML (2011)
Tur, G., Tur, D., Schapire, R.: Combining active and semi-supervised learning for spoken language understanding. Speech Commun. 45, 171–186 (2005)
Visual Paradigm (2021). https://www.visual-paradigm.com/scrum/how-to-use-scrum-board-for-agile-development/
Yih, W.-T., Chang, M.-W., He, X., Gao, J.: Semantic parsing via staged query graph generation: question answering with knowledge base. In: Proceedings of the ACL (2015)
Young, T., Hazarika, D., Cambria, E.: Recent Trends in Deep Learning Based NLP. IEEE Comput. Intell. Mag. 13(3), 55–75 (2018)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Das, S. et al. (2021). Agile Systems Engineering in Building Complex AI Systems. In: Lawless, W.F., Llinas, J., Sofge, D.A., Mittu, R. (eds) Engineering Artificially Intelligent Systems. Lecture Notes in Computer Science(), vol 13000. Springer, Cham. https://doi.org/10.1007/978-3-030-89385-9_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-89385-9_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89384-2
Online ISBN: 978-3-030-89385-9
eBook Packages: Computer ScienceComputer Science (R0)