Skip to main content
SpringerLink
Log in

Re-orienting Toward the Science of the Artificial: Engineering AI Systems

  • Chapter
  • First Online:
Systems Engineering and Artificial Intelligence

Abstract

AI-enabled systems are becoming more pervasive, yet system engineering techniques still face limitations in how AI systems are being deployed. This chapter provides a discussion of the implications of hierarchical component composition and the importance of data in bounding AI system performance and stability. Issues of interoperability and uncertainty are introduced and how they can impact emergent behaviors of AI systems are illustrated through the presentation of a natural language processing (NLP) system used to provide similarity comparisons of organizational corpora. Within the bounds of this discussion, we examine how the concepts from Design science can introduce additional rigor to AI complex system engineering.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Abukwaik, H., Abujayyab, M., Humayoun, S. R., & Rombach, D. (2016). Extracting conceptual interoperability constraints from API documentation using machine learning. Proceedings of the 38th International Conference on Software Engineering Companion, 701–703.

    Google Scholar 

  • Alpcan, T., Erfani, S. M., & Leckie, C. (2017). Toward the starting line: A systems engineering approach to strong AI. ArXiv:1707.09095.

    Google Scholar 

  • Ashmore, R., Calinescu, R., & Paterson, C. (2019). Assuring the machine learning lifecycle: Desiderata, methods, and challenges. ArXiv:1905.04223.

    Google Scholar 

  • Backlund, A. (2000). The definition of system. Kybernetes: The International Journal of Systems & Cybernetics, 29(4), 444–451.

    Google Scholar 

  • Baskerville, R. (2008). What design science is not. European Journal of Information Systems, 17(5), 441–443. https://doi.org/10.1057/ejis.2008.45

    Article  Google Scholar 

  • Belani, H., Vukovic, M., & Car, Ž. (2019). Requirements Engineering Challenges in Building AI-Based Complex Systems. 2019 IEEE 27th International Requirements Engineering Conference Workshops (REW), 252–255.

    Google Scholar 

  • Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan), 993–1022.

    Google Scholar 

  • Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. Proceedings of IEEE Big Data.

    Google Scholar 

  • Brings, J., Daun, M., Keller, K., Obe, P. A., & Weyer, T. (2020). A systematic map on verification and validation of emergent behavior in software engineering research. Future Generation Computer Systems, 112, 1010–1037.

    Article  Google Scholar 

  • Buisson, B., & Lakehal, D. (2019). Towards an integrated machine-learning framework for model evaluation and uncertainty quantification. Nuclear Engineering and Design, 354, 110197.

    Google Scholar 

  • Carleton, A. D., Harper, E., Menzies, T., Xie, T., Eldh, S., & Lyu, M. R. (2020). The AI Effect: Working at the Intersection of AI and SE. IEEE Software, 37(4), 26–35.

    Article  Google Scholar 

  • Cobb, A. D., & Jalaian, B. (2020). Scaling Hamiltonian Monte Carlo Inference for Bayesian Neural Networks with Symmetric Splitting. ArXiv:2010.06772.

    Google Scholar 

  • Cohen, J. E., & Newmans, C. M. (1985). When will a large complex system be stable? Journal of Theoretical Biology, 113, 153–156.

    Article  MathSciNet  Google Scholar 

  • D’Ambrogio, A., & Durak, U. (2016). Setting systems and simulation life cycle processes side by side. IEEE International Symposium on Systems Engineering (ISSE), 2016, 1–7.

    Google Scholar 

  • De Michell, G., & Gupta, R. K. (1997). Hardware/software co-design. Proceedings of the IEEE, 85(3), 349–365.

    Article  Google Scholar 

  • Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. ArXiv:1810.04805.

    Google Scholar 

  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778.

    Google Scholar 

  • Hevner, A. R., March, S. T., Park, J., & Ram, S. (2004). Design Science in Information Systems Research. MIS Quarterly, 28(1), 75–105. https://doi.org/10.2307/25148625

    Article  Google Scholar 

  • I.S.C. Committee. (1990). IEEE Standard Glossary of Software Engineering Terminology (IEEE Std 610.12–1990). Los Alamitos. CA IEEE Comput. Soc.

    Google Scholar 

  • Jalaian, B., Lee, M., & Russell, S. (2019). Uncertain Context: Uncertainty Quantification in Machine Learning. AI Magazine, 39(4).

    Google Scholar 

  • Khomh, F., Adams, B., Cheng, J., Fokaefs, M., & Antoniol, G. (2018). Software engineering for machine-learning applications: The road ahead. IEEE Software, 35(5), 81–84.

    Article  Google Scholar 

  • Kläs, M., & Jöckel, L. (2020). A Framework for Building Uncertainty Wrappers for AI/ML-Based Data-Driven Components. International Conference on Computer Safety, Reliability, and Security, 315–327.

    Google Scholar 

  • Kuras, M. L., & White, B. E. (2005). Engineering Enterprises Using Complex-System Engineering. INCOSE International Symposium, 15(1), 251–265.

    Article  Google Scholar 

  • Lewis, G. A., Morris, E., Simanta, S., & Wrage, L. (2008). Why standards are not enough to guarantee end-to-end interoperability. Seventh International Conference on Composition-Based Software Systems (ICCBSS 2008), 164–173.

    Google Scholar 

  • Li, Y. H., & Jain, A. K. (1998). Classification of text documents. The Computer Journal, 41(8), 537–546.

    Article  Google Scholar 

  • Li, Z., Sim, C. H., & Low, M. Y. H. (2006). A survey of emergent behavior and its impacts in agent-based systems. 2006 4th IEEE International Conference on Industrial Informatics, 1295–1300.

    Google Scholar 

  • Lwakatare, L. E., Raj, A., Bosch, J., Olsson, H., & Crnkovic, I. (2019). A Taxonomy of Software Engineering Challenges for Machine Learning Systems: An Empirical Investigation (pp. 227–243). https://doi.org/10.1007/978-3-030-19034-7_14

  • Maaten, L. van der, & Hinton, G. (2008). Visualizing data using t-SNE. Journal of Machine Learning Research, 9(Nov), 2579–2605.

    Google Scholar 

  • Maier, M. W., Rainey, L. B., & Tolk, A. (2015). The role of modeling and simulation in system of systems development. Modeling and Simulation Support for System of Systems Engineering Applications, 11–41.

    Google Scholar 

  • Mali, N., & Bojewar, S. (2015). A Survey of ETL Tools. International Journal of Computer Techniques, 2(5), 20–27.

    Google Scholar 

  • Markus, M. L., Majchrzak, A., & Gasser, L. (2002). A design theory for systems that support emergent knowledge processes. MIS Quarterly, 179–212.

    Google Scholar 

  • May, R. M. (1972). Will a large complex system be stable? Nature, 238(5364), 413–414.

    Article  Google Scholar 

  • May, R. M. (1976). Simple mathematical models with very complicated dynamics. Nature, 261(5560), 459–467.

    Article  Google Scholar 

  • McDuff, D., Cheng, R., & Kapoor, A. (2018). Identifying Bias in AI using Simulation. ArXiv:1810.00471 [Cs, Stat]. http://arxiv.org/abs/1810.00471

  • Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient estimation of word representations in vector space. ArXiv:1301.3781.

    Google Scholar 

  • Mittal, S., & Rainey, L. (2014). Harnessing Emergence: The design and control of emergent behavior in system of systems engineering. Proceedings of the Summer Simulation Multi-Conference.

    Google Scholar 

  • Mittal, Saurabh. (2019). New frontiers in modeling and simulation in complex systems engineering: The case of synthetic emergence. In Summer of Simulation (pp. 173–194). Springer.

    Google Scholar 

  • Newman, M. E. (2011). Complex systems: A survey. ArXiv:1112.1440.

    Google Scholar 

  • Nilsson, J. (2019). System of systems interoperability machine learning model [PhD Thesis]. Lule\aa University of Technology.

    Google Scholar 

  • Nilsson, J., Sandin, F., & Delsing, J. (2019). Interoperability and machine-to-machine translation model with mappings to machine learning tasks. 2019 IEEE 17th International Conference on Industrial Informatics (INDIN), 1, 284–289.

    Google Scholar 

  • Ning, C., & You, F. (2019). Optimization under uncertainty in the era of big data and deep learning: When machine learning meets mathematical programming. Computers & Chemical Engineering, 125, 434–448.

    Article  Google Scholar 

  • Ottino, J. M. (2004). Engineering complex systems. Nature, 427(6973), 399–399.

    Article  Google Scholar 

  • Probst, P., Bischl, B., & Boulesteix, A.-L. (2018). Tunability: Importance of Hyperparameters of Machine Learning Algorithms.

    Google Scholar 

  • Rehurek, R., & Sojka, P. (2010). Software framework for topic modelling with large corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks.

    Google Scholar 

  • Röder, M., Both, A., & Hinneburg, A. (2015). Exploring the Space of Topic Coherence Measures. Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pp. 399–408. https://doi.org/10.1145/2684822.2685324

  • Russell, S., Haddad, M., Bruni, M., & Granger, M. (2010). Organic Evolution and the Capability Maturity of Business Intelligence. AMCIS, 501.

    Google Scholar 

  • Russell, S., Moskowitz, I., & Raglin, A. (2017). Human Information Interaction, Artificial Intelligence, and Errors. In Autonomy and Artificial Intelligence: A Threat or Savior? (pp. 71–101). https://doi.org/10.1007/978-3-319-59719-5_4

  • Russell, S., & Moskowitz, I. S. (2016, March 4). Human Information Interaction, Artificial Intelligence, and Errors. 2016 AAAI Spring Symposium Series. 2016 AAAI Spring Symposium. https://www.aaai.org/ocs/index.php/SSS/SSS16/paper/view/12767

  • Salama, A., Linke, A., Rocha, I. P., & Binnig, C. (2019). XAI: A Middleware for Scalable AI. DATA, 109–120.

    Google Scholar 

  • Schindel, W. D. (1996). System Engineering: An Overview of Complexity’s Impact. SAE Technical Paper.

    Google Scholar 

  • Schluse, M., Priggemeyer, M., Atorf, L., & Rossmann, J. (2018). Experimentable digital twins—Streamlining simulation-based systems engineering for industry 4.0. IEEE Transactions on Industrial Informatics, 14(4), 1722–1731.

    Google Scholar 

  • Simon, H. A. (1969). The sciences of the artificial. Cambridge, MA.

    Google Scholar 

  • Simon, H. A. (1988). The science of design: Creating the artificial. Design Issues, 67–82.

    Google Scholar 

  • Simon, H. A. (1991). The architecture of complexity. In Facets of systems science (pp. 457–476). Springer.

    Google Scholar 

  • Smith, L. N. (2018). A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay. ArXiv:1803.09820.

    Google Scholar 

  • Thomas, P. S., da Silva, B. C., Barto, A. G., Giguere, S., Brun, Y., & Brunskill, E. (2019). Preventing undesirable behavior of intelligent machines. Science, 366(6468), 999–1004.

    Article  Google Scholar 

  • Thurner, S., Hanel, R., & Klimek, P. (2018). Introduction to the theory of complex systems. Oxford University Press.

    Book  Google Scholar 

  • Tolk, A., Diallo, S., & Mittal, S. (2018). Complex systems engineering and the challenge of emergence. Emergent Behavior in Complex Systems Engineering: A Modeling and Simulation Approach, 79–97.

    Google Scholar 

  • Trinchero, R., Larbi, M., Torun, H. M., Canavero, F. G., & Swaminathan, M. (2018). Machine learning and uncertainty quantification for surrogate models of integrated devices with a large number of parameters. IEEE Access, 7, 4056–4066.

    Article  Google Scholar 

  • Wieringa, R. J. (2014). What Is Design Science? In R. J. Wieringa (Ed.), Design Science Methodology for Information Systems and Software Engineering (pp. 3–11). Springer. https://doi.org/10.1007/978-3-662-43839-8_1

  • Yang, Z., Yu, Y., You, C., Steinhardt, J., & Ma, Y. (2020). Rethinking bias-variance trade-off for generalization of neural networks. ArXiv:2002.11328.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Army Research Laboratory, Adelphi, MD, USA

    Stephen Russell & Brian Jalaian

  2. Naval Research Laboratory, Washington, DC, USA

    Ira S. Moskowitz

Authors
  1. Stephen Russell
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian Jalaian
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ira S. Moskowitz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Stephen Russell .

Editor information

Editors and Affiliations

  1. Paine College, Augusta, GA, USA

    William F. Lawless

  2. Information Technology Division, United States Naval Research Laboratory, Washington, DC, USA

    Ranjeev Mittu

  3. United States Naval Research Laboratory, Washington, DC, USA

    Donald A. Sofge

  4. Lockheed Martin Space Systems, King of Prussia, PA, USA

    Thomas Shortell

  5. Stevens Institute of Technology, Hoboken, NJ, USA

    Thomas A. McDermott

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Russell, S., Jalaian, B., Moskowitz, I.S. (2021). Re-orienting Toward the Science of the Artificial: Engineering AI Systems. In: Lawless, W.F., Mittu, R., Sofge, D.A., Shortell, T., McDermott, T.A. (eds) Systems Engineering and Artificial Intelligence . Springer, Cham. https://doi.org/10.1007/978-3-030-77283-3_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-77283-3_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-77282-6

  • Online ISBN: 978-3-030-77283-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation