Abstract
The innovative science of the future must be multi-domain and interconnected to usher in the next generation of “self-driving” laboratories enabling consequential discoveries and transformative inventions. Such a disparate and interconnected ecosystem of scientific instruments will need to evolve using a system-of-systems (SoS) approach. The key to enabling application integration with such an SoS will be the use of Software Development Kits (SDKs). Currently, SDKs facilitate scientific research breakthroughs via algorithmic automation, databases and storage, optimization and structure, pervasive environmental monitoring, among others. However, existing SDKs lack instrument-interoperability and reusability capabilities, do not effectively work in an open federated architectural environment, and are largely isolated within silos of the respective scientific disciplines. Inspired by the scalable SoS framework, this work proposes the development of INTERSECT-SDK to provide a coherent environment for multi-domain scientific applications to benefit from the open federated architecture in an interconnected ecosystem of instruments. This approach will decompose functionality into loosely coupled software services for interoperability among several solutions that do not scale beyond a single domain and/or application. Furthermore, the proposed environment will allow operational and managerial inter-dependence while providing opportunities for the researchers to reuse software components from other domains and build universal solution libraries. We demonstrate this research for microscopy use-case, where we show how INTERSECT-SDK is developing the tools necessary to enable advanced scanning methods and accelerate scientific discovery.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
HashiCorp’s Terraform: https://www.terraform.io/.
- 2.
References
Bartlett, R., Demeshko, I., Gamblin, T., et al.: xSDK foundations: toward an extreme-scale scientific software development kit. Supercomput. Front. Innov. 4, 69–82 (2017)
Baxter, S.M., Day, S.W., Fetrow, J.S., Reisinger, S.J.: Scientific software development is not an oxymoron. PLOS Comput. Biol. 2, 1–4 (2006)
De Bayser, M., Azevedo, L.G., Cerqueira, R.: ResearchOps: the case for DevOps in scientific applications. In: Proceedings of the 2015 IFIP/IEEE International Symposium on Integrated Network Management, IM 2015, pp. 1398–1404 (2015)
Brassil, J., Kopaliani, I.: Cloudjoin: experimenting at scale with hybrid cloud computing. In: 2020 IEEE 3rd 5G World Forum (5GWF), pp. 467–472 (2020)
CADES. CADES OpenStack cloud computing (2022)
Cataldo, M., Mockus, A., Roberts, J.A., Herbsleb, J.D.: Software dependencies, work dependencies, and their impact on failures. IEEE Trans. Software Eng. 35(6), 864–878 (2009)
Crawford, K., Whittaker, M., Elish, M.C., Barocas, S., Plasek, A., Ferryman, K.: The AI now report. The Social and Economic Implications of Artificial Intelligence Technologies in the Near-Term (2016)
da Silva, R.F., Casanova, H., Chard, K., et al.: Workflows community summit: advancing the state-of-the-art of scientific workflows management systems research and development. arXiv preprint arXiv:2106.05177 (2021)
Eick, S.G., Graves, T.L., Karr, A.F., Mockus, A., Schuster, P.: Visualizing software changes. IEEE Trans. Software Eng. 28, 396–412 (2002)
Google Developers. Protocol Buffers (2022)
Harbor. Harbor Website (2022)
Hazzan, O., Dubinsky, Y.: The agile manifesto. SpringerBriefs Comput. Sci. 9, 9–14 (2014)
Hines, J.: ORNL adds powerful AI appliances to computing portfolio - oak ridge leadership computing facility, August 2019. https://www.olcf.ornl.gov/2019/02/06/ornl-adds-powerful-ai-appliances-to-computing-portfolio/. Accessed 26 June 2022
Li, L., Fan, Y., Tse, M., Lin, K.-Y.: A review of applications in federated learning. Comput. Ind. Eng. 149, 106854 (2020)
Li, T., Sahu, A.K., Talwalkar, A., Smith, V.: Federated learning: challenges, methods, and future directions. IEEE Signal Process. Mag. 37, 50–60 (2020)
Li, W., Liewig, M.: A survey of AI accelerators for edge environment. In: Rocha, Á., Adeli, H., Reis, L.P., Costanzo, S., Orovic, I., Moreira, F. (eds.) WorldCIST 2020. AISC, vol. 1160, pp. 35–44. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-45691-7_4
Lwakatare, L.E., Kuvaja, P., Oivo, M.: Dimensions of DevOps. In: Lassenius, C., Dingsøyr, T., Paasivaara, M. (eds.) XP 2015. LNBIP, vol. 212, pp. 212–217. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-18612-2_19
Malviya-Thakur, A., Watson, G.: Dynamics of scientific software teams. Collegeville (2021)
Meyer, C., Dellby, N., Hachtel, J.A., Lovejoy, T., Mittelberger, A., Krivanek, O.: Nion swift: open source image processing software for instrument control, data acquisition, organization, visualization, and analysis using python. Microsc. Microanal. 25(S2), 122–123 (2019)
Mockus, A., Fielding, R.T., Herbsleb, J.D.: Two case studies of open source software development: Apache and Mozilla. ACM Trans. Softw. Eng. Methodol. 11, 309–346 (2002)
Mockus, A., Weiss, D.M.: Predicting risk of software changes. Bell Labs Tech. J. 5, 169–180 (2000)
Naughton, T., et al.: Software framework for federated science instruments. In: Nichols, J., Verastegui, B., Maccabe, A.B., Hernandez, O., Parete-Koon, S., Ahearn, T. (eds.) SMC 2020. CCIS, vol. 1315, pp. 189–203. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-63393-6_13
Nguyen-Hoan, L., Flint, S., Sankaranarayana, R.: A survey of scientific software development. In: Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement. Association for Computing Machinery (2010)
Nion Co., Nion Swift User’s Guide (2022)
Ophus, C., Ercius, P., Sarahan, M., Czarnik, C., Ciston, J.: Recording and using 4D-stem datasets in materials science. Microsc. Microanal. 20(S3), 62–63 (2014)
Pennycook, S.J., Nellist, P.D.: Scanning Transmission Electron Microscopy: Imaging and Analysis. Springer, New York (2011). https://doi.org/10.1007/978-1-4419-7200-2
Library Pyro. Github - irmen/pyro5: Pyro 5 - python remote objects for modern python versions (2022). https://github.com/irmen/Pyro5. Accessed 27 June 2022
Python Code Quality Authority (PyCQA). PyCQA’s Bandit GitHub repository (2022)
RabbitMQ. RabbitMQ Website (2022)
Roccapriore, K.M., Dyck, O., Oxley, M.P., Ziatdinov, M., Kalinin, S.V.: Automated experiment in 4D-STEM: exploring emergent physics and structural behaviors. ACS Nano (2022)
Segal, J., Morris, C.: Developing scientific software. IEEE Softw. 25, 18–20 (2008)
Slate. Slate: Kubernetes cluster with access to Summit (2021)
Somnath, S., et al.: Building an integrated ecosystem of computational and observational facilities to accelerate scientific discovery. In: Nichols, J., et al. (eds.) Smoky Mountains Computational Sciences and Engineering Conference, pp. 58–75. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-96498-6_4
Nion Swift. Nion swift, March 2022. https://github.com/nion-software/nionswift. Accessed 26 June 2022
Treveil, M., et al.: Introducing MLOps. O’Reilly Media, Sebastopol (2020)
Zager, R., Zager, J.: Ooda loops in cyberspace: a new cyber-defense model. Small Wars J. (2017)
Zhao, F., Niu, X., Huang, S.L., Zhang, L.: Reproducing scientific experiment with cloud DevOps. In: 2020 IEEE World Congress on Services (SERVICES), pp. 259–264 (2020)
Acknowledgement
This manuscript has been authored by UT-Battelle, LLC under Contract No. DEAC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Thakur, A.M. et al. (2022). Towards a Software Development Framework for Interconnected Science Ecosystems. In: Doug, K., Al, G., Pophale, S., Liu, H., Parete-Koon, S. (eds) Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation. SMC 2022. Communications in Computer and Information Science, vol 1690. Springer, Cham. https://doi.org/10.1007/978-3-031-23606-8_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-23606-8_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-23605-1
Online ISBN: 978-3-031-23606-8
eBook Packages: Computer ScienceComputer Science (R0)