Skip to main content

NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning

  • Conference paper
  • First Online:
Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation (SMC 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1512))

Included in the following conference series:

  • 1026 Accesses

Abstract

Integrating cloud services into advanced computing facilities provides significant new capabilities and offers several advantages over focusing solely on traditional high performance computing (HPC) workloads. The integration of cloud services is especially potent for workflows that fuse data streams, modeling and simulation (“modsim”), and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is aligning optimal capability, data, and user intent on the right resources for each step in a workflow. The National Renewable Energy Laboratory (NREL) Stratus service provides a basis for this alignment. Stratus layers the capabilities needed to make cloud services accessible to a lab-based scientific community on commercial offerings, and currently supports upward of 200 projects, ranging from Internet of Things (IoT) integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements, which enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow. We highlight multiple components, including sensor data intake, cleaning and transforming (edge/cloud suitable), generation of synthetic data through modsim, computationally heavy machine learning training and hyperparameter optimization (HPC suitable), and inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis of the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined in this paper allow for better understanding of new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    GreenGrass, IoT Core and IoT Device Management are also certified under the Payment Card Industry (PCI), System and Organization Controls (SOC 2) and Department of Defense Cloud Computing Security Requirements Guide (CC SRG) [1].

References

  1. AWS Services In Scope. https://aws.amazon.com/compliance/services-in-scope/. Accessed 26 May 2021

  2. Bennett, K., Robertson, J.: Remote sensing: leveraging cloud IoT and AI/ML services. Proceedings of the SPIE 11746, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, 117462L, 12 April 2021. https://doi.org/10.1117/12.2587754

  3. Borge, S., Poonia, N.: Review on Amazon web services, Google cloud provider and Microsoft windows Azure. Advance and Innovative Research, p. 53 (2020)

    Google Scholar 

  4. Botez, R., Strautiu, V., Ivanciu, I., Dobrota, V.: Containerized application for IoT devices: comparison between balenaCloud and Amazon web services approaches. In: 2020 International Symposium on Electronics and Telecommunications (ISETC), pp. 1–4 (2020). https://doi.org/10.1109/ISETC50328.2020.9301070

  5. Javed, A., Malhi, A., Främling, K.: Edge computing-based fault-tolerant framework: a case study on vehicular networks. In: 2020 International Wireless Communications and Mobile Computing (IWCMC), pp. 1541–1548 (2020). https://doi.org/10.1109/IWCMC48107.2020.9148269

  6. Liu, Y., Yang, C., Jiang, L., Xie, S., Zhang, Y.: Intelligent edge computing for IoT-based energy management in smart cities. IEEE Netw. 33(2), 111–117 (2019). https://doi.org/10.1109/MNET.2019.1800254

    Article  Google Scholar 

  7. Mesbahi, M.R., Rahmani, A.M., Hosseinzadeh, M.: Reliability and high availability in cloud computing environments: a reference roadmap. HCIS 8(1), 1–31 (2018). https://doi.org/10.1186/s13673-018-0143-8

    Article  Google Scholar 

  8. Muhammed, A., Ucuz, D.: Comparison of the IoT platform vendors, microsoft Azure, Amazon web services, and Google cloud, from users’ perspectives. In: 2020 8th International Symposium on Digital Forensics and Security (ISDFS), pp. 1–4 (2020). https://doi.org/10.1109/ISDFS49300.2020.9116254

  9. National Renewable Energy Laboratory’s Mission. https://www.nrel.gov/about/mission-programs.html#:~:text=NREL%20advances%20the%20science%20and,integrate%20and%20optimize%20energy%20systems. Accessed 24 May 2021

  10. Nguyen, D., Luckow, A., Duffy, E., Kennedy, K., Apon, A.: Evaluation of highly available cloud streaming systems for performance and price. In: 2018 18th IEEE/ACM International Symposium on Cluster Cloud and Grid Computing (CCGRID), pp. 360–363 (2018). https://doi.org/10.1109/CCGRID.2018.00056

  11. Pflanzner, T., Kertesz, A.: A survey of IoT cloud providers. In: 2016 39th International Convention on Information and Communication Technology Electronics and Microelectronics (MIPRO), pp. 730–735 (2016). https://doi.org/10.1109/MIPRO.2016.7522237

  12. Pham, T., Ristov, S., Fahringer, T.: Performance and behavior characterization of Amazon EC2 spot instances. In: 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pp. 73–81 (2018). https://doi.org/10.1109/CLOUD.2018.00017

  13. Mohammed Sadeeq, M., Abdulkareem, N.M., Zeebaree, S. R. M., Mikaeel Ahmed, D., Saifullah Sami, A., Zebari, R. R.: IoT and cloud computing issues, challenges and opportunities: a review. Qubahan Acad. J. 1(2), 1–7 (2021). https://doi.org/10.48161/qaj.v1n2a36

  14. Singh, V., Dutta, K.: Dynamic price prediction for Amazon spot instances. In: 2015 48th Hawaii International Conference on System Sciences, pp. 1513–1520 (2015). https://doi.org/10.1109/HICSS.2015.184

  15. White House FACT SHEET: President Biden Sets 2030 Greenhouse Gas Pollution Reduction Target Aimed at Creating Good-Paying Union Jobs and Securing U.S. Leadership on Clean Energy Technologies. https://www.whitehouse.gov/briefing-room/statements-releases/2021/04/22/fact-sheet-president-biden-sets-2030-greenhouse-gas-pollution-reduction-target-aimed-at-creating-good-paying-union-jobs-and-securing-u-s-leadership-on-clean-energy-technologies/. Accessed 31 Aug 2021

  16. Zhang, J., et al.: Energy-latency tradeoff for energy-aware offloading in mobile edge computing networks. IEEE Internet Things J. 5(4), 2633–2645 (2018). https://doi.org/10.1109/JIOT.2017.2786343

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Aaron Andersen .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rager, D., Andersen, A. (2022). NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning. In: Nichols, J., et al. Driving Scientific and Engineering Discoveries Through the Integration of Experiment, Big Data, and Modeling and Simulation. SMC 2021. Communications in Computer and Information Science, vol 1512. Springer, Cham. https://doi.org/10.1007/978-3-030-96498-6_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-96498-6_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-96497-9

  • Online ISBN: 978-3-030-96498-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics