Skip to main content
Book cover

International Conference on Service-Oriented Computing

ICSOC 2021: Service-Oriented Computing pp 238–253Cite as

Resource Management for TensorFlow Inference

  • 1318 Accesses

Part of the Lecture Notes in Computer Science book series (LNPSE,volume 13121)

Abstract

TensorFlow, a popular machine learning (ML) platform, allows users to transparently exploit both GPUs and CPUs to run their applications. Since GPUs are optimized for compute-intensive workloads (e.g., matrix calculus), they help boost executions, but introduce resource heterogeneity. TensorFlow neither provides efficient heterogeneous resource management nor allows for the enforcement of user-defined constraints on the execution time. Most of the works address these issues in the context of creating models on existing data sets (training phase), and only focus on scheduling algorithms. This paper focuses on the inference phase, that is, on the application of created models to predict the outcome on new data interactively, and presents a comprehensive resource management solution called ROMA (Resource Constrained ML Applications). ROMA is an extension of TensorFlow that (a) provides means to easily deploy multiple TensorFlow models in containers using Kubernetes b) allows users to set constraints on response times, (c) schedules the execution of requests on GPUs and CPUs using heuristics, and (d) dynamically refines the CPU core allocation by exploiting control theory. The assessment conducted on four real-world benchmark applications compares ROMA against four different systems and demonstrates a significant reduction ( \(75\%\)) in constraint violations and \(24\%\) saved resources on average.

This work has been partially supported by the SISMA national research project, which has been funded by the MIUR under the PRIN 2017 program (Contract 201752ENYB) and by the European Commission grant no. 825480 (H2020), SODALITE.

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-91431-8_15
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   99.00
Price excludes VAT (USA)
  • ISBN: 978-3-030-91431-8
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   129.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.

Notes

  1. 1.

    https://kubernetes.io.

  2. 2.

    Source code is available at https://github.com/deib-polimi/ROMA.

  3. 3.

    https://www.tensorflow.org/tfx/guide/serving.

  4. 4.

    https://www.docker.com.

  5. 5.

    https://github.com/NVIDIA/nvidia-docker.

  6. 6.

    This average execution time is computed by considering the different executions of the same model executable over a given time window.

  7. 7.

    In December 2020, the Kubernetes team announced that the Docker runtime will be considered deprecated in future versions [14]. Docker will not be removed from Kubernetes at least until late 2021. While the evaluation of ROMA in Sect. 5 is based on the described Docker-dependent implementation, we are already developing a version of ROMA that does not require Docker and that supports other container runtimes as, for example, containerd [6].

  8. 8.

    To avoid ambiguities, in this section a dynamic model is a mathematical representation of the controlled system, that is, of the ML application.

References

  1. Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: 12th Symposium on Operating Systems Design and Implementation, pp. 265–283. USENIX (2016)

    Google Scholar 

  2. Baresi, L., Guinea, S., Leva, A., Quattrocchi, G.: A discrete-time feedback controller for containerized cloud applications. In: Proceedings of the 2016 24th International Symposium on Foundations of Software Engineering, pp. 217–228. ACM (2016)

    Google Scholar 

  3. Baresi, L., Leva, A., Quattrocchi, G.: Fine-grained dynamic resource allocation for big-data applications. IEEE Trans. Softw. Eng. 47(8), 1668–1682 (2021)

    CrossRef  Google Scholar 

  4. Chen, L., Huo, X., Agrawal, G.: Accelerating MapReduce on a coupled CPU-GPU architecture. In: Hollingsworth, J.K. (ed.) SC Conference on High Performance Computing Networking, Storage and Analysis, pp. 1–11. IEEE/ACM (2012)

    Google Scholar 

  5. Chen, T., Li, M., et al.: MXNet: a Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems. arXiv (2015)

    Google Scholar 

  6. Containerd: an industry-standard container runtime with an emphasis on simplicity, robustness and portability (2021). https://containerd.io

  7. Ding, J., Cao, R., Saravanan, I., Morris, N., Stewart, C.: Characterizing service level objectives for cloud services: realities and myths. In: 2019 IEEE International Conference on Autonomic Computing (ICAC), pp. 200–206. IEEE (2019)

    Google Scholar 

  8. Farokhi, S., Lakew, E.B., Klein, C., Brandic, I., Elmroth, E.: Coordinating CPU and memory elasticity controllers to meet service response time constraints. In: 2015 International Conference on Cloud and Autonomic Computing, pp. 69–80 (2015)

    Google Scholar 

  9. Fedorov, R., Camerada, A., et al.: Estimating snow cover from publicly available images. IEEE Trans. Multimed. 18(6), 1187–1200 (2016)

    CrossRef  Google Scholar 

  10. Forbes: TensorFlow Turns 5 - Five Reasons Why it is the Most Popular ML Framework. http://tiny.cc/Forbes-TF (2020)

  11. He, K., Zhang, X., et al.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  12. Jahani, A., Lattuada, M., Ciavotta, M., Ardagna, D., Amaldi, E., Zhang, L.: Optimizing on-demand GPUs in the cloud for deep learning applications training. In: 2019 4th International Conference on Computing, Communications and Security (ICCCS), pp. 1–8 (2019)

    Google Scholar 

  13. Khalid, Y.N., Aleem, M., Prodan, R., Iqbal, M.A., Islam, M.A.: E-OSched: a load balancing scheduler for heterogeneous multicores. J. Supercomput. 74(10), 5399–5431 (2018)

    CrossRef  Google Scholar 

  14. Kubernetes: Don’t Panic: Kubernetes and Docker (2020). https://kubernetes.io/blog/2020/12/02/dont-panic-kubernetes-and-docker

  15. Kubernetes: Schedule GPUs (2020). https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/

  16. Lakew, E., Papadopoulos, A., Maggio, M., Klein, C., Elmroth, E.: KPI-agnostic control for fine-grained vertical elasticity. In: Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, pp. 589–598. IEEE (2017)

    Google Scholar 

  17. Mittal, S., Vetter, J.S.: A survey of CPU-GPU heterogeneous computing techniques. ACM Comput. Surv. 47(4), 69:1–69:35 (2015)

    Google Scholar 

  18. Nozal, R., Bosque, J.L., Beivide, R.: EngineCL: usability and performance in heterogeneous computing. Future Gener. Comput. Syst. 107, 522–537 (2020)

    CrossRef  Google Scholar 

  19. Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. Adv. Neural Inf. Proc. Syst. 32, 8024–8035 (2019)

    Google Scholar 

  20. Szegedy, C., et al.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)

    Google Scholar 

  21. Verma, A., Cherkasova, L., et al.: Deadline-based workload management for MapReduce environments: pieces of the performance puzzle. In: 2012 IEEE Network Operations and Management Symposium, pp. 900–905. IEEE (2012)

    Google Scholar 

  22. Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. IEEE Trans. Pattern Anal. Mach. Intell. 38(10), 1943–1955 (2015)

    CrossRef  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Dipartimento di Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, Italy

    Luciano Baresi, Giovanni Quattrocchi & Nicholas Rasi

Authors
  1. Luciano Baresi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Giovanni Quattrocchi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nicholas Rasi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Giovanni Quattrocchi .

Editor information

Editors and Affiliations

  1. Zayed University, Dubai, United Arab Emirates

    Hakim Hacid

  2. Technical University of Berlin, Berlin, Germany

    Odej Kao

  3. Informatica Automatica Gestio, Sapienza University of Rome, Rome, Italy

    Massimo Mecella

  4. Departement d'Informatique, University of Quebec, Montreal, QC, Canada

    Dr. Naouel Moha

  5. UNSW Sydney, Sydney, NSW, Australia

    Hye-young Paik

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Baresi, L., Quattrocchi, G., Rasi, N. (2021). Resource Management for TensorFlow Inference. In: Hacid, H., Kao, O., Mecella, M., Moha, N., Paik, Hy. (eds) Service-Oriented Computing. ICSOC 2021. Lecture Notes in Computer Science(), vol 13121. Springer, Cham. https://doi.org/10.1007/978-3-030-91431-8_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-91431-8_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-91430-1

  • Online ISBN: 978-3-030-91431-8

  • eBook Packages: Computer ScienceComputer Science (R0)