Skip to main content

Benchmarking Deep Learning Infrastructures by Means of TensorFlow and Containers

Part of the Lecture Notes in Computer Science book series (LNTCS,volume 11887)

Abstract

Ever growing interest and usage of deep learning rises a question on the performance of various infrastructures suitable for training of neural networks. We present here our approach and first results of tests performed with TensorFlow Benchmarks which use best practices for multi-GPU and distributed training. We pack the Benchmarks in Docker containers and execute them by means of uDocker and Singularity container tools on a single machine and in the HPC environment. The Benchmarks comprise a number of convolutional neural network models run across synthetic data and e.g. the ImageNet dataset. For the same Nvidia K80 GPU card we achieve the same performance in terms of processed images per second and similar scalability between 1-2-4 GPUs as presented by the TensorFlow developers. We therefore do not obtain statistically significant overhead due to the usage of containers in the multi-GPU case, and the approach of using TF Benchmarks in a Docker container can be applied across various systems.

Keywords

  • Benchmarks
  • TensorFlow
  • ConvNet
  • Containers

This is a preview of subscription content, access via your institution.

Buying options

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-34356-9_36
  • Chapter length: 12 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   84.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-34356-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   109.99
Price excludes VAT (USA)
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.
Fig. 5.
Fig. 6.
Fig. 7.

References

  1. Computational resource ForHLR-II available at Karlsruhe Institute of Technology. https://wiki.scc.kit.edu/hpc/index.php/Category:ForHLR. Accessed 12 Apr 2019

  2. DEEP as a Service (DEEPaaS) API. https://deepaas.readthedocs.io. Accessed 12 Apr 2019

  3. Docker Hub website. https://hub.docker.com. Accessed 12 Apr 2019

  4. ImageNet Object Localization Challenge at Kaggle. https://www.kaggle.com/c/imagenet-object-localization-challenge. Accessed 12 Apr 2019

  5. Nvidia Geforce GTX980 Ti specifications. https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-980-ti/specifications. Accessed 12 Apr 2019

  6. Nvidia Tesla K80 GPU Accelerator, Board Specification (2015). https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-product-literature/Tesla-K80-BoardSpec-07317-001-v05.pdf. Accessed 12 Apr 2019

  7. Results of TF Benchmarks runs, GitHub repository. https://github.com/adriangrupp/tf_cnn_benchmarks_evaluation. Accessed 12 Apr 2019

  8. Scripts for downloading and converting ImageNet data to TFRecord format. https://github.com/tensorflow/models/tree/master/research/inception/inception/data. Accessed 12 Apr 2019

  9. TensorFlow Benchmarks, GitHub repository. https://github.com/tensorflow/benchmarks/. Accessed 8 Apr 2019

  10. TensorFlow Benchmarks webpage. https://www.tensorflow.org/guide/performance/benchmarks. Accessed 8 Apr 2019

  11. TF Benchmarks fork for TF 1.10.0, KIT Gitlab. https://git.scc.kit.edu/tf-benchmarks/tf_cnn_tf_benchmarks. Accessed 12 Apr 2019

  12. udocker: A basic user tool to execute simple docker containers in batch or interactive systems without root privileges. GiHub repository. https://github.com/indigo-dc/udocker. Accessed 8 Apr 2019

  13. Baker, M.: Is there a reproducibility crisis? Nature 533, 452–454 (2016)

    CrossRef  Google Scholar 

  14. Gomes, J., et al.: Enabling rootless linux containers in multi-user environments: the udocker tool. Comput. Phys. Commun. 232, 84–97 (2018). https://doi.org/10.1016/j.cpc.2018.05.021

    CrossRef  Google Scholar 

  15. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2015)

    Google Scholar 

  16. Jacobsen, D.M., Canon, R.S.: Contain this, unleashing Docker for HPC. Cray User Group 2015 (2015)

    Google Scholar 

  17. Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report (2009). https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf

  18. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc. (2012). http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

  19. Kurtzer, G.M., Sochat, V., Bauer, M.W.: Singularity: scientific containers for mobility of compute. PLoS One (2017). https://doi.org/10.1371/journal.pone.0177459

    CrossRef  Google Scholar 

  20. Lin, T., et al.: Microsoft COCO: common objects in context. CoRR abs/1405.0312 (2014). http://arxiv.org/abs/1405.0312

  21. Merkel, D.: Docker: lightweight Linux containers for consistent development and deployment (2014). https://doi.org/10.1097/01.NND.0000320699.47006.a3

  22. Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: an ASR corpus based on public domain audio books, pp. 5206–5210, April 2015. https://doi.org/10.1109/ICASSP.2015.7178964

  23. Priedhorsky, R., Randles, T.C., Randles, T.: Charliecloud: unprivileged containers for user-defined software stacks in HPC. In: SC17: International Conference for High Performance Computing, Networking, Storage and Analysis (2017). https://doi.org/10.1145/3126908.3126925

  24. Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    MathSciNet  CrossRef  Google Scholar 

  25. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)

    Google Scholar 

  26. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: InceptionV3. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016)

    Google Scholar 

Download references

Acknowledgments

uDocker is being developed within the DEEP HybridDataCloud project, which receives funding from the European Union’s Horizon 2020 research and innovation program under agreement RIA 777435.

A part of this work was performed on the computational resource ForHLR-II funded by the Ministry of Science, Research and the Arts Baden-Wuerttemberg and DFG (“Deutsche Forschungsgemeinschaft”).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Valentin Kozlov .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Grupp, A., Kozlov, V., Campos, I., David, M., Gomes, J., López García, Á. (2019). Benchmarking Deep Learning Infrastructures by Means of TensorFlow and Containers. In: Weiland, M., Juckeland, G., Alam, S., Jagode, H. (eds) High Performance Computing. ISC High Performance 2019. Lecture Notes in Computer Science(), vol 11887. Springer, Cham. https://doi.org/10.1007/978-3-030-34356-9_36

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-34356-9_36

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-34355-2

  • Online ISBN: 978-3-030-34356-9

  • eBook Packages: Computer ScienceComputer Science (R0)