Skip to main content
SpringerLink
Log in

PyDTNN: A user-friendly and extensible framework for distributed deep learning

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

We introduce a framework for training deep neural networks on clusters of computers with the following appealing properties: (1) It is developed in Python, exposing an amiable interface that provides an accessible entry point for the newcomer; (2) it is extensible, offering a customizable tool for the more advanced user in deep learning; (3) it covers the main functionality appearing in convolutional neural networks; and (4) it delivers reasonable inter-node parallel performance exploiting data parallelism by leveraging MPI via MPI4Py for communication and NumPy for the efficient execution of (multithreaded) numerical kernels.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Notes

  1. The source code for PyDTNN is available at https://github.com/hpca-uji/PyDTNN.

References

  1. Tal B-N, Torsten H (2019) Demystifying parallel and distributed deep learning: an in-depth concurrency analysis. ACM Comput Surv 52(4):65

    Google Scholar 

  2. Chan E, Heimlich M, Purkayastha A, van de Geijn R (2007) Collective communication: theory, practice, and experience. Concurr Comput Pract Exp 19(13):1749–1783

    Article  Google Scholar 

  3. Kumar C, Sidd P, Patrice S (2006) High performance convolutional neural networks for document processing. In: International workshop on frontiers in handwriting recognition

  4. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778

  5. Pouyanfar S et al (2018) A survey on deep learning: algorithms, techniques, and applications. ACM Comput Surv 51(5):92:1–92:36

    Google Scholar 

  6. Karen S, Andrew Z (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arxiv:1409.1556

  7. Sze V, Chen Y-H, Yang T-J, Emer JS (2017) Efficient processing of deep neural networks: a tutorial and survey. Proc IEEE 105(12):2295–2329

    Article  Google Scholar 

  8. Aravind V, Andrew A, David G (2017) Parallel multi channel convolution using general matrix multiplication. In: 2017 IEEE 28th international conference on application-specific systems, architectures and processors (ASAP), pp 19–24

  9. You Y, et al. (2018) Large-batch training for LSTM and beyond. Technical Report UCB/EECS-2018-138, Electrical Engineering and Computer Sciences, University of California at Berkeley

  10. Yang Y, Igor G, Boris G (2017) Scaling SGD batch size to 32k for ImageNet training. arXiv preprint arxiv:1708.03888

Download references

Acknowledgements

This work was supported by Project TIN2017-82972-R from the Spanish Ministerio de Ciencia, Innovación y Universidades. M. F. Dolz was supported by project CDEIGENT/2018/014 from the Generalitat Valenciana.

Author information

Authors and Affiliations

  1. Universitat Jaume I, Castellón de la Plana, Spain

    Sergio Barrachina, Adrián Castelló, Mar Catalán, Manuel F. Dolz & Jose I. Mestre

Authors
  1. Sergio Barrachina
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Adrián Castelló
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mar Catalán
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Manuel F. Dolz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jose I. Mestre
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manuel F. Dolz.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barrachina, S., Castelló, A., Catalán, M. et al. PyDTNN: A user-friendly and extensible framework for distributed deep learning. J Supercomput 77, 9971–9987 (2021). https://doi.org/10.1007/s11227-021-03673-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-03673-z

Keywords

Search

Navigation