Artificial Intelligence Platform for Heterogeneous Computing

  • Haikuo ZhangEmail author
  • Zhonghua Lu
  • Ke Xu
  • Yuchen Pang
  • Fang Liu
  • Liandong Chen
  • Jue Wang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11344)


Since the birth of artificial intelligence, the theory and the technology have become more mature, and the application field is expanding. In this paper, we build an artificial intelligence platform for heterogeneous computing, which supports deep learning frameworks such as TensorFlow and Caffe. We describe the overall architecture of the AI platform for a GPU cluster. In the GPU cluster, based on the scheduling layer, we propose Yarn by the Slurm scheduler to not only improve the distributed TensorFlow plug-in for the Slurm scheduling layer but also to extend YARN to manage and schedule GPUs. The front-end of the high-performance AI platform has the attributes of availability, scalability and efficiency. Finally, we verify the convenience, scalability, and effectiveness of the AI platform by comparing the performance of single-chip and distributed versions for the TensorFlow, Caffe and YARN systems.


Artificial intelligence Hadoop Slurm Schedule TensorFlow Caffe 



This work was partly supported by the National Key R&D Program of China (No. 2017YFB0202202), the State Key Program of National Natural Science Foundation of China (No. 61702476).


  1. 1.
    Shanhai, W., Xinxing, J., Haiyan, Y.: Research on isolated word speech recognition based on deep learning neural network. Comput. Appl. Res. 32(8), 2289–2291 (2015)Google Scholar
  2. 2.
    Gantz, J., Reinsel, D.: Digital Universe in 2020 [EB/OL]. Accessed 1 Dec 2012
  3. 3.
    Christiansen, B., Garey, M., Hartung, I.: SlurmOveview [EB/OL]. Accessed 12 Dec 2017
  4. 4.
    Jeff, D., Rajat, M., et al.: (9 November 2015) TensorFlow: Large-scale machine learning on heterogeneous systems (PDF). Google Research. Accessed 10 Nov 2015Google Scholar
  5. 5.
    Cybulska, M.: Assessing yarn structure with image analysis methods1. Text. Res. J. 69(5), 369–373 (1999)CrossRefGoogle Scholar
  6. 6.
    Pacelli, M., Caldani, L., Paradiso, R.: Performances evaluation of piezoresistive fabric sensors as function of yarn structure. In: 2013 35th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6502–6505 (2013)Google Scholar
  7. 7.
    Ozturk, M., Nergis, B.U.: Determining the dependence of colour values on yarn structure. Color. Technol. 124(3), 145–150 (2008)CrossRefGoogle Scholar
  8. 8.
    Owens, J.D., Houston, M., Luebke, D., et al.: GPU computing. Proceed. IEEE 96(5), 879–899 (2008)CrossRefGoogle Scholar
  9. 9.
    Hou, S., Tan, M.T., Luo, X.G.: Application driven multi DSP processor array in high performance computing. Comput. Appl. Res. 28(4), 1336–1338 (2011)Google Scholar
  10. 10.
    Nickolls, J., Dally, W.J.: The GPU computing era. IEEE Micro 30(2), 56–69 (2010)CrossRefGoogle Scholar
  11. 11.
    Hwu, W.M.W.: Introduction - GPU computing gems emerald edition. GPU Comput. Gems Emerald Ed. 27, 599–600 (2011)Google Scholar
  12. 12.
    Pratx, G., Xing, L.: GPU computing in medical physics: a review. Med. Phys. 38(5), 2685–2697 (2011)CrossRefGoogle Scholar
  13. 13.
    Li, X.: A design of secret information system against internal network attack based on software container. Inf. Comput. 10, 109–111 (2016)Google Scholar
  14. 14.
    Ubal, R., Schaa, D., Jang, B., et al.: Multi2Sim: a simulation framework for CPU-GPU computing, pp. 335–344 (2012)Google Scholar
  15. 15.
  16. 16.
    Wei, J.: Research on massive transaction record query system based on Hadoop. Nanjing University of Posts and Telecommunications, Nanjing (2013)Google Scholar
  17. 17.
    Ghemawat, S., Gobioff, H., Leung, S.-T.: The Google file system. In: 19th ACM Symposium on Operating Systems Principles, October 2013Google Scholar
  18. 18.
    Wikipedia contributors. Apache Hadoop [EB/OL]. Accessed 13 Dec 2017
  19. 19.
    Braam, P.J.: The Lustre storage architecture. Cluster File Systems, Inc. (2004).
  20. 20.
    Wikipedia contributors. Lustre_(file_system) [EB/OL]. Accessed 31 Jan 2018
  21. 21.
    Wang, J., Gao F., Vazquez-Poletti, J.L., Li, J.: Preface of high performance computing or advanced modeling and simulation of materials. Comput. Phys. Commun. (211) (2017). (IF: 3.653)Google Scholar
  22. 22.
    Martin, A., Raponi, S., Combe, T., et al.: Docker ecosystem – vulnerability analysis. Comput. Commun. 122, 30–43 (2018)CrossRefGoogle Scholar
  23. 23.
    Slurm workload manager [EB/OL].
  24. 24.
    Wang, J., Liu, C., Huang, Y.: Auto tuning for new energy dispatch problem: A case study. Future Gener. Comput. Syst. 54, 501–506 (2016)CrossRefGoogle Scholar
  25. 25.
    HuaiTe, Z., et al.: Hadoop Authoritative Guide, 2nd edn. Tsinghua University Press, Beijing (2011)Google Scholar
  26. 26.
    Zhao, H., Zhang, Y., Bradford, P.D., et al.: Carbon nanotube yarn strain sensors. Nanotechnology 21(30), 305502 (2010)CrossRefGoogle Scholar
  27. 27.
    Ramesh, M.C., Rajamanickam, R., Jayaraman, S.: The Prediction of yarn tensile properties by using artificial neural networks. J. Text. Inst. Proceed. Abstr. 86(3), 459–469 (1995)CrossRefGoogle Scholar
  28. 28.
    Debo, L.: Research of GPU cluster system based on YARN, Sun Yat-sen University, Guangzhou (2014)Google Scholar
  29. 29.
    Schwarz, E.R.: Certain aspects of yarn structure. Text. Res. J. 21(3), 125–136 (1951)CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Haikuo Zhang
    • 1
    • 2
    • 3
    Email author
  • Zhonghua Lu
    • 1
    • 2
  • Ke Xu
    • 1
    • 2
  • Yuchen Pang
    • 4
  • Fang Liu
    • 1
  • Liandong Chen
    • 5
  • Jue Wang
    • 1
  1. 1.Computer Network Information CenterChinese Academy of SciencesBeijingChina
  2. 2.University of Chinese Academy of SciencesBeijingChina
  3. 3.China Internet Network Information CenterBeijingChina
  4. 4.University of Illinois at Urbana-ChampaignChampaignUSA
  5. 5.State Grid Hebei Electric Power CompanyShijiazhuangChina

Personalised recommendations