Advertisement

Requirements for an Enterprise AI Benchmark

  • Cedric Bourrasset
  • France Boillod-Cerneux
  • Ludovic Sauge
  • Myrtille Deldossi
  • Francois Wellenreiter
  • Rajesh BordawekarEmail author
  • Susan Malaika
  • Jean-Armand Broyelle
  • Marc West
  • Brian Belgodere
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11135)

Abstract

Artificial Intelligence (AI) is now the center of attention for many industries, ranging from private companies to academic institutions. While domains of interest and AI applications vary, one concern remains unchanged for everyone: How to determine if an end-to-end AI solution is performant? As AI is spreading to more industries, what metrics might be the reference for AI applications and benchmarks in the enterprise space? This paper intends to answer some of these questions. At present, the AI benchmarks either focus on evaluating deep learning approaches or infrastructure capabilities. Unfortunately, these approaches don’t capture end-to-end performance behavior of enterprise AI workloads. It is also clear that there is not one reference metric that will be suitable for all AI applications nor all existing platforms. We will first present the state of the art regarding the current basic and most popular AI benchmarks. We will then present the main characteristics of AI workloads from various industrial domains. Finally, we will focus on the needs for ongoing and future industry AI benchmarks and conclude on the gaps to improve AI benchmarks for enterprise workloads.

References

  1. 1.
  2. 2.
    Abadi, M., et al.: TensorFlow: a system for large-scale machine learning. In: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation, OSDI 2016, pp. 265–283. USENIX Association, Berkeley (2016). http://dl.acm.org/citation.cfm?id=3026877.3026899
  3. 3.
    DKP et al.: OSU micro-benchmarks. http://mvapich.cse.ohio-state.edu/benchmarks
  4. 4.
    Coleman, C.A., et al.: DAWNBench: an end-to-end deep learning benchmark and competition. In: Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017) (2017)Google Scholar
  5. 5.
    Dongarra, J.J., Heroux, M.A., Luszczek, P.: HPCG benchmark: a new metric for ranking high performance computing systems. Technical report UT-EECS-15-736, November 2015Google Scholar
  6. 6.
    Heroux, M.A., Dongarra, J.J., Luszczek, P.: HPCG technical specification. Technical report SAND2013-8752, October 2013Google Scholar
  7. 7.
    Luszczek, P., et al.: S12 – the HPC challenge (HPCC) benchmark suite. In: Proceedings of SC 2006, November 2006Google Scholar
  8. 8.
    McCalpin, J.D.: Memory bandwidth and machine balance in current high performance computers. In: IEEE Computer Society Technical Committee on Computer Architecture (TCCA) Newsletter, December 1995Google Scholar
  9. 9.
    Petitet, A., Whaley, R.C., Dongarra, J.J., Cleary, A.: HPL - a portable implementation of the high-performance Linpack benchmark for distributed-memory computers. http://www.netlib.org/benchmark/hpl/
  10. 10.
    Bench Research: Deep Bench (2018). https://github.com/baidu-research/DeepBench
  11. 11.
    Bench Research: ML Perf (2018). https://mlperf.org/
  12. 12.

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Cedric Bourrasset
    • 1
  • France Boillod-Cerneux
    • 1
  • Ludovic Sauge
    • 1
  • Myrtille Deldossi
    • 1
  • Francois Wellenreiter
    • 1
  • Rajesh Bordawekar
    • 2
    Email author
  • Susan Malaika
    • 2
  • Jean-Armand Broyelle
    • 2
  • Marc West
    • 2
  • Brian Belgodere
    • 2
  1. 1.BULL ATOSMontpellierFrance
  2. 2.IBM CorporationYorktown HeightsUSA

Personalised recommendations