Abstract
The digital revolution, rapidly decreasing storage cost, and remarkable results achieved by state of the art machine learning (ML) methods are driving widespread adoption of ML approaches. While notable recent efforts to benchmark ML methods for canonical tasks exist, none of them address the challenges arising with the increasing pervasiveness of end-to-end ML deployments. The challenges involved in successfully applying ML methods in diverse enterprise settings extend far beyond efficient model training.
In this paper, we present our work in benchmarking advanced data analytics systems and lay the foundation towards an industry standard machine learning benchmark. Unlike previous approaches, we aim to cover the complete end-to-end ML pipeline for diverse, industry-relevant application domains rather than evaluating only training performance. To this end, we present reference implementations of complete ML pipelines including corresponding metrics and run rules, and evaluate them at different scales in terms of hardware, software, and problem size.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
Self-Monitoring, Analysis and Reporting Technology.
- 5.
- 6.
- 7.
- 8.
- 9.
- 10.
References
MLPerf (2018). https://mlperf.org/, https://mlperf.org/
Abadi, M., Barham, P., Chen, J., et al.: Tensorflow: a system for large-scale machine learning. In: OSDI, pp. 265–283 (2016)
Amatriain, X.: Building industrial-scale real-world recommender systems. In: RecSys, pp. 7–8 (2012)
Baru, C., et al.: Discussion of BigBench: a proposed industry standard performance benchmark for big data. In: Nambiar, R., Poess, M. (eds.) TPCTC 2014. LNCS, vol. 8904, pp. 44–63. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-15350-6_4
Bennett, J., Lanning, S., et al.: The netflix prize. In: Proceedings of KDD Cup and Workshop, p. 35 (2007)
Bhandarkar, M.: AdBench: a complete data pipeline benchmark for modern data pipelines. In: TPCTC, pp. 107–120 (2016)
Boden, C., Rabl, T., Markl, V.: Distributed machine learning-but at what cost. In: Machine Learning Systems Workshop at Conference on Neural Information Processing Systems (2017)
Boden, C., Spina, A., Rabl, T., Markl, V.: Benchmarking data flow systems for scalable machine learning. In: SIGMOD Workshop on Algorithms and Systems for MapReduce and Beyond, pp. 5:1–5:10. BeyondMR (2017)
Chen, T., Guestrin, C.: Xgboost: a scalable tree boosting system. In: ACM SigKDD, pp. 785–794 (2016)
Chen, T., Li, M., Li, Y., et al.: Mxnet: a flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 (2015)
Chigurupati, A., Thibaux, R., Lassar, N.: Predicting hardware failure using machine learning. In: Reliability and Maintainability Symposium (RAMS), pp. 1–6 (2016)
Chowdhury, B., Rabl, T., Saadatpanah, P., Du, J., Jacobsen, H.-A.: A BigBench implementation in the hadoop ecosystem. In: Rabl, T., Jacobsen, H.-A., Raghunath, N., Poess, M., Bhandarkar, M., Baru, C. (eds.) WBDB 2013. LNCS, vol. 8585, pp. 3–18. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10596-3_1
Chui, M., et al.: Notes from the AI Frontier - insights from hundreds of use cases. Technical report, McKinsey Global Institute (2018)
Coleman, C., et al.: Analysis of DAWNBench, a time-to-accuracy machine learning performance benchmark. CoRR abs/1806.01427 (2018)
Deng, D., Fernandez, R.C., Abedjan, Z., et al.: The data civilizer system. In: CIDR (2017)
Gao, W., Zhan, J., Wang, L., et al.: BigDataBench: a dwarf-based big data and AI benchmark suite. CoRR abs/1802.0 (2018)
Ghazal, A., et al.: BigBench: towards an industry standard benchmark for big data analytics. In: SIGMOD (2013)
Jain, R.: The Art of Computer Systems Performance Analysis - Techniques for Experimental Design, Measurement, Simulation, and Modeling. Wiley Professional Computing. Wiley, Hoboken (1991)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
LeCun, Y.: The mnist database of handwritten digits (1998). http://yann.lecun.com/exdb/mnist/
Liu, Y., Zhang, H., Zeng, L., Wu, W., Zhang, C.: Mlbench: benchmarking machine learning services against human experts. PVLDB 11(10), 1220–1232 (2018)
McKinney, W., et al.: Data structures for statistical computing in Python. In: Proceedings of the 9th Python in Science Conference, pp. 51–56 (2010)
Meng, X., Bradley, J., Yavuz, B., et al.: Mllib: machine learning in apache spark. J. Mach. Learn. Res. 17(1), 1235–1241 (2016)
Narang, S.: DeepBench (2016). https://svail.github.io/DeepBench/
Paszke, A., Gross, S., Chintala, S., et al.: Automatic differentiation in pytorch. In: NIPS Autodiff Decision Workshop (2017)
Pedregosa, F., Varoquaux, G., Gramfort, A., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Poess, M., Rabl, T., Jacobsen, H.A.: Analysis of TPC-DS: the first standard benchmark for SQL-based big data systems. In: Proceedings of the 2017 Symposium on Cloud Computing (2017)
Polyzotis, N., Roy, S., Whang, S.E., Zinkevich, M.: Data management challenges in production machine learning. In: SIGMOD, pp. 1723–1726 (2017)
Rabl, T., Frank, M., Danisch, M., Gowda, B., Jacobsen, H.-A.: Towards a complete BigBench implementation. In: Rabl, T., Sachs, K., Poess, M., Baru, C., Jacobson, H.-A. (eds.) WBDB 2015. LNCS, vol. 8991, pp. 3–11. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-20233-4_1
Rabl, T., Frank, M., Danisch, M., Jacobsen, H.A., Gowda, B.: The vision of BigBench 2.0. In: Proceedings of the Fourth Workshop on Data Analytics in the Cloud DanaC 2015, pp. 3:1–3:4. ACM, New York (2015)
Rabl, T., Frank, M., Sergieh, H.M., Kosch, H.: A data generator for cloud-scale benchmarking. In: Nambiar, R., Poess, M. (eds.) TPCTC 2010. LNCS, vol. 6417, pp. 41–56. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-18206-8_4
Rabl, T., Poess, M.: Parallel data generation for performance analysis of large, complex RDBMS. In: DBTest 2011, p. 5 (2011)
Schelter, S., Palumbo, A., Quinn, S., Marthi, S., Musselman, A.: Samsara: declarative machine learning on distributed dataflow systems. In: NIPS Workshop MLSystems (2016)
Shah, V., Kumar, A.: The ML data prep zoo: towards semi-automatic data preparation for ML. In: DEEM, pp. 11:1–11:4 (2019)
Stevens, C.E.: Information TPC-Cechnology - ATA/ATAPI Command Set - 2 (ACS-2). Technical report, ANSI INCITS (2011)
Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: ICCV (2017)
Transaction Processing Performance Council: TPC-Energy Specification, version 1.5.0 (2012)
Transaction Processing Performance Council: TPC Pricing Specification, version 2.5.0 (2019)
Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: HotCloud, p. 10 (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Rabl, T. et al. (2020). ADABench - Towards an Industry Standard Benchmark for Advanced Analytics. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking for the Era of Cloud(s). TPCTC 2019. Lecture Notes in Computer Science(), vol 12257. Springer, Cham. https://doi.org/10.1007/978-3-030-55024-0_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-55024-0_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-55023-3
Online ISBN: 978-3-030-55024-0
eBook Packages: Computer ScienceComputer Science (R0)