Data Shuffling Minimizing Approach for Apache Spark Programs
Conference paper
First Online:
Abstract
This article discusses a way to optimize the Apache Spark program by reducing the number of transformations with wide dependencies and, as a result, the number of data shuffles. This is achieved by combining sequential data processing algorithms in chains based on common key fields, as well as grouping the data which is stored in resilient distributed structures i.e., Spark SQL Datasets, according to the keys by which the processing takes place.
Keywords
Apache spark Big data Hadoop Optimization ScalaReferences
- 1.Apache Spark. http://spark.apache.org/
- 2.Karau, H., Warren, R.: High Performance Spark (2017)Google Scholar
- 3.Banerjee, S.: Apache Spark and Amazon s3 gotchas and best practices (2016)Google Scholar
- 4.Chambers, B.: Spark: The Definitive Guide (2017)Google Scholar
- 5.Wide vs Narrow Dependencies (2017). https://github.com/roh-gar/scala-spark-4/wiki/Wide-vs-Narrow-Dependencies
- 6.Zaharia, M., Wendell, P., Konwinski, A., Karau, H.: Learning Spark (2015)Google Scholar
- 7.Cloudera: Spark Execution Model. https://www.cloudera.com/docu-mentation/enterprise/5-9-x/top-ics/cdh_ig_spark_apps.html#spark_exec_model
- 8.Rizay, S., Leserson, U., Owen, S., Wills, J.: Spark for Professionals: Modern Patterns of Big Data Processing (2018)Google Scholar
- 9.Khazaei, T.: Spark Performance Tuning: A Checklist (2017)Google Scholar
- 10.Hu, R., Wang, Z., Fan, W., Agarwal, S.: Cost Based Optimizer in Apache Spark 2.2 (2017)Google Scholar
- 11.Armbrust, M., Huai, Y., Liang, C., Xin, R., Zaharia, M.: Deep Dive into Spark. SQL’s Catalyst Optimizer (2015)Google Scholar
- 12.Armbrust, M., Xin, R.S., Lian, C., Huai, Y.: Spark SQL: Relational Data Processing in Spark (2015)Google Scholar
- 13.Catalyst Optimizer. https://databricks.com/glossary/catalyst-optimizer
- 14.Moisan, Y.: Spark performance tuning from the trenches (2018)Google Scholar
- 15.Apache. «Tuning Spark». https://spark.apache.org/docs/latest/tuning.html
- 16.The Scala Programming Language. https://www.scala-lang.org/
- 17.Apache Hive. https://hive.apache.org/
- 18.Apache Phoenix. https://phoenix.apache.org
Copyright information
© Springer Nature Switzerland AG 2020