Advertisement

Data Shuffling Minimizing Approach for Apache Spark Programs

  • Maksim PopovEmail author
  • Pavel D. Drobintsev
Conference paper
Part of the Lecture Notes in Networks and Systems book series (LNNS, volume 95)

Abstract

This article discusses a way to optimize the Apache Spark program by reducing the number of transformations with wide dependencies and, as a result, the number of data shuffles. This is achieved by combining sequential data processing algorithms in chains based on common key fields, as well as grouping the data which is stored in resilient distributed structures i.e., Spark SQL Datasets, according to the keys by which the processing takes place.

Keywords

Apache spark Big data Hadoop Optimization Scala 

References

  1. 1.
  2. 2.
    Karau, H., Warren, R.: High Performance Spark (2017)Google Scholar
  3. 3.
    Banerjee, S.: Apache Spark and Amazon s3 gotchas and best practices (2016)Google Scholar
  4. 4.
    Chambers, B.: Spark: The Definitive Guide (2017)Google Scholar
  5. 5.
  6. 6.
    Zaharia, M., Wendell, P., Konwinski, A., Karau, H.: Learning Spark (2015)Google Scholar
  7. 7.
  8. 8.
    Rizay, S., Leserson, U., Owen, S., Wills, J.: Spark for Professionals: Modern Patterns of Big Data Processing (2018)Google Scholar
  9. 9.
    Khazaei, T.: Spark Performance Tuning: A Checklist (2017)Google Scholar
  10. 10.
    Hu, R., Wang, Z., Fan, W., Agarwal, S.: Cost Based Optimizer in Apache Spark 2.2 (2017)Google Scholar
  11. 11.
    Armbrust, M., Huai, Y., Liang, C., Xin, R., Zaharia, M.: Deep Dive into Spark. SQL’s Catalyst Optimizer (2015)Google Scholar
  12. 12.
    Armbrust, M., Xin, R.S., Lian, C., Huai, Y.: Spark SQL: Relational Data Processing in Spark (2015)Google Scholar
  13. 13.
  14. 14.
    Moisan, Y.: Spark performance tuning from the trenches (2018)Google Scholar
  15. 15.
  16. 16.
    The Scala Programming Language. https://www.scala-lang.org/
  17. 17.
  18. 18.

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Peter the Great St. Petersburg Polytechnic UniversitySaint PetersburgRussia

Personalised recommendations