Skip to main content
SpringerLink
Log in

An Adaptive Data Partitioning Scheme for Accelerating Exploratory Spark SQL Queries

  • Conference paper
  • First Online:
Database Systems for Advanced Applications (DASFAA 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10177))

Included in the following conference series:

Abstract

For data analysis, it’s useful to explore the data set with a sequence of queries, frequently using the results from the previous queries to shape the next queries. Thus, data used in the previous queries are often reused, at least in part, in the next queries. This fact may be used to accelerate queries with data partitioning, a widely used technique that enables skipping the irrelevant data for better I/O performance. For getting effective partitions which are likely to cover the query workload in the future, we propose an adaptive partitioning scheme, combining the data-driven metrics and user-driven metrics to guide the data partitioning as well as a heuristic model using the metric plugin system to support different exploratory patterns. For partition storage and management, we propose an effective partition index structure for quickly searching for appropriate partitions to answer queries. The system is quite helpful in improving the performance of exploratory queries.

The work is supported by the NSFC (No.61370080, No.61170007) and the Shanghai Innovation Action Project (Grant No.16DZ1100200).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Apache Hive. https://hive.apache.org/

  2. Early Journal Content Data Bundle. http://dfr.jstor.org/

  3. Sloan Digital Sky Surver (SkyServer). http://cas.sdss.org/dr8/en/

  4. TPC-H, Benchmark Specification. http://www.tpc.org/tpch/

  5. Aly, A.M., Mahmood, A.R., Hassan, M.S., Aref, W.G., Ouzzani, M., Elmeleegy, H., Qadah, T.: Aqwa: adaptive query workload aware partitioning of big spatial data. Proc. VLDB Endowment 8(13), 2062–2073 (2015)

    Article  Google Scholar 

  6. Armbrust, M., Xin, R.S., Lian, C., Huai, Y., Liu, D., Bradley, J.K., Meng, X., Kaftan, T., Franklin, M.J., Ghodsi, A., et al.: Spark SQL: Relational data processing in spark. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 1383–1394. ACM, New York (2015)

    Google Scholar 

  7. Ester, M., Kriegel, H.P., Sander, J., Xu, X., et al.: A density-based algorithm for discovering clusters in large spatial databases with noise (1996)

    Google Scholar 

  8. Halim, F., Idreos, S., Karras, P., Yap, R.H.: Stochastic database cracking: towards robust adaptive indexing in main-memory column-stores. Proc. VLDB Endowment 5(6), 502–513 (2012)

    Article  Google Scholar 

  9. Idreos, S., Kersten, M.L., Manegold, S., et al.: Database cracking. In: CIDR, vol. 3, pp. 1–8 (2007)

    Google Scholar 

  10. Jiang, L., Nandi, A.: Snaptoquery: providing interactive feedback during exploratory query specification. Proc. VLDB Endowment 8(11), 1250–1261 (2015)

    Article  Google Scholar 

  11. Nandi, A., Jagadish, H.: Guided interaction: rethinking the query-result paradigm. Proc. VLDB Endowment 4(12), 1466–1469 (2011)

    Google Scholar 

  12. Paurat, D., Garnett, R., Gärtner, T.: Interactive exploration of larger pattern collections: a case study on a cocktail dataset. In: Workshop on Interactive Data Exploration and Analytics. IDEA (2014)

    Google Scholar 

  13. Peng, S., Gu, J., Wang, X.S., Rao, W., Yang, M., Cao, Y.: Cost-based optimization of logical partitions for a query workload in a hadoop data warehouse. In: Chen, L., Jia, Y., Sellis, T., Liu, G. (eds.) APWeb 2014. LNCS, vol. 8709, pp. 559–567. Springer, Cham (2014). doi:10.1007/978-3-319-11116-2_52

    Google Scholar 

  14. Preparata, F.P., Shamos, M.: Computational Geometry: An Introduction. Springer, New York (2012)

    MATH  Google Scholar 

  15. Qin, W., Idreos, S.: Adaptive data skipping in main-memory systems. In: ACM SIGMOD International Conference on Management of Data (2016)

    Google Scholar 

  16. Sun, L., Franklin, M.J., Krishnan, S., Xin, R.S.: Fine-grained partitioning for aggressive data skipping. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 1115–1126. ACM, New York (2014)

    Google Scholar 

  17. Vartak, M., Rahman, S., Madden, S., Parameswaran, A., Polyzotis, N.: SeeDB: efficient data-driven visualization recommendations to support visual analytics. Proc. VLDB Endowment 8(13), 2182–2193 (2015)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science, Fudan University, Shanghai, China

    Chenghao Guo, Zhigang Wu, Zhenying He & X. Sean Wang

  2. Shanghai Key Laboratory of Data Science, Shanghai, China

    Chenghao Guo, Zhigang Wu, Zhenying He & X. Sean Wang

Authors
  1. Chenghao Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhigang Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhenying He
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. X. Sean Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to X. Sean Wang .

Editor information

Editors and Affiliations

  1. Arizona State University , Tempe - Phoenix, Arizona, USA

    Selçuk Candan

  2. Hong Kong University of Science and Tech , Hong Kong, China

    Lei Chen

  3. Aalborg University , Aalborg, Denmark

    Torben Bach Pedersen

  4. University of New South Wales , Sydney, New South Wales, Australia

    Lijun Chang

  5. The University of Queensland , Brisbane, Queensland, Australia

    Wen Hua

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Guo, C., Wu, Z., He, Z., Wang, X.S. (2017). An Adaptive Data Partitioning Scheme for Accelerating Exploratory Spark SQL Queries. In: Candan, S., Chen, L., Pedersen, T., Chang, L., Hua, W. (eds) Database Systems for Advanced Applications. DASFAA 2017. Lecture Notes in Computer Science(), vol 10177. Springer, Cham. https://doi.org/10.1007/978-3-319-55753-3_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-55753-3_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-55752-6

  • Online ISBN: 978-3-319-55753-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation