ATUN-HL: Auto Tuning of Hybrid Layouts Using Workload and Data Characteristics

  • Rana Faisal MunirEmail author
  • Alberto Abelló
  • Oscar Romero
  • Maik Thiele
  • Wolfgang Lehner
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11019)


Ad-hoc analysis implies processing data in near real-time. Thus, raw data (i.e., neither normalized nor transformed) is typically dumped into a distributed engine, where it is generally stored into a hybrid layout. Hybrid layouts divide data into horizontal partitions and inside each partition, data are stored vertically. They keep statistics for each horizontal partition and also support encoding (i.e., dictionary) and compression to reduce the size of the data. Their built-in support for many ad-hoc operations (i.e., selection, projection, aggregation, etc.) makes hybrid layouts the best choice for most operations.

Horizontal partition and dictionary sizes of hybrid layouts are configurable and can directly impact the performance of analytical queries. Hence, their default configuration cannot be expected to be optimal for all scenarios. In this paper, we present ATUN-HL (Auto TUNing Hybrid Layouts), which based on a cost model and given the workload and the characteristics of data, finds the best values for these parameters. We prototyped ATUN-HL for Apache Parquet, which is an open source implementation of hybrid layouts in Hadoop Distributed File System, to show its effectiveness. Our experimental evaluation shows that ATUN-HL provides on average 85% of all the potential performance improvement, and 1.2x average speedup against default configuration.


Big data Hybrid storage layouts Auto tuning Parquet 



This research has been funded by the European Commission through the Erasmus Mundus Joint Doctorate “Information Technologies for Business Intelligence - Doctoral College” (IT4BI-DC), and the GENESIS project, funded by the Spanish Ministerio de Ciencia e Innovación under project TIN2016-79269-R.


  1. 1.
    Abedjan, Z., Golab, L., Naumann, F.L.: Data profiling: a tutorial. In: SIGMOD Conference. ACM (2017)Google Scholar
  2. 2.
    Alagiannis, I., Idreos, S., Ailamaki, A.: H2O: a hands-free adaptive store. In: SIGMOD Conference. ACM (2014)Google Scholar
  3. 3.
    Azim, T., Karpathiotakis, M., Ailamaki, A.: Recache: reactive caching for fast analytics over heterogeneous data. PVLDB 11(3), 324–337 (2017)Google Scholar
  4. 4.
    Bian, H., et al.: Wide table layout optimization based on column ordering and duplication. In: SIGMOD Conference. ACM (2017)Google Scholar
  5. 5.
    Cardenas, A.F.: Analysis and performance of inverted data base structures. Commun. ACM 18(5), 253–263 (1975)MathSciNetCrossRefGoogle Scholar
  6. 6.
    Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)CrossRefGoogle Scholar
  7. 7.
    Ferreira, M., Paiva, J., Bravo, M., Rodrigues, L.E.T.: Smartfetch: efficient support for selective queries. In: CloudCom. IEEE Computer Society (2015)Google Scholar
  8. 8.
    Han, J., Cheng, H., Xin, D., Yan, X.: Frequent pattern mining: current status and future directions. Data Min. Knowl. Discov. 15(1), 55–86 (2007)MathSciNetCrossRefGoogle Scholar
  9. 9.
    Herodotou, H., Babu, S.: Profiling, what-if analysis, and cost-based optimization of mapreduce programs. PVLDB 4(11), 1111–1122 (2011)Google Scholar
  10. 10.
    Herodotou, H., et al.: Starfish: a self-tuning system for big data analytics. In: CIDR (2011)Google Scholar
  11. 11.
    Jindal, A., Quiané-Ruiz, J., Dittrich, J.: Trojan data layouts: right shoes for a running elephant. In: SoCC. ACM (2011)Google Scholar
  12. 12.
    Li, Y., Patel, J.M.: Widetable: an accelerator for analytical data processing. PVLDB 7(10), 907–918 (2014)Google Scholar
  13. 13.
    Moerkotte, G.: Small materialized aggregates: a light weight index structure for data warehousing. In: VLDB, pp. 476–487 (1998)Google Scholar
  14. 14.
    Munir, R.F., Abelló, A., Romero, O., Thiele, M., Lehner, W.: A cost-based storage format selector for materialization in big data frameworks. CoRR, abs/1806.03901 (2018)Google Scholar
  15. 15.
    Munir, R.F., Romero, O., Abelló, A., Bilalli, B., Thiele, M., Lehner, W.: ResilientStore: a heuristic-based data format selector for intermediate results. In: Bellatreche, L., Pastor, Ó., Almendros Jiménez, J.M., Aït-Ameur, Y. (eds.) MEDI 2016. LNCS, vol. 9893, pp. 42–56. Springer, Cham (2016). Scholar
  16. 16.
    Shvachko, K.V.: HDFS scalability: the limits to growth. Login 35(2), 6–16 (2010)Google Scholar
  17. 17.
    Sun, L., Franklin, M.J., Krishnan, S., Xin, R.S.: Fine-grained partitioning for aggressive data skipping. In: SIGMOD Conference. ACM (2014)Google Scholar
  18. 18.
    Sun, L., Franklin, M.J., Wang, J., Wu, E.: Skipping-oriented partitioning for columnar layouts. PVLDB 10(4), 421–432 (2016)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Rana Faisal Munir
    • 1
    • 2
    Email author
  • Alberto Abelló
    • 1
  • Oscar Romero
    • 1
  • Maik Thiele
    • 2
  • Wolfgang Lehner
    • 2
  1. 1.Universitat Politècnica de Catalunya (UPC)BarcelonaSpain
  2. 2.Technische Universität Dresden (TUD)DresdenGermany

Personalised recommendations