Abstract
Progressive queries (PQ) are a new type of queries emerging from numerous contemporary database applications such as e-commerce, social network, business intelligence, and decision support. Such a PQ is formulated on the fly in several steps via a set of inter-related step-queries (SQ). In our previous work, we presented a framework to process a restricted type of PQs. However, how to process generic PQs remains an open problem. In this paper, we develop a novel technique to efficiently process generic PQs based on materialized views. The SQs of an in-process PQ can utilize the results of previous SQs not only from the same PQ but also from other in-process and completed PQs. The key idea is to create a multiple query dependency graph (MQDG), which captures the data source dependency relationships among SQs from multiple PQs. A mathematic model is developed to estimate the benefit of keeping the result of an SQ as a materialized view (critical SQ/node) based on the MQDG. The kept materialized views are used to improve the performance of the future SQs. Strategies for constructing the MQDG and identifying the critical SQs for materialization by using the MQDG are presented. To manage the storage of the materialized views, we introduce two approaches – one employs a greedy method and the other adopts a dynamic programming (DP) based method. Strategies are also suggested to reduce the input problem size for the DP procedure. Experimental results demonstrate that our technique is quite promising in efficiently processing PQs.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
If k=0, there is no existing critical node in cns. Assume \(\sum \limits _{j=1}^{0} (...) = 0\) in such a case.
If s=0, the leading new critical node cannot fit in the free space. Condition (5) is trivially true.
Informally, X contains the first m new critical nodes \(c_{i_{1}}\), \(c_{i_{2}}\), ..., \(c_{i_{m}}\) from list c s+1, c s+2, ..., c s that can fit in the remaining space of cns (after removing the space taken by c 1, ..., c s ) to the maximum capacity.
Assume that any critical node that cannot fit in the current space has been removed from consideration.
References
Babu, S., & Bizarro, P. (2005). Adaptive query processing in the looking glass. In Proceedings of CIDR conference (pp. 238–249).
Agrawal, S., Chaudhuri, S., Narasayya, V. (2000). Automated selection of materialized views and indexes in SQL databases. In Proceedings of VLDB conference (pp. 391–398).
Agarwal, P.K., Xie, J., Yang, J., Yu, H. (2006). Scalable continuous query processing by tracking hotspots. In Proceedings of VLDB conference, (pp. 31–42).
Antoshenkov, G. (1993). Dynamic query optimization in RDB/VMS. In Proceedings of ICDE conference (pp. 538–547).
Arion, A., Benzaken, V., Manolescu, I., Papakonstantinou, Y. (2007). Structured materialized views for XML queries. In Proceedings of VLDB conference (pp. 87–98).
Balmin, A., Beyer, K.-S., Cochrane, R., Pirahesh, H. (1997). A Framework for using materialized Xpath views in xml query processing. In Proceedings of VLDB conference (pp. 136–145).
Baralis, E., Paraboschi, S., Teniente, E. (1998). Materialized view selection for multidimensional datasets. In Proceedings of VLDB conference (pp. 488–499).
Blakeley, J.A., Larson, P.-Å., Tompa, F.W. (1986). Efficiently updating materialized views. In Proceedings of SIGMOD conference (pp. 61–71).
Babu, S. (2005). Adaptive query processing in data stream management systems. Ph.D. Dissert., Stanford Univ.
Folkert, N., Gupta, A., et al. (2005). Optimizing refresh of a set of materialized views. In Proceedings of VLDB conference.
Goldstein, J., & Larson, P-Å (2001). Optimizing queries using materialized views: A practical, scalable solution. In Proceedings of SIGMOD conference (pp. 331–342).
Gupta, A., Mumick, I.S., Ross, K.-A. (1995). Adapting materialized views after redefinitions. In Proceedings of SIGMOD conference (pp. 211–222).
Lee, J.H., Whang, K.Y., Lim, H.S., Lee, B.S., Heo, J.S. (2010). Progressive processing of continuous range queries in hierarchical wireless sensor networks. IEICE Transactions, 93(7), 1832–1847.
Jang, J.S.R., & Lee, H.R. (2008). A general framework of progressive filtering and its application to query by singing/humming. IEEE Transactions on Audio, Speech and Language Processing, 16(2), 350–358.
Jiang, H., Gao, D., Li, W.S. (2008). Exploiting correlation and parallelism of materialized-view recommendation for distributed data warehouses. In Proceedings of ICDE conference (pp. 276–285).
Jorg, T., & DeBloch, S. (2008). Towards generating etl processes for incremental loading. In Proceedings of IDEAS’08, (pp. 101–110).
Kabra, N., & DeWitt, D.J. (1998). Efficient mid-query re-optimization of sub-optimal query execution plans. In Proceedings of SIGMOD’98 (pp. 106–117).
Kache, H., Han, W.S., Markl, V., Raman, V., Ewen, S. (2006). POP/FED: Progressive query optimization for federated queries in DB2. In Proceedings of VLDB conference (pp. 1175–1178).
Lim, H.-S., Lee, L.-G., Lee, M.-J., Whang, K.-Y., Song, I.-Y. (2006). Continuous query processing in data streams using duality of data and queries. In Proceedings of SIGMOD conference (pp. 313–324).
Liu, L., & Pu, C. (1997). Dynamic query processing in DIOM. IEEE Data Engineering Bull, 20(3), 30–37.
Liu, Z., & Chen, Y. (2008). Answering keyword queries on xml using materialized views. In Proceedings of ICDE conference (pp. 1501–1503).
Lu, H., Tan, K.-L., Dao, S. (1995). The fittest survives: An adaptive approach to query optimization. In Proceedings of VLDB conference (pp. 251–262).
Luo, G. (2007). Partial materialized views. In Proceedings of ICDE conference (pp. 756–765).
Markl, V., Raman, V., Simmen, D.E., Lohman, G.M., Pirahesh, H. (2004). Robust query processing through progressive optimization. In Proceedings of SIGMOD conference (pp. 659–670).
Mistry, H., Roy, P., Sudarshan, S., Ramamritham, K. (2001). Materialized view selection and maintenance using multi-query optimization. In Proceedings of SIGMOD, (pp. 307-318).
Mokbel, M.F. (2004). Continuous query processing in spatio-temporal databases. In Proceedings of EDBT workshops (pp. 100–111).
Papadias, D., Tao, Y., Fu, G., Seeger, B. (2003). An optimal and progressive algorithm for skyline queries. In Proceedings of SIGMOD conference (pp. 467–478).
Park, C.-S., Kim, M.-H., Lee, Y.-J. (2001). Rewriting OLAP queries using materialized views and dimension hierarchies in data warehouses. In Proceedings of ICDE conference (pp. 515–523).
Phan, T., & Li, W.S. (2008). Dynamic materialization of query views for data warehouse workloads. In Proceedings of ICDE conference (pp. 436–445).
Raghavan, V., & Rundensteiner, E.-A. (2010). Progressive result generation for multi-criteria decision support queries. In Proceedings of ICDE conference (pp. 733–744).
Re, C., & Suciu, D. (2007). Materialized views in probabilistic databases for information exchange and query optimization. In Proceedings of VLDB Conference, (pp. 51–62).
Simitsis, A.P., Vassiliadis, Sellis, T. (2005). State-space optimization of ETL workflows. IEEE Transactions on Knowledge and Data Engineering, 17(10), 1404–1409.
Tang, N., Yu, J.X., Tamer zsu, M., Choi, B., Wong, K.F. (2008). Multiple materialized view selection for xpath query rewriting. In Proceedings of ICDE conference (pp. 873-882).
Tiakas, E., Papadopoulos, A.-N., Manolopoulos, Y. (2011). Progressive processing of subspace dominating queries. VLDB Journal, 20(6), 921–948.
Vassiliadis, P., Vagena, Z., Skiadopoulos, S., Karayannidis, N., Sellis, T. (2001). ARKTOS: towards the modeling, design, control and execution of ETL processes. Information System, 26(2001), 537–561.
Vassiliadis, P., Simitsis, A., Georgantas, P., Terrovitis, M., Skiadopoulos, S. (2005). A generic and customizable framework for the design of ETL scenarios. Information System, 30(2005), 492– 525.
Xu, W., & Ozsoyoglu, Z.M. (2005). Rewriting XPath queries using materialized views. In Proceedings of VLDB conference (pp. 121–132).
Yang, J., Karlapalem, K., Li, Q. (1997). Algorithms for materialized view design in data warehousing environment. In Proceedings of VLDB conference (pp. 136–145).
Zhou, J., Larson, P., Goldstein, J., Ding, L. (2007). Dynamic materialized views. In Proceedings of ICDE conference (pp. 526– 535).
Zhu, C., Zhu, Q., Zuzarte, C. (2010). Efficient processing of monotonic linear progressive queries via dynamic materialized views. In Proceedings of CASCON conference (pp. 224–237).
Zhu, C., Zhu, Q., Zuzarte, C., Ma, W. (2011). A materialized-view based technique to optimize progressive queries via dependency analysis. In Proceedings of CASCON conference (pp. 60– 73).
Zhu, Q., Medjahed, B., Sharma, A., Huang, H. (2008). The collective index: A technique for efficient processing of progressive queries. The Computer Journal, 51(6), 662–676.
Acknowledgment
This work was partially supported by the IBM Canada Software Laboratory and The University of Michigan. The preliminary results of this work were presented at the 2011 Conference of the IBM Centre for Advanced Studies on Collaborative Research – CASCON’11 (Zhu et al. 2011), Toronto, Nov. 7-10, 2011.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhu, C., Zhu, Q., Zuzarte, C. et al. Optimization of generic progressive queries based on dependency analysis and materialized views. Inf Syst Front 18, 205–231 (2016). https://doi.org/10.1007/s10796-014-9517-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10796-014-9517-2