Abstract
Language-integrated query based on comprehension syntax is a powerful technique for safe database programming, and provides a basis for advanced techniques such as query shredding or query flattening that allow efficient programming with complex nested collections. However, the foundations of these techniques are lacking: although SQL, the most widely-used database query language, supports heterogeneous queries that mix set and multiset semantics, these important capabilities are not supported by known correctness results or implementations that assume homogeneous collections. In this paper we study language-integrated query for a heterogeneous query language \(\mathcal {NRC}_{\lambda }( Set,Bag )\) that combines set and multiset constructs. We show how to normalize and translate queries to SQL, and develop a novel approach to querying heterogeneous nested collections, based on the insight that “local” query subexpressions that calculate nested subcollections can be “lifted” to the top level analogously to lambda-lifting for local function definitions.
Chapter PDF
Similar content being viewed by others
References
Benedikt, M., Pradic, P.: Generating collection transformations from proofs. Proc. ACM Program. Lang. 5(POPL) (Jan 2021), https://doi.org/10.1145/3434295
Blelloch, G.E.: Vector Models for Data-Parallel Computing. MIT Press (1990)
Buneman, P., Libkin, L., Suciu, D., Tannen, V., Wong, L.: Comprehension syntax. SIGMOD Record 23 (1994)
Buneman, P., Naqvi, S., Tannen, V., Wong, L.: Principles of programming with complex objects and collection types. Theor. Comput. Sci. 149(1) (1995). https://doi.org/10.1016/0304-3975(95)00024-Q
Cao, B., Badia, A.: SQL query optimization through nested relational algebra. ACM Trans. Database Syst. 32(3), 18–es (Aug 2007). https://doi.org/10.1145/1272743.1272748
Carette, J., Kiselyov, O., Shan, C.: Finally tagless, partially evaluated: Tagless staged interpreters for simpler typed languages. J. Funct. Program. 19(5), 509–543 (2009). https://doi.org/10.1017/S0956796809007205
Cheney, J., Lindley, S., Wadler, P.: A practical theory of language-integrated query. In: ICFP (2013). https://doi.org/10.1145/2500365.2500586
Cheney, J., Lindley, S., Wadler, P.: Query shredding: efficient relational evaluation of queries over nested multisets. In: SIGMOD. pp. 1027–1038. ACM (2014). https://doi.org/10.1145/2588555.2612186
Cheney, J., Lindley, S., Wadler, P.: Query shredding: Efficient relational evaluation of queries over nested multisets (extended version). CoRR abs/1404.7078 (2014), http://arxiv.org/abs/1404.7078
Chu, S., Weitz, K., Cheung, A., Suciu, D.: HoTTSQL: Proving query rewrites with univalent SQL semantics. In: PLDI. pp. 510–524. ACM (2017). https://doi.org/10.1145/3062341.3062348
Cooper, E.: The script-writer’s dream: How to write great SQL in your own language, and be sure it will succeed. In: DBPL (2009). https://doi.org/10.1007/978-3-642-03793-1_3
Cooper, E., Lindley, S., Wadler, P., Yallop, J.: Links: web programming without tiers. In: FMCO (2007). https://doi.org/10.1007/978-3-540-74792-5_12
Copeland, G., Maier, D.: Making Smalltalk a database system. SIGMOD Rec. 14(2) (1984)
Fegaras, L.: An algebra for distributed big data analytics. J. Funct. Program. 27, e27 (2017). https://doi.org/10.1017/S0956796817000193
Fegaras, L., Maier, D.: Optimizing object queries using an effective calculus. ACM Trans. Database Syst. 25(4), 457–516 (2000)
Fegaras, L., Srinivasan, C., Rajendran, A., Maier, D.: lambda-DB: An ODMG-based object-oriented DBMS. In: Chen, W., Naughton, J.F., Bernstein, P.A. (eds.) SIGMOD. p. 583. ACM (2000). https://doi.org/10.1145/342009.335494
Fehrenbach, S., Cheney, J.: Language-integrated provenance. Science of Computer Programming 155, 103–145 (2018)
Foster, J.N., Green, T.J., Tannen, V.: Annotated XML: queries and provenance. In: PODS. pp. 271–280 (2008)
Fowler, S., Harding, S., Sharman, J., Cheney, J.: Cross-tier web programming for curated databases: a case study. International Journal of Digital Curation 15(1) (2020). https://doi.org/10.2218/ijdc.v15i1.717, pre-print presented at IDCC 2020
Gibbons, J., Henglein, F., Hinze, R., Wu, N.: Relational algebra by way of adjunctions. Proc. ACM Program. Lang. 2(ICFP) (Jul 2018). https://doi.org/10.1145/3236781
Giorgidze, G., Grust, T., Schreiber, T., Weijers, J.: Haskell boards the Ferry - database-supported program execution for Haskell. In: IFL. pp. 1–18. No. 6647 in LNCS, Springer-Verlag (2010)
Giorgidze, G., Grust, T., Ulrich, A., Weijers, J.: Algebraic data types for language-integrated queries. In: DDFP. pp. 5–10 (2013)
Green, T.J., Karvounarakis, G., Tannen, V.: Provenance semirings. In: PODS (2007)
Grust, T., Mayr, M., Rittinger, J.: Let SQL drive the XQuery workhorse (XQuery join graph isolation). In: EDBT. pp. 147–158 (2010). https://doi.org/10.1145/1739041.1739062
Grust, T., Mayr, M., Rittinger, J., Schreiber, T.: Ferry: Database-supported program execution. In: SIGMOD (June 2009)
Grust, T., Rittinger, J., Schreiber, T.: Avalanche-safe LINQ compilation. PVLDB 3(1) (2010)
Grust, T., Rittinger, J., Teubner, J.: Pathfinder: XQuery off the relational shelf. IEEE Data Eng. Bull. 31(4) (2008)
Grust, T., Scholl, M.H.: How to comprehend queries functionally. J. Intell. Inf. Syst. 12(2-3), 191–218 (1999). https://doi.org/10.1023/A:1008705026446
Grust, T., Ulrich, A.: First-class functions for first-order database engines. In: DBPL (2013), http://arxiv.org/abs/1308.0158
Johnsson, T.: Lambda lifting: Treansforming programs to recursive equations. In: FPCA. pp. 190–203 (1985). https://doi.org/10.1007/3-540-15975-4_37
Katsushima, T., Kiselyov, O.: Language-integrated query with ordering, grouping and outer joins (poster paper). In: PEPM. pp. 123–124 (2017)
Kiselyov, O., Katsushima, T.: Sound and efficient language-integrated query - maintaining the ORDER. In: APLAS 2017. pp. 364–383 (2017). https://doi.org/10.1007/978-3-319-71237-6_18
Libkin, L., Wong, L.: Query languages for bags and aggregate functions. J. Comput. Syst. Sci. 55(2) (1997). https://doi.org/10.1006/jcss.1997.1523
Lindley, S., Cheney, J.: Row-based effect types for database integration. In: TLDI (2012). https://doi.org/10.1145/2103786.2103798
Lindley, S., Wadler, P.: The audacity of hope: Thoughts on reclaiming the database dream. In: ESOP (2010)
Meijer, E., Beckman, B., Bierman, G.M.: LINQ: reconciling object, relations and XML in the .NET framework. In: SIGMOD (2006). https://doi.org/10.1145/1142473.1142552
Minamide, Y., Morrisett, J.G., Harper, R.: Typed closure conversion. In: POPL. pp. 271–283 (1996). https://doi.org/10.1145/237721.237791
Neumann, T., Kemper, A.: Unnesting arbitrary queries. In: Datenbanksysteme für Business, Technologie und Web (BTW). pp. 383–402 (2015)
Okura, R., Kameyama, Y.: Language-integrated query with nested data structures and grouping. In: FLOPS. pp. 139–158 (2020). https://doi.org/10.1007/978-3-030-59025-3_9
Paredaens, J., Van Gucht, D.: Converting nested algebra expressions into flat algebra expressions. ACM Trans. Database Syst. 17(1) (1992). https://doi.org/10.1145/128765.128768
Quill: Compile-time language integrated queries for Scala. Open source project, https://github.com/getquill/quill
Ricciotti, W., Cheney, J.: Mixing set and bag semantics. In: DBPL. pp. 70–73 (2019). https://doi.org/10.1145/3315507.3330202
Ricciotti, W., Cheney, J.: Strongly normalizing higher-order relational queries. In: FSCD. pp. 28:1–28:22 (2020). https://doi.org/10.4230/LIPIcs.FSCD.2020.28
Ricciotti, W., Cheney, J.: Query lifting: Language-integrated query for heterogeneous nested collections. ArXiv e-prints (2021), https://arxiv.org/abs/2101.04102
Russell, C.: Bridging the object-relational divide. Queue 6 (May 2008). https://doi.org/10.1145/1394127.1394139
Schek, H., Scholl, M.H.: The relational model with relation-valued attributes. Inf. Syst. 11(2), 137–147 (1986). https://doi.org/10.1016/0306-4379(86)90003-7
Stolarek, J., Cheney, J.: Language-integrated provenance in Haskell. The Art, Science, and Engineering of Programming 2(3), A11 (2018)
Suzuki, K., Kiselyov, O., Kameyama, Y.: Finally, safely-extensible and efficient language-integrated query. In: PEPM. pp. 37–48 (2016). https://doi.org/10.1145/2847538.2847542
Syme, D.: Leveraging .NET meta-programming components from F#: integrated queries and interoperable heterogeneous execution. In: ML Workshop (2006)
Trinder, P., Wadler, P.: Improving list comprehension database queries. In: TENCON ’89. (1989). https://doi.org/10.1109/TENCON.1989.176921
Ulrich, A.: A Ferry-based query backend for the Links programming language. Master’s thesis, University of Tübingen (2011)
Ulrich, A.: Query Flattening and the Nested Data Parallelism Paradigm. Ph.D. thesis, University of Tübingen, Germany (2019)
Ulrich, A., Grust, T.: The flatter, the better: Query compilation based on the flattening transformation. In: SIGMOD. pp. 1421–1426. ACM (2015). https://doi.org/10.1145/2723372.2735359
Van den Bussche, J.: Simulation of the nested relational algebra by the flat relational algebra, with an application to the complexity of evaluating powerset algebra expressions. Theor. Comput. Sci. 254(1-2) (2001)
Wong, L.: Normal forms and conservative extension properties for query languages over collection types. J. Comput. Syst. Sci. 52(3) (1996). https://doi.org/10.1006/jcss.1996.0037
Wong, L.: Kleisli, a functional query system. J. Funct. Program. 10(1) (2000). https://doi.org/10.1017/S0956796899003585
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
Copyright information
© 2021 The Author(s)
About this paper
Cite this paper
Ricciotti, W., Cheney, J. (2021). Query Lifting. In: Yoshida, N. (eds) Programming Languages and Systems. ESOP 2021. Lecture Notes in Computer Science(), vol 12648. Springer, Cham. https://doi.org/10.1007/978-3-030-72019-3_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-72019-3_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72018-6
Online ISBN: 978-3-030-72019-3
eBook Packages: Computer ScienceComputer Science (R0)