Abstract
Data wrangling is the process whereby data is cleaned and integrated for analysis. Data wrangling, even with tool support, is typically a labour intensive process. One aspect of data wrangling involves carrying out format transformations on attribute values, for example so that names or phone numbers are represented consistently. Recent research has developed techniques for synthesising format transformation programs from examples of the source and target representations. This is valuable, but still requires a user to provide suitable examples, something that may be challenging in applications in which there are huge data sets or numerous data sources. In this paper we investigate the automatic discovery of examples that can be used to synthesise format transformation programs. In particular, we propose an approach to identifying candidate data examples and validating the transformations that are synthesised from them. The approach is evaluated empirically using data sets from open government data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
It can be seen that the complexity of Algorithm 1 is \(\mathcal {O}(nm)\) where n is the number of attributes of S and m is the number of attributes of T. This is due to the cross product between the columns of the two data sets (i.e. the two for loops from the beginning of the algorithm). We do not analyse here the complexity of the other algorithms used in our experiments as this has been done in the original papers. Nor do we emphasize on the impact of input size on the overall solution. In our experiments, the run-time of Algorithm 1, pertaining examples generation alone, did not exceed one second for any of the datasets used.
- 3.
- 4.
- 5.
- 6.
- 7.
References
Chu, X., Ilyas, I.F., Papotti, P.: Holistic data cleaning: putting violations into context. In: ICDE 2013, pp. 458–469 (2013)
Fan, W.: Dependencies revisited for improving data quality. In: PODS 2008, pp. 159–170, 9–11 June 2008
Fan, W.: Data quality: from theory to practice. SIGMOD Rec. 44(3), 7–18 (2015)
Fan, W., Li, J., Ma, S., Tang, N., Yu, W.: Towards certain fixes with editing rules and master data. VLDB J. 21(2), 213–238 (2012)
Furche, T., Gottlob, G., Libkin, L., Orsi, G., Paton, N.W.: Data wrangling for big data: challenges and opportunities. In: EDBT, pp. 473–478 (2016)
Gulwani, S.: Automating string processing in spreadsheets using input-output examples. In: POPL, pp. 317–330 (2011)
Heer, J., Hellerstein, J.M., Kandel, S.: Predictive interaction for data transformation. In: CIDR 2015, 4–7 January 2015
Kandel, S., Paepcke, A., Hellerstein, J., Heer, J.: Wrangler: interactive visual specification of data transformation scripts. In: CHI, pp. 3363–3372 (2011)
Papenbrock, T., Naumann, F.: A hybrid approach to functional dependency discovery. In: SIGMOD Conference, pp. 821–833. ACM (2016)
Rahm, E., Bernstein, P.A.: A survey of approaches to automatic schema matching. VLDBJ 10(4), 334–350 (2001)
Raman, V., Hellerstein, J.M.: Potter’s wheel: an interactive data cleaning system. In: VLDB 2001, pp. 381–390, 11–14 September 2001
Singh, R.: BlinkFill: semi-supervised programming by example for syntactic string transformations. PVLDB 9(10), 816–827 (2016)
Jia, X., Fan, W., Geerts, F., Kementsietsidis, A.: Conditional functional dependencies for capturing data inconsistencies. TODS 33(1), 6:1–6:48 (2008)
Wu, B., Knoblock, C.A.: An iterative approach to synthesize data transformation programs. In: IJCAI 2015, pp. 1726–1732, 25–31 July 2015
Yakout, M., Elmagarmid, A.K., Neville, J., Ouzzani, M., Ilyas, I.F.: Guided data repair. PVLDB 4(5), 279–289 (2011)
Acknowledgement
This work has been made possible by funding from the UK Engineering and Physical Sciences Research council, whose support we are pleased to acknowledge.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Bogatu, A., Paton, N.W., Fernandes, A.A.A. (2017). Towards Automatic Data Format Transformations: Data Wrangling at Scale. In: Calì, A., Wood, P., Martin, N., Poulovassilis, A. (eds) Data Analytics. BICOD 2017. Lecture Notes in Computer Science(), vol 10365. Springer, Cham. https://doi.org/10.1007/978-3-319-60795-5_4
Download citation
DOI: https://doi.org/10.1007/978-3-319-60795-5_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-60794-8
Online ISBN: 978-3-319-60795-5
eBook Packages: Computer ScienceComputer Science (R0)