Empowering Domain Experts to Preprocess Massive Distributed Datasets

Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 389)


In recent years, the amount of data is growing extensively. In companies, spreadsheets are one common approach to conduct data processing and statistical analysis. However, especially when working with massive amounts of data, spreadsheet applications have their limitations. To cope with this issue, we introduce a human-in-the-loop approach for scalable data preprocessing using sampling. In contrast to state-of-the-art approaches, we also consider conflict resolution and recommendations based on data not contained in the sample itself. We implemented a fully functional prototype and conducted a user study with 12 participants. We show that our approach delivers a significantly higher error correction than comparable approaches which only consider the sample dataset.


Data cleaning Human-in-the-loop Interactive data preprocessing 



This research was performed in the project ‘IMPORT’ as part of the Software Campus program, which is funded by the German Federal Ministry of Education and Research (BMBF) under Grant No.: 01IS17051.


  1. 1.
    Abedjan, Z., et al.: Detecting data errors: where are we and what needs to be done? Proc. VLDB Endow. 9(12), 993–1004 (2016)CrossRefGoogle Scholar
  2. 2.
    Bendre, M., et al.: Anti-freeze for large and complex spreadsheets: asynchronous formula computation. In: Proceedings of the International Conference on Management of Data (SIGMOD) (2019)Google Scholar
  3. 3.
    Cypher, A. (ed.): Watch What I Do - Programming by Demonstration. MIT Press, Cambridge (1993)Google Scholar
  4. 4.
    Dua, D., Graff, C.: UCI machine learning repository (2017).
  5. 5.
    Fayyad, U.M., Piatetsky-Shapiro, G., Smyth, P.: From data mining to knowledge discovery in databases. AI Mag. 17(3), 37 (1996)Google Scholar
  6. 6.
    Gandel, S.: Damn Excel! How the ‘most important software application of all time’ is ruining the world (2013).
  7. 7.
    Gulwani, S., Marron, M.: NLyze: interactive programming by natural language for spreadsheet data analysis and manipulation. In: Proceedings of the International Conference on Management of Data (SIGMOD) (2014)Google Scholar
  8. 8.
    Gulwani, S., et al.: Spreadsheet data manipulation using examples. Commun. ACM 55(8), 97–105 (2012)CrossRefGoogle Scholar
  9. 9.
    International Business Machines Corporation: Transforming the Common Spreadsheet: A Smarter Approach to Budgeting, Planning and Forecasting, Technical report (2009)Google Scholar
  10. 10.
    Kandel, S., et al.: Wrangler: interactive visual specification of data transformation scripts. In: Proceedings of the Conference on Human Factors in Computing Systems (CHI) (2011)Google Scholar
  11. 11.
    Kemper, H.G., et al.: Datenbereitstellung und -modellierung. In: Business Intelligence - Grundlagen und praktische Anwendungen: Eine Einführung in die IT-basierte Managementunterstützung (2010)Google Scholar
  12. 12.
    Lohr, S.L.: Sampling: Design and Analysis. Brooks/Cole (2009)Google Scholar
  13. 13.
    Mack, K., et al.: Characterizing scalability issues in spreadsheet software using online forums. In: Extended Abstracts of the Conference on Human Factors in Computing Systems (CHI EA) (2018)Google Scholar
  14. 14.
    Moore, S.: Gartner says more than 40 percent of data science tasks will be automated by 2020 (2017).
  15. 15.
    Reimann, P., Schwarz, H., Mitschang, B.: A pattern approach to conquer the data complexity in simulation workflow design. In: Meersman, R., et al. (eds.) OTM 2014. LNCS, vol. 8841, pp. 21–38. Springer, Heidelberg (2014). Scholar
  16. 16.
    Rekatsinas, T., et al.: HoloClean - holistic data repairs with probabilistic inference. Proc. VLDB Endow. 10(11) (2017)Google Scholar
  17. 17.
    Shearer, C.: The CRISP-DM model: the new blueprint for data mining. J. Data Warehouse. 5(4) (2000)Google Scholar
  18. 18.
    Stodder, D.: Visual Analytics for Making Smarter Decisions Faster. Technical report, TDWI, Renton, WA, USA (2015)Google Scholar
  19. 19.
    Wache, H., et al.: Ontology-based integration of information - a survey of existing approaches. In: Proceedings of the Workshop on Ontologies and Information Sharing, International Joint Conference on Artificial Intelligence (IJCAI) (2001)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  1. 1.Institute for Parallel and Distributed SystemsUniversity of StuttgartStuttgartGermany

Personalised recommendations