Relaxed Functional Dependency Discovery in Heterogeneous Data Lakes

  • Rihan HaiEmail author
  • Christoph Quix
  • Dan Wang
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11788)


Functional dependencies are important for the definition of constraints and relationships that have to be satisfied by every database instance. Relaxed functional dependencies (RFDs) can be used for data exploration and profiling in datasets with lower data quality. In this work, we present an approach for RFD discovery in heterogeneous data lakes. More specifically, the goal of this work is to find RFDs from structured, semi-structured, and graph data. Our solution brings novelty to this problem in the following aspects: (1) We introduce a generic metamodel to the problem of RFD discovery, which allows us to define and detect RFDs for data stored in heterogeneous sources in an integrated manner. (2) We apply clustering techniques during RFD discovery for partitioning and pruning. (3) We performed an intensive evaluation with nine datasets, which shows that our approach is effective for discovering meaningful RFDs, reducing redundancy, and detecting inconsistent data.



The authors would like to thank the German Research Foundation DFG for the kind support within the Cluster of Excellence “Internet of Production” (Project ID: EXC 2023/390621612).


  1. 1.
    Bassée, R., Wijsen, J.: Neighborhood dependencies for prediction. In: Cheung, D., Williams, G.J., Li, Q. (eds.) PAKDD 2001. LNCS (LNAI), vol. 2035, pp. 562–567. Springer, Heidelberg (2001). Scholar
  2. 2.
    Caruccio, L., Deufemia, V., Polese, G.: Relaxed functional dependencies - a survey of approaches. IEEE Trans. Knowl. Data Eng. 28(1), 147–165 (2016)CrossRefGoogle Scholar
  3. 3.
    Cong, G., Fan, W., Geerts, F., Jia, X., Ma, S.: Improving data quality: consistency and accuracy. In: Proceedings of the VLDB, pp. 315–326 (2007)Google Scholar
  4. 4.
    Fassetti, F., Fazzinga, B.: Approximate functional dependencies for XML data. In: Proceedings of the ADBIS (2007)Google Scholar
  5. 5.
    Hai, R., Geisler, S., Quix, C.: Constance: an intelligent data lake system. In: Proceedings of the SIGMOD, pp. 2097–2100. ACM (2016)Google Scholar
  6. 6.
    Hai, R., Quix, C.: Rewriting of plain SO tgds into nested tgds. Proc. VLDB Endow. 12(11), 1526–1538 (2019)CrossRefGoogle Scholar
  7. 7.
    Hai, R., Quix, C., Kensche, D.: Nested schema mappings for integrating JSON. In: Trujillo, J.C., et al. (eds.) ER 2018. LNCS, vol. 11157, pp. 397–405. Springer, Cham (2018). Scholar
  8. 8.
    Hai, R., Quix, C., Zhou, C.: Query rewriting for heterogeneous data lakes. In: Benczúr, A., Thalheim, B., Horváth, T. (eds.) ADBIS 2018. LNCS, vol. 11019, pp. 35–49. Springer, Cham (2018). Scholar
  9. 9.
    Huhtala, Y., et al.: TANE: an efficient algorithm for discovering functional and approximate dependencies. Comput. J. 42(2), 100–111 (1999)CrossRefGoogle Scholar
  10. 10.
    Kensche, D., Quix, C., Li, X., Li, Y., Jarke, M.: Generic schema mappings for composition and query answering. Data Knowl. Eng. 68(7), 599–621 (2009)CrossRefGoogle Scholar
  11. 11.
    Kruse, S., Naumann, F.: Efficient discovery of approximate dependencies. Proc. VLDB Endow. 11(7), 759–772 (2018)CrossRefGoogle Scholar
  12. 12.
    Liu, J., Li, J., Liu, C., Chen, Y.: Discover dependencies from data - a review. IEEE Trans. Knowl. Data Eng. 24(2), 251–264 (2012)CrossRefGoogle Scholar
  13. 13.
    Pelleg, D., Moore, A.W., et al.: X-means: extending k-means with efficient estimation of the number of clusters. In: Proceedings of the ICML, pp. 727–734 (2000)Google Scholar
  14. 14.
    Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987)CrossRefGoogle Scholar
  15. 15.
    Ward Jr., J.H.: Hierarchical grouping to optimize an objective function. J. Am. Stat. Assoc. 58(301), 236–244 (1963)MathSciNetCrossRefGoogle Scholar
  16. 16.
    Yao, H., Hamilton, H.J., Butz, C.J.: FD\(\_\)Mine: discovering functional dependencies in a database using equivalences. In: Proceedings of the ICDM, pp. 729–732 (2002)Google Scholar
  17. 17.
    Yu, C., Jagadish, H.V.: XML schema refinement through redundancy detection and normalization. VLDB J. 17(2), 203–223 (2008)CrossRefGoogle Scholar
  18. 18.
    Yu, Y., Heflin, J.: Extending functional dependency to detect abnormal data in RDF graphs. In: Aroyo, L., et al. (eds.) ISWC 2011. LNCS, vol. 7031, pp. 794–809. Springer, Heidelberg (2011). Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.RWTH Aachen UniversityAachenGermany
  2. 2.Hochschule Niederrhein, University of Applied SciencesKrefeldGermany
  3. 3.Fraunhofer Institute for Applied Information Technology FITSankt AugustinGermany

Personalised recommendations