Advertisement

Knowledge and Information Systems

, Volume 44, Issue 2, pp 253–277 | Cite as

A masking index for quantifying hidden glitches

  • Laure Berti-ÉquilleEmail author
  • Ji Meng Loh
  • Tamraparni Dasu
Regular Paper

Abstract

Data glitches are errors in a dataset. They are complex entities that often span multiple attributes and records. When they co-occur in data, the presence of one type of glitch can hinder the detection of another type of glitch. This phenomenon is called masking. In this paper, we define two important types of masking and propose a novel, statistically rigorous indicator called masking index for quantifying the hidden glitches. We outline four cases of masking: outliers masked by missing values, outliers masked by duplicates, duplicates masked by missing values, and duplicates masked by outliers. The masking index is critical for data quality profiling and data exploration. It enables a user to measure the extent of masking and hence the confidence in the data. In this sense, it is a valuable data quality index for choosing an anomaly detection method that is best suited for the glitches that are present in any given dataset. We demonstrate the utility and effectiveness of the masking index by intensive experiments on synthetic and real-world datasets.

Keywords

Anomaly detection Masking Duplicate record identification  Missing values Outlier detection 

References

  1. 1.
    Acuna E, Rodriguez CA (2004) Meta analysis study of outlier detection methods in classification, IPSIGoogle Scholar
  2. 2.
    Barnett V, Lewis T (1994) Outliers in statistical data. Wiley, New YorkGoogle Scholar
  3. 3.
    Ben-Gal I (2005) Outlier detection. In: Maimon O, Rockach L (eds) Data mining and knowledge discovery handbook: a complete guide for practitioners and researchers. Kluwer, DordrechtGoogle Scholar
  4. 4.
    Berti-Equille L, Dasu T, Srivastava D (2011) Discovery of complex glitch patterns: a novel approach to quantitative data cleaning, ICDE, pp 733–744Google Scholar
  5. 5.
    Blake R, Mangiameli P (2011) The effects and interactions of data quality and problem complexity on classification. J Data Inf Qual 2(2):8:1–8:28Google Scholar
  6. 6.
    Dasu T, Loh JM (2012) Statistical distortion: consequences of data cleaning. PVLDB 5(11):1674–1683Google Scholar
  7. 7.
    Davies L, Gather U (1993) The identification of multiple outliers. J Am Stat Assoc 88(423):782–792CrossRefMathSciNetGoogle Scholar
  8. 8.
    Efron B (1979) Bootstrap methods: another look at the jackknife. Ann Stat 7:1–26CrossRefMathSciNetGoogle Scholar
  9. 9.
    Hawkins D (1980) Identification of outliers. Chapman and Hall, LondonCrossRefzbMATHGoogle Scholar
  10. 10.
    Iglewics B, Martinez J (1982) Outlier detection using robust measures of scale. J Stat Comput Simul 15:285–293CrossRefGoogle Scholar
  11. 11.
    Kushmerick N (1999) Learning to remove internet advertisements. In: Proceedings of the third annual conference on autonomous agents, AGENTS ’99, pp 175–181Google Scholar
  12. 12.
    Rao CR (1973) Linear statistical inference and its applications. Wiley, New YorkCrossRefzbMATHGoogle Scholar
  13. 13.
    Xiong H, Pandey G, Steinbach M, Kumar V (2006) Enhancing data analysis with noise removal. IEEE Trans Knowl Data Eng 18(2):304–319CrossRefGoogle Scholar

Copyright information

© Springer-Verlag London 2014

Authors and Affiliations

  • Laure Berti-Équille
    • 1
    • 2
    Email author
  • Ji Meng Loh
    • 3
  • Tamraparni Dasu
    • 4
  1. 1.IRD ESPACE DEVMontpellierFrance
  2. 2.Qatar Computing Research InstituteWest Bay, DohaQatar
  3. 3.Department of Mathematical SciencesNew Jersey Institute of TechnologyNewarkUSA
  4. 4.AT&T Labs-ResearchBedminsterUSA

Personalised recommendations