Statistical Quality Control of Warehouse Data

Abstract

The increasing popularity of data warehouse systems reflects the rising requirement to make strategic use of data integrated from heterogeneous sources. While the research subject of schema integration has been extensively discussed for many years, data integration has been neglected up to the recent past. Data integration often reveals deficiencies of data quality, e. g. inconsistency, redundancy, and incompleteness. Up to now, there are hardly any mellow methods for data quality control. In this paper, we propose an adaptation of statistical process control (SPC), a technique well-established in manufacturing for several decades, to the data quality field. After reviewing basic concepts of SPC, we introduce an appropriate SPC-oriented algorithm for data quality control. By means of several scenarios, we demonstrate the applicability of our approach. Finally, we integrate our concepts into a system architecture for data quality management.