A New Tool for Automated Quality Control of Environmental Time Series (AutoQC4Env) in Open Web Services
- 141 Downloads
We report on the development of a new software tool (AutoQC4Env) for automated quality control (QC) of environmental time series data. Novel features of this tool include a flexible Python software architecture, which makes it easy for users to configure the sequence of tests as well as their statistical parameters, and a statistical concept to assign each value a probability of being a valid data point. There are many occasions when it is necessary to inspect the quality of environmental data sets, from first quality checks during real-time sampling and data transmission to assessing the quality and consistency of long-term monitoring data from measurement stations. Erroneous data can have a substantial impact on the statistical data analysis and, for example, lead to wrong estimates of trends. Existing QC workflows largely rely on individual investigator knowledge and have been constructed from practical considerations and with a least theoretical foundation. The statistical framework that is being developed in AutoQC4Env aims to complement traditional data quality assessments and provide environmental researchers with a tool that is easy to use but also based on current statistical knowledge.
KeywordsAutoQC4Env tool Quality control Environmental time series
This work has been performed and funded as part of the IntelliAQ project under ERC-2017-ADG#787576 grant at the Jülich Super Computing Centre, Forschungszentrum Jülich. The TOAR community and various national environmental agencies are gratefully acknowledged for providing data and collaborating on the development of the TOAR database. Sabine Schröder and Lukas Leufen helped with the data analysis and software infrastructure.
- 2.IOOS-US: U.S. Integrated Ocean Observing System: a blueprint for full capability. Version 1.0. Technical report (2010)Google Scholar
- 4.Schultz, Martin G., et.al.: Tropospheric Ozone Assessment Report: database and metrics data of global surface ozone observations. Elem. Sci. Anth. 5, 58 (2017). https://doi.org/10.1525/elementa.244