Advertisement

A New Tool for Automated Quality Control of Environmental Time Series (AutoQC4Env) in Open Web Services

  • Najmeh KaffashzadehEmail author
  • Felix Kleinert
  • Martin G. Schultz
Conference paper
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 373)

Abstract

We report on the development of a new software tool (AutoQC4Env) for automated quality control (QC) of environmental time series data. Novel features of this tool include a flexible Python software architecture, which makes it easy for users to configure the sequence of tests as well as their statistical parameters, and a statistical concept to assign each value a probability of being a valid data point. There are many occasions when it is necessary to inspect the quality of environmental data sets, from first quality checks during real-time sampling and data transmission to assessing the quality and consistency of long-term monitoring data from measurement stations. Erroneous data can have a substantial impact on the statistical data analysis and, for example, lead to wrong estimates of trends. Existing QC workflows largely rely on individual investigator knowledge and have been constructed from practical considerations and with a least theoretical foundation. The statistical framework that is being developed in AutoQC4Env aims to complement traditional data quality assessments and provide environmental researchers with a tool that is easy to use but also based on current statistical knowledge.

Keywords

AutoQC4Env tool Quality control Environmental time series 

Notes

Acknowledgements

This work has been performed and funded as part of the IntelliAQ project under ERC-2017-ADG#787576 grant at the Jülich Super Computing Centre, Forschungszentrum Jülich. The TOAR community and various national environmental agencies are gratefully acknowledged for providing data and collaborating on the development of the TOAR database. Sabine Schröder and Lukas Leufen helped with the data analysis and software infrastructure.

References

  1. 1.
    Durre, I., Menne, M.J., Vose, R.S.: Strategies for evaluating quality assurance procedures. J. Appl. Meteorol. Climatol. 47(6), 1785–1791 (2008).  https://doi.org/10.1175/2007JAMC1706.1CrossRefGoogle Scholar
  2. 2.
    IOOS-US: U.S. Integrated Ocean Observing System: a blueprint for full capability. Version 1.0. Technical report (2010)Google Scholar
  3. 3.
    Key, R.M., Schirnick, C., Velo, A., Tanhua, T., van Heuven, S., Olsen, A.: Quality control procedures and methods of the CARINA database. Earth. Syst. Sci. Data. 2(1), 35–49 (2010).  https://doi.org/10.5194/essd-2-35-2010CrossRefGoogle Scholar
  4. 4.
    Schultz, Martin G., et.al.: Tropospheric Ozone Assessment Report: database and metrics data of global surface ozone observations. Elem. Sci. Anth. 5, 58 (2017).  https://doi.org/10.1525/elementa.244
  5. 5.
    Taylor, J.R., Loescher, H.L.: Automated quality control methods for sensor data: a novel observatory approach. Biogeosciences 10(7), 4957–4971 (2013).  https://doi.org/10.5194/bg-10-4957-2013CrossRefGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • Najmeh Kaffashzadeh
    • 1
    Email author
  • Felix Kleinert
    • 1
  • Martin G. Schultz
    • 1
  1. 1.Jülich Supercomputing Centre, Forschungszentrum Jülich GmbHJülichGermany

Personalised recommendations