Abstract
Quality control of long-term monitoring data of thousands and millions of individual records as present in meteorological data is cumbersome. In such data series, sensor drifts, stalled values, and scale shifts may occur and potentially result in flawed conclusions if not noticed and handled properly. However, there is no established standard procedure to perform quality control of high-frequency meteorological data. In this paper, we outline a procedure to remove sensor drift in high-frequency data series using the example of 15-year-long sets of hourly relative humidity (RH) data from 28 stations subdivided into 202 individual sensor operation periods. The procedure involves basic quality control, relative homogeneity testing, and drift removal. Significant sensor drifts were observed in 40.6 % of all sensor operation periods. The drifts varied between data series and depended in a complex, usually inconsistent way on absolute RH values; within single series for instance, a drift could be negative in the lower RH range and positive in the upper RH range. Detrending changed RH values by, on average, 1.96 %. For one fifth of the detrended data, adjustments were 2.75 % and more of the measured value, and in one tenth 4.75 % and more. Overall, drifts were strongest for RH values close to 100 %. The detrending procedure proved to effectively remove sensor drifts. The principles of the procedure also apply to other meteorological parameters and more generally to any time series of data for which comparable reference data are available.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Aguilar, E., Auer, I., Brunet, M., Peterson, T. C., & Wieringa, J. (2003). Guidelines on climate metadata and homogenization. WMO-TD 53. WMO-TD no. 1186, pp. 51
Begert, M., Schlegel, T., & Kirchhofer, W. (2005). Homogeneous temperature and precipitation series of Switzerland from 1864 to 2000. International Journal of Climatology, 25(1), 65–80. doi:10.1002/joc.1118.
Begert, M., Zenklusen, E., Häberli, C., Appenzeller, C., & Klok, L. (2008). An automated procedure to detect discontinuities; performance assessment and application to a large European climate data set. Meteorologische Zeitschrift, 17(5), 663–672.
Brandsma, T., & Konnen, G. P. (2006). Application of nearest-neighbor resampling for homogenizing temperature records on a daily to sub-daily level. International Journal of Climatology, 26(1), 75–89. doi:10.1002/joc.1236.
Conrad, V., & Pollack, L. W. (1950). Methods in climatology (2nd ed.). Cambridge: Harvard University Press.
Costa, A. C., & Soares, A. (2009). Homogenization of climate data: review and new perspectives using geostatistics. Mathematical Geosciences, 41(3), 291–305. doi:10.1007/s11004-008-9203-3.
Della-Marta, P. M., & Wanner, H. (2006). A method of homogenizing the extremes and mean of daily temperature measurements. Journal of Climate, 19(17), 4179–4197. doi:10.1175/jcli3855.1.
Innes, J. L. (1995). Theoretical and practical criteria for the selection of ecosystem monitoring plots in Swiss forests. Environmental Monitoring and Assessment, 36(3), 271–294.
Jakob, P., Sutter, F., Waldner, P., & Schneiter, G. (2007). Designing a flexible ict framework for processing remote gauging-data information. In J. M. Gómez, M. Sonnenschein, M. Müller, H. Welsch, & C. Rautenstrauch (Eds.), Technologies in environmental engineering (pp. 211–220). Berlin: Springer. Environmental Science and Engineering.
Kramer, J. P., & Boyer, J. S. (1995). Water relations of plants and soils (2nd ed.). San Diego: Academic.
Peterson, T. C., & Easterling, D. R. (1994). Creation of homogeneous composite climatological reference series. International Journal of Climatology, 14(6), 671–679. doi:10.1002/joc.3370140606.
Trewin, B. C., & Trevitt, A. C. F. (1996). The development of composite temperature records. International Journal of Climatology, 16(11), 1227–1242. doi:10.1002/(sici)1097-0088(199611)16:11<1227::aid-joc82>3.0.co;2-p.
von Arx, G., Dobbertin, M., & Rebetez, M. (2012). Spatio-temporal effects of forest canopy on understory microclimate in a long-term experiment in Switzerland. Agricultural and Forest Meteorology. doi:10.1016/j.agrformet.2012.07.018.
Acknowledgments
We thank Gustav Schneiter for meteorological data collection and sharing his expertise of local station situation and Peter Jakob for help with the data base. Michael Begert is appreciated for his valuable advice and input to quality control, data adjustment, and the text. An anonymous reviewer made valuable suggestions to a previous version of this paper. This project was supported by a grant within the program “Forest and Climate Change” from the Swiss Federal Office for the Environment FOEN and Swiss Federal Research Institute for Forest, Snow and Landscape Research WSL.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
von Arx, G., Dobbertin, M. & Rebetez, M. Detecting and correcting sensor drifts in long-term weather data. Environ Monit Assess 185, 4483–4489 (2013). https://doi.org/10.1007/s10661-012-2831-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10661-012-2831-6