Detecting and correcting sensor drifts in long-term weather data
- 323 Downloads
Quality control of long-term monitoring data of thousands and millions of individual records as present in meteorological data is cumbersome. In such data series, sensor drifts, stalled values, and scale shifts may occur and potentially result in flawed conclusions if not noticed and handled properly. However, there is no established standard procedure to perform quality control of high-frequency meteorological data. In this paper, we outline a procedure to remove sensor drift in high-frequency data series using the example of 15-year-long sets of hourly relative humidity (RH) data from 28 stations subdivided into 202 individual sensor operation periods. The procedure involves basic quality control, relative homogeneity testing, and drift removal. Significant sensor drifts were observed in 40.6 % of all sensor operation periods. The drifts varied between data series and depended in a complex, usually inconsistent way on absolute RH values; within single series for instance, a drift could be negative in the lower RH range and positive in the upper RH range. Detrending changed RH values by, on average, 1.96 %. For one fifth of the detrended data, adjustments were 2.75 % and more of the measured value, and in one tenth 4.75 % and more. Overall, drifts were strongest for RH values close to 100 %. The detrending procedure proved to effectively remove sensor drifts. The principles of the procedure also apply to other meteorological parameters and more generally to any time series of data for which comparable reference data are available.
KeywordsDetrending Homogeneity testing Quality control Relative humidity Sensor drift Swiss Long-Term Forest Ecosystem Research Programme LWF
We thank Gustav Schneiter for meteorological data collection and sharing his expertise of local station situation and Peter Jakob for help with the data base. Michael Begert is appreciated for his valuable advice and input to quality control, data adjustment, and the text. An anonymous reviewer made valuable suggestions to a previous version of this paper. This project was supported by a grant within the program “Forest and Climate Change” from the Swiss Federal Office for the Environment FOEN and Swiss Federal Research Institute for Forest, Snow and Landscape Research WSL.
- Aguilar, E., Auer, I., Brunet, M., Peterson, T. C., & Wieringa, J. (2003). Guidelines on climate metadata and homogenization. WMO-TD 53. WMO-TD no. 1186, pp. 51Google Scholar
- Conrad, V., & Pollack, L. W. (1950). Methods in climatology (2nd ed.). Cambridge: Harvard University Press.Google Scholar
- Jakob, P., Sutter, F., Waldner, P., & Schneiter, G. (2007). Designing a flexible ict framework for processing remote gauging-data information. In J. M. Gómez, M. Sonnenschein, M. Müller, H. Welsch, & C. Rautenstrauch (Eds.), Technologies in environmental engineering (pp. 211–220). Berlin: Springer. Environmental Science and Engineering.CrossRefGoogle Scholar
- Kramer, J. P., & Boyer, J. S. (1995). Water relations of plants and soils (2nd ed.). San Diego: Academic.Google Scholar
- von Arx, G., Dobbertin, M., & Rebetez, M. (2012). Spatio-temporal effects of forest canopy on understory microclimate in a long-term experiment in Switzerland. Agricultural and Forest Meteorology. doi: 10.1016/j.agrformet.2012.07.018.