An Automated Anomaly Detection Procedure for Hourly Observed Precipitation in Near-Real Time Application

Abstract

For setting up an early warning system or for making a decision of disaster mitigation, real-time precipitation observation collected by in situ stations are necessary. However, the data easily become corrupted because of sensor failure or communication error. The simulations of a relevant hydrologic model under an extreme situation could be ridiculous due to incorrect precipitation observations as model inputs. Anomaly detection approaches are essential to flag anomalous data that not conform to expected behavior from normal data in near-real time. Manual inspection was used as an anomaly detection approach, but no longer applicable for big data and near-real time application due to manpower limitation. Therefore, this study proposes an automated anomaly detection procedure for precipitation including neighboring station selection and spatial consistency checking. First, the neighboring stations are not manually selected by a fixed distance, but automatically selected by the self-organizing maps (SOM) with recent historical records. Second, the target observation pairwise compares against concurrent neighboring observations to flag inconsistent data as anomalies. Because a suitable estimation of the target station is unnecessary in this procedure, it is applicable in areas with high spatial and temporal variability.