Statistics and Computing

, Volume 24, Issue 4, pp 597–613 | Cite as

Online signal extraction by robust regression in moving windows with data-adaptive width selection

SCARM—Slope Comparing Adaptive Repeated Median
Article

Abstract

Online (also ‘real-time’ or ‘sequential’) signal extraction from noisy and outlier-interfered data streams is a basic but challenging goal. Fitting a robust Repeated Median (Siegel in Biometrika 69:242–244, 1982) regression line in a moving time window has turned out to be a promising approach (Davies et al. in J. Stat. Plan. Inference 122:65–78, 2004; Gather et al. in Comput. Stat. 21:33–51, 2006; Schettlinger et al. in Biomed. Eng. 51:49–56, 2006). The level of the regression line at the rightmost window position, which equates to the current time point in an online application, is then used for signal extraction. However, the choice of the window width has a large impact on the signal extraction, and it is impossible to predetermine an optimal fixed window width for data streams which exhibit signal changes like level shifts and sudden trend changes. We therefore propose a robust test procedure for the online detection of such signal changes. An algorithm including the test allows for online window width adaption, meaning that the window width is chosen w.r.t. the current data situation at each time point. Comparison studies show that our new procedure outperforms an existing Repeated Median filter with automatic window width selection (Schettlinger et al. in Int. J. Adapt. Control Signal Process. 24:346–362, 2010).

Keywords

Repeated Median regression Data streams Signal change detection Real time 

References

  1. Bernholt, T., Fried, R.: Computing the update of the repeated median regression line in linear time. Inf. Process. Lett. 88(3), 111–117 (2003) CrossRefMATHMathSciNetGoogle Scholar
  2. Bernholt, T., Fried, R., Gather, U., Wegener, I.: Modified repeated median filters. Stat. Comput. 16, 177–192 (2006) CrossRefMathSciNetGoogle Scholar
  3. Borowski, M., Schettlinger, K., Gather, U.: Multivariate real time signal processing by a robust adaptive regression filter. Commun. Stat., Simul. Comput. 38(2), 426–440 (2009) CrossRefMATHMathSciNetGoogle Scholar
  4. Chen, C., Chan, J., Gerlach, R., Hsieh, W.: A comparison of estimators for regression models with change points. Stat. Comput. 21(3), 395–414 (2011) CrossRefMATHMathSciNetGoogle Scholar
  5. Davies, P., Gather, U.: Breakdown and groups. Ann. Stat. 33(3), 977–1035 (2005) CrossRefMATHMathSciNetGoogle Scholar
  6. Davies, P., Fried, R., Gather, U.: Robust signal extraction for on-line monitoring data. J. Stat. Plan. Inference 122, 65–78 (2004). Special Issue: Contemporary Data Analysis: Theory and Methods in Honor of John W. Tukey CrossRefMATHMathSciNetGoogle Scholar
  7. Fried, R.: Robust filtering of time series with trends. J. Nonparametr. Stat. 16(3–4), 313–328 (2004) CrossRefMATHMathSciNetGoogle Scholar
  8. Fried, R.: On the robust detection of edges in time series filtering. Comput. Stat. Data Anal. 52, 1063–1074 (2007) CrossRefMATHGoogle Scholar
  9. Fried, R., Bernholt, T., Gather, U.: Repeated median and hybrid filters. Comput. Stat. Data Anal. 50(9), 2313–2338 (2006) CrossRefMATHMathSciNetGoogle Scholar
  10. Fried, R., Einbeck, J., Gather, U.: Weighted repeated median smoothing and filtering. J. Am. Stat. Assoc. 102, 1300–1308 (2007) CrossRefMATHMathSciNetGoogle Scholar
  11. Fried, R., Schettlinger, K., Borowski, M.: Getting started with robfilter (2012a). http://www.statistik.tu-dortmund.de/1543.html
  12. Fried, R., Schettlinger, K., Borowski, M., robfilter: Robust time series filters (2012b). http://CRAN.R-project.org/package=robfilter, with manual
  13. Gather, U., Fried, R.: Methods and algorithms for robust filtering. In: Proceedings in Computational Statistics (COMPSTAT 2004), pp. 159–170. Physica, Heidelberg (2004) CrossRefGoogle Scholar
  14. Gather, U., Schettlinger, K., Fried, R.: Online signal extraction by robust linear regression. Comput. Stat. 21, 33–51 (2006) CrossRefMATHMathSciNetGoogle Scholar
  15. Gelper, S., Schettlinger, K., Croux, C., Gather, U.: Robust online scale estimation in time series: a model-free approach. J. Stat. Plan. Inference 139(2), 335–349 (2009) CrossRefMATHMathSciNetGoogle Scholar
  16. Rousseeuw, P., Hubert, M.: Regression-free and robust estimation of scale for bivariate data. Comput. Stat. Data Anal. 21(1), 67–85 (1996) CrossRefMATHMathSciNetGoogle Scholar
  17. Rousseeuw, P., Leroy, A.: Robust Regression and Outlier Detection. Wiley, New York (1987) CrossRefMATHGoogle Scholar
  18. Schettlinger, K.: Signal and variability extraction for online monitoring in intensive care. Ph.D. thesis, Faculty of Statistics, TU Dortmund University (2009) Google Scholar
  19. Schettlinger, K., Fried, R., Gather, U.: Robust filters for intensive care monitoring: beyond the running median. Biomed. Eng. 51, 49–56 (2006) CrossRefGoogle Scholar
  20. Schettlinger, K., Fried, R., Gather, U.: Real time signal processing by adaptive repeated median filters. Int. J. Adapt. Control Signal Process. 24, 346–362 (2010) MATHMathSciNetGoogle Scholar
  21. Siegel, A.: Robust regression using repeated medians. Biometrika 69, 242–244 (1982) CrossRefMATHGoogle Scholar
  22. Ylvisaker, D.: Test resistance. J. Am. Stat. Assoc. 72, 551–556 (1977) CrossRefMathSciNetGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  1. 1.Faculty of StatisticsTU Dortmund UniversityDortmundGermany

Personalised recommendations