Skip to main content
SpringerLink
Log in

A new outlier detection method considering outliers as model errors

  • Technical Article
  • Published:
Experimental Techniques Aims and scope Submit manuscript

Abstract

Outlier detection is an important task for fitting a model to a set of data. Two different outlier detection approaches are given as tests for outliers and robust methods. For these approaches, usually outliers are considered as additive bias terms neglected in the original adjustment model. However, there is another approach that outlier is considered as a model error in the Gauss–Markov model. This model error is represented as an unknown parameter. As it cannot be known before which observation includes outlier; this method is applied on the data for each observation separately and tested with t-test or F-test. It is successful if the sample includes only one outlier. To detect multiple outliers more successfully, in this article, a new outlier detection method is introduced. In this method, all the possible combinations of multiple outliers are considered as model errors and it is accepted that the smallest variance of them gives the solution for a certain number of outliers, then the estimated model errors are detected by comparing with a critical value. The critical value is chosen as 3σo. To compare the results of the new method, with those of the Least Median of Squares (LMS) and Huber M-estimators, Monte Carlo simulation technique is used for linear regression. The Mean Success Rate is proposed to measure the reliabilities of the methods. We showed that the new method is robust and includes the property of high breakdown point as LMS; and more efficient than LMS.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Baarda, W., A Testing Procedure for Use in Geodetic Networks, Publication on Geodesy, New series 2, no. 5., Netherlands Geodetic Commission, Delft, (1968).

  2. Pope, A.J., The Statistics of Residuals and the Outlier Detection of Outliers, NOAA Technical Report, NOS 65, NGS 1, Rockville, MD (1976).

  3. Heck, B., “Der Einfluss einzelner Beobachtungen auf das Ergebnis einer und die Suche nach Ausreissern in den Beobachtungen,” AVN 88: 17–34 (1981).

    Google Scholar 

  4. Pelzer, H., Geodatische Netze in Landes–und İngenieurvermessung II- Oberprüfung von Ausgleichungsmodellen Konrad Wittwer Verlag-Stuttgart (1985).

  5. Koch, K.R., Parameter Estimation and Hypothesis Testing in Linear Models, 2nd Edition, Springer-Verlag, Berlin (1999).

    Book  Google Scholar 

  6. Baselga, S., “Critical Limitation in Use of τ Test for Gross Error Detection,” Journal of Surveying Engineering 133(2): 52–55 (2007).

    Article  Google Scholar 

  7. Huber, P.J., “Robust Estimation of a Location Parameter,” The Annals of Mathematical Statistics 35: 73–101 (1964).

    Article  Google Scholar 

  8. Huber, P.J., Robust Statistics, John Wiley and Sons, New York (1981).

    Book  Google Scholar 

  9. Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., and Stahel, W.A., Robust Statistics: The Approach Based on Influence Functions, John Wiley and Sons, New York (1986).

    Google Scholar 

  10. Rousseeuw, P.J., and Leroy, A.M., Robust Regression and Outlier Detection, John Wiley and Sons, New York (1987).

    Book  Google Scholar 

  11. Koch, K.R., “Robuste Parameterschaetzung,” AVN 103: 1–18 (1996).

    Google Scholar 

  12. Yang, Y., He, H., and Xu, G., “Adaptively Robust Filtering for Kinematic Geodetic Positioning,” Journal of Geodesy 75(2/3): 109–116 (2001).

    Article  Google Scholar 

  13. Yang, Y., Song, L., and Xu, T., “Robust Estimator for Correlated Observations Based on Bifactor Equivalent Weights,” Journal of Geodesy 76(6–7): 353–358 (2002).

    Article  Google Scholar 

  14. Hekimoglu, S., “Do Robust Methods Identify Outliers More Reliably than Conventional Tests for Outliers?” ZFV 5(3): 174–180 (2005).

    Google Scholar 

  15. Xu, P.L., “Sign-constrained Robust Least Squares, Subjective Breakdown Point and the Effect of Weights of Observations on Robustness,” Journal of Geodesy 79: 146–159 (2005).

    Article  Google Scholar 

  16. Koch, K.R., “Outlier Detection in Observations Including Leverage Points by Monte Carlo Simulations,” AVN 114(10): 330–336 (2007).

    Google Scholar 

  17. Erenoglu, R.C., and Hekimoglu, S., “Efficiency of robust methods and tests for outliers for geodetic adjustment models,” Acta. Geodaetica et Geophysica Hungarica 45(4): 426–439 (2010).

    Article  Google Scholar 

  18. Gui, Q., Gong, Y., Li, G., and Li, B. “A Bayesian Approach to the Detection of Gross Errors Based on Posterior Probability,” Journal of Geodesy 81(10): 651–659 (2007).

    Article  Google Scholar 

  19. Gui, Q., Li, X., Gong, Y., Li, B., and Li, G., “A Bayesian Unmasking Method for Locating Multiple Gross Errors Based on Posterior Probabilities of Classification Variables,” Journal of Geodesy 85(4): 191–203 (2011).

    Article  Google Scholar 

  20. Sun, W., “A New Method for Localization of Gross Errors,” Survey Review 32: 344–358 (1994).

    Article  Google Scholar 

  21. Cen, M., Li, Z., Ding, X., and Zhuo, J., “Gross Error Diagnostics Before Least Squares Adjustment of Observations,” Journal of Geodesy 77(1): 503–513 (2003).

    Article  Google Scholar 

  22. Xia, X.T.,Wang, Z.Y., and Gao, Y.S., “Gross Error Detection Using Fuzzy-Set Theory,” Proceedings of the 3rd International Symposium Instrumentation Science and Technology, Vol. 1, Xian, China, 120–127, 2004.

  23. Schaffrin, B, and Zewen, W., “Multiplicative Outlier Search Using HomBLUP and An Equivalence Theorem,” Manuscripta Geodaetica 20(1): 21–26 (1994).

    Google Scholar 

  24. Cook, R.D., and Weisberg, S., Residuals and Influence in Regression, Chapman and Hall, New York (1982).

    Google Scholar 

  25. Schall, R., and Dunne, T.T., “A Unified Approach to Outliers in the General Linear Model,” Sankhya, The Indian Journal of Statistics B15: 157–167 (1988).

    Google Scholar 

  26. Hekimoglu, S., and Koch, K.R., “How Can Reliability of the Robust Methods be Measured?”, Proc., Altan, M.O., and Gründig, L. (eds), Third Turkish-German Joint Geodetic Days, Istanbul, 1, 179–196 (1999).

  27. Chatterjee, S., and Hadi, A.S., Sensitivity Analysis in Linear Regression, John Wiley and Sons, New York (1988).

    Book  Google Scholar 

  28. Hadi, A.S., and Simonoff, J.S., “Procedures for the Identification of Multiple Outliers in Linear Models,” Journal of the American Statistical Association 88: 1264–1272 (1993).

    Article  Google Scholar 

  29. Rousseeuw, P.J., “Least Median of Squares Regression,” Journal of the American Statistical Association 79: 871–880 (1984).

    Article  Google Scholar 

  30. Hampel, F.R., “Beyond Location Parameters: Robust Concepts and Methods,” Bulletin of the International Statistical Institute 46(1): 375–382 (1975).

    Google Scholar 

  31. Hekimoglu, S., and Erenoglu, R.C., “A New Robust Method in Linear Regression,” Journal of Surveying Engineering 135(3): 1–5 (2009).

    Article  Google Scholar 

  32. Hekimoglu, S., and Koch, K.R., “How Can Reliability of Tests for Outliers be Measured?” AVN 107(2): 247–254 (2000).

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Geomatic Engineering, Yildiz Technical University, Istanbul, Turkey

    S. Hekimoglu & B. Erdogan

  2. Department of Geomatics Engineering, Canakkale Onsekiz Mart University, Canakkale, Turkey

    R. C. Erenoglu

Authors
  1. S. Hekimoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. B. Erdogan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R. C. Erenoglu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S. Hekimoglu.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hekimoglu, S., Erdogan, B. & Erenoglu, R.C. A new outlier detection method considering outliers as model errors. Exp Tech 39, 57–68 (2015). https://doi.org/10.1111/j.1747-1567.2012.00876.x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1111/j.1747-1567.2012.00876.x

Keywords

Search

Navigation