Skip to main content
Log in

Improved critical values for extreme normalized and studentized residuals in Gauss–Markov models

Journal of Geodesy Aims and scope Submit manuscript

Abstract

We investigate extreme studentized and normalized residuals as test statistics for outlier detection in the Gauss–Markov model possibly not of full rank. We show how critical values (quantile values) of such test statistics are derived from the probability distribution of a single studentized or normalized residual by dividing the level of error probability by the number of residuals. This derivation neglects dependencies between the residuals. We suggest improving this by a procedure based on the Monte Carlo method for the numerical computation of such critical values up to arbitrary precision. Results for free leveling networks reveal significant differences to the values used so far. We also show how to compute those critical values for non-normal error distributions. The results prove that the critical values are very sensitive to the type of error distribution.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Aliosmanoglu S, Akyilmaz O (2001) A comparison between statistical and fuzzy techniques in outlier detection. International association of geodesy symposia symposia, vol 125. Springer, Berlin, pp 382–387

  • Alkhatib H, Neumann I, Kutterer H (2009) Uncertainty modeling of random and systematic errors by means of Monte Carlo and fuzzy techniques. J Appl Geod 3(2): 67–79

    Google Scholar 

  • Andersen R (2008) Modern methods for robust regression. Sage Publications, London

    Google Scholar 

  • Baarda W (1968) A testing procedure for use in geodetic networks. Netherlands Geodetic Commission, Publication on Geodesy, vol 2, No. 5, Delft, The Netherlands

  • Box GEP, Muller ME (1958) A note on the generation of random normal deviates. Ann. Math. Stat. 29(2): 610–611

    Article  Google Scholar 

  • Hawkins D (1980) Identification of outliers. Chapman and Hall, London

    Google Scholar 

  • Koch KR (1999) Parameter estimation and hypothesis testing in linear models. Springer, Berlin

    Google Scholar 

  • Koch KR (2007) Outlier detection in observations including leverage points by Monte Carlo simulations. Allgemeine Vermessungsnachrichten. VDE Verlag Berlin Offenbach

  • Lemeshko BY, Lemeshko SB (2005) Extending the application of Grubbs-type tests in rejecting anomalous measurements. Meas Tech 48(6): 536–547

    Google Scholar 

  • Lehmann R (1994) Adjustment in non-linear models by means of the adaptive Monte-Carlo-Integration. Allgemeine Vermessungsnachrichten, vol 7/1994. Herbert Wichmann Verlag GmbH Heidelberg (in German)

  • Lehmann R (2010) Normalized residuals—how large is too large? Allgemeine Vermessungsnachrichten, vol 2/2010. VDE Verlag Berlin Offenbach (in German)

  • Lehmann R (2012) Geodetic error calculus by the scale contaminated normal distribution. Allgemeine Vermessungsnachrichten. VDE Verlag Berlin Offenbach (in German) (in press)

  • Lehmann R, Scheffler T (2011) Monte Carlo based data snooping with application to a geodetic network. J Appl Geod 5(3–4): 123–134

    Google Scholar 

  • Pope AJ (1976) The statistics of residuals and the detection of outliers. NOAA Technical Report NOS65 NGS1, US Department of Commerce, National Geodetic Survey Rockville, Maryland

  • Stefansky W (1972) Rejecting outliers in factorial designs. Technometrics 14: 469–479

    Article  Google Scholar 

  • Tanizaki H (2004) Computational methods in statistics and econometrics. Marcel Dekker, New York

    Book  Google Scholar 

  • Verma SP, Quiroz-Ruiz A, Díaz-González L (2008) Critical values for 33 discordancy test variants for outliers in normal samples up to sizes 1000, and applications in quality control in Earth Sciences. Revista Mexicana de Ciencias Geológicas 25(1): 82–96

    Google Scholar 

  • Wieser A (2001) Robust and fuzzy techniques for parameter estimation and quality assessment in GPS. Ph.D. dissertation, Graz University of Technology, Austria

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Rüdiger Lehmann.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lehmann, R. Improved critical values for extreme normalized and studentized residuals in Gauss–Markov models. J Geod 86, 1137–1146 (2012). https://doi.org/10.1007/s00190-012-0569-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00190-012-0569-0

Keywords

Navigation