Accreditation and Quality Assurance

, Volume 14, Issue 4, pp 185–192 | Cite as

Non-parametric estimation of reference intervals in small non-Gaussian sample sets

  • Johan Bjerner
  • Elvar Theodorsson
  • Eivind Hovig
  • Anders Kallner
General Paper


This study aimed at validating common bootstrap algorithms for reference interval calculation.We simulated 1500 random sets of 50–120 results originating from eight different statistical distributions. In total, 97.5 percentile reference limits were estimated from bootstrapping 5000 replicates, with confidence limits obtained by: (a) normal, (b) from standard error, (c) bootstrap percentile (as in RefVal) (d) BCa, (e) basic, or (f) student methods. Reference interval estimates obtained with ordinary bootstrapping and confidence intervals by percentile method were accurate for distributions close to normality and devoid of outliers, but not for log-normal distributions with outliers. Outlier removal and transformation to normality improved reference interval estimation, and the basic method was superior in such cases. In conclusions, if the neighborhood of the relevant percentile contains non-normally distributed results, bootstrapping fails. The distribution of bootstrap estimates should be plotted, and a non-normal distribution should warrant transformation or outlier removal.


Reference intervals Bootstrap Re-sampling Algorithm Non-parametric Percentile Confidence intervals Gaussian Distribution 



Thanks to the staff at Rikshospitalet-Radiumhospitalet, who provided us access to their computers for simulations during the Easter vacation!

Supplementary material

769_2009_490_MOESM1_ESM.pdf (541 kb)
Supplementary material 1 (PDF 541 kb)


  1. 1.
    Rustad P, Felding P, Lahti A (2004) Proposal for guidelines to establish common biological reference intervals in large geographical areas for biochemical quantities measured frequently in serum and plasma. Clin Chem Lab Med 42:783–791. doi: 10.1515/CCLM.2004.131 CrossRefGoogle Scholar
  2. 2.
    Poulsen OM, Holst E, Christensen JM (1997) Calculation, application of coverage intervals for biological reference values. A supplement to the approved IFCC recommendation (1987) on the theory of reference values. Pure Appl Chem 69:1601–1611. doi: 10.1351/pac199769071601 CrossRefGoogle Scholar
  3. 3.
    Solberg HE (1995) RefVal: a program implementing the recommendations of the International Federation of Clinical Chemistry on the statistical treatment of reference values. Comput Methods Prog Bio 48:247–256. doi: 10.1016/0169-2607(95)01697-X CrossRefGoogle Scholar
  4. 4.
    Linnet K (2008) CBstat: a program for statistical analysis in clinical biochemistry. (Accessed June 2008)
  5. 5.
    Solberg HE (2004) The IFCC recommendation on estimation of reference intervals. The RefVal program. Clin Chem Lab Med 42:710–714. doi: 10.1515/CCLM.2004.121 CrossRefGoogle Scholar
  6. 6.
    Linnet K (2000) Nonparametric estimation of reference intervals by simple and bootstrapbased procedures. Clin Chem 46:867–869Google Scholar
  7. 7.
    Henderson AR (2005) The bootstrap: a technique for data-driven statistics. Using computerintensive analyses to explore experimental data. Clin Chim Acta 359:1–26. doi: 10.1016/j.cccn.2005.04.002 CrossRefGoogle Scholar
  8. 8.
    Bjerner J, Høgetveit A, Akselberg K, et al (2008) Upper reference limits for carcinoembryonic antigen (CEA), CA125, MUC1, alfa-foeto-protein (AFP), neuron specific enolase (NSE) and CA19.9 from the NORIP study. Scand J Clin Lab Invest 68:703–713. doi: 10.1080/00365510802126836 Google Scholar
  9. 9.
    Bjerner J (2007) Age-dependent biochemical quantities: an approach for calculating reference intervals. Scand J Clin Lab Invest 67:707–722. doi: 10.1080/00365510701342070 CrossRefGoogle Scholar
  10. 10.
    Horn PS, Pesce AJ, Copeland BE (1998) A robust approach to reference interval estimation and evaluation. Clin Chem 44:622–631Google Scholar
  11. 11.
    R Development Core Team (2007). R: A language and environment for statistical computing. R Foundation for statistical computing, Vienna, Austria. http://www.rproject-org/ Accessed June 2008
  12. 12.
    Canty AJ (1998) An S-plus library for resampling methods. Computing Science and Statistics. In: Proceedings of the 30th symposium on the interface, pp 236–241Google Scholar
  13. 13.
    Carpenter J, Bithell J (1999) Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Stat Med 19:1141–1164. doi: 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO;2-F CrossRefGoogle Scholar
  14. 14.
    Efron B (1981) Nonparametric standard errors and confidence intervals (with Discussion). Can J Stat 9:139–172. doi: 10.2307/3314608 CrossRefGoogle Scholar
  15. 15.
    Hesterberg T, Monaghan S, Moore DS, Clipston A, Epstein R (2003) Bootstrap methods and permutation tests. The practice of business statistics, WH Freeman and Company, San FransiscoGoogle Scholar
  16. 16.
    Schwertman NC, Silva RD (2007) Identifying outliers with sequential fences. Comput Stat Data Anal 51:3800–3810. doi: 10.1016/j.csda.2006.01.019 CrossRefGoogle Scholar
  17. 17.
    Short T (2008) Rpad: interactive, web-based analysis package, webpage and GUI designer. and Accessed June 2008

Copyright information

© Springer-Verlag 2009

Authors and Affiliations

  • Johan Bjerner
    • 1
    • 2
  • Elvar Theodorsson
    • 3
  • Eivind Hovig
    • 4
  • Anders Kallner
    • 5
  1. 1.Department of Medical BiochemistryRikshospitalet Medical CenterOsloNorway
  2. 2.Dr. Fürst Medical LaboratoryOsloNorway
  3. 3.IKE/Clinical ChemistryLinköping University HospitalLinköpingSweden
  4. 4.Bioinformatics Core Facility, Institute of Cancer ResearchNorwegian Radium Hospital, Rikshospitalet University HospitalOsloNorway
  5. 5.Department of Clinical ChemistryKarolinska University HospitalStockholmSweden

Personalised recommendations