Perception & Psychophysics

, Volume 63, Issue 8, pp 1314–1329 | Cite as

The psychometric function: II. Bootstrap-based confidence intervals and sampling

  • Felix A. Wichmann
  • N. Jeremy Hill


The psychometric function relates an observer’s performance to an independent variable, usually a physical quantity of an experimental stimulus. Even if a model is successfully fit to the data and its goodness of fit is acceptable, experimenters require an estimate of the variability of the parameters to assess whether differences across conditions are significant. Accurate estimates of variability are difficult to obtain, however, given the typically small size of psychophysical data sets: Traditional statistical techniques are only asymptotically correct and can be shown to be unreliable in some common situations. Here and in our companion paper (Wichmann & Hill, 2001), we suggest alternative statistical techniques based on Monte Carlo resampling methods. The present paper’s principal topic is the estimation of the variability of fitted parameters and derived quantities, such as thresholds and slopes. First, we outline the basic bootstrap procedure and argue in favor of the parametric, as opposed to the nonparametric, bootstrap. Second, we describe how the bootstrap bridging assumption, on which the validity of the procedure depends, can be tested. Third, we show how one’s choice of sampling scheme (the placement of sample points on the stimulus axis) strongly affects the reliability of bootstrap confidence intervals, and we make recommendations on how to sample the psychometric function efficiently. Fourth, we show that, under certain circumstances, the (arbitrary) choice of the distribution function can exert an unwanted influence on the size of the bootstrap confidence intervals obtained, and we make recommendations on how to avoid this influence. Finally, we introduce improved confidence intervals (bias corrected and accelerated) that improve on the parametric and percentile-based bootstrap confidence intervals previously used. Software implementing our methods is available.


Sampling Scheme Bootstrap Method Psychometric Function Bootstrap Estimate Bootstrap Confidence Interval 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. Cox, D. R., & Hinkley, D. V. (1974).Theoretical statistics. London: Chapman and Hall.Google Scholar
  2. Davison, A. C., & Hinkley, D. V. (1997).Bootstrap methods and their application. Cambridge: Cambridge University Press.Google Scholar
  3. Efron, B. (1979). Bootstrap methods: Another look at the jackknife.Annals of Statistics,7, 1–26.CrossRefGoogle Scholar
  4. Efron, B. (1982).The jackknife, the bootstrap and other resampling plans (CBMS-NSF Regional Conference Series in Applied Mathematics). Philadelphia: Society for Industrial and Applied Mathematics.Google Scholar
  5. Efron, B. (1987). Better bootstrap confidence intervals.Journal of the American Statistical Association,82, 171–200.CrossRefGoogle Scholar
  6. Efron, B. (1988). Bootstrap confidence intervals: Good or bad?Psychological Bulletin,104, 293–296.CrossRefGoogle Scholar
  7. Efron, B., & Gong, G. (1983). A leisurely look at the bootstrap, the jackknife, and cross-validation.American Statistician,37, 36–48.CrossRefGoogle Scholar
  8. Efron, B., & Tibshirani, R. (1991). Statistical data analysis in the computer age.Science,253, 390–395.PubMedCrossRefGoogle Scholar
  9. Efron, B., & Tibshirani, R. J. (1993).An introduction to the bootstrap. New York: Chapman and Hall.Google Scholar
  10. Finney, D. J. (1952).Probit analysis (2nd ed.). Cambridge: Cambridge University Press.Google Scholar
  11. Finney, D. J. (1971).Probit analysis. (3rd ed.). Cambridge: Cambridge University Press.Google Scholar
  12. Foster, D. H., & Bischof, W. F. (1987). Bootstrap variance estimators for the parameters of small-sample sensory-performance functions.Biological Cybernetics,57, 341–347.PubMedCrossRefGoogle Scholar
  13. Foster, D. H., & Bischof, W. F. (1991). Thresholds from psychometric functions: Superiority of bootstrap to incremental and probit variance estimators.Psychological Bulletin,109, 152–159.CrossRefGoogle Scholar
  14. Foster, D. H., & Bischof, W. F. (1997). Bootstrap estimates of the statistical accuracy of thresholds obtained from psychometric functions.Spatial Vision,11, 135–139.PubMedCrossRefGoogle Scholar
  15. Hall, P. (1988). Theoretical comparison of bootstrap confidence intervals (with discussion).Annals of Statistics,16, 927–953.CrossRefGoogle Scholar
  16. Hill, N. J. (2001a, May).An investigation of bootstrap interval coverage and sampling efficiency in psychometric functions. Poster presented at the Annual Meeting of the Vision Sciences Society, Sarasota, FL.Google Scholar
  17. Hill, N. J. (2001b).Testing hypotheses about psychometric functions: An investigation of some confidence interval methods, their validity, and their use in assessing optimal sampling strategies. Forthcoming doctoral dissertation, Oxford University.Google Scholar
  18. Hill, N. J., & Wichmann, F. A. (1998, April).A bootstrap method for testing hypotheses concerning psychometric functions. Paper presented at the Computers in Psychology, York, U.K.Google Scholar
  19. Hinkley, D. V. (1988). Bootstrap methods.Journal of the Royal Statistical Society B,50, 321–337.Google Scholar
  20. Kendall, M. K., & Stuart, A. (1979).The advanced theory of statistics: Vol. 2. Inference and relationship. New York: Macmillan.Google Scholar
  21. Lam, C. F., Mills, J. H., & Dubno, J. R. (1996). Placement of observations for the efficient estimation of a psychometric function.Journal of the Acoustical Society of America,99, 3689–3693.PubMedCrossRefGoogle Scholar
  22. Maloney, L. T. (1990). Confidence intervals for the parameters of psychometric functions.Perception & Psychophysics,47, 127–134.Google Scholar
  23. McKee, S. P., Klein, S. A., & Teller, D. Y. (1985). Statistical properties of forced-choice psychometric functions: Implications of probit analysis.Perception & Psychophysics,37, 286–298.Google Scholar
  24. Rasmussen, J. L. (1987). Estimating correlation coefficients: Bootstrap and parametric approaches.Psychological Bulletin,101, 136–139.CrossRefGoogle Scholar
  25. Rasmussen, J. L. (1988). Bootstrap confidence intervals: Good or bad. Comments on Efron (1988) and Strube (1988) and further evaluation.Psychological Bulletin,104, 297–299.CrossRefGoogle Scholar
  26. Strube, M. J. (1988). Bootstrap type I error rates for the correlation coefficient: An examination of alternate procedures.Psychological Bulletin,104, 290–292.CrossRefGoogle Scholar
  27. Treutwein, B. (1995, August).Error estimates for the parameters of psychometric functions from a single session. Poster presented at the European Conference of Visual Perception, Tübingen.Google Scholar
  28. Treutwein, B., & Strasburger, H. (1999, September).Assessing the variability of psychometric functions. Paper presented at the European Mathematical Psychology Meeting, Mannheim.Google Scholar
  29. Wichmann, F. A., & Hill, N. J. (2001). The psychometric function: I. Fitting, sampling, and goodness of fit.Perception & Psychophysics,63, 1293–1313.CrossRefGoogle Scholar

Copyright information

© Psychonomic Society, Inc. 2001

Authors and Affiliations

  1. 1.University of OxfordOxfordEngland

Personalised recommendations