Skip to main content
SpringerLink
Log in

On the number of significant findings to be expected by chance

  • Published:
Psychometrika Aims and scope Submit manuscript

Abstract

When multiple significance tests are computed, a certain number of “significant” findings will emerge simply because of chance fluctuations. In the present paper, some factors affecting the number of nominally significant results are elaborated and a general method is suggested which permits unbiased inference as to the significance of a set of findings,as a set. The method advocated employs a high speed computer to generate empirically a sampling distribution tailormade to a particular data matrix. The method is illustrated in the case of dichotomous response to inventory items, where it is found that the statistical model still often used as a basis for estimation is overly conservative. Some problems in the application of the method are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Arthur, A. O. Random digit generation.Computing News, 1956,4, 85: 5–85: 14.

    Google Scholar 

  2. Brozek, J. and Tiede, K. Reliable and questionable significance in a series of statistical tests.Psychol. Bull., 1952,49, 339–341.

    Google Scholar 

  3. Johnson, D. L. Generating and testing pseudo-random numbers on the IBM Type 701.Mathematical tables and other aids to computation, 1956,10, 8–13.

    Google Scholar 

  4. Jones, L. V. and Fiske, D. W. Models for testing the significance of combined results.Psychol. Bull., 1953,50, 375–382.

    Google Scholar 

  5. Latscha, R. Tests of significance in a 2 × 2 contingency table: extension of Finney's table.Biometrika, 1953,40, 74–86.

    Google Scholar 

  6. Meyer, H. A. (Ed.)Symposium on Monte Carlo methods. New York: Wiley, 1956.

    Google Scholar 

  7. Mosteller, F. and Bush, R. R. Selected quantitative techniques. In G. Lindzey (Ed.),Handbook of social psychology. Cambridge, Mass.: Addison-Wesley, 1954. Pp. 289–334.

    Google Scholar 

  8. Sakoda, J. M., Cohen, B. H. and Beall, G. Test of significance for a series of statistical tests.Psychol. Bull., 1954,51, 172–175.

    Google Scholar 

  9. Tukey, J. W. Comparing two small samples on many items.Memo. Rep. 54, Statist. Res. Group, Princeton Univ., 1954.

  10. Wilkinson, B. A statistical consideration in psychological research.Psychol. Bull., 1951,48, 156–158.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of California, Berkeley

    Jack Block

Authors
  1. Jack Block
    View author publications

    You can also search for this author in PubMed Google Scholar

Additional information

This investigation, part of a larger research project, was supported primarily by research grant M-1078 from the National Institute of Mental Health of the United States Public Health Service. Gratitude is also extended to the National Science Foundation for making available research time on the IBM Model 701 computer. The present paper has benefited greatly from the comments on an earlier version offered by John Tukey and an editor ofPsychometrika.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Block, J. On the number of significant findings to be expected by chance. Psychometrika 25, 369–380 (1960). https://doi.org/10.1007/BF02289754

Download citation

  • Received:

  • Revised:

  • Issue Date:

  • DOI: https://doi.org/10.1007/BF02289754

Keywords

Search

Navigation