Quality & Quantity

, Volume 42, Issue 1, pp 53–73

All Bachelors are Unmarried Men (p <  0.05)

Article

Abstract

This paper adds to the list of criticisms against null hypothesis significance testing (NHST). I argue that when researchers do not analyze the conceptual relations among their variables, they may fail to distinguish between logical implications and empirical relations. It does not make sense to use significance testing on hypotheses involving conceptually related phenomena. The widespread lack of conceptual clarification also leads to very small effect sizes in psychology because it causes study participants to understand the stimulus material in different ways. Therefore, they answer in an inconsistent way. Researchers show an extremely low degree of ambition when they seek to show that psychological phenomena differ from chance, or when they try to disprove a hypothesis claiming that a psychological phenomenon does not exist. I see significance testing as a poor solution to the problem of tiny effect sizes in psychology. I recommend that psychological researchers be more explicit both about their main hypotheses and their auxiliary hypotheses. As examples, I analyse all quantitative articles in Issue 1, 2005 of the Journal of Health Psychology.

Keywords

significance testing nonempirical conceptual analysis pseudoempirical quantitative analysis 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Bagozzi R.P., Ascione F.J., Mannebach M.A. (2005). Inter-role in hospital-based pharmacy and therapeutics committee decision making. Journal of Health Psychology 10: 45–64CrossRefGoogle Scholar
  2. Bem D.J. (1996) Ganzfeld phenomena. In: Stein G (eds). Encyclopedia of the Paranormal. Prometheus Books, Buffalo NY, pp. 291–296Google Scholar
  3. Berkson J. (1938). Some difficulties of interpretation encountered in the application of the chi-square test. Journal of the American Statistical Association 33: 526–542CrossRefGoogle Scholar
  4. Cohen J. (1990). Things I have learned (so far). American Psychologist 45: 1304–1312CrossRefGoogle Scholar
  5. Cohen J. (1994). The earth is round (p < .05). American Psychologist 49: 997–1003CrossRefGoogle Scholar
  6. Falk R., Greenbaum C.W. (1995). Significance tests die hard: the persistence of a probabilistic misconception. Theory and Psychology 5: 75–98CrossRefGoogle Scholar
  7. Harlow L.L., Mulaik S.A., Steiger J.H. (1997). What if there were no Significance Tests?. Mahwah NJ, Lawrence Erlbaum AssociatesGoogle Scholar
  8. Heider F. (1958). The Psychology of Interpersonal Relations. New York, WileyGoogle Scholar
  9. Helstrup T., Rognes W., Vollmer F. (1999). Psychologic and memory. Scandinavian Journal of Psychology 40(Suppl. 1): 1–138Google Scholar
  10. Hubbard R., Parsa R.A., Luthy M.R. (1997). The spread of statistical significance testing in psychology. The case of the Journal of Applied Psychology, 1917–1994. Theory and Psychology 7: 545–554Google Scholar
  11. Ingledew D.K., Wray J.L., Markland D., Hardy L. (2005). goal perceptions and affective well-being. Journal of Health Psychology 10: 101–122CrossRefGoogle Scholar
  12. Insel K.C., Meek P.M., Leventhal H. (2005). Differences in illness representation among pulmonary patients and their providers. Journal of Health Psychology 10: 147–162CrossRefGoogle Scholar
  13. Iwasaki Y., Mannell R., Smale B.J., Butcher J. (2005). of leisure participation in predicting stress coping and health among police and emergency response service workers. Journal of Health Psychology 10: 79–99Google Scholar
  14. Jones L. (1955). Statistics and research design. Annual Review of Psychology 6: 405–430CrossRefGoogle Scholar
  15. Kahneman, D. Tversky, A. (1990). Prospect theory: an analysis of under risk. In: P. K. Moser (ed.), Rationality in Action: Approaches vol. New York, NY: Cambridge University Press, pp. 140–170Google Scholar
  16. Kenny D. (1994). Interpersonal Perception: A Social Relations Analysis. New York, Guilford PressGoogle Scholar
  17. Kish L. (1959). Some statistical problems in research design. American Sociological Review 24: 328–338CrossRefGoogle Scholar
  18. Krueger J. (2001). Null hypothesis significance testing: on the survival of a flawed method. American Psychologist 56: 16–26CrossRefGoogle Scholar
  19. Kukla A. (1989). Nonempirical issues in psychology. American 44: 785–794Google Scholar
  20. Marks D.F. (2000a). A pragmatic basis for judging models and theories in health psychology: the axiomatic method Open peer commentary. Journal of Health Psychology 10(1): 5–176CrossRefGoogle Scholar
  21. Marks D.F. (2000b). Editorial. Journal of Health Psychology 5: 131–132CrossRefGoogle Scholar
  22. Marks, D. F. (ed.) (2005). Journal of Health Psychology 10(1).Google Scholar
  23. Mccabe M.P., Judicibus M.D. (2005). The effects of economic disadvantage on psychological well-being and quality of life among people with multiple sclerosis. Journal of Health Psychology 10: 163–173CrossRefGoogle Scholar
  24. Meehl P.E. (1978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology 46: 806–834CrossRefGoogle Scholar
  25. Meehl P.E. (1997). The problem is epistemology, not statistics: replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. In: Harlow L., Mulaik S., Steiger J. (eds). What if There were no Significance Tests?. Erlbaum, Mahwah, NJ, pp. 393–425Google Scholar
  26. Oakes M. (1986). Statistical Inference: A Commentary for the Social and Behavioral Sciences. New York, WileyGoogle Scholar
  27. Ogden J. (2003). Some problems with social cognition models: a pragmatic and conceptual analysis. Health Psychology 22: 431–434CrossRefGoogle Scholar
  28. Ossorio P.G. (1991). Naive baseball theory. Psychological Inquiry 2: 352–355CrossRefGoogle Scholar
  29. Parrot W.G., Harré R. (1991). Smedslundian suburbs in the city of the case of embarrassment. Psychological Inquiry 2: 358–360CrossRefGoogle Scholar
  30. Pervin L. (1991). The pseudoempirical in psychology and the case for Psychologic. Psychological Inquiry 2(4): 325–382CrossRefGoogle Scholar
  31. Putnam, H. (1975). The analytic and the synthetic. In: H. Putnam (ed.), Philosophical Papers: Mind, Language and Reality, vol. 2. Cambridge: Cambridge University Press, pp. 33–69.Google Scholar
  32. Rivers S.E., Salovey P., Pizarro D.A., Pizarro J., Schneider T.R. (2005). Message framing and pap test utilization among women attending a community health clinic. Journal of Health Psychology 10: 65–77CrossRefGoogle Scholar
  33. Rothman K.J., Greenland S. (1998). Approaches to statistical analysis. In: Rothman K.J., Greenland S. (eds). Modern Epidemiology, 2nd edn. Lippincott Williams,sss Wilkins, London pp. 183–199Google Scholar
  34. Rozeboom W.W. (1960). The fallacy of the null-hypothesis significance test. Psychological Bulletin 67: 416–428CrossRefGoogle Scholar
  35. Schmidt F.L. (1996). Statistical significance testing and cumulative knowledge in psychology: implications for training of researchers. Psychological Methods 1: 115–129CrossRefGoogle Scholar
  36. Shotter J. (1991). Measuring blindly and speculating loosely: But is a the answer?. Psychological Inquiry 2: 363–365CrossRefGoogle Scholar
  37. Shweder R.A. (1991). On pseudoempiricism, pseudodeductionism, and common sense. Psychological Inquiry 2: 366–370CrossRefGoogle Scholar
  38. Smedslund G. (2000). A pragmatic basis for judging models and theories in health psychology: The axiomatic method (target paper). Journal of Health Psychology 5: 133–149Google Scholar
  39. Smedslund J. (1978). Bandura’s theory of self-efficacy: a set of common sense theorems. Scandinavian Journal of Psychology 19: 1–14Google Scholar
  40. Smedslund, J. (1984). What is necessarily true in psychology? In: J. R. Royce and L. P. Mos (eds.), Annals of Theoretical Psychology. Plenum Press, New York London, pp. 241–272Google Scholar
  41. Smedslund J. (1987a). Ebbinghaus, the illusionist: how psychology came to look like an experimental science. Passauer Schriften zur Psychologiegeschichte 5: 225–239Google Scholar
  42. Smedslund J. (1987b). The epistemic status of inter-item correlations in Eysenck’s Personality Questionnaire: the a priori versus the empirical in psychological data. Scandinavian Journal of Psychology 28: 42–55Google Scholar
  43. Smedslund J. (1988). Psycho-Logic. Springer Verlag, Berlin Heidelberg New YorkGoogle Scholar
  44. Smedslund J. (1991). The pseudoempirical in psychology and the case for Psychologic. Psychological Inquiry 2: 325–338CrossRefGoogle Scholar
  45. Smedslund J. (1995). Auxiliary versus theoretical hypotheses and ordinary versus scientific language. Human Development 38: 174–178CrossRefGoogle Scholar
  46. Smedslund, J. (1997a). Is the ‘psychologic’ of trust universal? In: Niemeier S., Dirven R. (eds) The Language of Emotions. Conceptualization, Expression, and Theoretical Foundation. Amsterdam/Philadelphia: John Benjamins Publishing Company, pp. 3–13.Google Scholar
  47. Smedslund J. (1997b). The Structure of Psychological Common Sense. Lawrence Erlbaum Associates Publishers, Mahwah, New Jersey, LondonGoogle Scholar
  48. Smedslund J. (1999). Psychologic and the study of memory. Scandinavian Journal of Psychology 40: 3–17Google Scholar
  49. Smedslund J. (2002). From hypothesis-testing psychology to procedure-testing psychologic. Review of General Psychology 6: 51–72CrossRefGoogle Scholar
  50. Smyth M.M. (2001). Fact making in psychology: the voice of the textbook. Theory and Psychology 11: 609–636CrossRefGoogle Scholar
  51. Sterne, J. A. & Davey Smith, G. (2001). Sifting the evidence – what’s wrong with significance tests? British Medical Journal 322(January): 226–231Google Scholar
  52. Storm, L. & Ertel, S. (2001). Does psi exist? Comments on Milton and Wiseman’s (1999): meta-analysis of ganzfeld research. Psychological Bulletin 127: 424–433.Google Scholar
  53. The Free Dictionary (2005). Retrieved December 21, 2005, from http:// www. thefreedictionary.com/helplessnessGoogle Scholar
  54. Wallach L., Wallach M.A. (1999). Why is experimentation in psychology often senseless?. Scandinavian Journal of Psychology 40: 103–106Google Scholar
  55. Wallach M.A., Wallach L. (1998a). Of surrogacy, circularity, causality and near-tautologies: a response. Theory and Psychology 8: 213–217CrossRefGoogle Scholar
  56. Wallach M.A., Wallach L. (1998b). When experiments serve purpose: misguided research in mainstream psychology. Theory and Psychology 8: 183–194CrossRefGoogle Scholar

Copyright information

© Springer Science + Business Media B.V. 2006

Authors and Affiliations

  1. 1.Norwegian Knowledge Centre for the Health ServicesOsloNorway

Personalised recommendations