Annals of Behavioral Medicine

, Volume 27, Issue 2, pp 138–145 | Cite as

Effect sizes and statistical testing in the determination of clinical significance in behavioral medicine research

Article

Abstract

Background: The interpretation of clinical significance continues to be an obstacle for researchers in behavioral medicine.Purpose: To review selected behavioral medicine research to critically examine the perception among investigators that behavioral effects on health are small based on common metrics of clinical significance.Methods: Using quantitative findings from recent behavioral medicine research in medical and psychiatric journals, we explored results in terms of several statistical metrics to assess potential clinical significance: r coefficients, risk ratios, risk difference measures, and attributable risk.Results: Translated into r coefficients, even established health predictors such as smoking, obesity, and fitness had only modest effects (rs = .03-.22), and the range of effect sizes were comparable with those based on psychological predictors including depression and stress-reactivity (rs = .06-.22). In contrast, effects for both classes of predictors were suggestive of clinical significance based on public health statistics.Conclusions: Our choice of statistics for defining “small” and “large” effect sizes affects the perceived importance of behavioral health findings. In the assessment of health outcomes with low incidence rates, effects expressed as correlations using even the most robust predictors will often appear small. In these instances, we challenge researchers to move beyond conventional data analysis approaches and to expand their clinical interpretation efforts by employing additional statistical methods favored in medicine and public health.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. (1).
    Cohen J: The earth is round (p <.05).American Psychologist. 1994,49:997–1003.CrossRefGoogle Scholar
  2. (2).
    Cohen J: Things I have learned (so far).American Psychologist. 1990,45:1304–1312.CrossRefGoogle Scholar
  3. (3).
    Cohen J: The statistical power of abnormal-social psychological research.Journal of Abnormal and Social Psychology. 1962,65:145–153.PubMedCrossRefGoogle Scholar
  4. (4).
    Rosenthal R, Rubin DB: A note on percent variance explained as a measure of the importance of effects.Journal of Applied Social Psychology. 1979,9:395–396.CrossRefGoogle Scholar
  5. (5).
    Rosenthal R: How are we doing in soft psychology?American Psychologist. 1990,45:775–777.CrossRefGoogle Scholar
  6. (6).
    Rosenthal R, Rubin DB: A simple general purpose display of magnitude of experimental effect.Journal of Education Psychology. 1982,74:166–169.CrossRefGoogle Scholar
  7. (7).
    Rosnow RL, Rosenthal R: Focused tests of significance and effect size estimation in counseling psychology.Journal of Counseling Psychology. 1988,35:203–208.CrossRefGoogle Scholar
  8. (8).
    Rosnow RL, Rosenthal R: Computing contrasts, effect sizes, and counternulls on other people’s published data: General procedures for research consumers.Psychological Methods. 1996,1: 331–340.CrossRefGoogle Scholar
  9. (9).
    Moher D, Dulerg CS, Wells GA: Statistical power, sample size, and their reporting in randomized clinical trials.Journal of the American Medical Association. 1994,272:122–124.PubMedCrossRefGoogle Scholar
  10. (10).
    Cohen J:Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc., 1988.Google Scholar
  11. (11).
    Schmidt FL: Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers.Psychological Methods. 1996,2:115–129.CrossRefGoogle Scholar
  12. (12).
    Murray LW, Dosser DA Jr.: How significant is a significant difference? Problems with the measurement of magnitude of effect.Journal of Counseling Psychology. 1987,34:68–72.CrossRefGoogle Scholar
  13. (13).
    Meehl PE: Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald and the slow progress of soft psychology.Journal of Consulting and Clinical Psychology. 1978,46:806–834.CrossRefGoogle Scholar
  14. (14).
    Thompson WD: Statistical criteria in the interpretation of epidemiologic data.American Journal of Public Health. 1987,77:191–194.PubMedCrossRefGoogle Scholar
  15. (15).
    Greenland S, Schlesselman JJ, Criqui MH: The fallacy of employing standardized regression coefficient and correlations as measures of effect.American Journal of Epidemiology. 1986,123:203–208.PubMedGoogle Scholar
  16. (16).
    Willenheimer R: Statistical significance versus clinical relevance in cardiovascular medicine.Progress in Cardiovascular Disease. 2001,44:155–167.CrossRefGoogle Scholar
  17. (17).
    Rosenthal R, Rosnow RL, Rubin DB:Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach. New York: Cambridge University Press, 2000.Google Scholar
  18. (18).
    Ozer DJ: Correlation and the coefficient of determination.Psychological Bulletin. 1985,97:307–315.CrossRefGoogle Scholar
  19. (19).
    Abelson RP: A variance explanation paradox: When a little is a lot.Psychological Bulletin. 1985,97:129–133.CrossRefGoogle Scholar
  20. (20).
    Sedlmeier P, Gigerenzer G: Do studies of statistical power have an effect on the power of studies?Psychological Bulletin. 1989,105:309–316.CrossRefGoogle Scholar
  21. (21).
    Rosnow RL, Rosenthal R: Statistical procedures and the justification of knowledge in psychological science.American Psychologist. 1989,44:1276–1284.CrossRefGoogle Scholar
  22. (22).
    Frasure-Smith N, Lesperance F, Talajic M: Depression following myocardial infarction. Impact on 6-month survival.Journal of the American Medical Association. 1993,270:1819–1825.PubMedCrossRefGoogle Scholar
  23. (23).
    Steering Committee of the Physicians’ Health Study Research Group: Preliminary report: Findings from the aspirin component of the ongoing physicians’ health study.New England Journal of Medicine. 1988,318:262–264.CrossRefGoogle Scholar
  24. (24).
    Rexrode KM, Carey VJ, Hennekens CH, et al.: abdominal adiposity and coronary heart disease in women.Journal of the American Medical Association. 1998,280:1843–1848.PubMedCrossRefGoogle Scholar
  25. (25).
    The DPP Research Group: The Diabetes Prevention Program: Recruitment methods and results.Controlled Clinical Trials. 2002,23:157–171.CrossRefGoogle Scholar
  26. (26).
    Smith ML, Glass GV: Meta-analysis of psychotherapy outcome studies.American Psychologist. 1977,32:752–760.PubMedCrossRefGoogle Scholar
  27. (27).
    Vaccarino V, Kasl SV, Abramson J, Krumholz HM: Depressive symptoms and risk of functional decline and death in patients with heart failure.Journal of the American College of Cardiology. 2001,38:199–205.PubMedCrossRefGoogle Scholar
  28. (28).
    Blair SD, Kampert JB, Kohl HW, et al.: Influences of cardiorespiratory fitness and other precursors on cardiovascular disease and all-cause mortality in men and women.Journal of the American Medical Association. 1996,276:205–210.PubMedCrossRefGoogle Scholar
  29. (29).
    Huang Z, Willett WC, Manson JE, et al.: Body weight, weight change, and risk for hypertension in women.Annals of Internal Medicine. 1998,128:81–8.PubMedGoogle Scholar
  30. (30).
    Wingard DL, Suarez L, Barrett-Connor E: The sex differential in mortality from all cause and ischemic heart disease.American Journal of Epidemiology. 1983,117:165–172.PubMedGoogle Scholar
  31. (31).
    Everson SA, Lynch JW, Kaplan GA, et al.: Stress-induced blood pressure reactivity and incident stroke in middle-aged men.Stroke. 2001,32:1263–1270.PubMedGoogle Scholar
  32. (32).
    Sauer WH, Berlin JA, Kimmel SE: Selective serotonin reuptake inhibitors and myocardial infarction.Circulation. 2001,104:1894–1898.PubMedCrossRefGoogle Scholar
  33. (33).
    Centers for Disease Control Vietnam Experience Study: Health status of Vietnam veterans: 1. Psychosocial characteristics.Journal of the American Medical Association. 1988,259:2701–2707.CrossRefGoogle Scholar
  34. (34).
    Miura K, Daviglus ML, Dyer AR, et al.: Relationship of blood pressure to 25-year mortality due to coronary heart disease, cardiovascular diseases, and all causes in young adult men.Archives of Internal Medicine. 2001,161:1501–1508.PubMedCrossRefGoogle Scholar
  35. (35).
    Anda R, Williamson D, Jones D, et al.: Depressed affect, hopelessness, and the risk of ischemic heart disease in a cohort of U.S. adults.Epidemiology. 1993,4:285–294.PubMedCrossRefGoogle Scholar
  36. (36).
    Davidson K, Jonas BS, Dixon KE, Markovitz JH: Do depression symptoms predict early hypertension incidence in young adults in the CARDIA study?Archives of Internal Medicine. 2000,160:1495–1500.PubMedCrossRefGoogle Scholar
  37. (37).
    Huff D:How to Lie With Statistics. New York: Norton, 1954.Google Scholar
  38. (38).
    Kronmal RA, Cain KC, Ye Z, Omenn GS: Total serum cholesterol levels and mortality risk as a function of age.Archives of Internal Medicine. 1993,153:1065–1073.PubMedCrossRefGoogle Scholar
  39. (39).
    Altman DG, Deeks J, Sacket DL: Odds ratios should be avoided when events are common.British Medical Journal. 1998,317:1318.PubMedGoogle Scholar
  40. (40).
    Zhang JMB, Yu KF: What’s the relative risk?: A method of correcting the odds ratio in cohort studies of common outcomes.Journal of the American Medical Association. 1998,280:1690–1691.PubMedCrossRefGoogle Scholar
  41. (41).
    Pfahlberg A, Gefeller O, Uter W: Reporting of attributable risk in epidemiologic studies.Epidemiology. 2001,12:4745.CrossRefGoogle Scholar
  42. (42).
    Uter W, Pfablberg A: The application of methods to quantify attributable risk in medical practice.Statistical Methods in Medical Research. 2001,10:231–237.PubMedCrossRefGoogle Scholar
  43. (43).
    Everson SA, Goldberg DE, Kaplan GA, Julkunen J, Salonen JT: Anger expression and incident hypertension.Psychosomatic Medicine. 1998,60:730–735.PubMedGoogle Scholar
  44. (44).
    Rosenthal R, Rosnow RL:Essentials of Behavioral Research: Methods and Data Analysis. New York:McGraw-Hill, 1991.Google Scholar
  45. (45).
    Kazdin AE:Research Design in Clinical Psychology. Allyn & Bacon, 1992.Google Scholar
  46. (46).
    Rosenthal R, DiMatteo MR: Meta analysis: Recent developments in quantitative methods for literature reviews.Annual Review of Psychology. 2000,52:59–82.CrossRefGoogle Scholar
  47. (47).
    Buyse M, Piedbois P, Piedbois Y, Carlson RW: Meta-analysis: methods, strengths, and weaknesses.Oncology. 2000,14:437–443.PubMedGoogle Scholar
  48. (48).
    Bailar JC: The promise and problems of meta-analysis.New England Journal of Medicine. 1997,337:559–561.PubMedCrossRefGoogle Scholar
  49. (49).
    Bradbury J: Transferring research from the laboratory to the clinic.Lancet. 1996,347:1103.PubMedCrossRefGoogle Scholar
  50. (50).
    Burke JD, Pincus HA, Pardes H: The clinician-researcher in psychiatry.American Journal of Psychiatry. 1986,143:968–975.PubMedGoogle Scholar

Copyright information

© The Society of Behavioral Medicine 2004

Authors and Affiliations

  1. 1.University of CaliforniaSan Diego

Personalised recommendations