Skip to main content
SpringerLink
Log in

Effect sizes and statistical testing in the determination of clinical significance in behavioral medicine research

  • Published:
Annals of Behavioral Medicine

Abstract

Background: The interpretation of clinical significance continues to be an obstacle for researchers in behavioral medicine.Purpose: To review selected behavioral medicine research to critically examine the perception among investigators that behavioral effects on health are small based on common metrics of clinical significance.Methods: Using quantitative findings from recent behavioral medicine research in medical and psychiatric journals, we explored results in terms of several statistical metrics to assess potential clinical significance: r coefficients, risk ratios, risk difference measures, and attributable risk.Results: Translated into r coefficients, even established health predictors such as smoking, obesity, and fitness had only modest effects (rs = .03-.22), and the range of effect sizes were comparable with those based on psychological predictors including depression and stress-reactivity (rs = .06-.22). In contrast, effects for both classes of predictors were suggestive of clinical significance based on public health statistics.Conclusions: Our choice of statistics for defining “small” and “large” effect sizes affects the perceived importance of behavioral health findings. In the assessment of health outcomes with low incidence rates, effects expressed as correlations using even the most robust predictors will often appear small. In these instances, we challenge researchers to move beyond conventional data analysis approaches and to expand their clinical interpretation efforts by employing additional statistical methods favored in medicine and public health.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Cohen J: The earth is round (p <.05).American Psychologist. 1994,49:997–1003.

    Article  Google Scholar 

  2. Cohen J: Things I have learned (so far).American Psychologist. 1990,45:1304–1312.

    Article  Google Scholar 

  3. Cohen J: The statistical power of abnormal-social psychological research.Journal of Abnormal and Social Psychology. 1962,65:145–153.

    Article  PubMed  CAS  Google Scholar 

  4. Rosenthal R, Rubin DB: A note on percent variance explained as a measure of the importance of effects.Journal of Applied Social Psychology. 1979,9:395–396.

    Article  Google Scholar 

  5. Rosenthal R: How are we doing in soft psychology?American Psychologist. 1990,45:775–777.

    Article  Google Scholar 

  6. Rosenthal R, Rubin DB: A simple general purpose display of magnitude of experimental effect.Journal of Education Psychology. 1982,74:166–169.

    Article  Google Scholar 

  7. Rosnow RL, Rosenthal R: Focused tests of significance and effect size estimation in counseling psychology.Journal of Counseling Psychology. 1988,35:203–208.

    Article  Google Scholar 

  8. Rosnow RL, Rosenthal R: Computing contrasts, effect sizes, and counternulls on other people’s published data: General procedures for research consumers.Psychological Methods. 1996,1: 331–340.

    Article  Google Scholar 

  9. Moher D, Dulerg CS, Wells GA: Statistical power, sample size, and their reporting in randomized clinical trials.Journal of the American Medical Association. 1994,272:122–124.

    Article  PubMed  CAS  Google Scholar 

  10. Cohen J:Statistical Power Analysis for the Behavioral Sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc., 1988.

    Google Scholar 

  11. Schmidt FL: Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers.Psychological Methods. 1996,2:115–129.

    Article  Google Scholar 

  12. Murray LW, Dosser DA Jr.: How significant is a significant difference? Problems with the measurement of magnitude of effect.Journal of Counseling Psychology. 1987,34:68–72.

    Article  Google Scholar 

  13. Meehl PE: Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald and the slow progress of soft psychology.Journal of Consulting and Clinical Psychology. 1978,46:806–834.

    Article  Google Scholar 

  14. Thompson WD: Statistical criteria in the interpretation of epidemiologic data.American Journal of Public Health. 1987,77:191–194.

    Article  PubMed  CAS  Google Scholar 

  15. Greenland S, Schlesselman JJ, Criqui MH: The fallacy of employing standardized regression coefficient and correlations as measures of effect.American Journal of Epidemiology. 1986,123:203–208.

    PubMed  CAS  Google Scholar 

  16. Willenheimer R: Statistical significance versus clinical relevance in cardiovascular medicine.Progress in Cardiovascular Disease. 2001,44:155–167.

    Article  CAS  Google Scholar 

  17. Rosenthal R, Rosnow RL, Rubin DB:Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach. New York: Cambridge University Press, 2000.

    Google Scholar 

  18. Ozer DJ: Correlation and the coefficient of determination.Psychological Bulletin. 1985,97:307–315.

    Article  Google Scholar 

  19. Abelson RP: A variance explanation paradox: When a little is a lot.Psychological Bulletin. 1985,97:129–133.

    Article  Google Scholar 

  20. Sedlmeier P, Gigerenzer G: Do studies of statistical power have an effect on the power of studies?Psychological Bulletin. 1989,105:309–316.

    Article  Google Scholar 

  21. Rosnow RL, Rosenthal R: Statistical procedures and the justification of knowledge in psychological science.American Psychologist. 1989,44:1276–1284.

    Article  Google Scholar 

  22. Frasure-Smith N, Lesperance F, Talajic M: Depression following myocardial infarction. Impact on 6-month survival.Journal of the American Medical Association. 1993,270:1819–1825.

    Article  PubMed  CAS  Google Scholar 

  23. Steering Committee of the Physicians’ Health Study Research Group: Preliminary report: Findings from the aspirin component of the ongoing physicians’ health study.New England Journal of Medicine. 1988,318:262–264.

    Article  Google Scholar 

  24. Rexrode KM, Carey VJ, Hennekens CH, et al.: abdominal adiposity and coronary heart disease in women.Journal of the American Medical Association. 1998,280:1843–1848.

    Article  PubMed  CAS  Google Scholar 

  25. The DPP Research Group: The Diabetes Prevention Program: Recruitment methods and results.Controlled Clinical Trials. 2002,23:157–171.

    Article  Google Scholar 

  26. Smith ML, Glass GV: Meta-analysis of psychotherapy outcome studies.American Psychologist. 1977,32:752–760.

    Article  PubMed  CAS  Google Scholar 

  27. Vaccarino V, Kasl SV, Abramson J, Krumholz HM: Depressive symptoms and risk of functional decline and death in patients with heart failure.Journal of the American College of Cardiology. 2001,38:199–205.

    Article  PubMed  CAS  Google Scholar 

  28. Blair SD, Kampert JB, Kohl HW, et al.: Influences of cardiorespiratory fitness and other precursors on cardiovascular disease and all-cause mortality in men and women.Journal of the American Medical Association. 1996,276:205–210.

    Article  PubMed  CAS  Google Scholar 

  29. Huang Z, Willett WC, Manson JE, et al.: Body weight, weight change, and risk for hypertension in women.Annals of Internal Medicine. 1998,128:81–8.

    PubMed  CAS  Google Scholar 

  30. Wingard DL, Suarez L, Barrett-Connor E: The sex differential in mortality from all cause and ischemic heart disease.American Journal of Epidemiology. 1983,117:165–172.

    PubMed  CAS  Google Scholar 

  31. Everson SA, Lynch JW, Kaplan GA, et al.: Stress-induced blood pressure reactivity and incident stroke in middle-aged men.Stroke. 2001,32:1263–1270.

    PubMed  CAS  Google Scholar 

  32. Sauer WH, Berlin JA, Kimmel SE: Selective serotonin reuptake inhibitors and myocardial infarction.Circulation. 2001,104:1894–1898.

    Article  PubMed  CAS  Google Scholar 

  33. Centers for Disease Control Vietnam Experience Study: Health status of Vietnam veterans: 1. Psychosocial characteristics.Journal of the American Medical Association. 1988,259:2701–2707.

    Article  Google Scholar 

  34. Miura K, Daviglus ML, Dyer AR, et al.: Relationship of blood pressure to 25-year mortality due to coronary heart disease, cardiovascular diseases, and all causes in young adult men.Archives of Internal Medicine. 2001,161:1501–1508.

    Article  PubMed  CAS  Google Scholar 

  35. Anda R, Williamson D, Jones D, et al.: Depressed affect, hopelessness, and the risk of ischemic heart disease in a cohort of U.S. adults.Epidemiology. 1993,4:285–294.

    Article  PubMed  CAS  Google Scholar 

  36. Davidson K, Jonas BS, Dixon KE, Markovitz JH: Do depression symptoms predict early hypertension incidence in young adults in the CARDIA study?Archives of Internal Medicine. 2000,160:1495–1500.

    Article  PubMed  CAS  Google Scholar 

  37. Huff D:How to Lie With Statistics. New York: Norton, 1954.

    Google Scholar 

  38. Kronmal RA, Cain KC, Ye Z, Omenn GS: Total serum cholesterol levels and mortality risk as a function of age.Archives of Internal Medicine. 1993,153:1065–1073.

    Article  PubMed  CAS  Google Scholar 

  39. Altman DG, Deeks J, Sacket DL: Odds ratios should be avoided when events are common.British Medical Journal. 1998,317:1318.

    PubMed  CAS  Google Scholar 

  40. Zhang JMB, Yu KF: What’s the relative risk?: A method of correcting the odds ratio in cohort studies of common outcomes.Journal of the American Medical Association. 1998,280:1690–1691.

    Article  PubMed  CAS  Google Scholar 

  41. Pfahlberg A, Gefeller O, Uter W: Reporting of attributable risk in epidemiologic studies.Epidemiology. 2001,12:4745.

    Article  Google Scholar 

  42. Uter W, Pfablberg A: The application of methods to quantify attributable risk in medical practice.Statistical Methods in Medical Research. 2001,10:231–237.

    Article  PubMed  CAS  Google Scholar 

  43. Everson SA, Goldberg DE, Kaplan GA, Julkunen J, Salonen JT: Anger expression and incident hypertension.Psychosomatic Medicine. 1998,60:730–735.

    PubMed  CAS  Google Scholar 

  44. Rosenthal R, Rosnow RL:Essentials of Behavioral Research: Methods and Data Analysis. New York:McGraw-Hill, 1991.

    Google Scholar 

  45. Kazdin AE:Research Design in Clinical Psychology. Allyn & Bacon, 1992.

  46. Rosenthal R, DiMatteo MR: Meta analysis: Recent developments in quantitative methods for literature reviews.Annual Review of Psychology. 2000,52:59–82.

    Article  Google Scholar 

  47. Buyse M, Piedbois P, Piedbois Y, Carlson RW: Meta-analysis: methods, strengths, and weaknesses.Oncology. 2000,14:437–443.

    PubMed  CAS  Google Scholar 

  48. Bailar JC: The promise and problems of meta-analysis.New England Journal of Medicine. 1997,337:559–561.

    Article  PubMed  Google Scholar 

  49. Bradbury J: Transferring research from the laboratory to the clinic.Lancet. 1996,347:1103.

    Article  PubMed  CAS  Google Scholar 

  50. Burke JD, Pincus HA, Pardes H: The clinician-researcher in psychiatry.American Journal of Psychiatry. 1986,143:968–975.

    PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of California, San Diego

    Thomas Rutledge Ph.D. & Cathy Loh Ph.D.

Authors
  1. Thomas Rutledge Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Cathy Loh Ph.D.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thomas Rutledge Ph.D..

About this article

Cite this article

Rutledge, T., Loh, C. Effect sizes and statistical testing in the determination of clinical significance in behavioral medicine research. ann. behav. med. 27, 138–145 (2004). https://doi.org/10.1207/s15324796abm2702_9

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1207/s15324796abm2702_9

Keywords

Search

Navigation