Skip to main content

Distinguishing true from false positives in genomic studies: p values

Abstract

Distinguishing true from false positive findings is a major challenge in human genetic epidemiology. Several strategies have been devised to facilitate this, including the positive predictive value (PPV) and a set of epidemiological criteria, known as the “Venice” criteria. The PPV measures the probability of a true association, given a statistically significant finding, while the Venice criteria grade the credibility based on the amount of evidence, consistency of replication and protection from bias. A vast majority of journals use significance thresholds to identify the true positive findings. We studied the effect of p value thresholds on the PPV and used the PPV and Venice criteria to define usable thresholds of statistical significance. Theoretical and empirical analyses of data published on AlzGene show that at a nominal p value threshold of 0.05 most “positive” findings will turn out to be false if the prior probability of association is below 0.10 even if the statistical power of the study is higher than 0.80. However, in underpowered studies (0.25) with a low prior probability of 1 × 10−3, a p value of 1 × 10−5 yields a high PPV (>96 %). Here we have shown that the p value threshold of 1 × 10−5 gives a very strong evidence of association in almost all studies. However, in the case of a very high prior probability of association (0.50) a p value threshold of 0.05 may be sufficient, while for studies with very low prior probability of association (1 × 10−4; genome-wide association studies for instance) 1 × 10−7 may serve as a useful threshold to declare significance.

This is a preview of subscription content, access via your institution.

Fig. 1

References

  1. 1.

    Hirschhorn JN, Lohmueller K, Byrne E, Hirschhorn K. A comprehensive review of genetic association studies. Genet Med. 2002;4(2):45–61.

    PubMed  Article  CAS  Google Scholar 

  2. 2.

    Abou-Sleiman PM, Hanna MG, Wood NW. Genetic association studies of complex neurological diseases. J Neurol Neurosurg Psychiatr. 2006;77(12):1302–4. doi:10.1136/jnnp.2005.082024.

    PubMed  Article  CAS  Google Scholar 

  3. 3.

    Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124.

    PubMed  Article  Google Scholar 

  4. 4.

    Wacholder S, Chanock S, Garcia-Closas M, El Ghormli L, Rothman N. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. J Natl Cancer Inst. 2004;96(6):434–42.

    PubMed  Article  Google Scholar 

  5. 5.

    Weitkunat R, Kaelin E, Vuillaume G, Kallischnigg G. Effectiveness of strategies to increase the validity of findings from association studies: size versus replication. BMC Med Res Methodol. 2010;10:47. doi:10.1186/1471-2288-10-47.

    PubMed  Article  Google Scholar 

  6. 6.

    Lucke JF. A critique of the false-positive report probability. Genet Epidemiol. 2009;33(2):145–50. doi:10.1002/gepi.20363.

    PubMed  Article  Google Scholar 

  7. 7.

    Matullo G, Berwick M, Vineis P. Gene-environment interactions: how many false positives? J Natl Cancer Inst. 2005;97(8):550–1. doi:10.1093/jnci/dji122.

    PubMed  Article  Google Scholar 

  8. 8.

    Rebbeck TR, Ambrosone CB, Bell DA, Chanock SJ, Hayes RB, Kadlubar FF, et al. SNPs, haplotypes, and cancer: applications in molecular epidemiology. Cancer Epidemiol Biomark Prev. 2004;13(5):681–7.

    CAS  Google Scholar 

  9. 9.

    Thomas DC, Clayton DG. Betting odds and genetic associations. J Natl Cancer Inst. 2004;96(6):421–3.

    PubMed  Article  Google Scholar 

  10. 10.

    Moonesinghe R, Khoury MJ, Janssens AC. Most published research findings are false-but a little replication goes a long way. PLoS Med. 2007;4(2):e28. doi:10.1371/journal.pmed.0040028.

    PubMed  Article  Google Scholar 

  11. 11.

    Ioannidis JP, Boffetta P, Little J, O’Brien TR, Uitterlinden AG, Vineis P, et al. Assessment of cumulative evidence on genetic associations: interim guidelines. Int J Epidemiol. 2008;37(1):120–32. doi:10.1093/ije/dym159.

    PubMed  Article  Google Scholar 

  12. 12.

    International HapMap C. The international HapMap project. Nature. 2003;426(6968):789–96. doi:10.1038/nature02168nature02168.

    Article  Google Scholar 

  13. 13.

    Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical. J R Statist Soc B. 1995;57(1):289–300.

    Google Scholar 

  14. 14.

    Gordon A, Glazko G, Qiu X, Yakovlev A. Control of the mean number of false discoveries, Bonferroni and stability of multiple testing. Ann Appl Stat. 2007;1:179–90.

    Article  Google Scholar 

  15. 15.

    Khoury MJ, Bertram L, Boffetta P, Butterworth AS, Chanock SJ, Dolan SM, et al. Genome-wide association studies, field synopses, and the development of the knowledge base on genetic variation and human diseases. Am J Epidemiol. 2009;170(3):269–79. doi:10.1093/aje/kwp119.

    PubMed  Article  Google Scholar 

  16. 16.

    Ridley J, Kolm N, Freckelton RP, Gage MJ. An unexpected influence of widely used significance thresholds on the distribution of reported p values. J Evol Biol. 2007;20(3):1082–9. doi:10.1111/j.1420-9101.2006.01291.x.

    PubMed  Article  CAS  Google Scholar 

  17. 17.

    Bertram L, McQueen MB, Mullin K, Blacker D, Tanzi RE. Systematic meta-analyses of Alzheimer disease genetic association studies: the AlzGene database. Nat Genet. 2007;39(1):17–23. doi:10.1038/ng1934.

    PubMed  Article  CAS  Google Scholar 

  18. 18.

    Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–60. doi:10.1136/bmj.327.7414.557327/7414/557.

    PubMed  Article  Google Scholar 

  19. 19.

    DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–88. doi:0197-2456(86)90046-2.

    PubMed  Article  CAS  Google Scholar 

  20. 20.

    Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443–57. doi:10.1002/sim.2380.

    PubMed  Article  Google Scholar 

  21. 21.

    Kavvoura FK, McQueen MB, Khoury MJ, Tanzi RE, Bertram L, Ioannidis JP. Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer’s disease. Am J Epidemiol. 2008;168(8):855–65. doi:10.1093/aje/kwn206.

    PubMed  Article  Google Scholar 

  22. 22.

    Ioannidis JP. Calibration of credibility of agnostic genome-wide associations. Am J Med Genet B Neuropsychiatr Genet. 2008;147B(6):964–72. doi:10.1002/ajmg.b.30721.

    PubMed  Article  Google Scholar 

  23. 23.

    Ioannidis JP, Tarone R, McLaughlin JK. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology. 2011;22(4):450–6. doi:10.1097/EDE.0b013e31821b506e.

    PubMed  Article  Google Scholar 

  24. 24.

    Rothman KJ. Epidemiology: an introduction. 1st ed. Oxford: Oxford University Press; 2002.

    Google Scholar 

  25. 25.

    Panagiotou OA, Ioannidis JP, Genome-Wide Significance Project. What should the genome-wide significance threshold be? Empirical replication of borderline genetic associations. Int J Epidemiol. 2011;. doi:10.1093/ije/dyr178.

    PubMed  Google Scholar 

  26. 26.

    Biotechnology Kaiser J. Researcher, two universities sued over validity of prostate cancer test. Science. 2009;325(5947):1484. doi:10.1126/science.325_1484.

    Article  Google Scholar 

  27. 27.

    van Duijn CM. STROBE-ME too! Eur J Epidemiol. 2011;26(10):761–2. doi:10.1007/s10654-011-9628-8.

    PubMed  Article  Google Scholar 

  28. 28.

    Sanna S, Li B, Mulas A, Sidore C, Kang HM, Jackson AU, et al. Fine mapping of five Loci associated with low-density lipoprotein cholesterol detects variants that double the explained heritability. PLoS Genet. 2011;7(7):e1002198. doi:10.1371/journal.pgen.1002198PGENETICS-D-11-00557.

    PubMed  Article  CAS  Google Scholar 

  29. 29.

    Boseley S. Six men in intensive care after drug trial goes wrong. The Guardian. 2006.

  30. 30.

    Farrer LA, Cupples LA, Haines JL, Hyman B, Kukull WA, Mayeux R, et al. Effects of age, sex, and ethnicity on the association between apolipoprotein E genotype and Alzheimer disease. A meta-analysis. APOE and Alzheimer disease meta analysis consortium. JAMA. 1997;278(16):1349–56.

    PubMed  Article  CAS  Google Scholar 

Download references

Acknowledgments

The study was supported by grants from the Centre for Medical Systems Biology (CMSB) and ENGAGE Consortium, grant agreement HEALTH-F4-2007-201413. The genetics databases used for this project have been made possible by the kind support of the Cure Alzheimer’s Fund (CAF), the Michael J. Fox Foundation (MJFF) for Parkinson’s Research, the National Alliance for Research on Schizophrenia and Depression (NARSAD), Prize4Life, and EMD Serono. C.M.L. is supported by the Fidelity Biosciences Research Initiative. L.Bertram is supported by the German Ministry for Education and Research (BMBF).

Conflict of interest

The authors declare no competing interests.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Cornelia M. van Duijn.

Additional information

Linda Broer, Christina M. Lill, Maaike Schuur are shared first author.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOC 364 kb)

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Broer, L., Lill, C.M., Schuur, M. et al. Distinguishing true from false positives in genomic studies: p values. Eur J Epidemiol 28, 131–138 (2013). https://doi.org/10.1007/s10654-012-9755-x

Download citation

Keywords

  • Venice Criteria
  • Significance thresholds
  • “-Omics”
  • Alzheimer’s disease