Advertisement

European Journal of Epidemiology

, Volume 28, Issue 2, pp 131–138 | Cite as

Distinguishing true from false positives in genomic studies: p values

  • Linda Broer
  • Christina M. Lill
  • Maaike Schuur
  • Najaf Amin
  • Johannes T. Roehr
  • Lars Bertram
  • John P. A. Ioannidis
  • Cornelia M. van Duijn
METHODS

Abstract

Distinguishing true from false positive findings is a major challenge in human genetic epidemiology. Several strategies have been devised to facilitate this, including the positive predictive value (PPV) and a set of epidemiological criteria, known as the “Venice” criteria. The PPV measures the probability of a true association, given a statistically significant finding, while the Venice criteria grade the credibility based on the amount of evidence, consistency of replication and protection from bias. A vast majority of journals use significance thresholds to identify the true positive findings. We studied the effect of p value thresholds on the PPV and used the PPV and Venice criteria to define usable thresholds of statistical significance. Theoretical and empirical analyses of data published on AlzGene show that at a nominal p value threshold of 0.05 most “positive” findings will turn out to be false if the prior probability of association is below 0.10 even if the statistical power of the study is higher than 0.80. However, in underpowered studies (0.25) with a low prior probability of 1 × 10−3, a p value of 1 × 10−5 yields a high PPV (>96 %). Here we have shown that the p value threshold of 1 × 10−5 gives a very strong evidence of association in almost all studies. However, in the case of a very high prior probability of association (0.50) a p value threshold of 0.05 may be sufficient, while for studies with very low prior probability of association (1 × 10−4; genome-wide association studies for instance) 1 × 10−7 may serve as a useful threshold to declare significance.

Keywords

Venice Criteria Significance thresholds “-Omics” Alzheimer’s disease 

Notes

Acknowledgments

The study was supported by grants from the Centre for Medical Systems Biology (CMSB) and ENGAGE Consortium, grant agreement HEALTH-F4-2007-201413. The genetics databases used for this project have been made possible by the kind support of the Cure Alzheimer’s Fund (CAF), the Michael J. Fox Foundation (MJFF) for Parkinson’s Research, the National Alliance for Research on Schizophrenia and Depression (NARSAD), Prize4Life, and EMD Serono. C.M.L. is supported by the Fidelity Biosciences Research Initiative. L.Bertram is supported by the German Ministry for Education and Research (BMBF).

Conflict of interest

The authors declare no competing interests.

Supplementary material

10654_2012_9755_MOESM1_ESM.doc (364 kb)
Supplementary material 1 (DOC 364 kb)

References

  1. 1.
    Hirschhorn JN, Lohmueller K, Byrne E, Hirschhorn K. A comprehensive review of genetic association studies. Genet Med. 2002;4(2):45–61.PubMedCrossRefGoogle Scholar
  2. 2.
    Abou-Sleiman PM, Hanna MG, Wood NW. Genetic association studies of complex neurological diseases. J Neurol Neurosurg Psychiatr. 2006;77(12):1302–4. doi: 10.1136/jnnp.2005.082024.PubMedCrossRefGoogle Scholar
  3. 3.
    Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2(8):e124. doi: 10.1371/journal.pmed.0020124.PubMedCrossRefGoogle Scholar
  4. 4.
    Wacholder S, Chanock S, Garcia-Closas M, El Ghormli L, Rothman N. Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. J Natl Cancer Inst. 2004;96(6):434–42.PubMedCrossRefGoogle Scholar
  5. 5.
    Weitkunat R, Kaelin E, Vuillaume G, Kallischnigg G. Effectiveness of strategies to increase the validity of findings from association studies: size versus replication. BMC Med Res Methodol. 2010;10:47. doi: 10.1186/1471-2288-10-47.PubMedCrossRefGoogle Scholar
  6. 6.
    Lucke JF. A critique of the false-positive report probability. Genet Epidemiol. 2009;33(2):145–50. doi: 10.1002/gepi.20363.PubMedCrossRefGoogle Scholar
  7. 7.
    Matullo G, Berwick M, Vineis P. Gene-environment interactions: how many false positives? J Natl Cancer Inst. 2005;97(8):550–1. doi: 10.1093/jnci/dji122.PubMedCrossRefGoogle Scholar
  8. 8.
    Rebbeck TR, Ambrosone CB, Bell DA, Chanock SJ, Hayes RB, Kadlubar FF, et al. SNPs, haplotypes, and cancer: applications in molecular epidemiology. Cancer Epidemiol Biomark Prev. 2004;13(5):681–7.Google Scholar
  9. 9.
    Thomas DC, Clayton DG. Betting odds and genetic associations. J Natl Cancer Inst. 2004;96(6):421–3.PubMedCrossRefGoogle Scholar
  10. 10.
    Moonesinghe R, Khoury MJ, Janssens AC. Most published research findings are false-but a little replication goes a long way. PLoS Med. 2007;4(2):e28. doi: 10.1371/journal.pmed.0040028.PubMedCrossRefGoogle Scholar
  11. 11.
    Ioannidis JP, Boffetta P, Little J, O’Brien TR, Uitterlinden AG, Vineis P, et al. Assessment of cumulative evidence on genetic associations: interim guidelines. Int J Epidemiol. 2008;37(1):120–32. doi: 10.1093/ije/dym159.PubMedCrossRefGoogle Scholar
  12. 12.
    International HapMap C. The international HapMap project. Nature. 2003;426(6968):789–96. doi: 10.1038/nature02168nature02168.CrossRefGoogle Scholar
  13. 13.
    Benjamini Y, Hochberg Y. Controlling the false discovery rate: a practical. J R Statist Soc B. 1995;57(1):289–300.Google Scholar
  14. 14.
    Gordon A, Glazko G, Qiu X, Yakovlev A. Control of the mean number of false discoveries, Bonferroni and stability of multiple testing. Ann Appl Stat. 2007;1:179–90.CrossRefGoogle Scholar
  15. 15.
    Khoury MJ, Bertram L, Boffetta P, Butterworth AS, Chanock SJ, Dolan SM, et al. Genome-wide association studies, field synopses, and the development of the knowledge base on genetic variation and human diseases. Am J Epidemiol. 2009;170(3):269–79. doi: 10.1093/aje/kwp119.PubMedCrossRefGoogle Scholar
  16. 16.
    Ridley J, Kolm N, Freckelton RP, Gage MJ. An unexpected influence of widely used significance thresholds on the distribution of reported p values. J Evol Biol. 2007;20(3):1082–9. doi: 10.1111/j.1420-9101.2006.01291.x.PubMedCrossRefGoogle Scholar
  17. 17.
    Bertram L, McQueen MB, Mullin K, Blacker D, Tanzi RE. Systematic meta-analyses of Alzheimer disease genetic association studies: the AlzGene database. Nat Genet. 2007;39(1):17–23. doi: 10.1038/ng1934.PubMedCrossRefGoogle Scholar
  18. 18.
    Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–60. doi: 10.1136/bmj.327.7414.557327/7414/557.PubMedCrossRefGoogle Scholar
  19. 19.
    DerSimonian R, Laird N. Meta-analysis in clinical trials. Control Clin Trials. 1986;7(3):177–88. doi: 0197-2456(86)90046-2.PubMedCrossRefGoogle Scholar
  20. 20.
    Harbord RM, Egger M, Sterne JA. A modified test for small-study effects in meta-analyses of controlled trials with binary endpoints. Stat Med. 2006;25(20):3443–57. doi: 10.1002/sim.2380.PubMedCrossRefGoogle Scholar
  21. 21.
    Kavvoura FK, McQueen MB, Khoury MJ, Tanzi RE, Bertram L, Ioannidis JP. Evaluation of the potential excess of statistically significant findings in published genetic association studies: application to Alzheimer’s disease. Am J Epidemiol. 2008;168(8):855–65. doi: 10.1093/aje/kwn206.PubMedCrossRefGoogle Scholar
  22. 22.
    Ioannidis JP. Calibration of credibility of agnostic genome-wide associations. Am J Med Genet B Neuropsychiatr Genet. 2008;147B(6):964–72. doi: 10.1002/ajmg.b.30721.PubMedCrossRefGoogle Scholar
  23. 23.
    Ioannidis JP, Tarone R, McLaughlin JK. The false-positive to false-negative ratio in epidemiologic studies. Epidemiology. 2011;22(4):450–6. doi: 10.1097/EDE.0b013e31821b506e.PubMedCrossRefGoogle Scholar
  24. 24.
    Rothman KJ. Epidemiology: an introduction. 1st ed. Oxford: Oxford University Press; 2002.Google Scholar
  25. 25.
    Panagiotou OA, Ioannidis JP, Genome-Wide Significance Project. What should the genome-wide significance threshold be? Empirical replication of borderline genetic associations. Int J Epidemiol. 2011;. doi: 10.1093/ije/dyr178.PubMedGoogle Scholar
  26. 26.
    Biotechnology Kaiser J. Researcher, two universities sued over validity of prostate cancer test. Science. 2009;325(5947):1484. doi: 10.1126/science.325_1484.CrossRefGoogle Scholar
  27. 27.
    van Duijn CM. STROBE-ME too! Eur J Epidemiol. 2011;26(10):761–2. doi: 10.1007/s10654-011-9628-8.PubMedCrossRefGoogle Scholar
  28. 28.
    Sanna S, Li B, Mulas A, Sidore C, Kang HM, Jackson AU, et al. Fine mapping of five Loci associated with low-density lipoprotein cholesterol detects variants that double the explained heritability. PLoS Genet. 2011;7(7):e1002198. doi: 10.1371/journal.pgen.1002198PGENETICS-D-11-00557.PubMedCrossRefGoogle Scholar
  29. 29.
    Boseley S. Six men in intensive care after drug trial goes wrong. The Guardian. 2006.Google Scholar
  30. 30.
    Farrer LA, Cupples LA, Haines JL, Hyman B, Kukull WA, Mayeux R, et al. Effects of age, sex, and ethnicity on the association between apolipoprotein E genotype and Alzheimer disease. A meta-analysis. APOE and Alzheimer disease meta analysis consortium. JAMA. 1997;278(16):1349–56.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2013

Authors and Affiliations

  • Linda Broer
    • 1
  • Christina M. Lill
    • 2
    • 3
  • Maaike Schuur
    • 4
  • Najaf Amin
    • 1
  • Johannes T. Roehr
    • 2
  • Lars Bertram
    • 2
  • John P. A. Ioannidis
    • 5
    • 6
    • 7
    • 8
  • Cornelia M. van Duijn
    • 1
  1. 1.Department of EpidemiologyErasmus University Medical Center RotterdamRotterdamThe Netherlands
  2. 2.Neuropsychiatric Genetics Group, Department of Vertebrate GenomicsMax Planck Institute for Molecular GeneticsBerlinGermany
  3. 3.Department of NeurologyUniversity Medical Center of the Johannes Gutenberg-UniversityMainzGermany
  4. 4.Department of NeurologyErasmus University Medical Center RotterdamRotterdamThe Netherlands
  5. 5.Department of MedicineStanford Prevention Research CenterStanfordUSA
  6. 6.Department of Health Research and PolicyStanford University School of MedicineStanfordUSA
  7. 7.Department of StatisticsStanford University School of Humanities and SciencesStanfordUSA
  8. 8.Department of Hygiene and EpidemiologyUniversity of Ioannina School of MedicineIoanninaGreece

Personalised recommendations