Human Genetics

, Volume 132, Issue 2, pp 167–178 | Cite as

Assessment of systematic effects of methodological characteristics on candidate genetic associations

  • Badr Aljasir
  • John P. A. Ioannidis
  • Alex Yurkiewich
  • David Moher
  • Julian P. T. Higgins
  • Paul Arora
  • Julian Little
Original Investigation


Candidate genetic association studies have been found to have a low replication rate in the past. Here, we aimed to assess whether aspects of reported methodological characteristics in genetic association studies may be related to the magnitude of effects observed. An observational, literature-based investigation of 511 case–control studies of genetic association studies indexed in 2007, was undertaken. Meta-regression analyses were used to assess the relationship between 23 reported methodological characteristics and the magnitude of genetic associations. The 511 studies had been conducted in 52 countries and were published in 220 journals (median impact factor 5.1). The multivariate meta-regression model of methodological characteristics plus disease category accounted for 17.2 % of the between-study variance in the magnitude of the reported genetic associations. Our findings are consistent with the view that better conducted and better reported genetic association research may lead to less inflated results.


Methodological Characteristic Wellcome Trust Case Control Consortium Significant Odds Ratio Genetic Epidemiology Study Report Sample Size 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



Diana Fox and Zahra Montazeri for administrative support. Knowledge Translation Branch, Canadian Institutes of Health Research (Grant Ref No: 200606KRS-162113-KRS-CECA-102805); Biotechnology, Genomics and Population Health Branch, Public Health Agency of Canada. National Guard Health Affairs, Saudi Arabia. Julian Little holds a Canada Research Chair in Human Genome Epidemiology. The funders had no role in the decision to submit the article or in its preparation.

Conflict of interest

The authors declare that they have no conflict of interest.

Supplementary material

439_2012_1237_MOESM1_ESM.docx (311 kb)
Supplementary material 1 (DOCX 311 kb)


  1. Attia J, Thakkinstian A, D’Este C (2003) Meta-analyses of molecular association studies: methodologic lessons for genetic epidemiology. J Clin Epidemiol 56:297–303PubMedCrossRefGoogle Scholar
  2. Bogardus ST Jr, Concato J, Feinstein AR (1999) Clinical epidemiological quality in molecular genetic research: the need for methodological standards. JAMA 281:1919–1926PubMedCrossRefGoogle Scholar
  3. Borenstein M, Hedges L, Higgins, JPT, Rothstein H (2009) Meta-Analysis, fixed effects vs. random effects. In: Introduction to Meta-analysis. J Wiley & Sons, Chichester, pp. 61–85. Available at Accessed 24 July 2012
  4. Brookmeyer R, Liang KY, Linet M (1986) Matched case-control designs and overmatched analyses. Am J Epidemiol 124:693–701PubMedGoogle Scholar
  5. Cardon LR, Bell JI (2001) Association study designs for complex diseases. Nat Rev Genet 2:91–99PubMedCrossRefGoogle Scholar
  6. Clark MF, Baudouin SV (2006) A systematic review of the quality of genetic association studies in human sepsis. Intensive Care Med 32:1706–1712PubMedCrossRefGoogle Scholar
  7. Committee on Assuring the Health of the Public in the 21st Century (2002) The Future of the Public’s Health in the 21st Century. Institute of Medicine, Washington DCGoogle Scholar
  8. Cooper DN, Nussbaum RL, Krawczak M (2002) Proposed guidelines for papers describing DNA polymorphism-disease associations. Hum Genet 110:207–208PubMedCrossRefGoogle Scholar
  9. Costanza MC (1995) Matching. Prev Med 24:425–433PubMedCrossRefGoogle Scholar
  10. Crow JF (1999) Hardy, Weinberg and language impediments. Genetics 152:821–825PubMedGoogle Scholar
  11. DeLisi LE, Faraone SV (2006) When is a “positive” association truly a “positive” in psychiatric genetics? A commentary based on issues debated at the World Congress of Psychiatric Genetics, Boston, October 12–18, 2005. Am J Med Genet B Neuropsychiatr Genet 141B:319–322PubMedCrossRefGoogle Scholar
  12. Dickersin K (2002) Systematic reviews in epidemiology: why are we so far behind? Int J Epidemiol 31:6–12PubMedCrossRefGoogle Scholar
  13. Dong LM, Potter JD, White E, Ulrich CM, Cardon LR, Peters U (2008) Genetic susceptibility to cancer: the role of polymorphisms in candidate genes. JAMA 299:2423–2436PubMedCrossRefGoogle Scholar
  14. Edwards AW (2008) G. H. Hardy (1908) and Hardy–Weinberg equilibrium. Genetics 179:1143–1150PubMedCrossRefGoogle Scholar
  15. European Bioinformatics Institute (2012) The European Genome-phenome Archive (EGA). Available at Accessed 1 June 2012
  16. GAIN Collaborative Research Group, Manolio TA, Rodriguez LL, Brooks L, Abecasis G, Collaborative Association Study of Psoriasis, Ballinger D, Daly M, Donnelly P, Faraone SV, International Multi-Center ADHD Genetics Project, Frazer K, Gabriel S, Gejman P, Molecular Genetics of Schizophrenia Collaboration, Guttmacher A, Harris EL, Insel T, Kelsoe JR, Bipolar Genome Study, Lander E, McCowin N, Mailman MD, Nabel E, Ostell J, Pugh E, Sherry S, Sullivan PF, Major Depression Stage 1 Genomewide Association in Population-Based Samples Study, Thompson JF, Warram J, Genetics of Kidneys in Diabetes (GoKinD) Study, Wholley D, Milos PM, Collins FS (2007) New models of collaboration in genome-wide association studies: the Genetic Association Information Network. Nat Genet 39:1045–1051Google Scholar
  17. Gambaro G, Anglani F, D’Angelo A (2000) Association studies of genetic polymorphisms and complex disease. Lancet 355:308–311PubMedCrossRefGoogle Scholar
  18. Garte S, Gaspari L, Alexandrie AK, Ambrosone C, Autrup H, Autrup JL, Baranova H, Bathum L, Benhamou S, Boffetta P, Bouchardy C, Breskvar K, Brockmoller J, Cascorbi I, Clapper ML, Coutelle C, Daly A, Dell’Omo M, Dolzan V, Dresler CM, Fryer A, Haugen A, Hein DW, Hildesheim A, Hirvonen A, Hsieh LL, Ingelman-Sundberg M, Kalina I, Kang D, Kihara M, Kiyohara C, Kremers P, Lazarus P, Le Marchand L, Lechner MC, van Lieshout EM, London S, Manni JJ, Maugard CM, Morita S, Nazar-Stewart V, Noda K, Oda Y, Parl FF, Pastorelli R, Persson I, Peters WH, Rannug A, Rebbeck T, Risch A, Roelandt L, Romkes M, Ryberg D, Salagovic J, Schoket B, Seidegard J, Shields PG, Sim E, Sinnet D, Strange RC, Stucker I, Sugimura H, To-Figueras J, Vineis P, Yu MC, Taioli E (2001) Metabolic gene polymorphism frequencies in control populations. Cancer Epidemiol Biomarkers Prev 10:1239–1248PubMedGoogle Scholar
  19. Gelernter J, Goldman D, Risch N (1993) The A1 allele at the D2 dopamine receptor gene and alcoholism. A reappraisal. JAMA 269:1673–1677PubMedCrossRefGoogle Scholar
  20. Genomics, Health and Society Working Group (2004) Genomics, Health and Society. Emerging Issues for Public Policy, Policy Research Initiative, Ottawa. Available at ISBN: 0-662-35154-1
  21. Gissler M, Hemminki E (1996) The danger of overmatching in studies of the perinatal mortality and birthweight of infants born after assisted conception. Eur J Obstet Gynecol Reprod Biol 69:73–75PubMedCrossRefGoogle Scholar
  22. Hall IP, Blakey JD (2005) Genetic association studies in Thorax. Thorax 60:357–359PubMedCrossRefGoogle Scholar
  23. Hegele RA (2002) SNP judgments and freedom of association. Arterioscler Thromb Vasc Biol 22:1058–1061PubMedCrossRefGoogle Scholar
  24. Higgins JP, Thompson SG (2002) Quantifying heterogeneity in a meta-analysis. Stat Med 21:1539–1558PubMedCrossRefGoogle Scholar
  25. Higgins JP, Thompson SG, Deeks JJ, Altman DG (2003) Measuring inconsistency in meta-analyses. BMJ 327:557–560PubMedCrossRefGoogle Scholar
  26. Hindorff LA, Sethupathy P, Junkins HA, Ramos EM, Mehta JP, Collins FS, Manolio TA (2009) Potential etiologic and functional implications of genome-wide association loci for human diseases and traits. Proc Natl Acad Sci USA 106:9362–9367PubMedCrossRefGoogle Scholar
  27. Hirschhorn JN, Lohmueller K, Byrne E, Hirschhorn K (2002) A comprehensive review of genetic association studies. Genet Med 4:45–61PubMedCrossRefGoogle Scholar
  28. Huizinga TW, Pisetsky DS, Kimberly RP (2004) Associations, populations, and the truth: recommendations for genetic association studies in Arthritis & Rheumatism. Arthritis Rheum 50:2066–2071PubMedCrossRefGoogle Scholar
  29. Ioannidis JP (2003) Genetic associations: false or true? Trends Mol Med 9:135–138PubMedCrossRefGoogle Scholar
  30. Ioannidis JP (2005a) Contradicted and initially stronger effects in highly cited clinical research. JAMA 294:218–228PubMedCrossRefGoogle Scholar
  31. Ioannidis JP (2005b) Why most published research findings are false. PLoS Med 2:e124PubMedCrossRefGoogle Scholar
  32. Ioannidis JP (2008) Why most discovered true associations are inflated. Epidemiology 19:640–648PubMedCrossRefGoogle Scholar
  33. Ioannidis JP, Trikalinos TA (2005) Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J Clin Epidemiol 58:543–549PubMedCrossRefGoogle Scholar
  34. Ioannidis JP, Ntzani EE, Trikalinos TA, Contopoulos-Ioannidis DG (2001) Replication validity of genetic association studies. Nat Genet 29:306–309PubMedCrossRefGoogle Scholar
  35. Ioannidis JP, Trikalinos TA, Ntzani EE, Contopoulos-Ioannidis DG (2003) Genetic associations in large versus small studies: an empirical assessment. Lancet 361:567–571PubMedCrossRefGoogle Scholar
  36. Ioannidis JP, Gwinn M, Little J, Higgins JP, Bernstein JL, Boffetta P, Bondy M, Bray MS, Brenchley PE, Buffler PA, Casas JP, Chokkalingam A, Danesh J, Smith GD, Dolan S, Duncan R, Gruis NA, Hartge P, Hashibe M, Hunter DJ, Jarvelin MR, Malmer B, Maraganore DM, Newton-Bishop JA, O’Brien TR, Petersen G, Riboli E, Salanti G, Seminara D, Smeeth L, Taioli E, Timpson N, Uitterlinden AG, Vineis P, Wareham N, Winn DM, Zimmern R, Khoury MJ, Human Genome Epidemiology Network and the Network of Investigator Networks (2006) A road map for efficient and reliable human genome epidemiology. Nat Genet 38:3–5Google Scholar
  37. Ioannidis JP, Thomas G, Daly MJ (2009) Validating, augmenting and refining genome-wide association signals. Nat Rev Genet 10:318–329PubMedCrossRefGoogle Scholar
  38. Ioannidis JP, Castaldi P, Evangelou E (2010) A compendium of genome-wide associations for cancer: critical synopsis and reappraisal. J Natl Cancer Inst 102:846–858PubMedCrossRefGoogle Scholar
  39. Ioannidis JP, Tarone R, McLaughlin JK (2011) The false-positive to false-negative ratio in epidemiologic studies. Epidemiology 22:450–456PubMedCrossRefGoogle Scholar
  40. Khoury MJ, Millikan R, Little J, Gwinn M (2004) The emergence of epidemiology in the genomics age. Int J Epidemiol 33:936–944PubMedCrossRefGoogle Scholar
  41. Khoury MJ, Davis R, Gwinn M, Lindegren ML, Yoon P (2005) Do we need genomic research for the prevention of common diseases with environmental causes? Am J Epidemiol 161:799–805PubMedCrossRefGoogle Scholar
  42. Khoury MJ, Gwinn M, Burke W, Bowen S, Zimmern R (2007) Will genomics widen or help heal the schism between medicine and public health? Am J Prev Med 33:310–317PubMedCrossRefGoogle Scholar
  43. Khoury MJ, Bertram L, Boffetta P, Butterworth AS, Chanock SJ, Dolan SM, Fortier I, Garcia-Closas M, Gwinn M, Higgins JP, Janssens AC, Ostell J, Owen RP, Pagon RA, Rebbeck TR, Rothman N, Bernstein JL, Burton PR, Campbell H, Chockalingam A, Furberg H, Little J, O’Brien TR, Seminara D, Vineis P, Winn DM, Yu W, Ioannidis JP (2009) Genome-wide association studies, field synopses, and the development of the knowledge base on genetic variation and human diseases. Am J Epidemiol 170:269–279PubMedCrossRefGoogle Scholar
  44. Lancet editorial (2003) In search of genetic precision. Lancet 361:357Google Scholar
  45. Lavedan C, Birznieks G, Dressman M, McCullough K, Paczkowski R, Torres R, Wolfgang C, Polymeropoulos M (2004) Translating the genome into individualized therapeutics. Drug Dev Res 62:371–382CrossRefGoogle Scholar
  46. Lawrence RW, Evans DM, Cardon LR (2005) Prospects and pitfalls in whole genome association studies. Philos Trans R Soc Lond B Biol Sci 360:1589–1595PubMedCrossRefGoogle Scholar
  47. Lee W, Bindman J, Ford T, Glozier N, Moran P, Stewart R, Hotopf M (2007) Bias in psychiatric case-control studies: literature survey. Br J Psychiatry 190:204–209PubMedCrossRefGoogle Scholar
  48. Little J (2004) Reporting and review of human genome epidemiology studies. In: Khoury MJ, Little J, Burke W (eds) Human genome epidemiology: a scientific foundation for using genetic information to improve health and prevent disease. Oxford University Press, New York, pp 168–192Google Scholar
  49. Little J, Higgins JPT (eds) (2006) The HuGENet™ HuGE Review Handbook, version 1.0. Available at Accessed 20 July 2011
  50. Little J, Bradley L, Bray MS, Clyne M, Dorman J, Ellsworth DL, Hanson J, Khoury M, Lau J, O’Brien TR, Rothman N, Stroup D, Taioli E, Thomas D, Vainio H, Wacholder S, Weinberg C (2002) Reporting, appraising, and integrating data on genotype prevalence and gene-disease associations. Am J Epidemiol 156:300–310PubMedCrossRefGoogle Scholar
  51. Little J, Higgins JP, Ioannidis JP, Moher D, Gagnon F, von Elm E, Khoury MJ, Cohen B, Davey-Smith G, Grimshaw J, Scheet P, Gwinn M, Williamson RE, Zou GY, Hutchings K, Johnson CY, Tait V, Wiens M, Golding J, van Duijn C, McLaughlin J, Paterson A, Wells G, Fortier I, Freedman M, Zecevic M, King R, Infante-Rivard C, Stewart A, Birkett N (2009) Strengthening the reporting of genetic association studies (STREGA): an extension of the STROBE Statement. Hum Genet 125:131–151PubMedCrossRefGoogle Scholar
  52. Lohmueller KE, Pearce CL, Pike M, Lander ES, Hirschhorn JN (2003) Meta-analysis of genetic association studies supports a contribution of common variants to susceptibility to common disease. Nat Genet 33:177–182PubMedCrossRefGoogle Scholar
  53. Mackay J, Taylor A (2006) Moving genetics into clinical cancer care: examples from BRCA gene testing and telemedicine. Breast 15(Suppl 2):S65–S70PubMedCrossRefGoogle Scholar
  54. Mailman MD, Feolo M, Jin Y, Kimura M, Tryka K, Bagoutdinov R, Hao L, Kiang A, Paschall J, Phan L, Popova N, Pretel S, Ziyabari L, Lee M, Shao Y, Wang ZY, Sirotkin K, Ward M, Kholodov M, Zbicz K, Beck J, Kimelman M, Shevelev S, Preuss D, Yaschenko E, Graeff A, Ostell J, Sherry ST (2007) The NCBI dbGaP database of genotypes and phenotypes. Nat Genet 39:1181–1186PubMedCrossRefGoogle Scholar
  55. Manolio TA, Brooks LD, Collins FS (2008) A HapMap harvest of insights into the genetics of common disease. J Clin Invest 118:1590–1605PubMedCrossRefGoogle Scholar
  56. Mayo O (2008) A century of Hardy–Weinberg equilibrium. Twin Res Hum Genet 11:249–256PubMedCrossRefGoogle Scholar
  57. McCarthy MI, Abecasis GR, Cardon LR, Goldstein DB, Little J, Ioannidis JP, Hirschhorn JN (2008) Genome-wide association studies for complex traits: consensus, uncertainty and challenges. Nat Rev Genet 9:356–369PubMedCrossRefGoogle Scholar
  58. Moher D, Pham B, Klassen TP, Schulz KF, Berlin JA, Jadad AR, Liberati A (2000) What contributions do languages other than English make on the results of meta-analyses? J Clin Epidemiol 53:964–972PubMedCrossRefGoogle Scholar
  59. Nat. Genet editorial (1999) Freely associating. Nat Genet 22:1–2Google Scholar
  60. National Institutes of Health (NIH) (2008) NIH Notice on Development of Data Sharing Policy for Sequence and Related Genomic Data. Available at Accessed 31 May 2012
  61. National Institutes of Health (NIH) (2009) Policy for Sharing of Data Obtained in NIH-Supported or Conducted Genome-Wide Association Studies. Available at Accessed 31 May 2012
  62. Pan Z, Trikalinos TA, Kavvoura FK, Lau J, Ioannidis JP (2005) Local literature bias in genetic epidemiology: an empirical evaluation of the Chinese literature. PLoS Med 2:e334PubMedCrossRefGoogle Scholar
  63. Patsopoulos NA, Analatos AA, Ioannidis JP (2005) Relative citation impact of various study designs in the health sciences. JAMA 293:2362–2366PubMedCrossRefGoogle Scholar
  64. Peters DL, Barber RC, Flood EM, Garner HR, O’Keefe GE (2003) Methodologic quality and genotyping reproducibility in studies of tumor necrosis factor -308 G–>A single nucleotide polymorphism and bacterial sepsis: implications for studies of complex traits. Crit Care Med 31:1691–1696PubMedCrossRefGoogle Scholar
  65. Peterson HB, Kleinbaum DG (1991) Interpreting the literature in obstetrics and gynecology: I. Key concepts in epidemiology and biostatistics. Obstet Gynecol 78:710–717PubMedGoogle Scholar
  66. Pham B, Klassen TP, Lawson ML, Moher D (2005) Language of publication restrictions in systematic reviews gave different results depending on whether the intervention was conventional or complementary. J Clin Epidemiol 58:769–776PubMedCrossRefGoogle Scholar
  67. Rebbeck TR, Martinez ME, Sellers TA, Shields PG, Wild CP, Potter JD (2004) Genetic variation and cancer: improving the environment for publication of association studies. Cancer Epidemiol Biomarkers Prev 13:1985–1986PubMedGoogle Scholar
  68. Rosenthal R, DiMatteo MR (2001) Meta-analysis: recent developments in quantitative methods for literature reviews. Annu Rev Psychol 52:59–82PubMedCrossRefGoogle Scholar
  69. Salanti G, Amountza G, Ntzani EE, Ioannidis JP (2005) Hardy–Weinberg equilibrium in genetic association studies: an empirical evaluation of reporting, deviations, and power. Eur J Hum Genet 13:840–848PubMedCrossRefGoogle Scholar
  70. Shastry BS (2005) Genetic diversity and new therapeutic concepts. J Hum Genet 50:321–328PubMedCrossRefGoogle Scholar
  71. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, Rennie D, Moher D, Becker BJ, Sipe TA, Thacker SB (2000) Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis of observational studies in epidemiology (MOOSE) group. JAMA 283:2008–2012PubMedCrossRefGoogle Scholar
  72. Sutton AJ, Higgins JP (2008) Recent developments in meta-analysis. Stat Med 27:625–650PubMedCrossRefGoogle Scholar
  73. Tabor HK, Risch NJ, Myers RM (2002) Candidate-gene approaches for studying complex genetic traits: practical considerations. Nat Rev Genet 3:391–397PubMedCrossRefGoogle Scholar
  74. Tan NC, Mulley JC, Berkovic SF (2004) Genetic association studies in epilepsy: “the truth is out there”. Epilepsia 45:1429–1442PubMedCrossRefGoogle Scholar
  75. Thomas DC, Haile RW, Duggan D (2005) Recent developments in genomewide association scans: a workshop summary and review. Am J Hum Genet 77:337–345PubMedCrossRefGoogle Scholar
  76. Trikalinos TA, Salanti G, Khoury MJ, Ioannidis JP (2006) Impact of violations and deviations in Hardy–Weinberg equilibrium on postulated gene-disease associations. Am J Epidemiol 163:300–309PubMedCrossRefGoogle Scholar
  77. Trikalinos TA, Salanti G, Zintzaras E, Ioannidis JP (2008) Meta-analysis methods. Adv Genet 60:311–334PubMedCrossRefGoogle Scholar
  78. Van Houwelingen HC, Zwinderman KH, Stijnen T (1993) A bivariate approach to meta-analysis. Stat Med 12:2273–2284PubMedCrossRefGoogle Scholar
  79. Van Houwelingen HC, Arends LR, Stijnen T (2002) Advanced methods in meta-analysis: multivariate approach and meta-regression. Stat Med 21:589–624PubMedCrossRefGoogle Scholar
  80. Vickers A, Goyal N, Harland R, Rees R (1998) Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials 19:159–166PubMedCrossRefGoogle Scholar
  81. Visscher PM, Brown MA, McCarthy MI, Yang J (2012) Five years of GWAS discovery. Am J Hum Genet 90:7–24PubMedCrossRefGoogle Scholar
  82. Wedzicha JN, Hall IP (2005) Genetic association studies in Thorax. Thorax 60:357CrossRefGoogle Scholar
  83. Wellcome Trust Case Control Consortium (2007) Genome-wide association study of 14,000 cases of seven common diseases and 3,000 shared controls. Nature 447:661–678CrossRefGoogle Scholar
  84. Yesupriya A, Evangelou E, Kavvoura FK, Patsopoulos NA, Clyne M, Walsh MC, Lin BK, Yu W, Gwinn M, Ioannidis JP, Khoury MJ (2008) Reporting of human genome epidemiology (HuGE) association studies: an empirical assessment. BMC Med Res Methodol 8:31PubMedCrossRefGoogle Scholar
  85. Yu W, Yesupriya A, Clyne M, Wulf A, Gwinn M, Khoury MJ (2007) HuGE Literature Finder. HuGE Navigator. Available at: Accessed 28 Dec 2007
  86. Yu W, Gwinn M, Clyne M, Yesupriya A, Khoury MJ (2008) A navigator for human genome epidemiology. Nat Genet 40:124–125PubMedCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Badr Aljasir
    • 1
    • 2
  • John P. A. Ioannidis
    • 3
    • 4
    • 5
  • Alex Yurkiewich
    • 1
  • David Moher
    • 1
    • 6
  • Julian P. T. Higgins
    • 7
  • Paul Arora
    • 8
  • Julian Little
    • 1
  1. 1.Department of Epidemiology and Community MedicineUniversity of OttawaOttawaCanada
  2. 2.National Guard Health AffairsWestern RegionJeddahSaudi Arabia
  3. 3.Department of Medicine, Stanford Prevention Research CenterStanford University School of MedicineStanfordUSA
  4. 4.Department of Health Research and PolicyStanford University School of MedicineStanfordUSA
  5. 5.Department of StatisticsStanford University School of Humanities and SciencesStanfordUSA
  6. 6.Ottawa Hospital Research InstituteOttawaCanada
  7. 7.MRC Biostatistics UnitCambridgeUK
  8. 8.Office for Biotechnology, Genomics and Population HealthPublic Health Agency of CanadaTorontoCanada

Personalised recommendations