Journal of Computer-Aided Molecular Design

, Volume 30, Issue 9, pp 651–668 | Cite as

D3R grand challenge 2015: Evaluation of protein–ligand pose and affinity predictions

  • Symon Gathiaka
  • Shuai Liu
  • Michael Chiu
  • Huanwang Yang
  • Jeanne A. Stuckey
  • You Na Kang
  • Jim Delproposto
  • Ginger Kubish
  • James B. DunbarJr.
  • Heather A. Carlson
  • Stephen K. Burley
  • W. Patrick Walters
  • Rommie E. Amaro
  • Victoria A. Feher
  • Michael K. Gilson


The Drug Design Data Resource (D3R) ran Grand Challenge 2015 between September 2015 and February 2016. Two targets served as the framework to test community docking and scoring methods: (1) HSP90, donated by AbbVie and the Community Structure Activity Resource (CSAR), and (2) MAP4K4, donated by Genentech. The challenges for both target datasets were conducted in two stages, with the first stage testing pose predictions and the capacity to rank compounds by affinity with minimal structural data; and the second stage testing methods for ranking compounds with knowledge of at least a subset of the ligand–protein poses. An additional sub-challenge provided small groups of chemically similar HSP90 compounds amenable to alchemical calculations of relative binding free energy. Unlike previous blinded Challenges, we did not provide cognate receptors or receptors prepared with hydrogens and likewise did not require a specified crystal structure to be used for pose or affinity prediction in Stage 1. Given the freedom to select from over 200 crystal structures of HSP90 in the PDB, participants employed workflows that tested not only core docking and scoring technologies, but also methods for addressing water-mediated ligand–protein interactions, binding pocket flexibility, and the optimal selection of protein structures for use in docking calculations. Nearly 40 participating groups submitted over 350 prediction sets for Grand Challenge 2015. This overview describes the datasets and the organization of the challenge components, summarizes the results across all submitted predictions, and considers broad conclusions that may be drawn from this collaborative community endeavor.


D3R Docking Scoring Free energy Ligand Protein 



This work was supported by National Institutes of Health (NIH) grant 1U01GM111528 for the Drug Design Data Resource (D3R) and U01 GM086873 to the Community Structure Activity Resource (CSAR). We are grateful to Dr. Seth Harris and to Genentech, Inc. for their collaboration and generous donation of the MAP4K4 dataset and Dr. Phil Hadjuk of Abbvie for the HSP90 dataset. We also thank OpenEye Scientific Software for generously donating the use of their software and Jenny Chong for her assistance in preparation of figures for the manuscript. The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. MKG has an equity interest in, and is a co-founder and scientific advisor of, VeraChem LLC. REA has equity interest in and is a co-founder of Actavalon, Inc., VAF has equity interest in Actavalon, Inc.

Supplementary material

10822_2016_9946_MOESM1_ESM.docx (3.2 mb)
Supplementary material 1 (DOCX 3274 kb)
10822_2016_9946_MOESM2_ESM.docx (130 kb)
Supplementary material 2 (DOCX 130 kb)
10822_2016_9946_MOESM3_ESM.docx (31 kb)
Supplementary material 3 (DOCX 31 kb)
10822_2016_9946_MOESM4_ESM.xlsx (234 kb)
Supplementary material 4 (XLSX 234 kb)
10822_2016_9946_MOESM5_ESM.docx (1.1 mb)
Supplementary material 5 (DOCX 1149 kb)
10822_2016_9946_MOESM6_ESM.xlsx (9 kb)
Supplementary material 6 (XLSX 9 kb)
10822_2016_9946_MOESM7_ESM.xlsx (11 kb)
Supplementary material 7 (XLSX 11 kb)


  1. 1.
    Smith RD, Dunbar JB Jr, Ung PM et al (2011) CSAR benchmark exercise of 2010: combined evaluation across all submitted scoring functions. J Chem Inf Model 51:2115–2131. doi: 10.1021/ci200269q CrossRefGoogle Scholar
  2. 2.
    Damm-Ganamet KL, Smith RD, Dunbar JB Jr et al (2013) CSAR benchmark exercise 2011–2012: evaluation of results from docking and relative ranking of blinded congeneric series. J Chem Inf Model 53:1853–1870. doi: 10.1021/ci400025f CrossRefGoogle Scholar
  3. 3.
    Smith RD, Damm-Ganamet KL, Dunbar JB Jr et al (2016) CSAR benchmark exercise 2013: evaluation of results from a combined computational protein design, docking, and scoring/ranking challenge. J Chem Inf Model 56:1022–1031. doi: 10.1021/acs.jcim.5b00387 CrossRefGoogle Scholar
  4. 4.
    Carlson HA, Smith RD, Damm-Ganamet KL et al (2016) CSAR 2014: a benchmark exercise using unpublished data from pharma. J Chem Inf Model 56:1063–1077. doi: 10.1021/acs.jcim.5b00523
  5. 5.
    Solit DB, Rosen N (2006) Hsp90: a novel target for cancer therapy. Curr Top Med Chem 6:1205–1214CrossRefGoogle Scholar
  6. 6.
    Travers J, Sharp S, Workman P (2012) HSP90 inhibition: two-pronged exploitation of cancer dependencies. Drug Discov Today 17:242–252. doi: 10.1016/j.drudis.2011.12.021 CrossRefGoogle Scholar
  7. 7.
    Crawford TD, Ndubaku CO, Chen H et al (2014) Discovery of selective 4-amino-pyridopyrimidine inhibitors of MAP4K4 using fragment-based lead identification and optimization. J Med Chem 57:3484–3493. doi: 10.1021/jm500155b CrossRefGoogle Scholar
  8. 8.
    Pearl LH, Prodromou C (2006) Structure and mechanism of the Hsp90 molecular chaperone machinery. Annu Rev Biochem 75:271–294. doi: 10.1146/annurev.biochem.75.103004.142738 CrossRefGoogle Scholar
  9. 9.
    Tembe BL, McCammon JA (1984) Ligand-receptor interactions. Comput Chem 8:281–283. doi: 10.1016/0097-8485(84)85020-2 CrossRefGoogle Scholar
  10. 10.
    Ndubaku CO, Crawford TD, Chen H et al (2015) Structure-based design of GNE-495, a potent and selective MAP4K4 inhibitor with efficacy in retinal angiogenesis. ACS Med Chem Lett 6:913–918. doi: 10.1021/acsmedchemlett.5b00174 CrossRefGoogle Scholar
  11. 11.
    Schroder P, Forster T, Kleine S et al (2015) Neuritogenic militarinone-inspired 4-hydroxypyridones target the stress pathway kinase MAP4K4. AngewChemIntEdEngl 54:12398–12403Google Scholar
  12. 12.
    Wang L, Stanley M, Boggs J et al (2014) Fragment-based identification and optimization of a class of potent pyrrolo[2,1-f][1, 2, 4]triazine MAP4K4 inhibitors. BioorgMedChemLett 24:4546–4552Google Scholar
  13. 13.
    Guimaraes CR, Rai BK, Munchhof MJ et al (2011) Understanding the impact of the P-loop conformation on kinase selectivity. J Chem Inf Model 51:1199–1204. doi: 10.1021/ci200153c CrossRefGoogle Scholar
  14. 14.
    Huth JR, Park C, Petros AM et al (2007) Discovery and design of novel HSP90 inhibitors using multiple fragment-based design strategies. Chem Biol Drug Des 70:1–12. doi: 10.1111/j.1747-0285.2007.00535.x CrossRefGoogle Scholar
  15. 15.
    Dunbar JB Jr, Smith RD, Damm-Ganamet KL et al (2013) CSAR data set release 2012: ligands, affinities, complexes, and docking decoys. J Chem Inf Model 53:1842–1852. doi: 10.1021/ci4000486 CrossRefGoogle Scholar
  16. 16.
    Bruncko M, Tahir SK, Song X et al (2010) N-aryl-benzimidazolones as novel small molecule HSP90 inhibitors. Bioorg Med Chem Lett 20:7503–7506. doi: 10.1016/j.bmcl.2010.10.010 CrossRefGoogle Scholar
  17. 17.
    Wang L, Stanley M, Boggs JW et al (2014) Fragment-based identification and optimization of a class of potent pyrrolo[2,1-f][1, 2, 4]triazine MAP4K4 inhibitors. Bioorg Med Chem Lett 24:4546–4552. doi: 10.1016/j.bmcl.2014.07.071 CrossRefGoogle Scholar
  18. 18.
    Molecular Operating Environment (MOE) 2013.08; Chemical Computing Group Inc., 1010 Sherbooke St. West, Suite #910, Montreal, QC, Canada, H3A 2R7, 2016.Google Scholar
  19. 19.
    Read RJ, Adams PD, Arendall WB 3rd et al (2011) A new generation of crystallographic validation tools for the protein data bank. Structure 19:1395–1412. doi: 10.1016/j.str.2011.08.006 CrossRefGoogle Scholar
  20. 20.
    Yang H, Peisach E, Westbrook JD et al (2016) DCC: a Swiss army knife for structure factor analysis and validation. J Appl Crystallogr 49:1081–1084. doi: 10.1107/S1600576716004428 CrossRefGoogle Scholar
  21. 21.
    OpenEye Toolkits 2016. Jun.1 OpenEye Scientific Software, Santa Fe, NM.
  22. 22.
    Mobley DL, Liu S, Lim NM et al (2014) Blind prediction of HIV integrase binding from the SAMPL4 challenge. J Comput Aided Mol Des 28:327–345. doi: 10.1007/s10822-014-9723-5 CrossRefGoogle Scholar
  23. 23.
    Brown SP, Muchmore SW, Hajduk PJ (2009) Healthy skepticism: assessing realistic model performance. Drug Discov Today 14:420–427CrossRefGoogle Scholar
  24. 24.
    Antosiewicz J, McCammon JA, Gilson MK (1994) Prediction of Ph-dependent properties of proteins. J Mol Biol 238:415–436. doi: 10.1006/jmbi.1994.1301 CrossRefGoogle Scholar
  25. 25.
    Klopman G, Li J-Y, Wang S, Dimayuga M (1994) Computer automated log P calculations based on an extended group contribution approach. Computer automated log P calculations based on an extended group contribution approach. J Chem Inf Comput Sci. 34:752–781. doi: 10.1021/ci00020a009
  26. 26.
    Ekins S, Bunin BA (2013) The Collaborative Drug Discovery (CDD) database. Methods Mol Biol 993:139–154. doi: 10.1007/978-1-62703-342-8_10 CrossRefGoogle Scholar
  27. 27.
    Yung-Chi C, Prusoff WH (1973) Relationship between the inhibition constant (KI) and the concentration of inhibitor which causes 50 per cent inhibition (I50) of an enzymatic reaction. Biochem Pharmacol 22:3099–3108. doi: 10.1016/0006-2952(73)90196-2 CrossRefGoogle Scholar
  28. 28.
    Muddana HS, Fenley AT, Mobley DL, Gilson MK (2014) The SAMPL4 host-guest blind prediction challenge: an overview. J Comput Aided Mol Des 28:305–317. doi: 10.1007/s10822-014-9735-1 CrossRefGoogle Scholar
  29. 29.
    Papadatos G, Gaulton A, Hersey A, Overington JP (2015) Activity, assay and target data curation and quality in the ChEMBL database. J Comput Aided Mol Des 29:885–896. doi: 10.1007/s10822-015-9860-5 CrossRefGoogle Scholar
  30. 30.
    Hawkins PC, Skillman AG, Warren GL et al (2010) Conformer generation with OMEGA: algorithm and validation using high quality structures from the Protein Databank and Cambridge Structural Database. J Chem Inf Model 50:572–584. doi: 10.1021/ci100031x CrossRefGoogle Scholar
  31. 31.
    Kumar A, Zhang KYJ (2016) A pose prediction approach based on ligand 3D shape similarity. J Comput Aided Mol Des 30:457–469. doi: 10.1007/s10822-016-9923-2 CrossRefGoogle Scholar
  32. 32.
    Kumar A, Zhang KYJ (2016) Prospective evaluation of shape similarity based pose prediction method in D3R Grand Challenge 2015. J Comput Aided Mol Des. doi: 10.1007/s10822-016-9931-2 Google Scholar
  33. 33.
    Rush TS 3rd, Grant JA, Mosyak L, Nicholls A (2005) A shape-based 3-D scaffold hopping method and its application to a bacterial protein–protein interaction. J Med Chem 48:1489–1495. doi: 10.1021/jm040163o CrossRefGoogle Scholar
  34. 34.
    Cozzetto D, Tramontano A (2005) Relationship between multiple sequence alignments and quality of protein comparative models. Proteins 58:151–157. doi: 10.1002/prot.20284 CrossRefGoogle Scholar
  35. 35.
    Kelley BP, Brown SP, Warren GL, Muchmore SW (2015) POSIT: flexible shape-guided docking for pose prediction. J Chem Inf Model. 55:1771-80. doi: 10.1021/acs.jcim.5b00142
  36. 36.
    Kim R, Skolnick J (2008) Assessment of programs for ligand binding affinity prediction. J Comput Chem 29:1316–1331. doi: 10.1002/jcc.20893 CrossRefGoogle Scholar
  37. 37.
    Gumbart JC, Roux B, Chipot C (2013) Efficient determination of protein–protein standard binding free energies from first principles. J Chem Theory Comput 9:3789–3798. doi: 10.1021/ct400273t CrossRefGoogle Scholar
  38. 38.
    Gumbart JC, Roux B, Chipot C (2013) Standard binding free energies from computer simulations: what is the best strategy? J Chem Theory Comput 9:794–802. doi: 10.1021/ct3008099 CrossRefGoogle Scholar
  39. 39.
    Gooljarsingh LT, Fernandes C, Yan K et al (2006) A biochemical rationale for the anticancer effects of Hsp90 inhibitors: slow, tight binding inhibition by geldanamycin and its analogues. Proc Natl Acad Sci 103:7625–7630. doi: 10.1073/pnas.0602650103 CrossRefGoogle Scholar
  40. 40.
    Gaulton A, Bellis LJ, Bento AP et al (2012) ChEMBL: a large-scale bioactivity database for drug discovery. Nucleic Acids Res 40:D1100–D1107. doi: 10.1093/nar/gkr777 CrossRefGoogle Scholar
  41. 41.
    Liu T, Lin Y, Wen X et al (2007) BindingDB: a web-accessible database of experimentally determined protein–ligand binding affinities. Nucleic Acids Res 35:D198–D201. doi: 10.1093/nar/gkl999 CrossRefGoogle Scholar
  42. 42.
    Kim S, Thiessen PA, Bolton EE et al (2016) PubChem substance and compound databases. Nucleic Acids Res 44:D1202–D1213. doi: 10.1093/nar/gkv951 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Symon Gathiaka
    • 1
  • Shuai Liu
    • 1
  • Michael Chiu
    • 1
  • Huanwang Yang
    • 2
  • Jeanne A. Stuckey
    • 3
  • You Na Kang
    • 3
  • Jim Delproposto
    • 3
  • Ginger Kubish
    • 3
  • James B. DunbarJr.
    • 4
  • Heather A. Carlson
    • 4
  • Stephen K. Burley
    • 2
    • 5
    • 6
  • W. Patrick Walters
    • 7
  • Rommie E. Amaro
    • 1
    • 8
  • Victoria A. Feher
    • 1
    • 8
    • 9
  • Michael K. Gilson
    • 1
    • 5
    • 8
  1. 1.Drug Design Data Resource, Center for Research in Biological SystemsUniversity of California, San DiegoLa JollaUSA
  2. 2.RCSB Protein Data Bank, Center for Integrative Proteomics Research, Institute for Quantitative Biomedicine, Department Chemistry and Chemical Biology, RutgersThe State University of New JerseyPiscatawayUSA
  3. 3.Center for Structural Biology, Life Sciences InstituteUniversity of MichiganAnn ArborUSA
  4. 4.Department of Medicinal ChemistryUniversity of MichiganAnn ArborUSA
  5. 5.Department of Pharmacy, Skaggs School of Pharmacy and Pharmaceutical SciencesUniversity of California, San DiegoLa JollaUSA
  6. 6.San Diego Supercomputer CenterUniversity of California, San DiegoLa JollaUSA
  7. 7.Relay TherapeuticsCambridgeUSA
  8. 8.Department of Chemistry and BiochemistryUniversity of California, San DiegoLa JollaUSA
  9. 9.Schrodinger, Inc.New YorkUSA

Personalised recommendations