Skip to main content
SpringerLink
Log in

Evaluation of the performance of 3D virtual screening protocols: RMSD comparisons, enrichment assessments, and decoy selection—What can we learn from earlier mistakes?

  • Published:
Journal of Computer-Aided Molecular Design Aims and scope Submit manuscript

Abstract

Within the last few years a considerable amount of evaluative studies has been published that investigate the performance of 3D virtual screening approaches. Thereby, in particular assessments of protein–ligand docking are facing remarkable interest in the scientific community. However, comparing virtual screening approaches is a non-trivial task. Several publications, especially in the field of molecular docking, suffer from shortcomings that are likely to affect the significance of the results considerably. These quality issues often arise from poor study design, biasing, by using improper or inexpressive enrichment descriptors, and from errors in interpretation of the data output. In this review we analyze recent literature evaluating 3D virtual screening methods, with focus on molecular docking. We highlight problematic issues and provide guidelines on how to improve the quality of computational studies. Since 3D virtual screening protocols are in general assessed by their ability to discriminate between active and inactive compounds, we summarize the impact of the composition and preparation of test sets on the outcome of evaluations. Moreover, we investigate the significance of both classic enrichment parameters and advanced descriptors for the performance of 3D virtual screening methods. Furthermore, we review the significance and suitability of RMSD as a measure for the accuracy of protein–ligand docking algorithms and of conformational space sub sampling algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Klebe G (2006) Drug Discovery Today 11:580–594

    Article  CAS  Google Scholar 

  2. Schneider G, Bohm H-J (2002) Drug Discovery Today 7:64–70

    CAS  Google Scholar 

  3. Chen H, Lyne PD, Giordanetto F, Lovell T, Li J (2006) J Chem Inf Model 46:401–415

    Article  CAS  Google Scholar 

  4. Cole JC, Murray CW, Nissink JWM, Taylor RD, Taylor R (2005) Proteins: Struct Funct Bioinf 60:325–332

    Article  CAS  Google Scholar 

  5. Kontoyianni M, McClellan LM, Sokol GS (2004) J Med Chem 47:558–565

    Article  CAS  Google Scholar 

  6. Steindl TM, Schuster D, Wolber G, Laggner C, Langer T (2006) J Comput Aided Mol Des 20:703–715

    Article  CAS  Google Scholar 

  7. Steindl TM, Schuster D, Laggner C, Chuang K, Hoffmann RD, Langer T (2007) J Chem Inf Model 47:563–571

    Article  CAS  Google Scholar 

  8. Steindl TM, Schuster D, Laggner C, Langer T (2006) J Chem Inf Model 46:2146–2157

    Article  CAS  Google Scholar 

  9. Sheng C, Zhang W, Ji H, Zhang M, Song Y, Xu H, Zhu J, Miao Z, Jiang Q, Yao J, Zhou Y, Zhu J, Lue J (2006) J Med Chem 49:2512–2525

    Article  CAS  Google Scholar 

  10. Güner OF (2000) Pharmacophore perception, development and use in drug design. International University Line, La Jolla, CA

    Google Scholar 

  11. Catalyst; Accelrys: San Diego, CA

  12. Verdonk ML, Berdini V, Hartshorn MJ, Mooij WTM, Murray CW, Taylor RD, Watson P (2004) J Chem Inf Comput Sci 44:793–806

    Article  CAS  Google Scholar 

  13. Weber L (2005) QSAR Comb Sci 24:809–823

    Article  CAS  Google Scholar 

  14. Wolber G, Langer T (2000) 13th European symposium on quantitative structure-activity relationships, Duesseldorf, Germany, Aug 27 - Sept 1, 2000

  15. Schueller A, Haehnke V, Schneider G (2007) QSAR Comb Sci 26:407–410

    Article  CAS  Google Scholar 

  16. Huang N, Shoichet BK, Irwin JJ (2006) J Med Chem 49:6789–6801

    Article  CAS  Google Scholar 

  17. Pan Y, Huang N, Cho S, MacKerell AD Jr (2003) J Chem Inf Comput Sci 43:267–272

    Article  CAS  Google Scholar 

  18. Jones G, Willett P, Glen RC (1995) J Mol Biol 245:43–53

    Article  CAS  Google Scholar 

  19. Jones G, Willett P, Glen RC, Leach AR, Taylor R (1997) J Mol Biol 267:727–748

    Article  CAS  Google Scholar 

  20. Irwin JJ, Shoichet BK (2005) J Chem Inf Model 45:177–182

    Article  CAS  Google Scholar 

  21. Edwards BS, Bologa C, Young SM, Balakin KV, Prossnitz ER, Savchuck NP, Sklar LA, Oprea TI (2005) Mol Pharmacol 68:1301–1310

    Article  CAS  Google Scholar 

  22. Berman HM, Westbrook J, Feng Z, Gilliland G, Bhat TN, Weissig H, Shindyalov IN, Bourne PE (2000) Nucleic Acids Res 28:235–242

    Article  CAS  Google Scholar 

  23. Wolber G, Langer T (2005) J Chem Inf Model 45:160–169

    Article  CAS  Google Scholar 

  24. Miller MA (2002) Nat Rev Drug Discovery 1:220–227

    Article  CAS  Google Scholar 

  25. Knox AJS, Meegan MJ, Carta G, Lloyd DG (2005) J Chem Inf Model 45:1908–1919

    Article  CAS  Google Scholar 

  26. Moe; Chemical Computing Group: Montreal, QC, Canada

  27. Pospisil P, Ballmer P, Scapozza L, Folkers G (2003) J Recept Signal Transduction 23:361–371

    Article  CAS  Google Scholar 

  28. Oellien F, Cramer J, Beyer C, Ihlenfeldt W-D, Selzer PM (2006) J Chem Inf Comput Sci 46:2342–2354

    CAS  Google Scholar 

  29. Willett P, Barnard JM, Downs GM (1998) J Chem Inf Comput Sci 38:983–996

    CAS  Google Scholar 

  30. Flower DR (1998) J Chem Inf Comput Sci 38:379–386

    Article  CAS  Google Scholar 

  31. Brandstetter H, Grams F, Glitz D, Lang A, Huber R, Bode W, Krell H-W, Engh RA (2001) J Biol Chem 276:17405–17412

    Article  CAS  Google Scholar 

  32. Yan X, Hollis T, Svinth M, Day P, Monzingo AF, Milne GWA, Robertus JD (1997) J Mol Biol 266:1043–1049

    Article  CAS  Google Scholar 

  33. Nederkoorn PHJ, Vernooijs P, Donne-Op den Kelder GM, Baerends EJ, Timmerman H (1994) J Mol Graph 12:242–256

    Article  CAS  Google Scholar 

  34. Tautomer; Molecular Networks GmbH: Erlangen, Germany

  35. Pospisil P, Ballmer P, Scapozza L, Folkers G (2004) 15th European symposium on Structure-Activity Relationships (QSAR) and molecular modelling, Istanbul, Turkey, Sept 5–10, 2004

  36. Li J, Ehlers T, Sutter J, Varma-O’Brien S, Kirchmair J (2007) J Chem Inf Model 47:1923–1932

    Article  CAS  Google Scholar 

  37. Kirchmair J, Laggner C, Wolber G, Langer T (2005) J Chem Inf Model 45:422–430

    Article  CAS  Google Scholar 

  38. Kirchmair J, Wolber G, Laggner C, Langer T (2006) J Chem Inf Model 46:1848–1861

    Article  CAS  Google Scholar 

  39. Onodera K, Satou K, Hirota H (2007) J Chem Inf Model 47:1609–1618

    Article  CAS  Google Scholar 

  40. Corina; Molecular Networks GmbH: Erlangen, Germany

  41. Carta G, Onnis V, Knox AJS, Fayne D, Lloyd DG (2006) J Comput Aided Mol Des 20:179–190

    Article  CAS  Google Scholar 

  42. Stahl M, Rarey M (2001) J Med Chem 44:1035–1042

    Article  CAS  Google Scholar 

  43. Clark RD, Strizhev A, Leonard JM, Blake JF, Matthew JB (2002) J Mol Graph Model 20:281–295

    Article  CAS  Google Scholar 

  44. Charifson PS, Corkery JJ, Murcko MA, Walters WP (1999) J Med Chem 42:5100–5109

    Article  CAS  Google Scholar 

  45. Schulz-Gasch T, Stahl M (2003) J Mol Mod 9:47–57

    CAS  Google Scholar 

  46. Warren GL, Andrews CW, Capelli A-M, Clarke B, LaLonde J, Lambert MH, Lindvall M, Nevins N, Semus SF, Senger S, Tedesco G, Wall ID, Woolven JM, Peishoff CE, Head MS (2006) J Med Chem 49:5912–5931

    Article  CAS  Google Scholar 

  47. Kellenberger E, Rodrigo J, Muller P, Rognan D (2004) Proteins: Struct, Funct, Bioinf 57:225–242

    Article  CAS  Google Scholar 

  48. Nissink JWM, Murray C, Hartshorn M, Verdonk ML, Cole JC, Taylor R (2002) Proteins: Struct, Funct, Genet 49:457–471

    Article  CAS  Google Scholar 

  49. Goto J, Kataoka R, Hirayama N (2004) J Med Chem 47:6804–6811

    Article  CAS  Google Scholar 

  50. Kristam R, Gillet VJ, Lewis RA, Thorner D (2005) J Chem Inf Model 45:461–476

    Article  CAS  Google Scholar 

  51. Kramer B, Rarey M, Lengauer T (1999) Proteins 37:228–241

    Article  CAS  Google Scholar 

  52. Rarey M, Kramer B, Lengauer T, Klebe G (1996) J Mol Biol 261:470–489

    Article  CAS  Google Scholar 

  53. Ewing TJA, Makino S, Skillman AG, Kuntz ID (2001) J Comput Aided Mol Des 15:411–428

    Article  CAS  Google Scholar 

  54. Ewing TJA, Kuntz ID (1997) J Comput Chem 18:1175–1189

    Article  CAS  Google Scholar 

  55. Friesner RA, Banks JL, Murphy RB, Halgren TA, Klicic JJ, Mainz DT, Repasky MP, Knoll EH, Shelley M, Perry JK, Shaw DE, Francis P, Shenkin PS (2004) J Med Chem 47:1739–1749

    Article  CAS  Google Scholar 

  56. Halgren TA, Murphy RB, Friesner RA, Beard HS, Frye LL, Pollard WT, Banks JL (2004) J Med Chem 47:1750–1759

    Article  CAS  Google Scholar 

  57. Venkatachalam CM, Jiang X, Oldfield T, Waldman M (2003) J Mol Graph Model 21:289–307

    Article  CAS  Google Scholar 

  58. Jain AN (2003) J Med Chem 46:499–511

    Article  CAS  Google Scholar 

  59. McGann MR, Almond HR, Nicholls A, Grant JA, Brown FK (2003) Biopolymers 68:76–90

    Article  CAS  Google Scholar 

  60. Kirchmair J, Ristic S, Eder K, Markt P, Wolber G, Laggner C, Langer T (2007) J Chem Inf Model 47:2182–2196

    Article  CAS  Google Scholar 

  61. Teodoro ML, Kavraki LE (2003) Curr Pharm Des 9:1635–1648

    Article  CAS  Google Scholar 

  62. Claussen H, Buning C, Rarey M, Lengauer T (2001) J Mol Biol 308:377–395

    Article  CAS  Google Scholar 

  63. Murray CW, Baxter CA, Frenkel AD (1999) J Comput Aided Mol Des 13:547–562

    Article  CAS  Google Scholar 

  64. Wang R, Wang S (2001) J Chem Inf Comput Sci 41:1422–1426

    Article  CAS  Google Scholar 

  65. Terp GE, Johansen BN, Christensen IT, Jorgensen FS (2001) J Med Chem 44:2333–2343

    Article  CAS  Google Scholar 

  66. Wang R, Lai L, Wang S (2002) J Comput Aided Mol Des 16:11–26

    Article  CAS  Google Scholar 

  67. Zavodszky MI, Sanschagrin PC, Kuhn LA, Korde RS, Kuhn LA (2003) J Comput Aided Mol Des 16:883–902

    Article  Google Scholar 

  68. Kroemer RT, Vulpetti A, McDonald JJ, Rohrer DC, Trosset J-Y, Giordanetto F, Cotesta S, McMartin C, Kihlen M, Stouten PFW (2004) J Chem Inf Comput Sci 44:871–881

    Article  CAS  Google Scholar 

  69. Abagyan RA, Totrov MM (1997) J Mol Biol 268:678–685

    Article  CAS  Google Scholar 

  70. Langer T, Hoffmann RD (2006) Pharmacophores and pharmacophore searches. Wiley-VCH, Weinheim

    Google Scholar 

  71. Triballeau N, Acher F, Brabet I, Pin J-P, Bertrand H-O (2005) J Med Chem 48:2534–2547

    Article  CAS  Google Scholar 

  72. Truchon J-F, Bayly CI (2007) J Chem Inf Model 47:488–508

    Article  CAS  Google Scholar 

  73. Jacobsson M, Liden P, Stjernschantz E, Bostroem H, Norinder U (2003) J Med Chem 46:5781–5789

    Article  CAS  Google Scholar 

  74. Shepherd AJ, Gorse D, Thornton JM (1999) Protein Sci 8:1045–1055

    CAS  Google Scholar 

  75. Gao H, Williams C, Labute P, Bajorath J (1999) J Chem Inf Comput Sci 39:164–168

    Article  CAS  Google Scholar 

  76. Martineau E, Aman AM, Kong X (2004) Accelrysworld. Accelrys, Inc., San Diego, CA

    Google Scholar 

  77. Guha R, Jurs Peter C (2005) J Chem Inf Model 45:65–73

    Article  CAS  Google Scholar 

  78. Weston J, Perez-Cruz F, Bousquet O, Chapelle O, Elisseeff A, Schoelkopf B (2003) Bioinformatics 19:764–771

    Article  CAS  Google Scholar 

  79. Ford MG (2003) 2nd international symposium on computational methods in toxicology and pharmacology integrating internet resources, Thessaloniki, Greece, 2003

  80. Bradley EK, Miller JL, Saiah E, Grootenhuis PDJ (2003) J Med Chem 46:4360–4364

    Article  CAS  Google Scholar 

  81. Bradley EK, Beroza P, Penzotti JE, Grootenhuis PDJ, Spellmeyer DC, Miller JL (2000) J Med Chem 43:2770–2774

    Article  CAS  Google Scholar 

  82. Matthews BW (1975) Biochim Biophys Acta 405:442–451

    CAS  Google Scholar 

  83. Frimurer TM, Bywater R, Nrum L, Lauritsen LN, Brunak S (2000) J Chem Inf Comput Sci 40:1315–1324

    Article  CAS  Google Scholar 

  84. Hecker EA, Duraiswami C, Andrea TA, Diller DJ (2002) J Chem Inf Comput Sci 42:1204–1211

    Article  CAS  Google Scholar 

  85. Diller DJ, Li R (2003) J Med Chem 46:4638–4647

    Article  CAS  Google Scholar 

  86. Diller DJ, Merz KM Jr (2001) Proteins: Struct, Funct, Genet 43:113–124

    Article  CAS  Google Scholar 

  87. Triballeau N, Acher F, Brabet I, Pin JP, Bertrand HO (2005) J Med Chem 48:2534–2547

    Article  CAS  Google Scholar 

  88. Sheridan RP, Singh SB, Fluder EM, Kearsley SK (2001) J Chem Inf Comput Sci 41:1395–1406

    Article  CAS  Google Scholar 

  89. Park SH, Goo JM, Jo CH (2004) Korean J Radiol 5:11–18

    Article  Google Scholar 

  90. McGaughey GB, Sheridan RP, Bayly CI, Culberson JC, Kreatsoulas C, Lindsley S, Maiorov V, Truchon J-F, Cornell WD (2007) J Chem Inf Model 47:1504–1519

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Inte:Ligand Software-Entwicklungs- und Consulting GmbH, Clemens Maria Hofbauer-Gasse 6, 2344, Maria Enzersdorf, Austria

    Johannes Kirchmair, Gerhard Wolber & Thierry Langer

  2. Department of Pharmaceutical Chemistry, Institute of Pharmacy and Center for Molecular Biosciences (CMBI), University of Innsbruck, Innrain 52, 6020, Innsbruck, Austria

    Patrick Markt & Thierry Langer

  3. Dipartimento Farmaco Chimico Tecnologico, Universita` degli Studi di Cagliari, 09124, Cagliari, Italy

    Simona Distinto

Authors
  1. Johannes Kirchmair
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patrick Markt
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Simona Distinto
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gerhard Wolber
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Thierry Langer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Thierry Langer.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kirchmair, J., Markt, P., Distinto, S. et al. Evaluation of the performance of 3D virtual screening protocols: RMSD comparisons, enrichment assessments, and decoy selection—What can we learn from earlier mistakes?. J Comput Aided Mol Des 22, 213–228 (2008). https://doi.org/10.1007/s10822-007-9163-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10822-007-9163-6

Keywords

Search

Navigation