Evaluation of the performance of 3D virtual screening protocols: RMSD comparisons, enrichment assessments, and decoy selection—What can we learn from earlier mistakes?

  • Johannes Kirchmair
  • Patrick Markt
  • Simona Distinto
  • Gerhard Wolber
  • Thierry LangerEmail author


Within the last few years a considerable amount of evaluative studies has been published that investigate the performance of 3D virtual screening approaches. Thereby, in particular assessments of protein–ligand docking are facing remarkable interest in the scientific community. However, comparing virtual screening approaches is a non-trivial task. Several publications, especially in the field of molecular docking, suffer from shortcomings that are likely to affect the significance of the results considerably. These quality issues often arise from poor study design, biasing, by using improper or inexpressive enrichment descriptors, and from errors in interpretation of the data output. In this review we analyze recent literature evaluating 3D virtual screening methods, with focus on molecular docking. We highlight problematic issues and provide guidelines on how to improve the quality of computational studies. Since 3D virtual screening protocols are in general assessed by their ability to discriminate between active and inactive compounds, we summarize the impact of the composition and preparation of test sets on the outcome of evaluations. Moreover, we investigate the significance of both classic enrichment parameters and advanced descriptors for the performance of 3D virtual screening methods. Furthermore, we review the significance and suitability of RMSD as a measure for the accuracy of protein–ligand docking algorithms and of conformational space sub sampling algorithms.


Virtual screening Evaluation of computational methods Pharmacophore modeling Protein–ligand docking Enrichment descriptors Decoy selection Virtual library design Tautomerism RMSD 


  1. 1.
    Klebe G (2006) Drug Discovery Today 11:580–594CrossRefGoogle Scholar
  2. 2.
    Schneider G, Bohm H-J (2002) Drug Discovery Today 7:64–70Google Scholar
  3. 3.
    Chen H, Lyne PD, Giordanetto F, Lovell T, Li J (2006) J Chem Inf Model 46:401–415CrossRefGoogle Scholar
  4. 4.
    Cole JC, Murray CW, Nissink JWM, Taylor RD, Taylor R (2005) Proteins: Struct Funct Bioinf 60:325–332CrossRefGoogle Scholar
  5. 5.
    Kontoyianni M, McClellan LM, Sokol GS (2004) J Med Chem 47:558–565CrossRefGoogle Scholar
  6. 6.
    Steindl TM, Schuster D, Wolber G, Laggner C, Langer T (2006) J Comput Aided Mol Des 20:703–715CrossRefGoogle Scholar
  7. 7.
    Steindl TM, Schuster D, Laggner C, Chuang K, Hoffmann RD, Langer T (2007) J Chem Inf Model 47:563–571CrossRefGoogle Scholar
  8. 8.
    Steindl TM, Schuster D, Laggner C, Langer T (2006) J Chem Inf Model 46:2146–2157CrossRefGoogle Scholar
  9. 9.
    Sheng C, Zhang W, Ji H, Zhang M, Song Y, Xu H, Zhu J, Miao Z, Jiang Q, Yao J, Zhou Y, Zhu J, Lue J (2006) J Med Chem 49:2512–2525CrossRefGoogle Scholar
  10. 10.
    Güner OF (2000) Pharmacophore perception, development and use in drug design. International University Line, La Jolla, CAGoogle Scholar
  11. 11.
    Catalyst; Accelrys: San Diego, CAGoogle Scholar
  12. 12.
    Verdonk ML, Berdini V, Hartshorn MJ, Mooij WTM, Murray CW, Taylor RD, Watson P (2004) J Chem Inf Comput Sci 44:793–806CrossRefGoogle Scholar
  13. 13.
    Weber L (2005) QSAR Comb Sci 24:809–823CrossRefGoogle Scholar
  14. 14.
    Wolber G, Langer T (2000) 13th European symposium on quantitative structure-activity relationships, Duesseldorf, Germany, Aug 27 - Sept 1, 2000Google Scholar
  15. 15.
    Schueller A, Haehnke V, Schneider G (2007) QSAR Comb Sci 26:407–410CrossRefGoogle Scholar
  16. 16.
    Huang N, Shoichet BK, Irwin JJ (2006) J Med Chem 49:6789–6801CrossRefGoogle Scholar
  17. 17.
    Pan Y, Huang N, Cho S, MacKerell AD Jr (2003) J Chem Inf Comput Sci 43:267–272CrossRefGoogle Scholar
  18. 18.
    Jones G, Willett P, Glen RC (1995) J Mol Biol 245:43–53CrossRefGoogle Scholar
  19. 19.
    Jones G, Willett P, Glen RC, Leach AR, Taylor R (1997) J Mol Biol 267:727–748CrossRefGoogle Scholar
  20. 20.
    Irwin JJ, Shoichet BK (2005) J Chem Inf Model 45:177–182CrossRefGoogle Scholar
  21. 21.
    Edwards BS, Bologa C, Young SM, Balakin KV, Prossnitz ER, Savchuck NP, Sklar LA, Oprea TI (2005) Mol Pharmacol 68:1301–1310CrossRefGoogle Scholar
  22. 22.
    Berman HM, Westbrook J, Feng Z, Gilliland G, Bhat TN, Weissig H, Shindyalov IN, Bourne PE (2000) Nucleic Acids Res 28:235–242CrossRefGoogle Scholar
  23. 23.
    Wolber G, Langer T (2005) J Chem Inf Model 45:160–169CrossRefGoogle Scholar
  24. 24.
    Miller MA (2002) Nat Rev Drug Discovery 1:220–227CrossRefGoogle Scholar
  25. 25.
    Knox AJS, Meegan MJ, Carta G, Lloyd DG (2005) J Chem Inf Model 45:1908–1919CrossRefGoogle Scholar
  26. 26.
    Moe; Chemical Computing Group: Montreal, QC, CanadaGoogle Scholar
  27. 27.
    Pospisil P, Ballmer P, Scapozza L, Folkers G (2003) J Recept Signal Transduction 23:361–371CrossRefGoogle Scholar
  28. 28.
    Oellien F, Cramer J, Beyer C, Ihlenfeldt W-D, Selzer PM (2006) J Chem Inf Comput Sci 46:2342–2354Google Scholar
  29. 29.
    Willett P, Barnard JM, Downs GM (1998) J Chem Inf Comput Sci 38:983–996Google Scholar
  30. 30.
    Flower DR (1998) J Chem Inf Comput Sci 38:379–386CrossRefGoogle Scholar
  31. 31.
    Brandstetter H, Grams F, Glitz D, Lang A, Huber R, Bode W, Krell H-W, Engh RA (2001) J Biol Chem 276:17405–17412CrossRefGoogle Scholar
  32. 32.
    Yan X, Hollis T, Svinth M, Day P, Monzingo AF, Milne GWA, Robertus JD (1997) J Mol Biol 266:1043–1049CrossRefGoogle Scholar
  33. 33.
    Nederkoorn PHJ, Vernooijs P, Donne-Op den Kelder GM, Baerends EJ, Timmerman H (1994) J Mol Graph 12:242–256CrossRefGoogle Scholar
  34. 34.
    Tautomer; Molecular Networks GmbH: Erlangen, GermanyGoogle Scholar
  35. 35.
    Pospisil P, Ballmer P, Scapozza L, Folkers G (2004) 15th European symposium on Structure-Activity Relationships (QSAR) and molecular modelling, Istanbul, Turkey, Sept 5–10, 2004Google Scholar
  36. 36.
    Li J, Ehlers T, Sutter J, Varma-O’Brien S, Kirchmair J (2007) J Chem Inf Model 47:1923–1932CrossRefGoogle Scholar
  37. 37.
    Kirchmair J, Laggner C, Wolber G, Langer T (2005) J Chem Inf Model 45:422–430CrossRefGoogle Scholar
  38. 38.
    Kirchmair J, Wolber G, Laggner C, Langer T (2006) J Chem Inf Model 46:1848–1861CrossRefGoogle Scholar
  39. 39.
    Onodera K, Satou K, Hirota H (2007) J Chem Inf Model 47:1609–1618CrossRefGoogle Scholar
  40. 40.
    Corina; Molecular Networks GmbH: Erlangen, GermanyGoogle Scholar
  41. 41.
    Carta G, Onnis V, Knox AJS, Fayne D, Lloyd DG (2006) J Comput Aided Mol Des 20:179–190CrossRefGoogle Scholar
  42. 42.
    Stahl M, Rarey M (2001) J Med Chem 44:1035–1042CrossRefGoogle Scholar
  43. 43.
    Clark RD, Strizhev A, Leonard JM, Blake JF, Matthew JB (2002) J Mol Graph Model 20:281–295CrossRefGoogle Scholar
  44. 44.
    Charifson PS, Corkery JJ, Murcko MA, Walters WP (1999) J Med Chem 42:5100–5109CrossRefGoogle Scholar
  45. 45.
    Schulz-Gasch T, Stahl M (2003) J Mol Mod 9:47–57Google Scholar
  46. 46.
    Warren GL, Andrews CW, Capelli A-M, Clarke B, LaLonde J, Lambert MH, Lindvall M, Nevins N, Semus SF, Senger S, Tedesco G, Wall ID, Woolven JM, Peishoff CE, Head MS (2006) J Med Chem 49:5912–5931CrossRefGoogle Scholar
  47. 47.
    Kellenberger E, Rodrigo J, Muller P, Rognan D (2004) Proteins: Struct, Funct, Bioinf 57:225–242CrossRefGoogle Scholar
  48. 48.
    Nissink JWM, Murray C, Hartshorn M, Verdonk ML, Cole JC, Taylor R (2002) Proteins: Struct, Funct, Genet 49:457–471CrossRefGoogle Scholar
  49. 49.
    Goto J, Kataoka R, Hirayama N (2004) J Med Chem 47:6804–6811CrossRefGoogle Scholar
  50. 50.
    Kristam R, Gillet VJ, Lewis RA, Thorner D (2005) J Chem Inf Model 45:461–476CrossRefGoogle Scholar
  51. 51.
    Kramer B, Rarey M, Lengauer T (1999) Proteins 37:228–241CrossRefGoogle Scholar
  52. 52.
    Rarey M, Kramer B, Lengauer T, Klebe G (1996) J Mol Biol 261:470–489CrossRefGoogle Scholar
  53. 53.
    Ewing TJA, Makino S, Skillman AG, Kuntz ID (2001) J Comput Aided Mol Des 15:411–428CrossRefGoogle Scholar
  54. 54.
    Ewing TJA, Kuntz ID (1997) J Comput Chem 18:1175–1189CrossRefGoogle Scholar
  55. 55.
    Friesner RA, Banks JL, Murphy RB, Halgren TA, Klicic JJ, Mainz DT, Repasky MP, Knoll EH, Shelley M, Perry JK, Shaw DE, Francis P, Shenkin PS (2004) J Med Chem 47:1739–1749CrossRefGoogle Scholar
  56. 56.
    Halgren TA, Murphy RB, Friesner RA, Beard HS, Frye LL, Pollard WT, Banks JL (2004) J Med Chem 47:1750–1759CrossRefGoogle Scholar
  57. 57.
    Venkatachalam CM, Jiang X, Oldfield T, Waldman M (2003) J Mol Graph Model 21:289–307CrossRefGoogle Scholar
  58. 58.
    Jain AN (2003) J Med Chem 46:499–511CrossRefGoogle Scholar
  59. 59.
    McGann MR, Almond HR, Nicholls A, Grant JA, Brown FK (2003) Biopolymers 68:76–90CrossRefGoogle Scholar
  60. 60.
    Kirchmair J, Ristic S, Eder K, Markt P, Wolber G, Laggner C, Langer T (2007) J Chem Inf Model 47:2182–2196CrossRefGoogle Scholar
  61. 61.
    Teodoro ML, Kavraki LE (2003) Curr Pharm Des 9:1635–1648CrossRefGoogle Scholar
  62. 62.
    Claussen H, Buning C, Rarey M, Lengauer T (2001) J Mol Biol 308:377–395CrossRefGoogle Scholar
  63. 63.
    Murray CW, Baxter CA, Frenkel AD (1999) J Comput Aided Mol Des 13:547–562CrossRefGoogle Scholar
  64. 64.
    Wang R, Wang S (2001) J Chem Inf Comput Sci 41:1422–1426CrossRefGoogle Scholar
  65. 65.
    Terp GE, Johansen BN, Christensen IT, Jorgensen FS (2001) J Med Chem 44:2333–2343CrossRefGoogle Scholar
  66. 66.
    Wang R, Lai L, Wang S (2002) J Comput Aided Mol Des 16:11–26CrossRefGoogle Scholar
  67. 67.
    Zavodszky MI, Sanschagrin PC, Kuhn LA, Korde RS, Kuhn LA (2003) J Comput Aided Mol Des 16:883–902CrossRefGoogle Scholar
  68. 68.
    Kroemer RT, Vulpetti A, McDonald JJ, Rohrer DC, Trosset J-Y, Giordanetto F, Cotesta S, McMartin C, Kihlen M, Stouten PFW (2004) J Chem Inf Comput Sci 44:871–881CrossRefGoogle Scholar
  69. 69.
    Abagyan RA, Totrov MM (1997) J Mol Biol 268:678–685CrossRefGoogle Scholar
  70. 70.
    Langer T, Hoffmann RD (2006) Pharmacophores and pharmacophore searches. Wiley-VCH, WeinheimGoogle Scholar
  71. 71.
    Triballeau N, Acher F, Brabet I, Pin J-P, Bertrand H-O (2005) J Med Chem 48:2534–2547CrossRefGoogle Scholar
  72. 72.
    Truchon J-F, Bayly CI (2007) J Chem Inf Model 47:488–508CrossRefGoogle Scholar
  73. 73.
    Jacobsson M, Liden P, Stjernschantz E, Bostroem H, Norinder U (2003) J Med Chem 46:5781–5789CrossRefGoogle Scholar
  74. 74.
    Shepherd AJ, Gorse D, Thornton JM (1999) Protein Sci 8:1045–1055Google Scholar
  75. 75.
    Gao H, Williams C, Labute P, Bajorath J (1999) J Chem Inf Comput Sci 39:164–168CrossRefGoogle Scholar
  76. 76.
    Martineau E, Aman AM, Kong X (2004) Accelrysworld. Accelrys, Inc., San Diego, CAGoogle Scholar
  77. 77.
    Guha R, Jurs Peter C (2005) J Chem Inf Model 45:65–73CrossRefGoogle Scholar
  78. 78.
    Weston J, Perez-Cruz F, Bousquet O, Chapelle O, Elisseeff A, Schoelkopf B (2003) Bioinformatics 19:764–771CrossRefGoogle Scholar
  79. 79.
    Ford MG (2003) 2nd international symposium on computational methods in toxicology and pharmacology integrating internet resources, Thessaloniki, Greece, 2003Google Scholar
  80. 80.
    Bradley EK, Miller JL, Saiah E, Grootenhuis PDJ (2003) J Med Chem 46:4360–4364CrossRefGoogle Scholar
  81. 81.
    Bradley EK, Beroza P, Penzotti JE, Grootenhuis PDJ, Spellmeyer DC, Miller JL (2000) J Med Chem 43:2770–2774CrossRefGoogle Scholar
  82. 82.
    Matthews BW (1975) Biochim Biophys Acta 405:442–451Google Scholar
  83. 83.
    Frimurer TM, Bywater R, Nrum L, Lauritsen LN, Brunak S (2000) J Chem Inf Comput Sci 40:1315–1324CrossRefGoogle Scholar
  84. 84.
    Hecker EA, Duraiswami C, Andrea TA, Diller DJ (2002) J Chem Inf Comput Sci 42:1204–1211CrossRefGoogle Scholar
  85. 85.
    Diller DJ, Li R (2003) J Med Chem 46:4638–4647CrossRefGoogle Scholar
  86. 86.
    Diller DJ, Merz KM Jr (2001) Proteins: Struct, Funct, Genet 43:113–124CrossRefGoogle Scholar
  87. 87.
    Triballeau N, Acher F, Brabet I, Pin JP, Bertrand HO (2005) J Med Chem 48:2534–2547CrossRefGoogle Scholar
  88. 88.
    Sheridan RP, Singh SB, Fluder EM, Kearsley SK (2001) J Chem Inf Comput Sci 41:1395–1406CrossRefGoogle Scholar
  89. 89.
    Park SH, Goo JM, Jo CH (2004) Korean J Radiol 5:11–18CrossRefGoogle Scholar
  90. 90.
    McGaughey GB, Sheridan RP, Bayly CI, Culberson JC, Kreatsoulas C, Lindsley S, Maiorov V, Truchon J-F, Cornell WD (2007) J Chem Inf Model 47:1504–1519CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media B.V. 2008

Authors and Affiliations

  • Johannes Kirchmair
    • 1
  • Patrick Markt
    • 2
  • Simona Distinto
    • 3
  • Gerhard Wolber
    • 1
  • Thierry Langer
    • 1
    • 2
    Email author
  1. 1.Inte:Ligand Software-Entwicklungs- und Consulting GmbHMaria EnzersdorfAustria
  2. 2.Department of Pharmaceutical Chemistry, Institute of Pharmacy and Center for Molecular Biosciences (CMBI)University of InnsbruckInnsbruckAustria
  3. 3.Dipartimento Farmaco Chimico TecnologicoUniversita` degli Studi di CagliariCagliariItaly

Personalised recommendations