Skip to main content


Log in

The role of human in the loop: lessons from D3R challenge 4

  • Published:
Journal of Computer-Aided Molecular Design Aims and scope Submit manuscript


The rapid development of new machine learning techniques led to significant progress in the area of computer-aided drug design. However, despite the enormous predictive power of new methods, they lack explainability and are often used as black boxes. The most important decisions in drug discovery are still made by human experts who rely on intuitions and simplified representation of the field. We used D3R Grand Challenge 4 to model contributions of human experts during the prediction of the structure of protein–ligand complexes, and prediction of binding affinities for series of ligands in the context of absence or abundance of experimental data. We demonstrated that human decisions have a series of biases: a tendency to focus on easily identifiable protein–ligand interactions such as hydrogen bonds, and neglect for a more distributed and complex electrostatic interactions and solvation effects. While these biases still allow human experts to compete with blind algorithms in some areas, the underutilization of the information leads to significantly worse performance in data-rich tasks such as binding affinity prediction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others


  1. Sunseri J, King JE, Francoeur PG, Koes DR (2019) J Comput Aided Mol Des 33(1):19

    Article  CAS  Google Scholar 

  2. Zhong F, Xing J, Li X, Liu X, Fu Z, Xiong Z, Lu D, Wu X, Zhao J, Tan X, Li F, Luo X, Li Z, Chen K, Zheng M, Jiang H (2018) Sci China Life Sci 61(10):1191

    Article  Google Scholar 

  3. Zhang L, Tan J, Han D, Zhu H (2017) Drug Discov Today 22(11):1680

    Article  Google Scholar 

  4. Macalino SJ, Gosu V, Hong S, Choi S (2015) Arch Pharm Res 38(9):1686

    Article  CAS  Google Scholar 

  5. Baig MH, Ahmad K, Rabbani G, Danishuddin M, Choi I (2018) Curr Neuropharmacol 16(6):740

    Article  CAS  Google Scholar 

  6. (2018) Nat Biomed Eng 2:709

  7. Holzinger A, Biemann C, Pattichis CS, Kell DB (2017) What do we need to build explainable AI systems for the medical domain? arXiv preprint arXiv

  8. Vassar R, Kovacs DM, Yan R, Wong PC (2009) J Neurosci 29(41):12787

    Article  CAS  Google Scholar 

  9. Kitchen DB, Decornez H, Furr JR, Bajorath J (2004) Nat Rev Drug Discov 3(11):935

    Article  CAS  Google Scholar 

  10. Liu J, Wang R (2015) J Chem Inf Model 55(3):475

    Article  CAS  Google Scholar 

  11. Guedes IA, Pereira FSS, Dardenne LE (2018) Front Pharmacol 9:1089

    Article  CAS  Google Scholar 

  12. Wilkinson RD, Williams R, Scott CJ, Burden RE (2015) Biol Chem 396(8):867

    Article  CAS  Google Scholar 

  13. Ameriks MK, Bembenek SD, Burdett MT, Choong IC, Edwards JP, Gebauer D, Gu Y, Karlsson L, Purkey HE, Staker BL, Sun S, Thurmond RL, Zhu J (2010) Bioorg Med Chem Lett 20(14):4060

    Article  CAS  Google Scholar 

  14. Wiener DK, Lee-Dutra A, Bembenek S, Nguyen S, Thurmond RL, Sun S, Karlsson L, Grice CA, Jones TK, Edwards JP (2010) Bioorg Med Chem Lett 20(7):2379

    Article  CAS  Google Scholar 

  15. Ameriks MK, Axe FU, Bembenek SD, Edwards JP, Gu Y, Karlsson L, Randal M, Sun S, Thurmond RL, Zhu J (2009) Bioorg Med Chem Lett 19(21):6131

    Article  CAS  Google Scholar 

  16. Levin NMB, Pintro VO, Bitencourt-Ferreira G, de Mattos BB, de Castro Silverio A, de Azevedo WF Jr (2018) Biophys Chem 235:1

    Article  CAS  Google Scholar 

  17. Zhang L, Ai HX, Li SM, Qi MY, Zhao J, Zhao Q, Liu HS (2017) Oncotarget 8(47):83142

    PubMed  PubMed Central  Google Scholar 

  18. Park H, Eom JW, Kim YH (2014) J Chem Inf Model 54(7):2139

    Article  CAS  Google Scholar 

  19. O'Boyle NM, Banck M, James CA, Morley C, Vandermeersch T, Hutchison GR (2011) J Cheminform 3(1):33

    Article  CAS  Google Scholar 

  20. Cheeseright T, Mackey M, Rose S, Vinter A (2006) J Chem Inf Model 46(2):665

    Article  CAS  Google Scholar 

  21. Stroganov OV, Novikov FN, Zeifman AA, Stroylov VS, Chilov GG (2011) Proteins 79(9):2693

    Article  CAS  Google Scholar 

  22. Stroganov OV, Novikov FN, Stroylov VS, Kulkov V, Chilov GG (2008) J Chem Inf Model 48(12):2371

    Article  CAS  Google Scholar 

  23. Ester M, Kriegel H-P, #246 S, Xu X. A density-based algorithm for discovering clusters a density-based algorithm for discovering clusters in large spatial databases with noise. Proceedings of the Second International Conference on Knowledge Discovery and Data Mining. Portland, Oregon: AAAI Press, 1996:226

  24. Novikov FN, Stroylov VS, Stroganov OV, Chilov GG (2010) J Mol Model 16(7):1223

    Article  CAS  Google Scholar 

  25. Machauer R, Laumen K, Veenstra S, Rondeau JM, Tintelnot-Blomley M, Betschart C, Jaton AL, Desrayaud S, Staufenbiel M, Rabe S, Paganetti P, Neumann U (2009) Bioorg Med Chem Lett 19(5):1366

    Article  CAS  Google Scholar 

  26. Gaulton A, Bellis LJ, Bento AP, Chambers J, Davies M, Hersey A, Light Y, McGlinchey S, Michalovich D, Al-Lazikani B, Overington JP (2012) Nucleic Acids Res 40:D1100

    Article  CAS  Google Scholar 

  27. Gaieb Z, Parks CD, Chiu M, Yang H, Shao C, Walters WP, Lambert MH, Nevins N, Bembenek SD, Ameriks MK, Mirzadegan T, Burley SK, Amaro RE, Gilson MK (2019) J Comput Aided Mol Des 33(1):1

    Article  CAS  Google Scholar 

  28. Team RC. R: A language and environment for statistical computing.

  29. The team. H2O: Scalable machine learning (2015),

  30. The team. h2o: R Interface for H2O (2015),

  31. Wolpert DH (1992) Neural Netw 5(2):241–259

    Article  Google Scholar 

  32. Breiman L (2001) Mach Learn 45(1):5

    Article  Google Scholar 

  33. Friedman JH (2001) Annals of statistics:1189

  34. LeCun Y, Bengio Y, Hinton G (2015) Nature 521(7553):436

    Article  CAS  Google Scholar 

  35. Carlson HA (2016) J Chem Inf Model 56(6):951

    Article  Google Scholar 

  36. Taylor R, Kennard O, Versichel W (1984) Acta Cryst B40:280–288

    Article  CAS  Google Scholar 

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Oleg V. Stroganov.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Stroganov, O.V., Novikov, F.N., Medvedev, M.G. et al. The role of human in the loop: lessons from D3R challenge 4. J Comput Aided Mol Des 34, 121–130 (2020).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: