Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

The SAMPL6 SAMPLing challenge: assessing the reliability and efficiency of binding free energy calculations


Approaches for computing small molecule binding free energies based on molecular simulations are now regularly being employed by academic and industry practitioners to study receptor-ligand systems and prioritize the synthesis of small molecules for ligand design. Given the variety of methods and implementations available, it is natural to ask how the convergence rates and final predictions of these methods compare. In this study, we describe the concept and results for the SAMPL6 SAMPLing challenge, the first challenge from the SAMPL series focusing on the assessment of convergence properties and reproducibility of binding free energy methodologies. We provided parameter files, partial charges, and multiple initial geometries for two octa-acid (OA) and one cucurbit[8]uril (CB8) host–guest systems. Participants submitted binding free energy predictions as a function of the number of force and energy evaluations for seven different alchemical and physical-pathway (i.e., potential of mean force and weighted ensemble of trajectories) methodologies implemented with the GROMACS, AMBER, NAMD, or OpenMM simulation engines. To rank the methods, we developed an efficiency statistic based on bias and variance of the free energy estimates. For the two small OA binders, the free energy estimates computed with alchemical and potential of mean force approaches show relatively similar variance and bias as a function of the number of energy/force evaluations, with the attach-pull-release (APR), GROMACS expanded ensemble, and NAMD double decoupling submissions obtaining the greatest efficiency. The differences between the methods increase when analyzing the CB8-quinine system, where both the guest size and correlation times for system dynamics are greater. For this system, nonequilibrium switching (GROMACS/NS-DS/SB) obtained the overall highest efficiency. Surprisingly, the results suggest that specifying force field parameters and partial charges is insufficient to generally ensure reproducibility, and we observe differences between seemingly converged predictions ranging approximately from 0.3 to 1.0 kcal/mol, even with almost identical simulations parameters and system setup (e.g., Lennard-Jones cutoff, ionic composition). Further work will be required to completely identify the exact source of these discrepancies. Among the conclusions emerging from the data, we found that Hamiltonian replica exchange—while displaying very small variance—can be affected by a slowly-decaying bias that depends on the initial population of the replicas, that bidirectional estimators are significantly more efficient than unidirectional estimators for nonequilibrium free energy calculations for systems considered, and that the Berendsen barostat introduces non-negligible artifacts in expanded ensemble simulations.

This is a preview of subscription content, log in to check access.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Code availability

Input files and setup scripts: https://github.com/samplchallenges/SAMPL6/tree/master/host_guest/SAMPLing/. Analysis scripts: https://github.com/samplchallenges/SAMPL6/tree/master/host_guest/Analysis/Scripts/. Analysis results: https://github.com/samplchallenges/SAMPL6/tree/master/host_guest/Analysis/SAMPLing/. Participant submissions: https://github.com/samplchallenges/SAMPL6/tree/master/host_guest/Analysis/Submissions/SAMPLing/


  1. 1.

    Shirts MR, Mobley DL, Brown SP (2010) Free-energy calculations in structure-based drug design. Drug Des. https://doi.org/10.1017/CBO9780511730412.007

  2. 2.

    Kuhn B, Tichý M, Wang L, Robinson S, Martin RE, Kuglstatter A, Benz J (2017) Prospective evaluation of free energy calculations for the prioritization of cathepsin L inhibitors. J Med Chem 60(6):2485–2497

  3. 3.

    Ciordia M, Pérez-Benito L, Delgado F, Trabanco AA, Tresadern G (2016) Application of free energy perturbation for the design of BACE1 inhibitors. J Chem Inf Model 56(9):1856–1871

  4. 4.

    Schindler C, Rippmann F, Kuhn D (2018) Relative binding affinity prediction of farnesoid X receptor in the D3R Grand Challenge 2 using FEP+. J Comput Aided Mol Des 32(1):265–272

  5. 5.

    Wang L, Wu Y, Deng Y, Kim B, Pierce L, Krilov G, Lupyan D, Robinson S, Dahlgren MK, Greenwood J et al (2015) Accurate and reliable prediction of relative ligand binding potency in prospective drug discovery by way of a modern free-energy calculation protocol and force field. J Am Chem Soc 137(7):2695–2703

  6. 6.

    Minh DD (2019) Alchemical Grid Dock (AlGDock): binding free energy calculations between flexible ligands and rigid receptors. J Comput Chem. https://doi.org/10.1002/jcc.26036

  7. 7.

    Capelli R, Carloni P, Parrinello M (2019) Exhaustive search of ligand binding pathways via volume-based metadynamics. J Phys Chem Lett 10:3495–3499

  8. 8.

    Irwin BW, Huggins DJ (2018) Estimating atomic contributions to hydration and binding using free energy perturbation. J Chem Theory Comput 14(6):3218–3227

  9. 9.

    Sherborne B, Shanmugasundaram V, Cheng AC, Christ CD, DesJarlais RL, Duca JS, Lewis RA, Loughney DA, Manas ES, McGaughey GB et al (2016) Collaborating to improve the use of free-energy and other quantitative methods in drug discovery. J Comput Aided Mol Des 30(12):1139–1141

  10. 10.

    Cournia Z, Allen B, Sherman W (2017) Relative binding free energy calculations in drug discovery: recent advances and practical considerations. J Chem Inf Model 57(12):2911–2937

  11. 11.

    Mobley DL, Gilson MK (2017) Predicting binding free energies: frontiers and benchmarks. Annu Rev Biophys 46:531–558

  12. 12.

    Gathiaka S, Liu S, Chiu M, Yang H, Stuckey JA, Kang YN, Delproposto J, Kubish G, Dunbar JB, Carlson HA et al (2016) D3R grand challenge 2015: evaluation of protein-ligand pose and affinity predictions. J Comput Aided Mol Des 30(9):651–668

  13. 13.

    Gaieb Z, Liu S, Gathiaka S, Chiu M, Yang H, Shao C, Feher VA, Walters WP, Kuhn B, Rudolph MG et al (2018) D3R Grand Challenge 2: blind prediction of protein-ligand poses, affinity rankings, and relative binding free energies. J Comput Aided Mol Des 32(1):1–20

  14. 14.

    Gaieb Z, Parks CD, Chiu M, Yang H, Shao C, Walters WP, Lambert MH, Nevins N, Bembenek SD, Ameriks MK et al (2019) D3R Grand Challenge 3: blind prediction of protein-ligand poses and affinity rankings. J Comput Aided Mol Des 33(1):1–18

  15. 15.

    Gilson MK, Given JA, Bush BL, McCammon JA (1997) The statistical-thermodynamic basis for computation of binding affinities: a critical review. Biophys J 72(3):1047–1069. https://doi.org/10.1016/S0006-3495(97)78756-3

  16. 16.

    Laio A, Parrinello M (2002) Escaping free-energy minima. Proc Natl Acad Sci 99(20):12562–12566

  17. 17.

    Barducci A, Bussi G, Parrinello M (2008) Well-tempered metadynamics: a smoothly converging and tunable free-energy method. Phys Rev Lett 100(2):020603

  18. 18.

    Swendsen RH, Wang JS (1986) Replica Monte Carlo simulation of spin-glasses. Phys Rev Lett 57(21):2607

  19. 19.

    Hukushima K, Nemoto K (1996) Exchange Monte Carlo method and application to spin glass simulations. J Phys Soc Jpn 65(6):1604–1608

  20. 20.

    Sugita Y, Kitao A, Okamoto Y (2000) Multidimensional replica-exchange method for free-energy calculations. J Chem Phys 113(15):6042–6051. https://doi.org/10.1063/1.1308516

  21. 21.

    Lyubartsev A, Martsinovski A, Shevkunov S, Vorontsov-Velyaminov P (1992) New approach to Monte Carlo calculation of the free energy: method of expanded ensembles. J Chem Phys 96(3):1776–1783

  22. 22.

    Leimkuhler B, Matthews C (2012) Rational construction of stochastic numerical methods for molecular sampling. Appl Math Res Express 2013(1):34–56

  23. 23.

    Fass J, Sivak D, Crooks G, Beauchamp K, Leimkuhler B, Chodera J (2018) Quantifying configuration-sampling error in Langevin simulations of complex molecular systems. Entropy 20(5):318

  24. 24.

    Shirts MR, Pande VS (2005) Comparison of efficiency and bias of free energies computed by exponential averaging, the Bennett acceptance ratio, and thermodynamic integration. J Chem Phys 122(14):144107

  25. 25.

    Yin J, Henriksen NM, Slochower DR, Shirts MR, Chiu MW, Mobley DL, Gilson MK (2017) Overview of the SAMPL5 host–guest challenge: are we doing better? J Comput Aided Mol Des 31(1):1–19

  26. 26.

    Rizzi A, Murkli S, McNeill JN, Yao W, Sullivan M, Gilson MK, Chiu MW, Isaacs L, Gibb BC, Mobley DL et al (2018) Overview of the SAMPL6 host–guest binding affinity prediction challenge. J Comput Aided Mol Des 32(10):937–963

  27. 27.

    Cabeza de Vaca I, Qian Y, Vilseck JZ, Tirado-Rives J, Jorgensen WL (2018) Enhanced Monte Carlo methods for modeling proteins including computation of absolute free energies of binding. J Chem Theory Comput 14(6):3279–3288

  28. 28.

    Deng N, Cui D, Zhang BW, Xia J, Cruz J, Levy R (2018) Comparing alchemical and physical pathway methods for computing the absolute binding free energy of charged ligands. Phys Chem Chem Phys 20(25):17081–17092

  29. 29.

    Shirts MR, Klein C, Swails JM, Yin J, Gilson MK, Mobley DL, Case DA, Zhong ED (2016) Lessons learned from comparing molecular dynamics engines on the SAMPL5 dataset. J Comput Aided Mol Des. https://doi.org/10.1007/s10822-016-9977-1

  30. 30.

    Loeffler HH, Bosisio S, Duarte Ramos Matos G, Suh D, Roux B, Mobley DL, Michel J (2018) Reproducibility of free energy calculations across different molecular simulation software packages. J Chem Theory Comput 14(11):5567–5582

  31. 31.

    Aldeghi M, Heifetz A, Bodkin MJ, Knapp S, Biggin PC (2016) Accurate calculation of the absolute free energy of binding for drug molecules. Chem Sci 7(1):207–218

  32. 32.

    Bhati AP, Wan S, Wright DW, Coveney PV (2016) Rapid, accurate, precise, and reliable relative free energy prediction using ensemble based thermodynamic integration. J Chem Theory Comput 13(1):210–222

  33. 33.

    Xie B, Nguyen TH, Minh DD (2017) Absolute binding free energies between T4 lysozyme and 141 small molecules: calculations based on multiple rigid receptor configurations. J Chem Theory Comput 13(6):2930–2944

  34. 34.

    Henriksen NM, Gilson MK (2017) Evaluating force field performance in thermodynamic calculations of cyclodextrin host–guest binding: water models, partial charges, and host force field parameters. J Chem Theory Comput 13(9):4253–4269

  35. 35.

    Gill SC, Lim NM, Grinaway PB, Rustenburg AS, Fass J, Ross GA, Chodera JD, Mobley DL (2018) Binding modes of ligands using enhanced sampling (BLUES): rapid decorrelation of ligand binding modes via nonequilibrium candidate Monte Carlo. J Phys Chem B 122(21):5579–5598

  36. 36.

    Miao Y, Feher VA, McCammon JA (2015) Gaussian accelerated molecular dynamics: unconstrained enhanced sampling and free energy calculation. J Chem Theory Comput 11(8):3584–3595

  37. 37.

    Pham TT, Shirts MR (2012) Optimal pairwise and non-pairwise alchemical pathways for free energy calculations of molecular transformation in solution phase. J Chem Phys 136(12):124120

  38. 38.

    Athènes M, Terrier P (2017) Estimating thermodynamic expectations and free energies in expanded ensemble simulations: systematic variance reduction through conditioning. J Chem Phys 146(19):194101

  39. 39.

    Nguyen TH, Minh DD (2016) Intermediate thermodynamic states contribute equally to free energy convergence: a demonstration with replica exchange. J Chem Theory Comput 12(5):2154–2161

  40. 40.

    Shenfeld DK, Xu H, Eastwood MP, Dror RO, Shaw DE (2009) Minimizing thermodynamic length to select intermediate states for free-energy calculations and replica-exchange simulations. Phys Rev E 80(4):046705

  41. 41.

    MacCallum JL, Muniyat MI, Gaalswyk K (2018) Online optimization of total acceptance in Hamiltonian replica exchange simulations. J Phys Chem B 122(21):5448–5457

  42. 42.

    Lindahl V, Lidmar J, Hess B (2018) Riemann metric approach to optimal sampling of multidimensional free-energy landscapes. Phys Rev E 98(2):023312

  43. 43.

    Martinsson A, Lu J, Leimkuhler B, Vanden-Eijnden E (2019) The simulated tempering method in the infinite switch limit with adaptive weight learning. J Stat Mech Theory Exp 2019(1):013207

  44. 44.

    Crooks GE (2007) Measuring thermodynamic length. Phys Rev Lett 99(10):100602

  45. 45.

    Sivak DA, Crooks GE (2012) Thermodynamic metrics and optimal paths. Phys Rev Lett 108(19):190602

  46. 46.

    Coleman RG, Sterling T, Weiss DR (2014) SAMPL4 & DOCK3. 7: lessons for automated docking procedures. J Comput Aided Mol Des 28(3):201–209

  47. 47.

    Eken Y, Patel P, Díaz T, Jones MR, Wilson AK (2018) SAMPL6 host–guest challenge: binding free energies via a multistep approach. J Comput Aided Mol Des 32(10):1097–1115

  48. 48.

    Hudson PS, Han K, Woodcock HL, Brooks BR (2018) Force matching as a stepping stone to QM/MM CB [8] host/guest binding free energies: a SAMPL6 cautionary tale. J Comput Aided Mol Des 32(10):983–999

  49. 49.

    Olsson MA, Ryde U (2017) Comparison of QM/MM methods to obtain ligand-binding free energies. J Comput Aided Mol Des 13(5):2245–2253

  50. 50.

    Zheng Z, Ucisik MN, Merz KM (2013) The movable type method applied to protein-ligand binding. J Comput Aided Mol Des 9(12):5526–5538

  51. 51.

    Bansal N, Zheng Z, Cerutti DS, Merz KM (2017) On the fly estimation of host–guest binding free energies using the movable type method: participation in the SAMPL5 blind challenge. J Comput Aided Mol Des 31(1):47–60

  52. 52.

    World Health Organization (2015) Guidelines for the treatment of malaria. World Health Organization, Geneva

  53. 53.

    Gibb CL, Gibb BC (2004) Well-defined, organic nanoenvironments in water: the hydrophobic effect drives a capsular assembly. J Am Chem Soc 126(37):11408–11409

  54. 54.

    Hillyer MB, Gibb CL, Sokkalingam P, Jordan JH, Ioup SE, Gibb BC (2016) Synthesis of water-soluble deep-cavity cavitands. Org Lett 18(16):4048–4051

  55. 55.

    Liu S, Ruspic C, Mukhopadhyay P, Chakrabarti S, Zavalij PY, Isaacs L (2005) The cucurbit [n] uril family: prime components for self-sorting systems. J Am Chem Soc 127(45):15959–15967

  56. 56.

    Mobley DL, Heinzelmann G, Henriksen NM, Gilson MK (2017) Predicting binding free energies: frontiers and benchmarks (a perpetual review). Department of Pharmaceutical Sciences, UCI, Irvine. https://escholarship.org/uc/item/9p37m6bq

  57. 57.

    Muddana HS, Gilson MK (2012) Prediction of SAMPL3 host–guest binding affinities: evaluating the accuracy of generalized force-fields. J Comput Aided Mol Des 26(5):517–525

  58. 58.

    Muddana HS, Fenley AT, Mobley DL, Gilson MK (2014) The SAMPL4 host–guest blind prediction challenge: an overview. J Comput Aided Mol Des 28(4):305–317

  59. 59.

    Ewell J, Gibb BC, Rick SW (2008) Water inside a hydrophobic cavitand molecule. J Phys Chem B 112(33):10272–10279

  60. 60.

    Rogers KE, Ortiz-Sánchez JM, Baron R, Fajer M, de Oliveira CAF, McCammon JA (2012) On the role of dewetting transitions in host–guest binding free energy calculations. J Chem Theory Comput 9(1):46–53. https://doi.org/10.1021/ct300515n

  61. 61.

    Mobley DL, Chodera JD, Dill KA (2006) On the use of orientational restraints and symmetry corrections in alchemical free energy calculations. J Chem Phys 125(8):084902

  62. 62.

    Chen W, Deng Y, Russell E, Wu Y, Abel R, Wang L (2018) Accurate calculation of relative binding free energies between ligands with different net charges. J Chem Theory Comput 14(12):6346–6358

  63. 63.

    Rocklin GJ, Mobley DL, Dill KA, Hünenberger PH (2013) Calculating the binding free energies of charged species based on explicit-solvent simulations employing lattice-sum methods: an accurate correction scheme for electrostatic finite-size effects. J Chem Phys 139(18):11B606-1

  64. 64.

    Lin YL, Aleksandrov A, Simonson T, Roux B (2014) An overview of electrostatic free energy computations for solutions and proteins. J Chem Theory Comput 10(7):2690–2709

  65. 65.

    Morgan BR, Massi F (2010) Accurate estimates of free energy changes in charge mutations. J Chem Theory Comput 6(6):1884–1893

  66. 66.

    McGann M (2011) FRED pose prediction and virtual screening accuracy. J Chem Inf Model 51(3):578–596. https://doi.org/10.1021/ci100436p

  67. 67.

    McGann M (2012) FRED and HYBRID docking performance on standardized datasets. J Comput Aided Mol Des 26(8):897–906. https://doi.org/10.1007/s10822-012-9584-8

  68. 68.

    Jakalian A, Bush BL, Jack DB, Bayly CI (2000) Fast, efficient generation of high-quality atomic. Charges AM1-BCC model: I. Method. J Comput Chem 21(2):132–146

  69. 69.

    Jakalian A, Jack DB, Bayly CI (2002) Fast, efficient generation of high-quality atomic charges. AM1-BCC model: II. Parameterization and validation. J Comput Chem 23(16):1623–1641. https://doi.org/10.1002/jcc.10128

  70. 70.

    Wang J, Wolf RM, Caldwell JW, Kollman PA, Case DA (2004) Development and testing of a general amber force field. J Comput Chem 25(9):1157–1174. https://doi.org/10.1002/jcc.20035

  71. 71.

    Jorgensen WL, Chandrasekhar J, Madura JD, Impey RW, Klein ML (1983) Comparison of simple potential functions for simulating liquid water. J Chem Phys 79(2):926–935. https://doi.org/10.1063/1.445869

  72. 72.

    Case D, Ben-Shalom I, Brozell S, Cerutti D, Cheatham T III, Cruzeiro V, Darden T, Duke R, Ghoreishi D, Gilson M, Gohlke H, Goetz A, Greene D, Harris R, Homeyer N, Izadi S, Kovalenko A, Kurtzman T, Lee T et al (2018) AMBER 18. University of California, San Francisco

  73. 73.

    Abraham MJ, Murtola T, Schulz R, Páll S, Smith JC, Hess B, Lindahl E (2015) GROMACS: high performance molecular simulations through multi-level parallelism from laptops to supercomputers. SoftwareX 1:19–25

  74. 74.

    Phillips JC, Braun R, Wang W, Gumbart J, Tajkhorshid E, Villa E, Chipot C, Skeel RD, Kale L, Schulten K (2005) Scalable molecular dynamics with NAMD. J Comput Chem 26(16):1781–1802

  75. 75.

    Eastman P, Swails J, Chodera JD, McGibbon RT, Zhao Y, Beauchamp KA, Wang LP, Simmonett AC, Harrigan MP, Stern CD et al (2017) OpenMM 7: rapid development of high performance algorithms for molecular dynamics. PLoS Computat Biol 13(7):e1005659. https://doi.org/10.1371/journal.pcbi.1005659

  76. 76.

    Papadourakis M, Bosisio S, Michel J (2018) Blinded predictions of standard binding free energies: lessons learned from the SAMPL6 challenge. J Comput Aided Mol Des 32(10):1047–1058

  77. 77.

    Dixon T, Lotz SD, Dickson A (2018) Predicting ligand binding affinity using on-and off-rates for the SAMPL6 SAMPLing challenge. J Comput Aided Mol Des 32(10):1001–1012

  78. 78.

    Shirts MR, Chodera JD (2008) Statistically optimal analysis of samples from multiple equilibrium states. J Chem Phys 129(12):124105. https://doi.org/10.1063/1.2978177

  79. 79.

    Rizzi A, Chodera J, Naden L, Beauchamp K, Grinaway P, Rustenburg B, Albanese S, Saladi S (2018) choderalab/yank: 0.20.1: exact treatment of PME electrostatics and optimizations. https://doi.org/10.5281/zenodo.1161274

  80. 80.

    Wang K, Chodera JD, Yang Y, Shirts MR (2013) Identifying ligand binding sites and poses using GPU-accelerated Hamiltonian replica exchange molecular dynamics. J Comput Aided Mol Des 27(12):989–1007

  81. 81.

    Beutler TC, Mark AE, van Schaik RC, Gerber PR, Van Gunsteren WF (1994) Avoiding singularities and numerical instabilities in free energy calculations based on molecular simulations. Chem Phys Lett 222(6):529–539

  82. 82.

    Chodera JD, Shirts MR (2011) Replica exchange and expanded ensemble simulations as Gibbs sampling: simple improvements for enhanced mixing. J Chem Phys 135(19):194110

  83. 83.

    Desgranges C, Delhommelle J (2012) Evaluation of the grand-canonical partition function using expanded Wang–Landau simulations. I. Thermodynamic properties in the bulk and at the liquid-vapor phase boundary. J Chem Phys 136(18):184107

  84. 84.

    Wang F, Landau D (2001) Efficient, multiple-range random walk algorithm to calculate the density of states. Phys Rev Lett 86(10):2050

  85. 85.

    Woods CJ, Mey AS, Calabro G, Julien M (2019) Sire molecular simulation framework. https://siremol.org

  86. 86.

    Andersen HC (1980) Molecular dynamics simulations at constant pressure and/or temperature. J Chem Phys 72(4):2384–2393

  87. 87.

    Tironi IG, Sperb R, Smith PE, van Gunsteren WF (1995) A generalized reaction field method for molecular dynamics simulations. J Chem Phys 102(13):5451–5459

  88. 88.

    Barker J, Watts R (1973) Monte Carlo studies of the dielectric properties of water-like models. Mol Phys 26(3):789–792

  89. 89.

    Bernardi R, Bhandarkar M, Bhatele BAA, Brunner R, Buelens F, Chipot C, Dalke A, Dixit S, Fiorin G, Freddolino P, Fu H, Grayson P, Gullingsrud J, Gursoy A, Hardy D, Harrison C, Hénin J, Humphrey W, Hurwitz D, Hynninen A, et al (2019) NAMD User’s guide. Version 2.12

  90. 90.

    Gapsys V, Michielssens S, Peters JH, de Groot BL, Leonov H (2015) Calculation of binding free energies. In: Molecular modeling of proteins. Springer, New York, pp 173–209

  91. 91.

    Boresch S, Tettinger F, Leitgeb M, Karplus M (2003) Absolute binding free energies: a quantitative approach for their calculation. J Phys Chem B 107(35):9535–9551. https://doi.org/10.1021/jp0217839

  92. 92.

    Crooks GE (1999) Entropy production fluctuation theorem and the nonequilibrium work relation for free energy differences. Phys Rev E 60(3):2721

  93. 93.

    Velez-Vega C, Gilson MK (2013) Overcoming dissipation in the calculation of standard binding free energies by ligand extraction. J Comput Chem 34(27):2360–2371

  94. 94.

    Henriksen NM, Fenley AT, Gilson MK (2015) Computational calorimetry: high-precision calculation of host–guest binding thermodynamics. J Chem Theory Comput 11(9):4377–4394

  95. 95.

    Donyapour N, Roussey NM, Dickson A (2019) REVO: resampling of ensembles by variation optimization. J Chem Phys 150:244112

  96. 96.

    Essmann U, Perera L, Berkowitz ML, Darden T, Lee H, Pedersen LG (1995) A smooth particle mesh Ewald method. J Chem Phys 103(19):8577–8593. https://doi.org/10.1063/1.470117

  97. 97.

    You W, Tang Z, Chang CeA (2019) Potential mean force from umbrella sampling simulations: what can we learn and what is missed? J Chem Theory Comput 15(4):2433–2443

  98. 98.

    Laury ML, Wang Z, Gordon AS, Ponder JW (2018) Absolute binding free energies for the SAMPL6 cucurbit [8] uril host–guest challenge via the AMOEBA polarizable force field. J Comput Aided Mol Des 32(10):1087–1095

  99. 99.

    Efron B (1987) Better bootstrap confidence intervals. J Am Stat Assoc 82(397):171–185

  100. 100.

    Monroe JI, Shirts MR (2014) Converging free energies of binding in cucurbit [7] uril and octa-acid host–guest systems from SAMPL4 using expanded ensemble simulations. J Comput Aided Mol Des 28(4):401–415

  101. 101.

    Crooks GE (2000) Path-ensemble averages in systems driven far from equilibrium. Phys Rev E 61(3):2361

  102. 102.

    Hummer G (2001) Fast-growth thermodynamic integration: error and efficiency analysis. J Chem Phys 114(17):7330–7337

  103. 103.

    Jarzynski C (1997) Nonequilibrium equality for free energy differences. Phys Rev Lett 78(14):2690

  104. 104.

    Chow KH, Ferguson DM (1995) Isothermal-isobaric molecular dynamics simulations with Monte Carlo volume sampling. Comput Phys Commun 91(1–3):283–289

  105. 105.

    Åqvist J, Wennerström P, Nervall M, Bjelic S, Brandsdal BO (2004) Molecular dynamics simulations of water and biomolecules with a Monte Carlo constant pressure algorithm. Chem Phys Lett 384(4–6):288–294

  106. 106.

    Berendsen HJ, van Gunsteren WF, DiNola A, Haak J (1984) Postma Jv, molecular dynamics with coupling to an external bath. J Chem Phys 81(8):3684–3690

  107. 107.

    Merz PT, Shirts MR (2018) Testing for physical validity in molecular simulations. PLoS ONE 13(9):e0202764

  108. 108.

    Shirts MR (2013) Simple quantitative tests to validate sampling from thermodynamic ensembles. J Chem Theory Comput 9(2):909–926

  109. 109.

    Lehmann EL, Casella G (2006) Theory of point estimation. Springer, New York

  110. 110.

    Chodera JD, Mobley DL (2013) Entropy-enthalpy compensation: role and ramifications in biomolecular ligand recognition and design. Annu Rev Biophys 42(1):121–142. https://doi.org/10.1146/annurev-biophys-083012-130318

  111. 111.

    Chodera JD, Swope WC, Pitera JW, Seok C, Dill KA (2007) Use of the weighted histogram analysis method for the analysis of simulated and parallel tempering simulations. J Chem Theory Comput 3(1):26–41

  112. 112.

    Murkli S, McNeill JN, Isaacs L (2019) Cucurbit [8] uril · guest complexes: blinded dataset for the SAMPL6 challenge. Supramol Chem 31(3):150–158

  113. 113.

    Sullivan MR, Yao W, Gibb BC (2019) The thermodynamics of guest complexation to octa-acid and tetra-endo-methyl octa-acid: reference data for the sixth statistical assessment of modeling of proteins and ligands (SAMPL6). Supramol Chem 31(3):184–189

  114. 114.

    Pohorille A, Jarzynski C, Chipot C (2010) Good practices in free-energy calculations. J Phys Chem B 114(32):10235–10253

  115. 115.

    Grossfield A, Patrone PN, Roe DR, Schultz AJ, Siderius DW, Zuckerman DM (2018) Best practices for quantification of uncertainty and sampling quality in molecular simulations [Article v1. 0]. Living J Comput Mol Sci. 1(1)

  116. 116.

    Bhati AP, Wan S, Hu Y, Sherborne B, Coveney PV (2018) Uncertainty quantification in alchemical free energy methods. J Chem Theory Comput 14(6):2867–2880

  117. 117.

    Balasubramanian V, Jensen T, Turilli M, Kasson P, Shirts M, Jha S (2018) Adaptive ensemble biomolecular simulations at scale. arXiv:180404736

  118. 118.

    Wang L, Berne B, Friesner RA (2012) On achieving high accuracy and reliability in the calculation of relative protein-ligand binding affinities. Proc Natl Acad Sci 109(6):1937–1942

  119. 119.

    Hedges L, Mey A, Laughton C, Gervasio F, Mulholland A, Woods C, Michel J (2019) BioSimSpace: an interoperable Python framework for biomolecular simulation. J Open Source Software 4(43):1831

  120. 120.

    Dakka J, Turilli M, Wright DW, Zasada SJ, Balasubramanian V, Wan S, Coveney PV, Jha S (2018) High-throughput binding affinity calculations at extreme scales. BMC Bioinformatics 19(S18)

  121. 121.

    Shelley JC, Cholleti A, Frye LL, Greenwood JR, Timlin MR, Uchimaya M (2007) Epik: a software program for pK a prediction and protonation state generation for drug-like molecules. J Comput Aided Mol Des 21(12):681–691. https://doi.org/10.1007/s10822-007-9133-z

  122. 122.

    Greenwood JR, Calkins D, Sullivan AP, Shelley JC (2010) Towards the comprehensive, rapid, and accurate prediction of the favorable tautomeric states of drug-like molecules in aqueous solution. J Comput Aided Mol Des 24(6–7):591–604. https://doi.org/10.1007/s10822-010-9349-1

  123. 123.

    Wang J, Wang W, Kollman PA, Case DA (2006) Automatic atom type and bond type perception in molecular mechanical calculations. J Mol Graph Model 25(2):247–260

  124. 124.

    Case D, Betz R, Cerutti D, Cheatham T III, Darden T, Duke R, Giese T, Gohlke H, Goetz A, Homeyer N, Izadi S, Janowski P, Kaus J, Kovalenko A, Lee T, LeGrand S, Li P, Lin C, Luchko T et al (2016) AMBER 16. University of California, San Francisco

  125. 125.

    Joung IS, Cheatham TE III (2008) Determination of alkali and halide monovalent ion parameters for use in explicitly solvated biomolecular simulations. J Phys Chem B 112(30):9020–9041

  126. 126.

    Chodera J, Rizzi A, Naden L, Beauchamp K, Grinaway P, Fass J, Rustenburg B, Ross GA, Simmonett A, Swenson DWH (2018) choderalab/openmmtools: 0.14.0: exact treatment of alchemical PME electrostatics, water cluster test system, optimizations. https://doi.org/10.5281/zenodo.1161149

  127. 127.

    McGibbon RT, Beauchamp KA, Harrigan MP, Klein C, Swails JM, Hernández CX, Schwantes CR, Wang LP, Lane TJ, Pande VS (2015) MDTraj: a modern open library for the analysis of molecular dynamics trajectories. Biophys J 109(8):1528–1532. https://doi.org/10.1016/j.bpj.2015.08.015

  128. 128.

    Parrinello M, Rahman A (1980) Crystal structure and pair potentials: a molecular-dynamics study. Phys Rev Lett 45(14):1196

  129. 129.

    Gapsys V, Michielssens S, Seeliger D, de Groot BL (2015) pmx: automated protein structure and topology generation for alchemical perturbations. J Comput Chem 36(5):348–354

  130. 130.

    Bennett CH (1976) Efficient estimation of free energy differences from Monte Carlo data. J Comput Phys 22(2):245–268. https://doi.org/10.1016/0021-9991(76)90078-4

  131. 131.

    Shirts MR, Bair E, Hooker G, Pande VS (2003) Equilibrium free energies from nonequilibrium measurements using maximum-likelihood methods. Phys Rev Lett 91(14):140601

  132. 132.

    Feller SE, Zhang Y, Pastor RW, Brooks BR (1995) Constant pressure molecular dynamics simulation: the Langevin piston method. J Chem Phys 103(11):4613–4621

  133. 133.

    Jakobsen AF (2005) Constant-pressure and constant-surface tension simulations in dissipative particle dynamics. J Chem Phys 122(12):124901

  134. 134.

    Liu P, Dehez F, Cai W, Chipot C (2012) A toolkit for the analysis of free-energy perturbation calculations. J Chem Theory Comput 8(8):2606–2616

  135. 135.

    Chodera JD (2016) A simple method for automated equilibration detection in molecular simulations. J Chem Theory Comput 12(4):1799–1805

  136. 136.

    Sheppard K, Khrapov S, Lipták G, Capellini R (2018) esvhd, Hugle, JPN, RENE-CORAIL X, Rose ME, jbrockmendel, bashtage/arch: Release 4.7. 2018. https://doi.org/10.5281/zenodo.2240590

Download references


AR and JDC acknowledge support from the Sloan Kettering Institute. JDC acknowledges support from NIH grant P30 CA008748. AR acknowledges partial support from the Tri-Institutional Program in Computational Biology and Medicine. DLM appreciates financial support from the National Institutes of Health (1R01GM108889-01 and 1R01GM124270-01A1) and the National Science Foundation (CHE 1352608). AR and JDC are grateful to OpenEye Scientific for providing a free academic software license for use in this work. MA was supported by a Postdoctoral Research Fellowship of the Alexander von Humboldt Foundation. VG and BLdG were supported by BioExcel CoE, a project funded by the European Union contract H2020-INFRAEDI-02-2018-823830. MKG thanks NIGMS (NIH) for partial support of this project (GM061300). The contents of this publication are solely the responsibility of the authors and do not necessarily represent the official views of the NIH. AD acknowledges support from the National Institutes of Health (R01GM130794) and the National Science Foundation (DMS 1761320). ZC and DN would like to thank Stamatia Zavitsanou, Michail Papadourakis and Chris Chipot for useful discussions. This work was further supported by computational time granted from the Greek Research & Technology Network (GRNET) in the National HPC facility—ARIS—under Project ID vspr001005/arp2/3. ZC acknowledges support of this work by the project “An Open-Access Research Infrastructure of Chemical Biology and Target-Based Screening Technologies for Human and Animal Health, Agriculture and the Environment (OPENSCREEN-GR)” (MIS 5002691), which is implemented under the Action “Reinforcement of the Research and Innovation Infrastructure”, funded by the Operational Programme “Competitiveness, Entrepreneurship and Innovation” (NSRF 2014-2020) and co-financed by Greece and the European Union (European Regional Development Fund). TJ and MRS acknowledge a CU Boulder Research and Innovation Seed Grant.


JDC was a member of the Scientific Advisory Board for Schrödinger, LLC during part of this study. JDC and DLM are current members of the Scientific Advisory Board of OpenEye Scientific Software. The Chodera laboratory receives or has received funding from multiple sources, including the National Institutes of Health, the National Science Foundation, the Parker Institute for Cancer Immunotherapy, Relay Therapeutics, Entasis Therapeutics, Silicon Therapeutics, EMD Serono (Merck KGaA), AstraZeneca, Vir Biotechnology, XtalPi, the Molecular Sciences Software Institute, the Starr Cancer Consortium, the Open Force Field Consortium, Cycle for Survival, a Louis V. Gerstner Young Investigator Award, and the Sloan Kettering Institute. A complete funding history for the Chodera lab can be found at http://choderalab.org/funding. MKG has an equity interest in, and is a cofounder and scientific advisor of VeraChem LLC.

Author information

Conceptualization: DLM, AR, JDC, MS, JM; Data Curation: AR; Formal Analysis: AR; Funding Acquisition: JDC, DLM, MRS, MKG, AD, BLdG, JM, ZC; Investigation: AR, TJ, DRS, MA, VG, AD, DN, SB, NMH, MP; Methodology: AR, DLM, MKG, JM, JDC; Project Administration: AR, DLM, JDC; Resources: JDC, MRS, MKG, ZC, JM, AD, BLdG; Software: AR; Supervision: JDC, MRS, MKG, DLM, JM, ZC, AD, BLdG; Visualization: AR, VG, TJ; Writing—Original Draft: AR; Writing—Review & Editing: AR, MKG, JDC, DLM, DRS, MRS, MA, VG, JM, DN, AD, ZC, BLdG.

Correspondence to Andrea Rizzi or David L. Mobley or John D. Chodera.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations

Electronic supplementary material

Below is the link to the electronic supplementary material.

Electronic supplementary material 1 (PDF 7176 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Rizzi, A., Jensen, T., Slochower, D.R. et al. The SAMPL6 SAMPLing challenge: assessing the reliability and efficiency of binding free energy calculations. J Comput Aided Mol Des (2020). https://doi.org/10.1007/s10822-020-00290-5

Download citation


  • SAMPL6
  • Host–guest
  • Binding affinity
  • Free energy calculations
  • Cucurbit[8]uril
  • Octa-acid
  • Sampling