Journal of Computer-Aided Molecular Design

, Volume 29, Issue 12, pp 1073–1086 | Cite as

Modeling error in experimental assays using the bootstrap principle: understanding discrepancies between assays using different dispensing technologies

  • Sonya M. Hanson
  • Sean Ekins
  • John D. ChoderaEmail author


All experimental assay data contains error, but the magnitude, type, and primary origin of this error is often not obvious. Here, we describe a simple set of assay modeling techniques based on the bootstrap principle that allow sources of error and bias to be simulated and propagated into assay results. We demonstrate how deceptively simple operations—such as the creation of a dilution series with a robotic liquid handler—can significantly amplify imprecision and even contribute substantially to bias. To illustrate these techniques, we review an example of how the choice of dispensing technology can impact assay measurements, and show how large contributions to discrepancies between assays can be easily understood and potentially corrected for. These simple modeling techniques—illustrated with an accompanying IPython notebook—can allow modelers to understand the expected error and bias in experimental datasets, and even help experimentalists design assays to more effectively reach accuracy and imprecision goals.


Error modeling Assay modeling Bootstrap principle Dispensing technologies Liquid handling Direct dispensing Acoustic droplet ejection 



The authors are grateful to Anthony Nichols (OpenEye) and Martin Stahl (Roche) for organizing the excellent 2013 Computer-Aided Drug Discovery Gordon Research Conference on the topic of “The Statistics of Drug Discovery”, as well as Terry Stouch for both his infinite patience and inspiring many of the ideas in this work. The authors are especially grateful to Cosma Shalizi for presenting a clear and lucid overview of the bootstrap principle to this audience, and we hope this contribution can further aid readers in the community in employing these principles in their work. The authors further acknowledge Adrienne Chow and Anthony Lozada of Tecan US for a great deal of assistance in understanding the nature of operation and origin of errors in automated liquid handling equipment. The authors thank Paul Czodrowski (Merck Darmstadt) for introducing us to IPython notebooks as a means of interactive knowledge transfer. JDC and SMH acknowledge support from the Sloan Kettering Institute, a Louis V. Gerstner Young Investigator Award, and NIH Grant P30 CA008748. SE acknowledges Joe Olechno and Antony Williams for extensive discussions on the topic, as well as the many scientists that responded to the various blog posts mentioned herein.

Compliance with ethical standards

Conflict of interest

The authors acknowledge no conflicts of interest, but wish to disclose that JDC is on the Scientific Advisory Board of Schrödinger and SE is an employee of Collaborative Drug Discovery.

Supplementary material

10822_2015_9888_MOESM1_ESM.ipynb (474 kb)
Supplementary material 1 (IPYNB 473 KB)
10822_2015_9888_MOESM2_ESM.html (806 kb)
Supplementary material 2 (HTML 806 KB)


  1. 1.
    Kozikowski BA (2003) J Biomol Screen 8:210CrossRefGoogle Scholar
  2. 2.
    Kozikowski BA (2003) J Biomol Screen 8:205CrossRefGoogle Scholar
  3. 3.
    Cheng X, Hochlowski J, Tang H, Hepp D, Beckner C, Kantor S, Schmitt R (2003) J Biomol Screen 8:292CrossRefGoogle Scholar
  4. 4.
    Waybright TJ, Britt JR, McCloud TG (2009) J Biomol Screen 14:708CrossRefGoogle Scholar
  5. 5.
    Harris D, Olechno J, Datwani S, Ellson R (2010) J Biomol Screen 15:86CrossRefGoogle Scholar
  6. 6.
    Grant RJ, Roberts K, Pointon C, Hodgson C, Womersley L, Jones DC, Tang E (2009) J Biomol Screen 14:452CrossRefGoogle Scholar
  7. 7.
    Tjernberg A (2005) J Biomol Screen 11:131CrossRefGoogle Scholar
  8. 8.
    Simeonov A, Jadhav A, Thomas CJ, Wang Y, Huang R, Southall NT, Shinn P, Smith J, Austin CP, Auld DS, Inglese J (2008) J Med Chem 51:2363CrossRefGoogle Scholar
  9. 9.
    Baell JB, Holloway GA (2010) J Med Chem 53:2719CrossRefGoogle Scholar
  10. 10.
    Di L, Kerns EH (2006) Drug Discov Today 11:446CrossRefGoogle Scholar
  11. 11.
    McGovern SL, Caselli E, Grigorieff N, Shoichet BK (2002) J Med Chem 45:1712CrossRefGoogle Scholar
  12. 12.
    McGovern SL, Shoichet BK (2003) J Med Chem 46:1478CrossRefGoogle Scholar
  13. 13.
    Feng BY, Shelat A, Doman TN, Guy RK, Shoichet BK (2005) Nat Chem Biol 1:146CrossRefGoogle Scholar
  14. 14.
    Feng BY, Shoichet BK (2006) J Med Chem 49:2151CrossRefGoogle Scholar
  15. 15.
    Busch M, Thorma HB, Kober I (2015) J Biomol Screen 18:744CrossRefGoogle Scholar
  16. 16.
    Chodera JD, Mobley DL (2013) Annu Rev Biophys 42:121CrossRefGoogle Scholar
  17. 17.
    Kramer C, Kalliokoski T, Gedeck P, Vulpetti A (2012) J Med Chem 55:5165CrossRefGoogle Scholar
  18. 18.
    Kalliokoski T, Kramer C, Vulpetti A, Gedeck P (2013) PLoS ONE 8:e61007CrossRefGoogle Scholar
  19. 19.
    Ekins S, Olechno J, Williams AJ (2013) PLoS ONE 8:e62325CrossRefGoogle Scholar
  20. 20.
    Barlaam BC, Ducray R (2009) Novel pyrimidine derivatives 965, uS20090054428 A1Google Scholar
  21. 21.
    Barlaam BC, Ducray R, Kettle JG (2010) Pyrimidine derivatives for inhibiting Eph receptors, uS7718653 B2Google Scholar
  22. 22.
    Xia G, Kumar SR, Masood R, Zhu S, Reddy R, Krasnoperov V, Quinn DI, Henshall SM, Sutherland RL, Pinski JK et al (2005) Cancer Res 65:4623CrossRefGoogle Scholar
  23. 23.
    Bardelle C, Cross D, Davenport S, Kettle JG, Ko EJ, Leach AG, Mortlock A, Read J, Roberts NJ, Robins P, Williams EJ (2008) Bioorg Med Chem Lett 18:2776CrossRefGoogle Scholar
  24. 24.
    Lowe D (2015) Drug assay numbers, all over the place. In the Pipeline:
  25. 25.
  26. 26.
    Ekins S (2013) What it took to get the paper out.
  27. 27.
    Palmgren J, Monkkonen J, Korjamo T, Hassinen A, Auriola S (2006) Eur J Pharm Biopharm 64:369CrossRefGoogle Scholar
  28. 28.
    Taylor JR (1997) An introduction to error analysis: the study of uncertainties in physical measurements. University Science Books, Mill ValleyGoogle Scholar
  29. 29.
  30. 30.
    Walling L, Carramanzana N, Schulz C, Romig T, Johnson M (2007) ASSAY Drug Dev Technol 5:265CrossRefGoogle Scholar
  31. 31.
    Weiss S, John G, Klimant I, Heinzle E (2002) Biotechnol Prog 18:821CrossRefGoogle Scholar
  32. 32.
    Mitre E, Schulze M, Cumme GA, Rossler F, Rausch T, Rhode H (2007) J Biomol Screen 12:361CrossRefGoogle Scholar
  33. 33.
    Tecan Genesis Operating Manual (2001).
  34. 34.
  35. 35.
    Dong H, Ouyang Z, Liu J, Jemal M (2006) J Assoc Lab Autom 11:60CrossRefGoogle Scholar
  36. 36.
    Gu H, Deng Y (2007) J Assoc Lab Autom 12:355CrossRefGoogle Scholar
  37. 37.
    Jones G (2015) J Comput Aided Mol Des 29:1CrossRefGoogle Scholar
  38. 38.
    Wingfield J (2012) Impact of acoustic dispensing on data quality in HTS and hit confirmation. Drug Discovery 2012. ManchesterGoogle Scholar
  39. 39.
    Olechno J, Ekins S, Williams AJ, Fischer-Colbrie M (2013) Direct improvement with direct dilution.
  40. 40.
    Olechno J, Ekins S, Williams AJ (2013) Sound dilutions.
  41. 41.
    Olechno J, Shieh J, Ellson R (2006) J Assoc Lab Autom 11:240CrossRefGoogle Scholar
  42. 42.
    Jones RE, Zheng W, McKew JC, Chen CZ (1094) J Lab Autom 221106821349:2013Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  1. 1.Computational Biology Program, Sloan Kettering InstituteMemorial Sloan Kettering Cancer CenterNew YorkUSA
  2. 2.Collaborations in ChemistryFuquay-VarinaUSA

Personalised recommendations