Modeling error in experimental assays using the bootstrap principle: understanding discrepancies between assays using different dispensing technologies
- 321 Downloads
All experimental assay data contains error, but the magnitude, type, and primary origin of this error is often not obvious. Here, we describe a simple set of assay modeling techniques based on the bootstrap principle that allow sources of error and bias to be simulated and propagated into assay results. We demonstrate how deceptively simple operations—such as the creation of a dilution series with a robotic liquid handler—can significantly amplify imprecision and even contribute substantially to bias. To illustrate these techniques, we review an example of how the choice of dispensing technology can impact assay measurements, and show how large contributions to discrepancies between assays can be easily understood and potentially corrected for. These simple modeling techniques—illustrated with an accompanying IPython notebook—can allow modelers to understand the expected error and bias in experimental datasets, and even help experimentalists design assays to more effectively reach accuracy and imprecision goals.
KeywordsError modeling Assay modeling Bootstrap principle Dispensing technologies Liquid handling Direct dispensing Acoustic droplet ejection
The authors are grateful to Anthony Nichols (OpenEye) and Martin Stahl (Roche) for organizing the excellent 2013 Computer-Aided Drug Discovery Gordon Research Conference on the topic of “The Statistics of Drug Discovery”, as well as Terry Stouch for both his infinite patience and inspiring many of the ideas in this work. The authors are especially grateful to Cosma Shalizi for presenting a clear and lucid overview of the bootstrap principle to this audience, and we hope this contribution can further aid readers in the community in employing these principles in their work. The authors further acknowledge Adrienne Chow and Anthony Lozada of Tecan US for a great deal of assistance in understanding the nature of operation and origin of errors in automated liquid handling equipment. The authors thank Paul Czodrowski (Merck Darmstadt) for introducing us to IPython notebooks as a means of interactive knowledge transfer. JDC and SMH acknowledge support from the Sloan Kettering Institute, a Louis V. Gerstner Young Investigator Award, and NIH Grant P30 CA008748. SE acknowledges Joe Olechno and Antony Williams for extensive discussions on the topic, as well as the many scientists that responded to the various blog posts mentioned herein.
Compliance with ethical standards
Conflict of interest
The authors acknowledge no conflicts of interest, but wish to disclose that JDC is on the Scientific Advisory Board of Schrödinger and SE is an employee of Collaborative Drug Discovery.
- 20.Barlaam BC, Ducray R (2009) Novel pyrimidine derivatives 965, uS20090054428 A1Google Scholar
- 21.Barlaam BC, Ducray R, Kettle JG (2010) Pyrimidine derivatives for inhibiting Eph receptors, uS7718653 B2Google Scholar
- 24.Lowe D (2015) Drug assay numbers, all over the place. In the Pipeline: http://blogs.sciencemag.org/pipeline/archives/2013/05/03/drug_assay_numbers_all_over_the_place
- 25.Evanko D (2013) Serial dilution woes. http://blogs.nature.com/methagora/2013/05/serial-dilution-woes.html
- 26.Ekins S (2013) What it took to get the paper out. http://www.collabchem.com/2013/05/03/what-it-took-to-get-the-paper-out/
- 28.Taylor JR (1997) An introduction to error analysis: the study of uncertainties in physical measurements. University Science Books, Mill ValleyGoogle Scholar
- 29.Shalizi C Simple simulation methods for quantifying uncertainty. https://www.grc.org/programs.aspx?year=2013&program=cadd, https://openeye.app.box.com/s/ebafecebpplobcdy1yxg, http://lanyrd.com/2013/grc-cadd-2013/slides
- 33.Tecan Genesis Operating Manual (2001). http://www.tecan.com
- 34.Echo 5XX Specifications (2011). http://www.labcyte.com/inc/lib/download_sheets?pkid=11
- 38.Wingfield J (2012) Impact of acoustic dispensing on data quality in HTS and hit confirmation. Drug Discovery 2012. ManchesterGoogle Scholar
- 39.Olechno J, Ekins S, Williams AJ, Fischer-Colbrie M (2013) Direct improvement with direct dilution. http://americanlaboratory.com/914-Application-Notes/142860-Direct-Improvement-With-Direct-Dilution/
- 40.Olechno J, Ekins S, Williams AJ (2013) Sound dilutions. https://theanalyticalscientist.com/issues/0713/sound-dilutions/
- 42.Jones RE, Zheng W, McKew JC, Chen CZ (1094) J Lab Autom 221106821349:2013Google Scholar