, Volume 2, Issue 4, pp 171–196 | Cite as

Statistical strategies for avoiding false discoveries in metabolomics and related experiments

  • David I. BroadhurstEmail author
  • Douglas B. KellEmail author

Many metabolomics, and other high-content or high-throughput, experiments are set up such that the primary aim is the discovery of biomarker metabolites that can discriminate, with a certain level of certainty, between nominally matched ‘case’ and ‘control’ samples. However, it is unfortunately very easy to find markers that are apparently persuasive but that are in fact entirely spurious, and there are well-known examples in the proteomics literature. The main types of danger are not entirely independent of each other, but include bias, inadequate sample size (especially relative to the number of metabolite variables and to the required statistical power to prove that a biomarker is discriminant), excessive false discovery rate due to multiple hypothesis testing, inappropriate choice of particular numerical methods, and overfitting (generally caused by the failure to perform adequate validation and cross-validation). Many studies fail to take these into account, and thereby fail to discover anything of true significance (despite their claims). We summarise these problems, and provide pointers to a substantial existing literature that should assist in the improved design and evaluation of metabolomics experiments, thereby allowing robust scientific conclusions to be drawn from the available data. We provide a list of some of the simpler checks that might improve one’s confidence that a candidate biomarker is not simply a statistical artefact, and suggest a series of preferred tests and visualisation tools that can assist readers and authors in assessing papers. These tools can be applied to individual metabolites by using multiple univariate tests performed in parallel across all metabolite peaks. They may also be applied to the validation of multivariate models. We stress in particular that classical p-values such as “p < 0.05”, that are often used in biomedicine, are far too optimistic when multiple tests are done simultaneously (as in metabolomics). Ultimately it is desirable that all data and metadata are available electronically, as this allows the entire community to assess conclusions drawn from them. These analyses apply to all high-dimensional ‘omics’ datasets.


statistics machine learning false discovery receiver–operator characteristic hypothesis testing statistical power Bonferroni correction bias overfitting cross validiation credit assignment visualisation 



We thank the BBSRC, MRC and BHF for financial support and many colleagues for useful discussions and examples.


  1. Adriaans P., Zantinge D. (1996) Data Mining. Addison-Wesley, Harlow, EssexGoogle Scholar
  2. Alsberg B.K., Kell D.B., Goodacre R. (1998) Variable selection in discriminant partial least-squares analysis. Anal. Chem. 70: 4126–4133Google Scholar
  3. Alsberg B.K., Woodward A.M., Winson M.K., Rowland J., Kell D.B. (1997) Wavelet denoising of infrared spectra. Analyst 122: 645–652Google Scholar
  4. Altman D.G. (2001) Systematic reviews of evaluations of prognostic variables. BMJ 323: 224–228PubMedGoogle Scholar
  5. Altman D.G., Deeks J.J. (2002) Meta-analysis, Simpson’s paradox, and the number needed to treat. BMC Med. Res. Methodol. 2: 3PubMedGoogle Scholar
  6. Anthony M., Biggs N. (1992) Computational Learning Theory. Cambridge University Press, CambridgeGoogle Scholar
  7. Baggerly K.A., Morris J.S., Coombes K.R. (2004) Reproducibility of SELDI-TOF protein patterns in serum: comparing datasets from different experiments. Bioinformatics 20: 777–785PubMedGoogle Scholar
  8. Baker S.G. (2003) The central role of receiver operating characteristic (ROC) curves in evaluating tests for the early detection of cancer. J. Natl. Cancer Inst. 95: 511–515PubMedCrossRefGoogle Scholar
  9. Baldi P., Long A.D. (2001) A Bayesian framework for the analysis of microarray expression data: regularized t-test and statistical inferences of gene changes. Bioinformatics 17: 509–519PubMedGoogle Scholar
  10. Barrow J.D., Silk J. (1995) The Left Hand of Creation: The Origin and Evolution of The Expanding Universe. Penguin, LondonGoogle Scholar
  11. Bellman R. (1961) Adaptive Control Processes: A Guided Tour. Princeton University Press, Princeton, NJGoogle Scholar
  12. Benjamini Y., Hochberg Y. (1995) Controlling the false discovery rate – a practical and powerful approach to multiple testing. J. Roy. Stat. Soc. B Met. 57: 289–300Google Scholar
  13. Bennett K., Demiriz A. (1998) Semi-supervised support vector machines. Adv. Neural Inf. Proc. Syst. 12: 368–374Google Scholar
  14. Bernardo J.M., Smith A.F.M. (2000) Bayesian Theory. Wiley, ChichesterGoogle Scholar
  15. Berry D.A. (1996) Statistics: A Bayesian Perspective. Duxbury Press, BelmontGoogle Scholar
  16. Berry M.J.A., Linoff G.S. (2000) Mastering the Art of Data Mining. Wiley, New YorkGoogle Scholar
  17. Bezdek J.C. and Pal, S.K. (Eds) (1992). Fuzzy Models for Pattern recognition: Methods That Search for Structures In Data. IEEE Press., New YorkGoogle Scholar
  18. Bland J.M., Altman D.G. (1995) Multiple significance tests: the Bonferroni method. BMJ 310: 170PubMedGoogle Scholar
  19. Bland M. (2000) An Introduction to Medical Statistics. Oxford University Press, OxfordGoogle Scholar
  20. Box G.E.P., Hunter W.G., Hunter J.S. (1978) Statistics for Experimenters. Wiley, New YorkGoogle Scholar
  21. Bradford Hill A., Hill I.D. (1991) Bradford Hill’s Principles of medical statistics 12. Edward Arnold, LondonGoogle Scholar
  22. Breiman L. (1966) The heuristics of instability in model selection. Ann. Statist. 24: 2350–2381Google Scholar
  23. Breiman L. (2001) Statistical modeling: The two cultures. Stat. Sci. 16: 199–215Google Scholar
  24. Brenner H., Gefeller O. (1997) Variation of sensitivity, specificity, likelihood ratios and predictive values with disease prevalence. Stat. Med. 16: 981–91PubMedGoogle Scholar
  25. Brent R. (1999) Functional genomics: learning to think about gene expression data. Curr. Biol. 9: R338–R341PubMedGoogle Scholar
  26. Brent R. (2000) Genomic biology. Cell 100: 169–183PubMedGoogle Scholar
  27. Brent R., Lok L. (2005) A fishing buddy for hypothesis generators. Science 308: 504–506PubMedGoogle Scholar
  28. Brereton R.G. (2003) Chemometrics: Data Analysis for the Laboratory and Chemical Plant. Wiley, New YorkGoogle Scholar
  29. Broadhurst D., Goodacre R., Jones A., Rowland J.J. Kell D.B. (1997) Genetic algorithms as a method for variable selection in multiple linear regression and partial least squares regression, with applications to pyrolysis mass spectrometry. Anal. Chim. Acta. 348: 71–86Google Scholar
  30. Brown M., Dunn W.B., Ellis D.I., Goodacre R., Handl J., Knowles J.D., O’Hagan S., Spasic I., Kell D.B. (2005) A metabolome pipeline: from concept to data to knowledge. Metabolomics 1: 35–46Google Scholar
  31. Cabena P., Hadjinian P., Stadler R., Verhees J., Zanasi A. (1998) Discovering Data Mining: From Concept to Implementation. Prentice Hall, Englewood Cliffs, NJGoogle Scholar
  32. Camacho D., de la Fuente A., Mendes P. (2005) The origins of correlations in metabolomics data. Metabolomics 1: 53–63Google Scholar
  33. Cascante M., Boros L.G., Comin-Anduix B., de Atauri P., Centelles J.J., Lee P.W. (2002) Metabolic control analysis in drug discovery and disease. Nat. Biotechnol. 20: 243–249PubMedGoogle Scholar
  34. Casella G., Berger R.L. (2002) Statistical Inference, 2. Duxbury, Pacific Grove, CAGoogle Scholar
  35. Catchpole G.S., Beckmann M., Enot D.P., Mondhe M., Zywicki B., Taylor J., Hardy N., Smith A., King R.D., Kell D.B., Fiehn O., Draper J. (2005) Hierarchical metabolomics demonstrates substantial compositional similarity between genetically modified and conventional potato crops. Proc. Natl. Acad. Sci. 102: 14458–14462PubMedGoogle Scholar
  36. Chatfield C. (1995) Model uncertainty, data mining and statistical inference. J. Roy. Stat. Soc. Ser. A 158: 419–466Google Scholar
  37. Chen M., Hofestädt R. (2006) A medical bioinformatics approach for metabolic disorders: biomedical data prediction, modeling, and systematic analysis. J. Biomed. Inform. 39: 147–159PubMedGoogle Scholar
  38. Chen V.C.P., Tsui K.L., Barton R.R., Meckesheimer M. (2006) A review on design, modeling and applications of computer experiments. IIE Trans. 38: 273–291Google Scholar
  39. Cleveland W.S. (1993) Visualizing Data. Hobart Press, Summit, NJGoogle Scholar
  40. Cleveland W.S. (1994) The Elements of Graphing Data. Hobart Press, Summit, NJGoogle Scholar
  41. Coello Coello C.A., van Veldhuizen D.A., Lamont G.B. (2002) Evolutionary Algorithms for Solving Multi-Objective Problems. Kluwer Academic Publishers, New YorkGoogle Scholar
  42. Conover W.J. (1980) Practical Nonparametric Statistics. Wiley, New YorkGoogle Scholar
  43. Cook R.J., Farewell V.T. (1996) Multiplicity considerations in the design and analysis of clinical trials. J. Roy. Stat. Soc. A 159: 93–110Google Scholar
  44. Cornfield J. (1966) Sequential trials, sequential analysis and likelihood rinciple. Am. Stat. 20: 18–23Google Scholar
  45. Cornish-Bowden A., Cárdenas M.L. (2000) From genome to cellular phenotype-a role for metabolic flux analysis? Nat. Biotechnol. 18: 267–269PubMedGoogle Scholar
  46. Crary S.B. (2002) Design of computer experiments for metamodel generation. Analog. Integr. Circ. Sig. Proc. 32: 7–16Google Scholar
  47. Cui X., Churchill G.A. (2003) Statistical tests for differential expression in cDNA microarray experiments. Genome Biol. 4: 210PubMedGoogle Scholar
  48. Dasgupta P., Chakrabarti P.P., DeSarkar S.C. (1999) Multiobjective Heuristic Search. Vieweg, BraunschweigGoogle Scholar
  49. Deb K. (2001) Multi-Objective Optimization Using Evolutionary Algorithms. Wiley, New YorkGoogle Scholar
  50. Deming S.N., Morgan S.L. (1993) Experimental Design: A Chemometric Approach. Elsevier, AmsterdamGoogle Scholar
  51. Demiriz A., Bennett K., Embrechts M.J. (1999) Semi-supervised clustering using genetic algorithms. In Dagli C.H., Buczak A.L., Ghosh J., Embrechts M.J., Ersoy O. (Eds.), Intelligent Engineering Systems Through Artificial Neural Networks. ASME Press, New York, pp. 809–814Google Scholar
  52. di Bernardo D., Thompson M.J., Gardner T.S., Chobot S.E., Eastwood E.L., Wojtovich A.P., Elliott S.J., Schaus S.E., Collins J.J. (2005) Chemogenomic profiling on a genome-wide scale using reverse-engineered gene networks. Nat. Biotechnol. 23: 377–383PubMedGoogle Scholar
  53. Diamandis E.P. (2004) Analysis of serum proteomic patterns for early cancer diagnosis: drawing attention to potential problems. J. Natl. Cancer Inst. 96: 353–356PubMedCrossRefGoogle Scholar
  54. Duda R.O., Hart P.E., Stork D.E. (2001) Pattern Classification, 2. John Wiley, LondonGoogle Scholar
  55. Duesberg P., Stindl R., Hehlmann R. (2000) Explaining the high mutation rates of cancer cells to drug and multidrug resistance by chromosome reassortments that are catalyzed by aneuploidy. Proc. Natl. Acad. Sci. USA 97: 14295–14300PubMedGoogle Scholar
  56. Eades P. (1984) A heuristic for graph drawing. Congressus Numerantium 42: 149–160Google Scholar
  57. Ebbels T.M.D., Buxton B.F., Jones D.T. (2006) springScape: visualisation of microarray and contextual bioinformatic data using spring embedding an ‘information landscape’. Bioinformatics 22, e99–e108PubMedGoogle Scholar
  58. Edwards A.W.F. (1992) Likelihood. Johns Hopkins University Press, BaltimoreGoogle Scholar
  59. Edwards D. (2000) Introduction to Graphical Modeling. 2nd ed. Springer, BerlinGoogle Scholar
  60. Efron B., Gong G. (1983) A Leisurely Look at the Bootstrap, the Jackknife, and Cross-Validation. Am. Stat. 37: 36–48Google Scholar
  61. Efron B., Tibshirani R. (2002) Empirical Bayes methods and false discovery rates for microarrays. Genet. Epidemiol. 23: 70–86PubMedGoogle Scholar
  62. Efron B., Tibshirani R.J. (1993) Introduction to the Bootstrap. Chapman and Hall, LondonGoogle Scholar
  63. Egan J.P. (1975) Signal Detection Theory and ROC Analysis. Academic Press, New YorkGoogle Scholar
  64. Ein-Dor L., Zuk O., Domany E. (2006) Thousands of samples are needed to generate a robust gene list for predicting outcome in cancer. Proc. Natl. Acad. Sci USA 103: 5923–5928PubMedGoogle Scholar
  65. Eriksson L., Johansson E., Kettaneh-Wold N., Wold S. (2001) Multi- and Megavariate Data Analysis: Principles and Applications. Umetrics Academy, UmeåGoogle Scholar
  66. Evans W.E., Johnson J.A. (2001) Pharmacogenomics: the inherited basis for interindividual differences in drug response. Annu. Rev. Genomics. Hum. Genet. 2: 9–39PubMedGoogle Scholar
  67. Evans W.E., Relling M.V. (1999) Pharmacogenomics: translating functional genomics into rational therapeutics. Science 286: 487–491PubMedGoogle Scholar
  68. Evans W.E., Relling M.V. (2004) Moving towards individualized medicine with pharmacogenomics. Nature 429: 464–468PubMedGoogle Scholar
  69. Everitt B.S. (1993) Cluster Analysis. Edward Arnold, LondonGoogle Scholar
  70. Farnum M.A., DesJarlais, R. and Agrafiotis, D.K. (2003). Molecular diversity in Gasteiger, J. (Ed.), Handbook of Cheminformatics: vol 4 From Data to Knowledge. Wiley/VCH, Weinheim, pp. 1640–1686Google Scholar
  71. Fell D.A. (1996) Understanding the Control of Metabolism. Portland Press, LondonGoogle Scholar
  72. Fielding A.H., Bell J.F. (1997) A review of methods for the assessment of prediction errors in conservation presence/absence models. Environ. Conserv. 24: 38–49Google Scholar
  73. Fortner B. (1995) The Data Handbook. 2nd ed. Springer, New YorkGoogle Scholar
  74. Frey H.C., Patil S.R. (2002) Identification and review of sensitivity analysis methods. Risk Anal. 22: 553–578PubMedGoogle Scholar
  75. Friendly M. (2000) Visualising Categorical Data. SAS Institute, Cary, NCGoogle Scholar
  76. Fruchterman T.M.J., Reingold E.M. (1991) Graph Drawing by Force-Directed Placement. Software –practice & experience 21: 1129–1164Google Scholar
  77. Gansner E.R., North S.C. (2000) An open graph visualization system and its applications to software engineering. Software: Practice and Experience 30: 1203–1233Google Scholar
  78. Gardner M.J., Altman D.G. (1989) Statistics with Confidence: Confidence Intervals And Statistical Guidelines. BMJ, LondonGoogle Scholar
  79. Gillet V.J., Khatib W., Willett P., Fleming P.J., Green D.V.S. (2002) Combinatorial library design using a multiobjective genetic algorithm. J. Chem. Inf. Comput. Sci. 42: 375–385PubMedGoogle Scholar
  80. Goble C.A., Stevens R., Ng G., Bechhofer S., Paton N.W., Baker P.G., Peim M., Brass A. (2001) Transparent access to multiple bioinformatics information sources. IBM. Syst. J. 40: 532–551CrossRefGoogle Scholar
  81. Goffeau A., Barrell B.G., Bussey H., Davis R.W., Dujon B., Feldmann H., Galibert F., Hoheisel J.D., Jacq C., Johnston M., Louis E.J., Mewes H.W., Murakami Y., Philippsen P., Tettelin H., Oliver S.G. (1996) Life With 6000 Genes. Science 274: 546–567PubMedGoogle Scholar
  82. Golbraikh A., Tropsha A. (2002) Beware of q2!. J. Mol. Graph Model 20: 269–276PubMedGoogle Scholar
  83. Golub T.R., Slonim D.K., Tamayo P., Huard C., Gaasenbeek M., Mesirov J.P., Coller H., Loh M.L., Downing J.R., Caligiuri M.A., Bloomfield C.D., Lander E.S. (1999) Molecular classification of cancer: Class discovery and class prediction by gene expression monitoring. Science 286: 531–537PubMedGoogle Scholar
  84. Goodacre R., Kell D.B. (2003) Evolutionary computation for the interpretation of metabolome data. In Harrigan G.G., Goodacre R. (Eds.), Metabolic Profiling: Its Role in Biomarker Discovery and Gene Function Analysis. Kluwer Academic Publishers, Boston, pp. 239–256Google Scholar
  85. Goodacre R., Neal M.J., Kell D.B. (1996) Quantitative analysis of multivariate data using artificial neural networks: a tutorial review and applications to the deconvolution of pyrolysis mass spectra. Z. Bakteriol. 284: 516–539Google Scholar
  86. Goodman S.N., Royall R. (1988) Evidence and scientific research. Am. J. Publ. Health 78: 1568–1574CrossRefGoogle Scholar
  87. Greenaway W., May J., Scaysbrook T., Whatley F.R. (1991) Identification by gas chromatography-mass spectrometry of 150 compounds in propolis. Z. Naturforsch. C 46: 111–121Google Scholar
  88. Grimes D.S. (2006) Are statins analogues of vitamin D? Lancet 368: 83–6PubMedGoogle Scholar
  89. Hand D., Mannila H., Smyth P. (2001) Principles of Data Mining. MIT Press, Cambridge, MAGoogle Scholar
  90. Handl, J., Kell, D.B. and Knowles, J. (2006). Multiobjective optimization in bioinformatics and computational biology. IEEE Trans Comput Biol Bioinformatics (in the press)Google Scholar
  91. Handl, J. and Knowles, J. (2004). Evolutionary Multiobjective Clustering. PPSN VIII, LNCS 3242, 1081–1091 (see Scholar
  92. Handl, J. and Knowles, J. (2006a) An evolutionary approach to multiobjective clustering. IEEE Trans Evol Comput (in press)Google Scholar
  93. Handl, J. and Knowles, J. (2006b). Semi-supervised feature selection via multiobjective optimization. International Joint Conference on Neural Networks (IJCNN 2006). Proc WCCI 2006, IEEE Press, pp. 6351–6358Google Scholar
  94. Handl J., Knowles J., Kell D.B. (2005) Computational cluster validation in post-genomic data analysis. Bioinformatics 21: 3201–3212PubMedGoogle Scholar
  95. Hanley J.A., McNeil B.J. (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143: 29–36PubMedGoogle Scholar
  96. Harrigan G.G., LaPlante R.H., Cosma G.N., Cockerell G., Goodacre R., Maddox J.F., Luyendyk J.P., Ganey P.E., Roth R.A. (2004) Application of high-throughput Fourier-transform infrared spectroscopy in toxicology studies: contribution to a study on the development of an animal model for idiosyncratic toxicity. Toxicol. Lett. 146: 197–205PubMedGoogle Scholar
  97. Hastie T., Tibshirani R., Friedman J. (2001) The Elements Of Statistical Learning: Data Mining, Inference and Prediction. Springer-Verlag, BerlinGoogle Scholar
  98. Heinrich R., Schuster S. (1996) The Regulation Of Cellular Systems. Chapman & Hall, New YorkGoogle Scholar
  99. Hicks C.R., Turner K.V. Jr (1999) Fundamental Concepts in the Design of Experiments. 5th ed. Oxford University Press, OxfordGoogle Scholar
  100. Hollander M., Wolfe D.A. (1973) Nonparametric Statistical Methods. Wiley, New YorkGoogle Scholar
  101. Horchner U., Kalivas J.H. (1995) Further investigation on a comparative study of simulated annealing and genetic algorithm for wavelength selection. Anal. Chim. Acta. 311: 1–13Google Scholar
  102. Horning E.C., Horning M.G. (1971) Metabolic profiles: gas-phase methods for analysis of metabolites. Clin Chem 17: 802–809PubMedGoogle Scholar
  103. Hubert L., Arabie P. (1985) Comparing partitions. J. Classif. 2: 193–218Google Scholar
  104. Hutchinson A. (1994) Algorithmic Learning. Clarendon Press, OxfordGoogle Scholar
  105. Ioannidis J.P. (2005a) Contradicted and initially stronger effects in highly cited clinical research. JAMA 294: 218–228Google Scholar
  106. Ioannidis J.P. (2005b) Why most published research findings are false. PLoS Med. 2, e124Google Scholar
  107. Ioannidis J.P., Ntzani E.E., Trikalinos T.A., Contopoulos-Ioannidis D.G. (2001) Replication validity of genetic association studies. Nat. Genet. 29: 306–309PubMedGoogle Scholar
  108. Ioannidis J.P., Trikalinos T.A. (2005) Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. J. Clin. Epidemiol. 58: 543–549PubMedGoogle Scholar
  109. Ioannidis J.P., Trikalinos T.A., Ntzani E.E., Contopoulos-Ioannidis D.G. (2003) Genetic associations in large versus small studies: an empirical assessment. Lancet 361: 567–571PubMedGoogle Scholar
  110. Jarvis R.M., Goodacre R. (2005) Genetic algorithm optimization for pre-processing and variable selection of spectroscopic data. Bioinformatics 21: 860–868PubMedGoogle Scholar
  111. Jellum E., Bjornson I., Nesbakken R., Johansson E., Wold S. (1981) Classification of human cancer cells by means of capillary gas chromatography and pattern recognition analysis. J. Chromatogr. 217: 231–237PubMedGoogle Scholar
  112. Jensen F.V. (2001) Bayesian Networks and Decision Graphs. Springer, BerlinGoogle Scholar
  113. Jolliffe I.T. (1986) Principal Component Analysis. Springer-Verlag, New YorkGoogle Scholar
  114. Judson R. (1997) Genetic algorithms and their use in chemistry. Rev. Comput. Chem. 10: 1–73Google Scholar
  115. Jung S.H. (2005) Sample size for FDR-control in microarray data analysis. Bioinformatics 21: 3097–104PubMedGoogle Scholar
  116. Kamada T., Kawai S. (1989) An algorithm for drawing general undirected graphs. Inf .Proc. Lett. 31: 7–15Google Scholar
  117. Kannel W.B. (1995) Range of serum cholesterol values in the population developing coronary artery disease. Am. J. Cardiol. 76: 69C–77CPubMedGoogle Scholar
  118. Kell D.B. (2002a) Genotype:phenotype mapping: genes as computer programs. Trends. Genet. 18: 555–559Google Scholar
  119. Kell D.B. (2002b) Metabolomics and machine learning: explanatory analysis of complex metabolome data using genetic programming to produce simple, robust rules. Mol. Biol. Rep. 29: 237–41Google Scholar
  120. Kell D.B. (2004) Metabolomics and systems biology: making sense of the soup. Curr. Op. Microbiol. 7: 296–307Google Scholar
  121. Kell D.B. (2006) Metabolomics, modelling and machine learning in systems biology: towards an understanding of the languages of cells . The 2005 Theodor Bücher lecture. FEBS J. 273: 873–894PubMedGoogle Scholar
  122. Kell D.B., Brown M., Davey H.M., Dunn W.B., Spasic I., Oliver S.G. (2005) Metabolic footprinting and Systems Biology: the medium is the message. Nat. Rev. Microbiol. 3: 557–565PubMedGoogle Scholar
  123. Kell D.B., Darby R.M., Draper J. (2001) Genomic computing: explanatory analysis of plant expression profiling data using machine learning. Plant. Physiol. 126: 943–951PubMedGoogle Scholar
  124. Kell D.B., King R.D. (2000) On the optimization of classes for the assignment of unidentified reading frames in functional genomics programmes: the need for machine learning. Trends Biotechnol. 18: 93–98PubMedGoogle Scholar
  125. Kell D.B., Knowles J.D. (2006) The role of modeling in systems biology. In Szallasi Z., Stelling J., Periwal V. (Eds.), System Modeling in Cellular Biology: From Concepts to Nuts and Bolts. MIT Press, Cambridge, pp. 3–18Google Scholar
  126. Kell D.B., Oliver S.G. (2004) Here is the evidence, now what is the hypothesis? The complementary roles of inductive and hypothesis-driven science in the post-genomic era. Bioessays 26: 99–105PubMedGoogle Scholar
  127. Kell D.B., Sonnleitner B. (1995) GMP - Good Modelling Practice: an essential component of good manufacturing practice. Trends Biotechnol. 13: 481–492Google Scholar
  128. Kell, D.B. and Welch, G.R. (1991). No turning back, Reductonism and Biological Complexity. Times Higher Educational Supplement 9th August, 15Google Scholar
  129. Kell D.B., Westerhoff H.V. (1986) Metabolic control theory: its role in microbiology and biotechnology. FEMS Microbiol. Rev. 39: 305–320Google Scholar
  130. Kemp, C., Griffiths, T., Stromsten, S. and Tenenbaum, J.B. (2003) Semi-supervised learning with trees. Adv. Neural Inf Proc Syst 16Google Scholar
  131. Kenny, L.C., Dunn, W.B., Ellis, D.I., Myers, J., Baker, P.N., The GOPEC Consortium and Kell, D.B. (2005) Novel biomarkers for pre-eclampsia detected using metabolomics and machine learning. Metabolomics 1, 227–234 - online DOI: 10.1007/s11306–005–0003–1Google Scholar
  132. Kim S.K., Lund J., Kiraly M., Duke K., Jiang M., Stuart J.M., Eizinger A., Wylie B.N., Davidson G.S. (2001) A gene expression map for Caenorhabditis elegans. Science 293: 2087–2092PubMedGoogle Scholar
  133. Kirkwood B.R., Sterne J.A.C. (2003) Essential Medical Statistics. Blackwell, OxfordGoogle Scholar
  134. Kirschenlohr H.L., Griffin J.L., Clarke S.C., Rhydwen R., Grace A.A., Schofield P.M., Brindle K.M., Metcalfe J.C. (2006) Proton NMR analysis of plasma is a weak predictor of coronary artery disease. Nat. Med. 12: 705–710PubMedGoogle Scholar
  135. Knowles, J.D. and Hughes, E.J. (2005). Multiobjective optimization on a budget of 250 evaluations. Evolutionary Multi-Criterion Optimization (EMO 2005), LNCS 3410, 176–190 Scholar
  136. Knowles, J.D., Watson, R.A. and Corne, D.W. (2001). Reducing local optima in single-objective problems by multi-objectivization in E. Zitzler et al., (ed.), Proc. 1st Int. Conf. on Evolutionary Multi-criterion Optimization (EMO’01), Springer, Berlin, pp. 269–283Google Scholar
  137. Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, pp. 1137–1143Google Scholar
  138. Kohonen T. (1989) Self-Organization and Associative Memory. Springer-Verlag, BerlinGoogle Scholar
  139. Kose F., Weckwerth W., Linke T., Fiehn O. (2001) Visualizing plant metabolomic correlation networks using clique-metabolite matrices. Bioinformatics 17: 1198–1208PubMedGoogle Scholar
  140. Koza J.R. (1992) Genetic Programming: On The Programming of Computers by Means Of Natural Selection. MIT Press, Cambridge, MassGoogle Scholar
  141. Koza J.R., Keane M.A., Streeter M.J., Mydlowec W., Yu J., Lanza G. (2003) Genetic Programming: Routine Human-Competitive Machine Intelligence. Kluwer, New YorkGoogle Scholar
  142. Kruse R., Gebhardt J., Klawonn F. (1994) Foundations of Fuzzy Systems. John Wiley, ChichesterGoogle Scholar
  143. Kruskal, J.B. and Seery, J.B. (1980). Designing network diagrams. Proc. 1st General Conf. on Social Graphics, pp. 22–50Google Scholar
  144. Krzanowski W.J. (1988) Principles of Multivariate Analysis: A User’s Perspective. Oxford Univeristy Press, OxfordGoogle Scholar
  145. Langdon W.B. (1998) Genetic Programming And Data Structures: Genetic Programming + Data Structures = Automatic Programming!. Kluwer, BostonGoogle Scholar
  146. Langley P., Simon H.A., Bradshaw G.L., Zytkow J.M. (1987) Scientific Discovery: Computational Exploration Of The Creative Processes. MIT Press, Cambridge, MAGoogle Scholar
  147. Leon A.C. (2004) Multiplicity-adjusted sample size requirements: a strategy to maintain statistical power with Bonferroni adjustments. J. Clin. Psychiatry 65: 1511–1514PubMedGoogle Scholar
  148. Li H.-X., Yen V.C. (1995) Fuzzy Sets And Fuzzy Decision-Making. CRC Press, Boca Raton, FloridaGoogle Scholar
  149. Li, T., Zhu, S., Li, Q., and Ogihara, M. (2003). Gene functional classification by semi-supervised learning from heterogeneous data. Proc ACM Symp. Appl. Computing. pp. 78–82Google Scholar
  150. Liang Y., Kelemen A. (2006) Associating phenotypes with molecular events: recent statistical advances and challenges underpinning microarray experiments. Funct .Integr Genomics 6: 1–13PubMedGoogle Scholar
  151. Linden A. (2006) Measuring diagnostic and predictive accuracy in disease management: an introduction to receiver operating characteristic (ROC) analysis. J. Eval. Clin. Pract. 12: 132–139PubMedGoogle Scholar
  152. Lucasius C.B., Beckers M.L.M., Kateman G. (1994) Genetic algorithms in wavelength selection – a comparative-study. Analytica Chimica Acta 286: 135–153Google Scholar
  153. Lucasius C.B., Kateman G. (1994) Understanding and using genetic algorithms .2. Representation, configuration and hybridization. Chemometrics and Intelligent Laboratory Systems 25: 99–145Google Scholar
  154. Mackay D.J.C. (2003) Information Theory, Inference and Learning Algorithms. Cambridge University Press, CambridgeGoogle Scholar
  155. Manly B.F.J. (1994) Multivariate Statistical Methods : A Primer. Chapman and Hall, LondonGoogle Scholar
  156. Martens H., Næs T. (1989) Multivariate Calibration. John Wiley, ChichesterGoogle Scholar
  157. Metz C.E. (1978) Basic principles of ROC analysis. Semin Nucl Med 8: 283–98PubMedGoogle Scholar
  158. Michalewicz Z., Fogel D.B. (2000) How to Solve it: Modern Heuristics. Springer-Verlag, HeidelbergGoogle Scholar
  159. Michalski R.S., Bratko I., Kubat M. (Eds) (1998) Machine Learning and Data Mining. Methods and applications, Wiley, ChichesterGoogle Scholar
  160. Michie D., Spiegelhalter D.J., Taylor C.C. (eds) (1994) Machine Learning Neural and Statistical Classification. Ellis Horwood, ChichesterGoogle Scholar
  161. Miller A.J. (1990) Subset Selection in Regression. Chapman and Hall, LondonGoogle Scholar
  162. Mitchell T.M. (1997) Machine Learning. McGraw Hill, New YorkGoogle Scholar
  163. Montgomery D.C. (2001) Design and Analysis of Experiments. 5th edition. Wiley, ChichesterGoogle Scholar
  164. Myers R.H., Montgomery D.C. (1995) Response Surface Methodology: Process and Product Optimization using Designed Experiments. Wiley, New YorkGoogle Scholar
  165. Natarajan S., Glick H., Criqui M., Horowitz D., Lipsitz S.R., Kinosian B. (2003) Cholesterol measures to identify and treat individuals at risk for coronary heart disease. Am. J. Prev. Med. 25: 50–7PubMedGoogle Scholar
  166. Needham C.J., Bradford J.R., Bulpitt A.J., Westhead D.R. (2006) Inference in Bayesian networks. Nat. Biotechnol. 24: 51–53PubMedGoogle Scholar
  167. Ntzani E.E., Ioannidis J.P. (2003) Predictive ability of DNA microarrays for cancer outcomes and correlates: an empirical assessment. Lancet 362: 1439–44PubMedGoogle Scholar
  168. O’Hagan S., Dunn W.B., Brown M., Knowles J.D., Kell D.B. (2005) Closed-loop, multiobjective optimisation of analytical instrumentation: gas-chromatography-time-of-flight mass spectrometry of the metabolomes of human serum and of yeast fermentations. Anal. Chem. 77: 290–303PubMedGoogle Scholar
  169. Oakley J.E., O’Hagan A. (2004) Probabilistic sensitivity analysis of complex models: a Bayesian approach. JR Stat. Soc. A 66: 751–769Google Scholar
  170. Obuchowski N.A., Lieber M.L., Wians F.H. Jr. (2004) ROC curves in clinical chemistry: uses, misuses, and possible solutions. Clin. Chem. 50: 1118–25PubMedGoogle Scholar
  171. Oinn T., Addis M., Ferris J., Marvin D., Senger M., Greenwood M., Carver T., Glover K., Pocock M.R., Wipat A., Li P. (2004) Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics 20: 3045–3054PubMedGoogle Scholar
  172. Oinn T., Li P., Kell D., Goble C., Goderis A., Greenwood M., Hull D., Stevens R., Turi D., Zhao J. (2006) Taverna/Mygrid: Aligning a Workflow System with the Life Sciences Community Workflows for eScience. Springer, Guildford, pp. 299–318Google Scholar
  173. Oliver S.G., Winson M.K., Kell D.B., Baganz F. (1998) Systematic functional analysis of the yeast genome. Trends Biotechnol. 16: 373–378PubMedGoogle Scholar
  174. Pearl J. (1988) Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San FranciscoGoogle Scholar
  175. Pearl J. (2000) Causality: Models, Reasoning and Inference. Cambridge University Press, CambridgeGoogle Scholar
  176. Peleg M., Yeh I., Altman R.B. (2002) Modelling biological processes using workflow and Petri Net models. Bioinformatics 18: 825–37PubMedGoogle Scholar
  177. Perneger T.V. (1998) What’s wrong with Bonferroni adjustments. BMJ 316: 1236–8PubMedGoogle Scholar
  178. Petricoin E.F. III, Ardekani A.M., Hitt B.A., Levine P.J., Fusaro V.A., Steinberg S.M., Mills G.B., Simone C., Fishman D.A., Kohn E.C., Liotta L.A. (2002) Use of proteomic patterns in serum to identify ovarian cancer. Lancet 359: 572–577PubMedGoogle Scholar
  179. Potter S.C., Clarke L., Curwen V., Keenan S., Mongin E., Searle S.M., Stabenau A., Storey R., Clamp M. (2004) The Ensembl analysis pipeline. Genome Res. 14: 934–941PubMedGoogle Scholar
  180. Raamsdonk L.M., Teusink B., Broadhurst D., Zhang N., Hayes A., Walsh M., Berden J.A., Brindle K.M., Kell D.B., Rowland J.J., Westerhoff H.V., van Dam K., Oliver S.G. (2001) A functional genomics strategy that uses metabolome data to reveal the phenotype of silent mutations. Nat. Biotechnol. 19: 45–50PubMedGoogle Scholar
  181. Ramoni M., Sabastini P. (1998) Theory and Practice of Bayesian Belief Networks. Edward Arnold, LondonGoogle Scholar
  182. Ransohoff D.F. (2004) Rules of evidence for cancer molecular-marker discovery and validation. Nat. Rev. Cancer 4: 309–314PubMedGoogle Scholar
  183. Ransohoff D.F. (2005) Bias as a threat to the validity of cancer molecular-marker research. Nat. Rev. Cancer 5: 142–149PubMedGoogle Scholar
  184. Ransohoff D.F., Feinstein A.R. (1978) Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. N. Engl. J. Med. 299: 926–930PubMedCrossRefGoogle Scholar
  185. Rapp P.E. (1993) Chaos in the neurosciences: cautionary tales from the frontier. Biologist 40: 89–94Google Scholar
  186. Raubertas R.F., Rodewald L.E., Humiston S.G., Szilagyi P.G. (1994) ROC curves for classification trees. Med. Decis. Making 14: 169–174PubMedGoogle Scholar
  187. Reiner A., Yekutieli D., Benjamini Y. (2003) Identifying differentially expressed genes using false discovery rate controlling procedures. Bioinformatics 19: 368–375PubMedGoogle Scholar
  188. Ressom H.W., Varghese R.S., Abdel-Hamid M., Eissa S.A., Saha D., Goldman L., Petricoin E.F., Conrads T.P., Veenstra T.D., Loffredo C.A., Goldman R. (2005) Analysis of mass spectral serum profiles for biomarker selection. Bioinformatics 21: 4039–4045PubMedGoogle Scholar
  189. Rifai N., Gillette M.A., Carr S.A. (2006) Protein biomarker discovery and validation: the long and uncertain path to clinical utility. Nat. Biotechnol. 24: 971–983PubMedGoogle Scholar
  190. Ringuest J.L. (1992) Multiobjective Optimization: Behavioral and Computational Considerations. Kluwer Academic Publishers, DordrechtGoogle Scholar
  191. Romano P., Marra D., Milanesi L. (2005) Web services and workflow management for biological resources. BMC Bioinformatics 6(Suppl 4), S24PubMedGoogle Scholar
  192. Rothman K.J., Greenland S. (1998) Modern Epidemiology. 2nd ed. Lippincott, Williams & Wilkins, PhiladelphiaGoogle Scholar
  193. Rowland J.J. (2003) Model selection methodology in supervised learning with evolutionary computation. Biosystems 72: 187–196PubMedGoogle Scholar
  194. Royall R. (1997) Statistical Evidence: A Likelihood Paradigm. Chapman and Hall/CRC, LondonGoogle Scholar
  195. Rud O.P. (2001) Data Mining Cookbook. Wiley, New YorkGoogle Scholar
  196. Sacks J., Welch W., Mitchell T., Wynn H. (1989) Design and analysis of computer experiments (with discussion). Statist Sci 4: 409–435Google Scholar
  197. Saltelli A., Tarantola S., Campolongo F., Ratt M. (2004) Sensitivity Analysis in Practice: A Guide to Assessing Scientific Models. Wiley, New YorkGoogle Scholar
  198. Sammon J.W. Jr. (1969) A nonlinear mapping for data structure analysis. IEEE Trans. Computers C-18: 401–409Google Scholar
  199. Schena M. (Ed) (2000) Microarray Biochip Technology. Eaton Publishing, Natick, MAGoogle Scholar
  200. Seasholtz M.B., Kowalski B. (1993) The parsimony principle applied to multivariate calibration. Anal. Chim. Acta 277: 165–177Google Scholar
  201. Seber G.A.F., Wild C.J. (1989) Nonlinear Regression. Wiley, New YorkGoogle Scholar
  202. Sehgal M.S., Gondal I., Dooley L.S. (2005) Collateral missing value imputation: a new robust missing value estimation algorithm for microarray data. Bioinformatics 21: 2417–2423PubMedGoogle Scholar
  203. Shaffer R.E., Small G.W. (1997) Learning optimization from nature – genetic algorithms and simulated annealing. Anal. Chem. 69, A236–A242Google Scholar
  204. Sharp S.J., Thompson S.G., Altman D.G. (1996) The relation between treatment benefit and underlying risk in meta-analysis. BMJ 313: 735–738PubMedGoogle Scholar
  205. Shipley B. (2001) Cause and Correlation in Biology: A User’s Guide to Path Analysis, Structural Equations and Causal Inference. Cambridge University Press, CambridgeGoogle Scholar
  206. Sokal R.R., Rohlf F.J. (1995) Biometry. 3rd edition. Freeman, New YorkGoogle Scholar
  207. Stephan C., Wesseling S., Schink T., Jung K. (2003) Comparison of eight computer programs for receiver-operating characteristic analysis. Clin. Chem. 49: 433–439PubMedGoogle Scholar
  208. Steuer R. (2006) On the analysis and interpretation of correlations in metabolomic data. Brief Bioinform. 7: 151–158PubMedGoogle Scholar
  209. Steuer R., Kurths J., Fiehn O., Weckwerth W. (2003) Observing and interpreting correlations in metabolomic networks. Bioinformatics 19: 1019–1026PubMedGoogle Scholar
  210. Stevens R., McEntire R., Goble C., Greenwood M., Zhao J., Wipat A., Li P. (2004) myGrid and the drug discovery process. DDT Biosilico. 4: 140–148Google Scholar
  211. Storey J.D. (2002) A direct approach to false discovery rates. J. Roy. Stat. Soc. B 64: 479–498Google Scholar
  212. Storey J.D., Tibshirani R. (2003) Statistical significance for genomewide studies. Proc. Natl. Acad. Sci. USA 100: 9440–5PubMedGoogle Scholar
  213. Tas A.C., van der Greef J. (1994) Mass spectrometric profiling and pattern recognition. Mass Spectrum Rev. 13: 155–181Google Scholar
  214. Todd J.A. (2006) Statistical false positive or true disease pathway? Nat. Genet. 38: 731–733PubMedGoogle Scholar
  215. Troyanskaya O., Cantor M., Sherlock G., Brown P., Hastie T., Tibshirani R., Botstein D., Altman R.B. (2001) Missing value estimation methods for DNA microarrays. Bioinformatics 17: 520–525PubMedGoogle Scholar
  216. Tu Y., Stolovitzky G., Klein U. (2002) Quantitative noise analysis for gene expression microarray experiments. Proc. Natl. Acad. Sci. USA 99: 14031–14036PubMedGoogle Scholar
  217. Tufte E.R. (2001) The Visual Display of Quantitative Information. 2nd ed. Graphics Press, Cheshire, CTGoogle Scholar
  218. Tukey J.W. (1977) Exploratory Data Analysis. Addison-Wesley, Reading, MAGoogle Scholar
  219. Urbanczyk-Wochniak E., Luedemann A., Kopka J., Selbig J., Roessner-Tunali U., Willmitzer L., Fernie A.R. (2003) Parallel analysis of transcript and metabolic profiles: a new approach in systems biology. EMBO Rep 4: 989–993PubMedGoogle Scholar
  220. Valiant L.G. (1984) A theory of the learnable. Comm ACM 27: 1134–1142Google Scholar
  221. van ′t Veer L.J., Dai H., van de Vijver M.J., He Y.D., Hart A.A., Mao M., Peterse H.L., van der Kooy K., Marton M.J., Witteveen A.T., Schreiber G.J., Kerkhoven R.M., Roberts C., Linsley P.S., Bernards R., Friend S.H. (2002) Gene expression profiling predicts clinical outcome of breast cancer. Nature 415: 530–536PubMedGoogle Scholar
  222. van de Vijver M.J., He Y.D., van ′t Veer L.J., Dai H., Hart A.A., Voskuil D.W., Schreiber G.J., Peterse J.L., Roberts C., Marton M.J., Parrish M., Atsma D., Witteveen A., Glas A., Delahaye L., van der Velde T., Bartelink H., Rodenhuis S., Rutgers E.T., Friend S.H., Bernards R. (2002) A gene-expression signature as a predictor of survival in breast cancer. N. Engl. J. Med. 347: 1999–2009PubMedGoogle Scholar
  223. van Rijsbergen C. (1979) Information Retrieval. Butterworth, LondonGoogle Scholar
  224. Van Veldhuizen D.A., Lamont G.B. (2000) Multiobjective evolutionary algorithms: analyzing the state-of-the-art. Evol Comput 8: 125–147PubMedGoogle Scholar
  225. Vapnik V.N. (1998) Statistical Learning Theory. Wiley, New YorkGoogle Scholar
  226. von Mering C., Krause R., Snel B., Cornell M., Oliver S.G., Fields S., Bork P. (2002) Comparative assessment of large-scale data sets of protein-protein interactions. Nature 417: 399–403Google Scholar
  227. Wacholder S., Chanock S., Garcia-Closas M., El Ghormli L., Rothman N. (2004) Assessing the probability that a positive report is false: an approach for molecular epidemiology studies. J. Natl. Cancer Inst. 96: 434–442PubMedGoogle Scholar
  228. Wang Y., Klijn J.G., Zhang Y., Sieuwerts A.M., Look M.P., Yang F., Talantov D., Timmermans M., Meijer-van Gelder M.E., Yu J., Jatkoe T., Berns E.M., Atkins D., Foekens J.A. (2005) Gene-expression profiles to predict distant metastasis of lymph-node-negative primary breast cancer. Lancet 365: 671–679PubMedGoogle Scholar
  229. Weckwerth W., Morgenthal K. (2005) Metabolomics: from pattern recognition to biological interpretation. Drug Discov. Today 10: 1551–1558PubMedGoogle Scholar
  230. Weiss S.H., Kulikowski C.A. (1991) Computer Systems that Learn: Classification and Prediction Methods from Statistics, Neural Networks, Machine Learning, and Expert Systems. Morgan Kaufmann Publishers, San Mateo, CAGoogle Scholar
  231. Weiss S.M., Indurkhya N. (1998) Predictive Data Mining. Morgan Kaufmann, San FranciscoGoogle Scholar
  232. Westerhoff H.V., Kell D.B. (1987) Matrix method for determining the steps most rate-limiting to metabolic fluxes in biotechnological processes. Biotechnol. Bioeng. 30: 101–107PubMedGoogle Scholar
  233. White H. (1992) Artificial Neural Networks: Approximation and Learning Theory. Blackwell, OxfordGoogle Scholar
  234. White T.A., Kell D.B. (2004) Comparative genomic assessment of novel broad-spectrum targets for antibacterial drugs. Comp. Func. Genomics 5: 304–327Google Scholar
  235. Wilkinson L. (1999) The Grammar of Graphics. Springer-Verlag, New YorkGoogle Scholar
  236. Williamson P.R., Gamble C., Altman D.G., Hutton J.L. (2005) Outcome selection bias in meta-analysis. Stat. Methods Med. Res. 14: 515–524PubMedGoogle Scholar
  237. Wold S., Trygg J., Berglund A., Antti H. (2001) Some recent developments in PLS modeling. Chemometr. Intell. Lab Syst. 58: 131–150Google Scholar
  238. Woodward M. (2000) Epidemiology: Study Design and Data analysis. Chapman and Hall/CRC, LondonGoogle Scholar
  239. Xie Y., Pan W., Khodursky A.B. (2005) A note on using permutation-based false discovery rate estimates to compare different analysis methods for microarray data. Bioinformatics 21: 4280–4288PubMedGoogle Scholar
  240. Zadeh L.A. (1965) Fuzzy sets. Information and Control 8: 338–353Google Scholar
  241. Zhang J.H., Chung T.D.Y., Oldenburg K.R. (1999) A simple statistical parameter for use in evaluation and validation of high throughput screening assays. J. Biomol. Screen. 4: 67–73PubMedGoogle Scholar
  242. Zhou X., Wang X., Dougherty E.R. (2003) Missing-value estimation using linear and non-linear regression with Bayesian gene selection. Bioinformatics 19: 2302–2307PubMedGoogle Scholar
  243. Zhou X.H., Obuchowski N.A., McClish D.K. (2002) Statistical Methods in Diagnostic Medicine. Wiley, New YorkGoogle Scholar
  244. Zitzler E. (1999) Evolutionary Algorithms for Multiobjective Optimization: Methods And Applications. Shaker Verlag, AachenGoogle Scholar
  245. Zupan J., Gasteiger J. (1993) Neural Networks for Chemists. Verlag Chemie, WeinheimGoogle Scholar
  246. Zweig M.H., Campbell G. (1993) Receiver-Operating Characteristic (ROC) plots - a fundamental evaluation tool in clinical medicine. Clin. Chem. 39: 561–577PubMedGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2006

Authors and Affiliations

  1. 1.School of ChemistryThe University of ManchesterManchesterUK
  2. 2.Manchester Centre for Integrative Systems Biology, The Manchester Interdisciplinary BiocentreThe University of ManchesterManchesterUK

Personalised recommendations