Sloppiness and the Geometry of Parameter Space

  • Brian K. Mannakee
  • Aaron P. Ragsdale
  • Mark K. Transtrum
  • Ryan N. Gutenkunst
Chapter

Abstract

When modeling complex biological systems, exploring parameter space is critical, because parameter values are typically poorly known a priori. This exploration can be challenging, because parameter space often has high dimension and complex structure. Recent work, however, has revealed universal structure in parameter space of models for nonlinear systems. In particular, models are often sloppy, with strong parameter correlations and an exponential range of parameter sensitivities. Here we review the evidence for universal sloppiness and its implications for parameter fitting, model prediction, and experimental design. In principle, one can transform parameters to alleviate sloppiness, but a parameterization-independent information geometry perspective reveals deeper universal structure. We thus also review the recent insights offered by information geometry, particularly in regard to sloppiness and numerical methods.

Keywords

Sloppiness Hessian Experimental design Bayesian ensembles Cost functions Information geometry 

References

  1. 1.
    Abdi, H., Williams, L.J.: Principal component analysis. Wiley Interdiscip Rev: Comput. Statist. 2(4), 433–459 (2010)CrossRefGoogle Scholar
  2. 2.
    Amari, S.I., Nagaoka, H.: Methods of Information Geometry, Translations of Mathematical Monographs, vol. 191. American Mathematical Society, New York (2000)Google Scholar
  3. 3.
    Apgar, J.F., Witmer, D.K., White, F.M., Tidor, B.: Sloppy models, parameter uncertainty, and the role of experimental design. Mol. Biosyst. 6(10), 1890–1900 (2010)Google Scholar
  4. 4.
    Barndorff-Nielsen, O., Cox, D., Reid, N.: The role of differential geometry in statistical theory. Int. Stat. Rev. 54(1), 83–96 (1986). doi:10.2307/1403260
  5. 5.
    Bates, D.M., Watts, D.G.: Relative curvature measures of nonlinearity. J. Roy. Stat. Soc. B 42, 1–25 (1980)MATHMathSciNetGoogle Scholar
  6. 6.
    Bates, D.M., Watts, D.G.: Parameter transformations for improved approximate confidence regions in nonlinear least squares. Ann. Stat. 9(6), 1152–1167 (1981)MATHMathSciNetCrossRefGoogle Scholar
  7. 7.
    Bates, D.M., Watts, D.G.: Nonlinear Regression Analysis and Its Applications, Wiley Series in Probability and Statistics, vol. 32. Wiley, New York (1988)Google Scholar
  8. 8.
    Bates, D.M., Hamilton, D.C., Watts, D.G.: Calculation of intrinsic and parameter-effects curvatures for nonlinear regression models. Commun. Stat. Simulat. 12(4), 469–477 (1983). doi:10.1080/03610918308812333 MATHCrossRefGoogle Scholar
  9. 9.
    Battogtokh, D., Asch, D., Case, M., Arnold, J., Schüttler, H.B.: An ensemble method for identifying regulatory circuits with special reference to the qa gene cluster of Neurospora crassa. Proc. Natl. Acad. Sci. USA 99(26), 16904–16909 (2002). doi:10.1073/pnas.262658899 CrossRefGoogle Scholar
  10. 10.
    Beale, E.M.L.: Confidence regions in non-linear estimation. J. Roy. Stat. Soc. B 22(1), 41–88 (1960)MATHMathSciNetGoogle Scholar
  11. 11.
    Beaumont, M.A., Zhang, W., Balding, D.J.: Approximate Bayesian computation in population genetics. Genetics 162(4), 2025–2035 (2002)Google Scholar
  12. 12.
    Birnbaum, A.: On the foundations of statistical inference. J. Am. Stat. Assoc. 57(298), 269–306 (1962). doi:10.2307/2281641 MATHMathSciNetCrossRefGoogle Scholar
  13. 13.
    Brown, K., Sethna, J.: Statistical mechanical approaches to models with many poorly known parameters. Phys. Rev. E 68(2), 021904 (2003). doi:10.1103/PhysRevE.68.021904
  14. 14.
    Brown, K.S., Hill, C.C., Calero, G.A., Myers, C.R., Lee, K.H., Sethna, J.P., Cerione, R.A.: The statistical mechanics of complex signaling networks: nerve growth factor signaling. Phys. Biol. 1(3–4), 184–195 (2004). doi:10.1088/1478-3967/1/3/006 CrossRefGoogle Scholar
  15. 15.
    Casey, F.P., Baird, D., Feng, Q., Gutenkunst, R.N., Waterfall, J.J., Myers, C.R., Brown, K.S., Cerione, R.A., Sethna, J.P.: Optimal experimental design in an epidermal growth factor receptor signalling and down-regulation model. IET Syst. Biol. 1(3), 190–202 (2007). doi:10.1049/iet-syb CrossRefGoogle Scholar
  16. 16.
    Chachra, R., Transtrum, M.K., Sethna, J.P.: Comment on Sloppy models, parameter uncertainty, and the role of experimental design. Mol. Biosyst. 7(8), 2522; author reply 2523–4 (2011). doi:10.1039/c1mb05046j
  17. 17.
    Chib, S., Greenberg, E.: Understanding the Metropolis Hastings algorithm. Am. Stat. 49(4), 327–335 (1995). doi:10.1080/00031305.1995.10476177 Google Scholar
  18. 18.
    Daniels, B.C., Chen, Y.J., Sethna, J.P., Gutenkunst, R.N., Myers, C.R.: Sloppiness, robustness, and evolvability in systems biology. Curr. Opin. Biotech. 19(4), 389–395 (2008). doi:10.1016/j.copbio.2008.06.008 CrossRefGoogle Scholar
  19. 19.
    De Smet, R., Marchal, K.: Advantages and limitations of current network inference methods. Nat. Rev. Microbiol. 8(10), 717–729 (2010). doi:10.1038/nrmicro2419
  20. 20.
    Del Moral, P., Doucet, A., Jasra, A.: Sequential Monte Carlo samplers. J. Roy. Stat. Soc. B. Met. 68(3), 411–436 (2006). doi:10.1111/j.1467-9868.2006.00553.x
  21. 21.
    Demidenko, E.: Criteria for global minimum of sum of squares in nonlinear regression. Comput. Stat. Data An. 51(3), 1739–1753 (2006). doi:10.1016/j.csda.2006.06.015
  22. 22.
    Efron, B., Hinkley, D.V.: Assessing the accuracy of the maximum likelihood estimator: observed versus expected Fisher information. Biometrika 65(3), 457–483 (1978). doi:10.1093/biomet/65.3.457 MATHMathSciNetCrossRefGoogle Scholar
  23. 23.
    Erguler, K., Stumpf, M.P.H.: Practical limits for reverse engineering of dynamical systems: a statistical analysis of sensitivity and parameter inferability in systems biology models. Mol. Biosyst. 7(5), 1593–1602 (2011). doi:10.1039/c0mb00107d CrossRefGoogle Scholar
  24. 24.
    Eydgahi, H., Chen, W.W., Muhlich, J.L., Vitkup, D., Tsitsiklis, J.N., Sorger, P.K.: Properties of cell death models calibrated and compared using Bayesian approaches. Mol. Syst. Biol. 9(644), 644 (2013). doi:10.1038/msb.2012.69 Google Scholar
  25. 25.
    Fernández Slezak, D., Suárez, C., Cecchi, G.A., Marshall, G., Stolovitzky, G.: When the optimal is not the best: parameter estimation in complex biological models. PloS One 5(10), e13,283 (2010). doi:10.1371/journal.pone.0013283
  26. 26.
    Fisher, R.A.: On the mathematical foundations of theoretical statistics. Philos. T. Roy. Soc. Lond. 222(594–604), 309–368 (1922). doi:10.1098/rsta.1922.0009 MATHCrossRefGoogle Scholar
  27. 27.
    Flaherty, P., Radhakrishnan, M.L., Dinh, T., Rebres, R.A., Roach, T.I., Jordan, M.I., Arkin, A.P.: A dual receptor crosstalk model of G-protein-coupled signal transduction. PLoS Comput. Biol. 4(9), e1000185 (2008). doi:10.1371/journal.pcbi.1000185
  28. 28.
    Girolami, M., Calderhead, B.: Riemann manifold Langevin and Hamiltonian Monte Carlo methods. J. Roy. Stat. Soc. B. Met. 73, 123–214 (2011). doi:10.1111/j.1467-9868.2010.00765.x MathSciNetCrossRefGoogle Scholar
  29. 29.
    Gunawardena, J.: Models in sytems biology: the parameter problem and the meaning of robustness. In: Lodhi H.M., Muggleton, S.H. (eds.) Elements of Computational Systems Biology, pp. 19–47. Wiley Hoboken (2010). doi:10.1002/9780470556757.ch2
  30. 30.
    Gutenkunst, R.: Sloppiness, Modeling, and Evolution in Biochemical Networks. Ph.D. thesis, Cornell University (2008). http://www.ecommons.cornell.edu/handle/1813/8206
  31. 31.
    Gutenkunst, R.N., Casey, F.P., Waterfall, J.J., Myers, C.R., Sethna, J.P.: Extracting falsifiable predictions from sloppy models. Ann. NY Acad. Sci. 1115, 203–211 (2007a). doi:10.1196/annals.1407.003
  32. 32.
    Gutenkunst, R.N., Waterfall, J.J., Casey, F.P., Brown, K.S., Myers, C.R., Sethna, J.P.: Universally sloppy parameter sensitivities in systems biology models. PLoS Comput. Biol. 3(10), e189 (2007b). doi:10.1371/journal.pcbi.0030189
  33. 33.
    Hagen, D.R., Apgar, J.F., White, F.M., Tidor, B.: Molecular BioSystems reply to comment on Sloppy models, parameter uncertainty, and the role of experimental design. Interface Focus pp. 2523–2524 (2011). doi:10.1039/c1mb05200d
  34. 34.
    Hagen, D.R., White, J.K., Tidor, B.: Convergence in parameters and predictions using computational experimental design. Interface Focus 3(4), 20130,008–20130,008 (2013). doi:10.1098/rsfs.2013.0008
  35. 35.
    Haines, L.M., O Brien, T.E., Clarke, G.P.Y.: Kurtosis and curvature measures for nonlinear regression models. Stat. Sinica 14(2), 547–570 (2004)Google Scholar
  36. 36.
    Hamilton, D.C., Watts, D.G., Bates, D.M.: Accounting for intrinsic nonlinearity in nonlinear regression parameter inference regions. Ann. Stat. 10(38), 393 (1982)MathSciNetGoogle Scholar
  37. 37.
    Hotelling, H.: Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 24(6), 417–441 (1933)CrossRefGoogle Scholar
  38. 38.
    Hug, S., Schmidl, D., Li, W.B., Greiter, M.B., Theis, F.J.: Bayesian model selection methods and their application to biological ODE systems. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)Google Scholar
  39. 39.
    Ivancevic, T.T.: Applied Differential Geometry: a Modern introduction. World Scientific, Singapore (2007)Google Scholar
  40. 40.
    Jaqaman, K., Danuser, G.: Linking data to models: data regression. Nat. Rev. Mol. Cell Bio. 7(11), 813–819 (2006). doi:10.1038/nrm2030 CrossRefGoogle Scholar
  41. 41.
    Kass, R.E.: The geometry of asymptotic inference. Stat. Sci. 4(3), 188–219 (1989)MATHMathSciNetCrossRefGoogle Scholar
  42. 42.
    Kirk, P., Silk, D., Stumpf, M.P.H.: Reverse engineering under uncertainty. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)Google Scholar
  43. 43.
    Kirk, P., Thorne, T., Stumpf, M.P.: Model selection in systems and synthetic biology. Curr. Opin. Biotech. 24(4), 767–774 (2013). doi:10.1016/j.copbio.2013.03.012 CrossRefGoogle Scholar
  44. 44.
    Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983). doi:10.1126/science.220.4598.671 MATHMathSciNetCrossRefGoogle Scholar
  45. 45.
    Kitano, H.: Systems biology: a brief overview. Science 295(5560), 1662–1664 (2002). doi:10.1126/science.1069492 CrossRefGoogle Scholar
  46. 46.
    Kitano, H.: Biological robustness. Nat. Rev. Genet. 5(11), 826–837 (2004). doi:10.1038/nrg1471 CrossRefGoogle Scholar
  47. 47.
    Kreutz, C., Timmer, J.: Systems biology: experimental design. FEBS J. 276(4), 923–942 (2009). doi:10.1111/j.1742-4658.2008.06843.x CrossRefGoogle Scholar
  48. 48.
    Le Novère, N., Bornstein, B., Broicher, A., Courtot, M., Donizelli, M., Dharuri, H., Li, L., Sauro, H., Schilstra, M., Shapiro, B., Snoep, J.L., Hucka, M.: BioModels Database: a free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems. Nucleic Acids Res. 34(Database issue), D689–91 (2006). doi:10.1093/nar/gkj092
  49. 49.
    Machta, B., Chachra, R., Transtrum, M., Sethna, J.: Parameter space compression underlies emergent theories and predictive models. Science 342(6158), 604–607 (2013). doi:10.1126/science.1238723 CrossRefGoogle Scholar
  50. 50.
    Marino, S., Hogue, I.B., Ray, C.J., Kirschner, D.E.: A methodology for performing global uncertainty and sensitivity analysis in systems biology. J. Theor. Biol. 254(1), 178–196 (2008). doi:10.1016/j.jtbi.2008.04.011 MathSciNetCrossRefGoogle Scholar
  51. 51.
    Marjoram, P., Molitor, J., Plagnol, V., Tavare, S.: Markov chain Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. USA 100(26), 15,324–8 (2003). doi:10.1073/pnas.0306899100
  52. 52.
    Meyer, P., Cokelaer, T., Chandran, D., Kim, K.H., Loh, P.R., Tucker, G., Lipson, M., Berger, B., Kreutz, C., Raue, A., Steiert, B., Timmer, J., Bilal, E., Sauro, H.M., Stolovitzky, G., Saez-Rodriguez, J.: Network topology and parameter estimation: from experimental design methods to gene regulatory network kinetics using a community based approach. BMC Syst. Biol. 8(1), 13 (2014). doi:10.1186/1752-0509-8-13 CrossRefGoogle Scholar
  53. 53.
    Moles, C.G., Mendes, P., Banga, J.R.: Parameter estimation in biochemical pathways: a comparison of global optimization methods. Genome Res. 13(11), 2467–2474 (2003). doi:10.1101/gr.1262503 CrossRefGoogle Scholar
  54. 54.
    Murray, M.K., Rice, J.W.: Differential Geometry and Statistics, Monographs on statistics and applied probability, vol. 48. Chapman & Hall, London (1993)Google Scholar
  55. 55.
    Myers, C.R., Gutenkunst, R.N., Sethna, J.P.: Python unleashed on systems biology. Comput. Sci. Eng. 9(3), 34–37 (2007). doi:10.1109/MCSE.2007.60 CrossRefGoogle Scholar
  56. 56.
    Pittendrigh, C.: On temperature independence in the clock system controlling emergence time in Drosophila. Proc. Natl. Acad. Sci. USA 40(10), 1018–1029 (1954)CrossRefGoogle Scholar
  57. 57.
    Rand, D.A., Shulgin, B.V., Salazar, D., Millar, A.J.: Design principles underlying circadian clocks. J. Roy. Soc. Interface 1(1), 119–130 (2004). doi:10.1098/rsif.2004.0014 CrossRefGoogle Scholar
  58. 58.
    Robertson, H.: The solution of a set of reaction rate equations. In: Walsh, J. (ed.) Numerical Analysis, an Introduction, pp. 178–182. Academ Press, London (1966)Google Scholar
  59. 59.
    Rodriguez-Fernandez, M., Mendes, P., Banga, J.R.: A hybrid approach for efficient and robust parameter estimation in biochemical pathways. Biosyst. 83(2–3), 248–265 (2006). doi:10.1016/j.biosystems.2005.06.016 CrossRefGoogle Scholar
  60. 60.
    Cedersund, G., Samuelsson, O., Ball, G., Tegnér, J., Gomez-Cabrero, D.: Optimization in biology parameter estimation and the associated optimization problem. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)Google Scholar
  61. 61.
    Savageau, M.A., Coelho, P.M.B.M., Fasani, R.A., Tolla, D.A., Salvador, A.: Phenotypes and tolerances in the design space of biochemical systems. Proc. Natl. Acad. Sci. USA 106(16), 6435–6440 (2009). doi:10.1073/pnas.0809869106 CrossRefGoogle Scholar
  62. 62.
    Seber, G.A.F., Wild, C.J.: Nonlinear Regression. Wiley, New York (1988)Google Scholar
  63. 63.
    Shah, M., Chitforoushzadeh, Z., Janes, K.A.: Statistical data analysis and modeling. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)Google Scholar
  64. 64.
    Sisson, S.A., Fan, Y., Tanaka, M.M.: Sequential Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. USA 104(6), 1760–1765 (2007). doi:10.1073/pnas.0607208104 MATHMathSciNetCrossRefGoogle Scholar
  65. 65.
    Spivak, M.: A Comprehensive Introduction to Differential Geometry. Publish or Perish (1979)Google Scholar
  66. 66.
    Sunnåker, M., Stelling, J.: Model extension and model selection. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)Google Scholar
  67. 67.
    Toni, T., Welch, D., Strelkowa, N., Ipsen, A., Stumpf, M.P.: Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems. J. Roy. Soc. Interface 6(31), 187–202 (2009). doi:10.1098/rsif.2008.0172 CrossRefGoogle Scholar
  68. 68.
    Tönsing, C., Timmer, J., Kreutz, C.: Cause and cure of sloppiness in ordinary differential equation models (2014). arXiv:1406.1734
  69. 69.
    Transtrum, M.K.: Geodesic Levenberg-Marquardt source code (2012). http://sourceforge.net/projects/geodesiclm/
  70. 70.
    Transtrum, M.K., Hart, G., Qiu, P.: Information topology identifies emergent model classes. arXiv:1409.6203 (2014)
  71. 71.
    Transtrum, M.K., Machta, B.B., Sethna, J.P.: Why are nonlinear fits to data so challenging? Phys. Rev. Lett. 104(6), 060,201 (2010). doi:10.1103/PhysRevLett.104.060201
  72. 72.
    Transtrum, M.K., Machta, B.B., Sethna, J.P.: Geometry of nonlinear least squares with applications to sloppy models and optimization. Phys. Rev. E 83(3), 036,701 (2011). doi:10.1103/PhysRevE.83.036701
  73. 73.
    Transtrum, M.K., Qiu, P.: Optimal experiment selection for parameter estimation in biological differential equation models. BMC Bioinf. 13, 181 (2012). doi:10.1186/1471-2105-13-181 CrossRefGoogle Scholar
  74. 74.
    Transtrum, M.K., Qiu, P.: Model reduction by manifold boundaries. Phys. Rev. Lett. 113(9), 098,701 (2014). doi:10.1103/PhysRevLett.113.098701
  75. 75.
    Transtrum, M.K., Sethna, J.P.: Improvements to the Levenberg-Marquardt algorithm for nonlinear least-squares minimization. arXiv:1201.5885 (2012)
  76. 76.
    Waterfall, J., Casey, F., Gutenkunst, R., Brown, K., Myers, C., Brouwer, P., Elser, V., Sethna, J.: Sloppy-Model Universality Class and the Vandermonde Matrix. Phy. Rev. Lett. 97(15) (2006). doi:10.1103/PhysRevLett.97.150601
  77. 77.
    Van Schepdael, A., Carlier, A., Geris, L.: Sensitivity analysis in the design of experiments. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)Google Scholar
  78. 78.
    von Dassow, G., Meir, E., Munro, E.M., Odell, G.M.: The segment polarity network is a robust developmental module. Nature 406(6792), 188–92 (2000). doi:10.1038/35018085 CrossRefGoogle Scholar
  79. 79.
    Xu, T.R., Vyshemirsky, V., Gormand, A., von Kriegsheim, A., Girolami, M., Baillie, G.S., Ketley, D., Dunlop, A.J., Milligan, G., Houslay, M.D., Kolch, W.: Inferring signaling pathway topologies from multiple perturbation measurements of specific biochemical species. Sci. Signal. 3(113), ra20 (2010). doi:10.1126/scisignal.2000517

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Brian K. Mannakee
    • 1
  • Aaron P. Ragsdale
    • 2
  • Mark K. Transtrum
    • 3
  • Ryan N. Gutenkunst
    • 4
  1. 1.Graduate Interdisciplinary Program in StatisticsUniversity of ArizonaTucsonUSA
  2. 2.Graduate Interdisciplinary Program in Applied MathematicsUniversity of ArizonaTucsonUSA
  3. 3.Department of Physics and AstronomyBrigham Young UniversityProvoUSA
  4. 4.Department of Molecular and Cellular BiologyTucsonUSA

Personalised recommendations