Skip to main content
SpringerLink
Log in

Sloppiness and the Geometry of Parameter Space

  • Chapter
  • First Online:
Uncertainty in Biology

Part of the book series: Studies in Mechanobiology, Tissue Engineering and Biomaterials ((SMTEB,volume 17))

Abstract

When modeling complex biological systems, exploring parameter space is critical, because parameter values are typically poorly known a priori. This exploration can be challenging, because parameter space often has high dimension and complex structure. Recent work, however, has revealed universal structure in parameter space of models for nonlinear systems. In particular, models are often sloppy, with strong parameter correlations and an exponential range of parameter sensitivities. Here we review the evidence for universal sloppiness and its implications for parameter fitting, model prediction, and experimental design. In principle, one can transform parameters to alleviate sloppiness, but a parameterization-independent information geometry perspective reveals deeper universal structure. We thus also review the recent insights offered by information geometry, particularly in regard to sloppiness and numerical methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Concurrent with Brown and Sethna’s work, Rand et al. independently noted an exponential spacing of eigenvalues for several circadian clock models, although Rand et al. focused their analysis on the stiffest few eigenvalues [57].

  2. 2.

    A log-normal prior that bounds a parameter \(\theta \) to be, with \(\approx \)95 % confidence, between \(\theta _0/F\) and \(\theta _0 \times F\) corresponds to an additional residual in the cost function (Eq. 11.1) of \(r = (\log \theta - \log \theta _0) / \log \sqrt{F}\). Such a residual adds \(1/(\log \sqrt{F})^2\) to the diagonal elements of the Hessian matrix, bounding the eigenvalues from below. In our case, \(F = 10^3\), so the eigenvalues must be greater than \(\approx \)0.084.

  3. 3.

    Note that this PCA was done on points sampled uniformly in parameter space, not data space, and this non-uniformity may bias the resulting eigenvalue summary of the model manifold. We expect, however, that this approach provides a good first approximation to the hierarchy of manifold widths that would be found by geodesics.

References

  1. Abdi, H., Williams, L.J.: Principal component analysis. Wiley Interdiscip Rev: Comput. Statist. 2(4), 433–459 (2010)

    Article  Google Scholar 

  2. Amari, S.I., Nagaoka, H.: Methods of Information Geometry, Translations of Mathematical Monographs, vol. 191. American Mathematical Society, New York (2000)

    Google Scholar 

  3. Apgar, J.F., Witmer, D.K., White, F.M., Tidor, B.: Sloppy models, parameter uncertainty, and the role of experimental design. Mol. Biosyst. 6(10), 1890–1900 (2010)

    Google Scholar 

  4. Barndorff-Nielsen, O., Cox, D., Reid, N.: The role of differential geometry in statistical theory. Int. Stat. Rev. 54(1), 83–96 (1986). doi:10.2307/1403260

    Google Scholar 

  5. Bates, D.M., Watts, D.G.: Relative curvature measures of nonlinearity. J. Roy. Stat. Soc. B 42, 1–25 (1980)

    MATH  MathSciNet  Google Scholar 

  6. Bates, D.M., Watts, D.G.: Parameter transformations for improved approximate confidence regions in nonlinear least squares. Ann. Stat. 9(6), 1152–1167 (1981)

    Article  MATH  MathSciNet  Google Scholar 

  7. Bates, D.M., Watts, D.G.: Nonlinear Regression Analysis and Its Applications, Wiley Series in Probability and Statistics, vol. 32. Wiley, New York (1988)

    Google Scholar 

  8. Bates, D.M., Hamilton, D.C., Watts, D.G.: Calculation of intrinsic and parameter-effects curvatures for nonlinear regression models. Commun. Stat. Simulat. 12(4), 469–477 (1983). doi:10.1080/03610918308812333

    Article  MATH  Google Scholar 

  9. Battogtokh, D., Asch, D., Case, M., Arnold, J., Schüttler, H.B.: An ensemble method for identifying regulatory circuits with special reference to the qa gene cluster of Neurospora crassa. Proc. Natl. Acad. Sci. USA 99(26), 16904–16909 (2002). doi:10.1073/pnas.262658899

    Article  Google Scholar 

  10. Beale, E.M.L.: Confidence regions in non-linear estimation. J. Roy. Stat. Soc. B 22(1), 41–88 (1960)

    MATH  MathSciNet  Google Scholar 

  11. Beaumont, M.A., Zhang, W., Balding, D.J.: Approximate Bayesian computation in population genetics. Genetics 162(4), 2025–2035 (2002)

    Google Scholar 

  12. Birnbaum, A.: On the foundations of statistical inference. J. Am. Stat. Assoc. 57(298), 269–306 (1962). doi:10.2307/2281641

    Article  MATH  MathSciNet  Google Scholar 

  13. Brown, K., Sethna, J.: Statistical mechanical approaches to models with many poorly known parameters. Phys. Rev. E 68(2), 021904 (2003). doi:10.1103/PhysRevE.68.021904

  14. Brown, K.S., Hill, C.C., Calero, G.A., Myers, C.R., Lee, K.H., Sethna, J.P., Cerione, R.A.: The statistical mechanics of complex signaling networks: nerve growth factor signaling. Phys. Biol. 1(3–4), 184–195 (2004). doi:10.1088/1478-3967/1/3/006

    Article  Google Scholar 

  15. Casey, F.P., Baird, D., Feng, Q., Gutenkunst, R.N., Waterfall, J.J., Myers, C.R., Brown, K.S., Cerione, R.A., Sethna, J.P.: Optimal experimental design in an epidermal growth factor receptor signalling and down-regulation model. IET Syst. Biol. 1(3), 190–202 (2007). doi:10.1049/iet-syb

    Article  Google Scholar 

  16. Chachra, R., Transtrum, M.K., Sethna, J.P.: Comment on Sloppy models, parameter uncertainty, and the role of experimental design. Mol. Biosyst. 7(8), 2522; author reply 2523–4 (2011). doi:10.1039/c1mb05046j

    Google Scholar 

  17. Chib, S., Greenberg, E.: Understanding the Metropolis Hastings algorithm. Am. Stat. 49(4), 327–335 (1995). doi:10.1080/00031305.1995.10476177

    Google Scholar 

  18. Daniels, B.C., Chen, Y.J., Sethna, J.P., Gutenkunst, R.N., Myers, C.R.: Sloppiness, robustness, and evolvability in systems biology. Curr. Opin. Biotech. 19(4), 389–395 (2008). doi:10.1016/j.copbio.2008.06.008

    Article  Google Scholar 

  19. De Smet, R., Marchal, K.: Advantages and limitations of current network inference methods. Nat. Rev. Microbiol. 8(10), 717–729 (2010). doi:10.1038/nrmicro2419

    Google Scholar 

  20. Del Moral, P., Doucet, A., Jasra, A.: Sequential Monte Carlo samplers. J. Roy. Stat. Soc. B. Met. 68(3), 411–436 (2006). doi:10.1111/j.1467-9868.2006.00553.x

    Google Scholar 

  21. Demidenko, E.: Criteria for global minimum of sum of squares in nonlinear regression. Comput. Stat. Data An. 51(3), 1739–1753 (2006). doi:10.1016/j.csda.2006.06.015

    Google Scholar 

  22. Efron, B., Hinkley, D.V.: Assessing the accuracy of the maximum likelihood estimator: observed versus expected Fisher information. Biometrika 65(3), 457–483 (1978). doi:10.1093/biomet/65.3.457

    Article  MATH  MathSciNet  Google Scholar 

  23. Erguler, K., Stumpf, M.P.H.: Practical limits for reverse engineering of dynamical systems: a statistical analysis of sensitivity and parameter inferability in systems biology models. Mol. Biosyst. 7(5), 1593–1602 (2011). doi:10.1039/c0mb00107d

    Article  Google Scholar 

  24. Eydgahi, H., Chen, W.W., Muhlich, J.L., Vitkup, D., Tsitsiklis, J.N., Sorger, P.K.: Properties of cell death models calibrated and compared using Bayesian approaches. Mol. Syst. Biol. 9(644), 644 (2013). doi:10.1038/msb.2012.69

    Google Scholar 

  25. Fernández Slezak, D., Suárez, C., Cecchi, G.A., Marshall, G., Stolovitzky, G.: When the optimal is not the best: parameter estimation in complex biological models. PloS One 5(10), e13,283 (2010). doi:10.1371/journal.pone.0013283

    Google Scholar 

  26. Fisher, R.A.: On the mathematical foundations of theoretical statistics. Philos. T. Roy. Soc. Lond. 222(594–604), 309–368 (1922). doi:10.1098/rsta.1922.0009

    Article  MATH  Google Scholar 

  27. Flaherty, P., Radhakrishnan, M.L., Dinh, T., Rebres, R.A., Roach, T.I., Jordan, M.I., Arkin, A.P.: A dual receptor crosstalk model of G-protein-coupled signal transduction. PLoS Comput. Biol. 4(9), e1000185 (2008). doi:10.1371/journal.pcbi.1000185

    Google Scholar 

  28. Girolami, M., Calderhead, B.: Riemann manifold Langevin and Hamiltonian Monte Carlo methods. J. Roy. Stat. Soc. B. Met. 73, 123–214 (2011). doi:10.1111/j.1467-9868.2010.00765.x

    Article  MathSciNet  Google Scholar 

  29. Gunawardena, J.: Models in sytems biology: the parameter problem and the meaning of robustness. In: Lodhi H.M., Muggleton, S.H. (eds.) Elements of Computational Systems Biology, pp. 19–47. Wiley Hoboken (2010). doi:10.1002/9780470556757.ch2

    Google Scholar 

  30. Gutenkunst, R.: Sloppiness, Modeling, and Evolution in Biochemical Networks. Ph.D. thesis, Cornell University (2008). http://www.ecommons.cornell.edu/handle/1813/8206

  31. Gutenkunst, R.N., Casey, F.P., Waterfall, J.J., Myers, C.R., Sethna, J.P.: Extracting falsifiable predictions from sloppy models. Ann. NY Acad. Sci. 1115, 203–211 (2007a). doi:10.1196/annals.1407.003

    Google Scholar 

  32. Gutenkunst, R.N., Waterfall, J.J., Casey, F.P., Brown, K.S., Myers, C.R., Sethna, J.P.: Universally sloppy parameter sensitivities in systems biology models. PLoS Comput. Biol. 3(10), e189 (2007b). doi:10.1371/journal.pcbi.0030189

    Google Scholar 

  33. Hagen, D.R., Apgar, J.F., White, F.M., Tidor, B.: Molecular BioSystems reply to comment on Sloppy models, parameter uncertainty, and the role of experimental design. Interface Focus pp. 2523–2524 (2011). doi:10.1039/c1mb05200d

    Google Scholar 

  34. Hagen, D.R., White, J.K., Tidor, B.: Convergence in parameters and predictions using computational experimental design. Interface Focus 3(4), 20130,008–20130,008 (2013). doi:10.1098/rsfs.2013.0008

    Google Scholar 

  35. Haines, L.M., O Brien, T.E., Clarke, G.P.Y.: Kurtosis and curvature measures for nonlinear regression models. Stat. Sinica 14(2), 547–570 (2004)

    Google Scholar 

  36. Hamilton, D.C., Watts, D.G., Bates, D.M.: Accounting for intrinsic nonlinearity in nonlinear regression parameter inference regions. Ann. Stat. 10(38), 393 (1982)

    MathSciNet  Google Scholar 

  37. Hotelling, H.: Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 24(6), 417–441 (1933)

    Article  Google Scholar 

  38. Hug, S., Schmidl, D., Li, W.B., Greiter, M.B., Theis, F.J.: Bayesian model selection methods and their application to biological ODE systems. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)

    Google Scholar 

  39. Ivancevic, T.T.: Applied Differential Geometry: a Modern introduction. World Scientific, Singapore (2007)

    Google Scholar 

  40. Jaqaman, K., Danuser, G.: Linking data to models: data regression. Nat. Rev. Mol. Cell Bio. 7(11), 813–819 (2006). doi:10.1038/nrm2030

    Article  Google Scholar 

  41. Kass, R.E.: The geometry of asymptotic inference. Stat. Sci. 4(3), 188–219 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  42. Kirk, P., Silk, D., Stumpf, M.P.H.: Reverse engineering under uncertainty. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)

    Google Scholar 

  43. Kirk, P., Thorne, T., Stumpf, M.P.: Model selection in systems and synthetic biology. Curr. Opin. Biotech. 24(4), 767–774 (2013). doi:10.1016/j.copbio.2013.03.012

    Article  Google Scholar 

  44. Kirkpatrick, S., Gelatt, C.D., Vecchi, M.P.: Optimization by simulated annealing. Science 220(4598), 671–680 (1983). doi:10.1126/science.220.4598.671

    Article  MATH  MathSciNet  Google Scholar 

  45. Kitano, H.: Systems biology: a brief overview. Science 295(5560), 1662–1664 (2002). doi:10.1126/science.1069492

    Article  Google Scholar 

  46. Kitano, H.: Biological robustness. Nat. Rev. Genet. 5(11), 826–837 (2004). doi:10.1038/nrg1471

    Article  Google Scholar 

  47. Kreutz, C., Timmer, J.: Systems biology: experimental design. FEBS J. 276(4), 923–942 (2009). doi:10.1111/j.1742-4658.2008.06843.x

    Article  Google Scholar 

  48. Le Novère, N., Bornstein, B., Broicher, A., Courtot, M., Donizelli, M., Dharuri, H., Li, L., Sauro, H., Schilstra, M., Shapiro, B., Snoep, J.L., Hucka, M.: BioModels Database: a free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems. Nucleic Acids Res. 34(Database issue), D689–91 (2006). doi:10.1093/nar/gkj092

    Google Scholar 

  49. Machta, B., Chachra, R., Transtrum, M., Sethna, J.: Parameter space compression underlies emergent theories and predictive models. Science 342(6158), 604–607 (2013). doi:10.1126/science.1238723

    Article  Google Scholar 

  50. Marino, S., Hogue, I.B., Ray, C.J., Kirschner, D.E.: A methodology for performing global uncertainty and sensitivity analysis in systems biology. J. Theor. Biol. 254(1), 178–196 (2008). doi:10.1016/j.jtbi.2008.04.011

    Article  MathSciNet  Google Scholar 

  51. Marjoram, P., Molitor, J., Plagnol, V., Tavare, S.: Markov chain Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. USA 100(26), 15,324–8 (2003). doi:10.1073/pnas.0306899100

    Google Scholar 

  52. Meyer, P., Cokelaer, T., Chandran, D., Kim, K.H., Loh, P.R., Tucker, G., Lipson, M., Berger, B., Kreutz, C., Raue, A., Steiert, B., Timmer, J., Bilal, E., Sauro, H.M., Stolovitzky, G., Saez-Rodriguez, J.: Network topology and parameter estimation: from experimental design methods to gene regulatory network kinetics using a community based approach. BMC Syst. Biol. 8(1), 13 (2014). doi:10.1186/1752-0509-8-13

    Article  Google Scholar 

  53. Moles, C.G., Mendes, P., Banga, J.R.: Parameter estimation in biochemical pathways: a comparison of global optimization methods. Genome Res. 13(11), 2467–2474 (2003). doi:10.1101/gr.1262503

    Article  Google Scholar 

  54. Murray, M.K., Rice, J.W.: Differential Geometry and Statistics, Monographs on statistics and applied probability, vol. 48. Chapman & Hall, London (1993)

    Google Scholar 

  55. Myers, C.R., Gutenkunst, R.N., Sethna, J.P.: Python unleashed on systems biology. Comput. Sci. Eng. 9(3), 34–37 (2007). doi:10.1109/MCSE.2007.60

    Article  Google Scholar 

  56. Pittendrigh, C.: On temperature independence in the clock system controlling emergence time in Drosophila. Proc. Natl. Acad. Sci. USA 40(10), 1018–1029 (1954)

    Article  Google Scholar 

  57. Rand, D.A., Shulgin, B.V., Salazar, D., Millar, A.J.: Design principles underlying circadian clocks. J. Roy. Soc. Interface 1(1), 119–130 (2004). doi:10.1098/rsif.2004.0014

    Article  Google Scholar 

  58. Robertson, H.: The solution of a set of reaction rate equations. In: Walsh, J. (ed.) Numerical Analysis, an Introduction, pp. 178–182. Academ Press, London (1966)

    Google Scholar 

  59. Rodriguez-Fernandez, M., Mendes, P., Banga, J.R.: A hybrid approach for efficient and robust parameter estimation in biochemical pathways. Biosyst. 83(2–3), 248–265 (2006). doi:10.1016/j.biosystems.2005.06.016

    Article  Google Scholar 

  60. Cedersund, G., Samuelsson, O., Ball, G., Tegnér, J., Gomez-Cabrero, D.: Optimization in biology parameter estimation and the associated optimization problem. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)

    Google Scholar 

  61. Savageau, M.A., Coelho, P.M.B.M., Fasani, R.A., Tolla, D.A., Salvador, A.: Phenotypes and tolerances in the design space of biochemical systems. Proc. Natl. Acad. Sci. USA 106(16), 6435–6440 (2009). doi:10.1073/pnas.0809869106

    Article  Google Scholar 

  62. Seber, G.A.F., Wild, C.J.: Nonlinear Regression. Wiley, New York (1988)

    Google Scholar 

  63. Shah, M., Chitforoushzadeh, Z., Janes, K.A.: Statistical data analysis and modeling. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)

    Google Scholar 

  64. Sisson, S.A., Fan, Y., Tanaka, M.M.: Sequential Monte Carlo without likelihoods. Proc. Natl. Acad. Sci. USA 104(6), 1760–1765 (2007). doi:10.1073/pnas.0607208104

    Article  MATH  MathSciNet  Google Scholar 

  65. Spivak, M.: A Comprehensive Introduction to Differential Geometry. Publish or Perish (1979)

    Google Scholar 

  66. Sunnåker, M., Stelling, J.: Model extension and model selection. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)

    Google Scholar 

  67. Toni, T., Welch, D., Strelkowa, N., Ipsen, A., Stumpf, M.P.: Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems. J. Roy. Soc. Interface 6(31), 187–202 (2009). doi:10.1098/rsif.2008.0172

    Article  Google Scholar 

  68. Tönsing, C., Timmer, J., Kreutz, C.: Cause and cure of sloppiness in ordinary differential equation models (2014). arXiv:1406.1734

  69. Transtrum, M.K.: Geodesic Levenberg-Marquardt source code (2012). http://sourceforge.net/projects/geodesiclm/

  70. Transtrum, M.K., Hart, G., Qiu, P.: Information topology identifies emergent model classes. arXiv:1409.6203 (2014)

  71. Transtrum, M.K., Machta, B.B., Sethna, J.P.: Why are nonlinear fits to data so challenging? Phys. Rev. Lett. 104(6), 060,201 (2010). doi:10.1103/PhysRevLett.104.060201

  72. Transtrum, M.K., Machta, B.B., Sethna, J.P.: Geometry of nonlinear least squares with applications to sloppy models and optimization. Phys. Rev. E 83(3), 036,701 (2011). doi:10.1103/PhysRevE.83.036701

  73. Transtrum, M.K., Qiu, P.: Optimal experiment selection for parameter estimation in biological differential equation models. BMC Bioinf. 13, 181 (2012). doi:10.1186/1471-2105-13-181

    Article  Google Scholar 

  74. Transtrum, M.K., Qiu, P.: Model reduction by manifold boundaries. Phys. Rev. Lett. 113(9), 098,701 (2014). doi:10.1103/PhysRevLett.113.098701

  75. Transtrum, M.K., Sethna, J.P.: Improvements to the Levenberg-Marquardt algorithm for nonlinear least-squares minimization. arXiv:1201.5885 (2012)

  76. Waterfall, J., Casey, F., Gutenkunst, R., Brown, K., Myers, C., Brouwer, P., Elser, V., Sethna, J.: Sloppy-Model Universality Class and the Vandermonde Matrix. Phy. Rev. Lett. 97(15) (2006). doi:10.1103/PhysRevLett.97.150601

  77. Van Schepdael, A., Carlier, A., Geris, L.: Sensitivity analysis in the design of experiments. In: Uncertainty in Biology, A Computational Modeling Approach. Springer, Chem (2016, this volume)

    Google Scholar 

  78. von Dassow, G., Meir, E., Munro, E.M., Odell, G.M.: The segment polarity network is a robust developmental module. Nature 406(6792), 188–92 (2000). doi:10.1038/35018085

    Article  Google Scholar 

  79. Xu, T.R., Vyshemirsky, V., Gormand, A., von Kriegsheim, A., Girolami, M., Baillie, G.S., Ketley, D., Dunlop, A.J., Milligan, G., Houslay, M.D., Kolch, W.: Inferring signaling pathway topologies from multiple perturbation measurements of specific biochemical species. Sci. Signal. 3(113), ra20 (2010). doi:10.1126/scisignal.2000517

    Google Scholar 

Download references

Acknowledgments

B.M. was supported by an ARCS Foundation Fellowship. A.R. was supported by NSF IGERT grant DGE-0654435. R.G. was supported by NSF grant DEB-1146074. We thank Alec Coffman for helpful discussions. R.G. and M.T. particularly thank Jim Sethna for his outstanding support and mentorship.

Author information

Authors and Affiliations

  1. Graduate Interdisciplinary Program in Statistics, University of Arizona, Tucson, AZ, USA

    Brian K. Mannakee

  2. Graduate Interdisciplinary Program in Applied Mathematics, University of Arizona, Tucson, AZ, USA

    Aaron P. Ragsdale

  3. Department of Physics and Astronomy, Brigham Young University, Provo, UT, USA

    Mark K. Transtrum

  4. Department of Molecular and Cellular Biology, Tucson, AZ, USA

    Ryan N. Gutenkunst

Authors
  1. Brian K. Mannakee
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aaron P. Ragsdale
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mark K. Transtrum
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ryan N. Gutenkunst
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryan N. Gutenkunst .

Editor information

Editors and Affiliations

  1. Biomechanics Research Unit, University of Liège, Liège 1, Belgium

    Liesbet Geris

  2. Center for Molecular Medicine, Karolinska University Hospital, Stockholm, Sweden

    David Gomez-Cabrero

Conflict of Interest

Conflict of Interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

Copyright information

© 2016 Springer International Publishing Switzerland

About this chapter

Cite this chapter

Mannakee, B.K., Ragsdale, A.P., Transtrum, M.K., Gutenkunst, R.N. (2016). Sloppiness and the Geometry of Parameter Space. In: Geris, L., Gomez-Cabrero, D. (eds) Uncertainty in Biology. Studies in Mechanobiology, Tissue Engineering and Biomaterials, vol 17. Springer, Cham. https://doi.org/10.1007/978-3-319-21296-8_11

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-21296-8_11

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-21295-1

  • Online ISBN: 978-3-319-21296-8

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation