Skip to main content
SpringerLink
Log in

On Design of Approximate Finite-Dimensional Estimators: The Bayesian View

  • Chapter
Book cover Mutual Impact of Computing Power and Control Theory

Abstract

Despite long experience with the use of probability for inductive reasoning, propagation of conditional probability using a reduced rather than sufficient data statistic remains to be a challenging puzzle. The paper analyses in detail the main obstacles on the way towards real-time recursive implementation of Bayesian parameter estimation. Three key issues are dealt with — reduction of excessive information, approximation of the ideal Bayesian solution and numerical implementation of the resulting estimator. In all the stages of design of an approximate Bayesian estimator, the uncertainty of the true conditional probability caused by using just limited computer resources is required to be reflected consistently in the posterior uncertainty of the unknown parameters.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abramowitz, M. and I.A. Stegun (Eds) (1964). Handbook of Mathematical Functions with Formulas, Graphs and Mathematical Tables. John Wiley, New York.

    Google Scholar 

  2. Amari, S. (1990). Differential-Geometrical Methods in Statistics,2nd revised edition. Springer-Verlag, New York.

    Google Scholar 

  3. Bahadur, R.R. (1954). Sufficiency and statistical decision functions. Ann. Math. Statist., 25, 423–462.

    Article  MathSciNet  Google Scholar 

  4. Barron, A.R. and R.L. Barron (1988). Statistical learning networks: a unifying view. Computing Science and Statistics: Proc. 20th Symp. on the Interface, Alexandria, Va., 192–203.

    Google Scholar 

  5. Barron, A.R. (1989). Statistical properties of artificial neural networks. Proc. 28th IEEE Conf. on Decision and Control,New York.

    Google Scholar 

  6. Barron, A.R. and C.H. Sheu (1991). Approximation of density functions by sequences of exponential families. Ann. Statist., 19 1347–1369.

    Article  MathSciNet  MATH  Google Scholar 

  7. Bernardo, J.M. (1987). Approximations in statistics from a decision theoretical viewpoint. In R. Viertl (Ed.), Probability and Bayesian Statistics. Plenum Press, New York.

    Google Scholar 

  8. Bertsekas, D. and S.E. Shreve (1978). Stochastic Optimal Control: The Discrete Time Case. Academic Press, New York.

    MATH  Google Scholar 

  9. Billings, S.A., H.B. Jamaluddin and S. Chen (1992). Properties of neural networks with applications to modelling non-linear dynamical systems. Int. J. Control, 55, 193–224.

    Article  MathSciNet  MATH  Google Scholar 

  10. Bohlin, T. (1970). Information pattern for linear discrete-time models with stochastic coefficients. IEEE Trans. Automatic Control, 15, 104–106.

    Article  Google Scholar 

  11. Breiman, L., J.H. Friedman, R.A. Olshen and C.J. Stone (1984). Classification and Regression Trees. Wadsworth, Belmont, Ca.

    MATH  Google Scholar 

  12. Brown, L.D. (1987). Fundamentals of Statistical Exponential Families. Institute of Mathematical Statistics, Hayward, Ca.

    Google Scholar 

  13. Byrnes, C.I. and A. Lindquist (Eds) (1986). Theory and Applications of Nonlinear Control Systems. North-Holland, Amsterdam.

    MATH  Google Scholar 

  14. Byrnes, C.I., C.F. Martin and R.E. Saeks (Eds) (1988). Analysis and Control of Nonlinear Systems. North-Holland, Amsterdam.

    Google Scholar 

  15. Cencov, N.N. (1982). Statistical Decision Rules and Optimal Inference. American Mathematical Society, Providence, R.I.

    Google Scholar 

  16. Cox, R.T. (1946). Probability, frequency and reasonable expectation. Am. J. Physics, 14, 1–13.

    Article  MATH  Google Scholar 

  17. Csiszár, I. (1975). I-divergence geometry of probability distributions and minimization problems. Ann. Probab., 3, 146–158.

    Article  MATH  Google Scholar 

  18. Davis, M.H.A. and P.P. Varaiya (1972). Information states for linear stochastic systems. J. Mathematical Analysis and Applications, 37, 384–402.

    Article  MathSciNet  MATH  Google Scholar 

  19. Davis, M.H.A. and S.I. Marcus (1981). An introduction to nonlinear filtering. In M. Hazewinkel and J.C. Willems (Eds), Stochastic Systems: The Mathematics of Filtering and Identification and Applications. Reidel, Dordrecht., 53–75.

    Chapter  Google Scholar 

  20. Dawid, A.P. (1979). Conditional independence in statistical theory (with discussion). J. R. Statist. Soc. B, 41, 1–31.

    MathSciNet  MATH  Google Scholar 

  21. de Boor, C. and J.R. Rice (1979). An adaptive algorithm for multivariate approximation giving optimal convergence rates. J. Approx. Theory,25, 337–359.

    Article  MathSciNet  MATH  Google Scholar 

  22. de Finetti, B. (1990). Theory of Probability, Vol. 1 and 2, Wiley Classics Library Edition. John Wiley, Chichester.

    Google Scholar 

  23. Dempster, A.P. (1967). Upper and lower probabilities induced by a multivalued mapping. Ann. Math. Statist., 38, 325–339.

    Article  MathSciNet  MATH  Google Scholar 

  24. Donoho, D.L. and I.M. Johnstone (1989). Projection-based approximation and a duality with kernel methods. Ann. Statist., 17, 435–475.

    Article  MathSciNet  Google Scholar 

  25. DiMasi, G.B. and T.J. Taylor (1991). A new approximation method for nonlinear filtering using nilpotent harmonic analysis. Proc. 30th IEEE Conf. on Decision and Control, Brighton, UK., 2750–2751.

    Chapter  Google Scholar 

  26. Ferguson, T.S. (1973). A Bayesian analysis of some nonparametric problems. Ann. Statist., 1, 209–230.

    Article  MathSciNet  MATH  Google Scholar 

  27. Fisher, R.A. (1922). On the mathematical foundations of theoretical statistics. Roy. Soc. Phil. Trans. A, 222 309–368.

    Article  MATH  Google Scholar 

  28. Florens, J.P., M. Mouchart and J.M. Rolin (1990). Elements of Bayesian Statistics. Marcel Dekker, New York.

    MATH  Google Scholar 

  29. Friedman, J.H. and W. Stuetzle (1981). Projection pursuit regression. J. Amer. Statist. Assoc., 76 817–823.

    Article  MathSciNet  Google Scholar 

  30. Friedman, J.H. (1991). Multivariate adaptive regression splines (with discussion). Ann. Statist., 19 1–141.

    Article  MathSciNet  MATH  Google Scholar 

  31. Grandy, W.T. (1985). Incomplete information and generalized inverse problems. In C.R. Smith and W.T. Grandy (Eds), Maximum-Entropy and Bayesian Methods in Inverse Problems. Reidel, Dordrecht., 1–19.

    Google Scholar 

  32. Halmos, P.R. and L.J. Savage (1949). Application of the Radon-Nikodym theorem to the theory of sufficient statistics. Ann. Math. Stat., 20, 225–241.

    Article  MathSciNet  MATH  Google Scholar 

  33. Hampel, F.R., E.M. Ronchetti, P.J. Rousseeuw and W.A. Stahel (1986). Robust Statistics: The Approach Based on Influence Functions. John Wiley, New York.

    MATH  Google Scholar 

  34. Hazewinkel, M. and J.C. Willems (Eds) (1981). Stochastic Systems: The Mathematics of Filtering and Identification and Applications. Reidel, Dordrecht.

    MATH  Google Scholar 

  35. Huber, P.J. (1981). Robust Statistics. John Wiley, New York.

    Book  MATH  Google Scholar 

  36. Huber, P.J. (1985). Projection pursuit. Ann. Statist., 13, 435–475.

    Article  MathSciNet  MATH  Google Scholar 

  37. Isidori, A. (1989). Nonlinear Control Systems: An Introduction, 2nd edition. Springer-Verlag, Berlin.

    Google Scholar 

  38. Jacobs, O.L.R. (Ed.) (1980). Analysis and Optimisation of Stochastic Systems. Academic Press, London.

    MATH  Google Scholar 

  39. Jaynes, E.T. (1979). Where do we stand on maximum entropy? In R.O. Levine and M. Tribus (Eds), The Maximum Entropy Formalism. MIT Press, Cambridge., 15–118.

    Google Scholar 

  40. Jaynes, E.T. (1984). Prior information and ambiguity in inverse problems. In D.W. McLaughlin (Ed.), Inverse Problems. American Mathematical Society, Providence, R.I.

    Google Scholar 

  41. Kárný, M., A. Halousková, J. Böhm, R. Kulhavý and P. Nedoma (1985). Design of linear quadratic adaptive control: theory and algorithms for practice. Supplement to the journal Kybernetika, 21, No. 1–4.

    Google Scholar 

  42. Kulhavý, R. (1990a). Recursive Bayesian estimation under memory limitation. Kybernetika, 26 1–16.

    MATH  Google Scholar 

  43. Kulhavý, R. (1990b). A Bayes-closed approximation of recursive nonlinear estimation. Int. J. Adaptive Control and Signal Processing,4, 271–285.

    Article  MATH  Google Scholar 

  44. Kulhavý, R. (1990с). Recursive nonlinear estimation: a geometric approach. Automatica, 26, 545–555.

    Article  MATH  Google Scholar 

  45. Kulhavý, R. (1992a). Recursive nonlinear estimation: geometry of a space of posterior densities. Automatica, 28, 313–323.

    Article  MATH  Google Scholar 

  46. Kulhavý, R. (1992b). Implementation of Bayesian parameter estimation in adaptive control and signal processing. Int. Conf. on Practical Bayesian Statistics, Nottingham, UK.

    Google Scholar 

  47. Kulhavý, R., I. Nagy and J. Spousta (1992). Towards real-time implementation of Bayesian parameter estimation. IFAC Symp. on Adaptive Systems in Control and Signal Processing, Grenoble, France, 263–268.

    Google Scholar 

  48. Kulhavý, R. and M.B. Zarrop (1992). On a general concept of forgetting. Submitted to Int. J. Control.

    Google Scholar 

  49. Kumar, P.R. (1985). A survey of some results in stochastic adaptive control. SIAM J. Control and Optimization, 23, 329–380.

    Article  MATH  Google Scholar 

  50. Kumar, P.R. and P.P. Varaiya (1986). Stochastic Systems: Estimation, Identification and Adaptive Control. Prentice Hall, Englewood Cliffs, N.J.

    Google Scholar 

  51. Kyburg, H.E. and H.E. Smokier (1964). Studies in Subjective Probability. John Wiley, New York.

    MATH  Google Scholar 

  52. Lauritzen, S.L. (1988). Extremal Families and Systems of Sufficient Statistics. Springer-Verlag, Berlin.

    Book  MATH  Google Scholar 

  53. Light, W.A. and E.W. Cheney (1985). Approximation Theory in Tensor Product Spaces. Springer-Verlag, New York.

    MATH  Google Scholar 

  54. Lindley, D.V. (1980). Approximate Bayesian methods. In J.M. Bernardo, M.H. DeGroot, D.V. Lindley and A.F.M. Smith (Eds), Bayesian Statistics 1. Valencia Univ. Press, Valencia, 223–245.

    Google Scholar 

  55. Narendra, K.S. and K. Parthasarathy (1990). Identification and control of dynamical systems using neural networks. IEEE Trans. Neural Networks, 1, 4–27.

    Article  Google Scholar 

  56. Peterka, V. (1981). Bayesian approach to system identification. In P. Eykhoff (Ed.), Trends and Progress in System Identification. Pergamon Press, Oxford. Chap. 8, 239–304.

    Google Scholar 

  57. Poggio, T. and F. Girosi (1989). A theory of networks for approximation and learning. A.I. Memo, No. 1140, Artificial Intelligence Laboratory, Massachusetts Inst. of Technology.

    Google Scholar 

  58. Poggio, T. and F. Girosi (1990). Networks for approximation and learning. Proc. IEEE, 78, 1481–1497.

    Article  Google Scholar 

  59. Rudin, W. (1987). Real and Complex Analysis, 3rd edition. McGraw-Hill, New York.

    MATH  Google Scholar 

  60. Sonner, R.M. and J.-J.E. Slotine (1991). Stable adaptive control and recursive identification using radial Gaussian networks. Proc. 30th IEEE Conf. on Decision and Control, Brighton, UK, 2116–2123.

    Chapter  Google Scholar 

  61. Savage, L.J. (1972). The Foundations of Statistics, 2nd edition. Dover Publications, New York.

    MATH  Google Scholar 

  62. Shafer, G. (1976). A Mathematical Theory of Evidence. Princeton Univ. Press, Princeton, N.J.

    MATH  Google Scholar 

  63. Shaw, J.E.H. (1988a). Aspects of numerical integration and summarisation. In J.M. Bernardo, M.H. DeGroot, D.V. Lindley and A.F.M. Smith (Eds), Bayesian Statistics 3. Oxford Univ. Press, Oxford, 411–428.

    Google Scholar 

  64. Shaw, J.E.H. (1988b). A quasirandom approach to integration in Bayesian statistics. Ann. Statist., 16, 895–914.

    Article  MATH  Google Scholar 

  65. Sorenson, H.W. (1974). Practical realization of nonlinear estimators. In D.G. Lainiotis (Ed.), Estimation Theory. American Elsevier, New York.

    Google Scholar 

  66. Sorenson, H.W. (1988). Recursive estimation for nonlinear dynamic systems. In J.C. Spall (Ed.), Bayesian Analysis of Time Series and Dynamic Models. Dekker, New York.

    Google Scholar 

  67. Striebel, C. (1965). Sufficient statistics in the optimal control of stochastic systems. J. Mathematical Analysis and Applications, 12, 576–592.

    Article  MathSciNet  MATH  Google Scholar 

  68. Tierney, L. and J.B. Kadane (1986). Accurate approximations for posterior moments and marginal densities. J. Amer. Statist. Assoc., 81, 82–86.

    Article  MathSciNet  MATH  Google Scholar 

  69. Tierney, L., R.E. Kass and J.B. Kadane (1989). Fully exponential Laplace approximations to expectations and variances of nonpositive functions. J. Amer. Statist. Assoc., 84, 710–716.

    Article  MathSciNet  MATH  Google Scholar 

  70. van Campenhout, J.M. and T.M. Cover (1981). Maximum entropy and conditional probability. IEEE Trans. Inform. Theory,IT-27, 483–489.

    Article  Google Scholar 

  71. West, M. and J. Harrison (1988). Bayesian Forecasting and Dynamic Models. Springer-Verlag, New York.

    Google Scholar 

  72. White, H. (1989). Learning in artificial neural networks: a statistical perspective. Neural Computation, 1, 425–464.

    Article  Google Scholar 

  73. Wiberg, D.M. and D.G. DeWolf (1991). A convergent approximation of the optimal parameter estimator. Proc. 30th IEEE Conf. on Decision and Control, Brighton, UK, 2017–2023.

    Chapter  Google Scholar 

  74. Willems, J.C. (1980). Some remarks on the concept of information state. In O.L.R. Jacobs (Ed.), Analysis and Optimization of Stochastic Systems. Academic Press, London, 285–295.

    Google Scholar 

  75. Zadeh, L.A. (1978). Fuzzy sets as a basis for a theory of possibility. Fuzzy Sets and Systems, 1, 3–28.

    Article  MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Control Systems Centre, UMIST, P.O. Box 88, Manchester, M60 1QD, UK

    Rudolf Kulhavý (SERC Visiting Fellow)

  2. Institute of Information Theory and Automation, Czech Academy of Sciences, P.O. Box 18, 182 08, Prague, Czechoslovakia

    Rudolf Kulhavý (SERC Visiting Fellow)

Authors
  1. Rudolf Kulhavý
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute of Information Theory and Automation, Academy of Sciences of the Czech Republic, Prague, Czech Republic

    M. Kárný

  2. School of Engineering and Information Sciences, University of Reading, Reading, UK

    K. Warwick

Rights and permissions

Reprints and permissions

Copyright information

© 1993 Springer Science+Business Media New York

About this chapter

Cite this chapter

Kulhavý, R. (1993). On Design of Approximate Finite-Dimensional Estimators: The Bayesian View. In: Kárný, M., Warwick, K. (eds) Mutual Impact of Computing Power and Control Theory. Springer, Boston, MA. https://doi.org/10.1007/978-1-4615-2968-2_2

Download citation

  • DOI: https://doi.org/10.1007/978-1-4615-2968-2_2

  • Publisher Name: Springer, Boston, MA

  • Print ISBN: 978-1-4613-6291-3

  • Online ISBN: 978-1-4615-2968-2

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation