The Duality of State and Observation in Probabilistic Transition Systems

  • Monica Dinculescu
  • Christopher Hundt
  • Prakash Panangaden
  • Joelle Pineau
  • Doina Precup
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7758)

Abstract

In this paper we consider the problem of representing and reasoning about systems, especially probabilistic systems, with hidden state. We consider transition systems where the state is not completely visible to an outside observer. Instead, there are observables that partly identify the state. We show that one can interchange the notions of state and observation and obtain what we call a dual system. In the case of deterministic systems, the double dual gives a minimal representation of the behaviour of the original system. We extend these ideas to probabilistic transition systems and to partially observable Markov decision processes (POMDPs).

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Arbib, M.A., Manes, E.G.: Machines in a category: An expository introduction. SIAM Review 16, 163–192 (1974)MathSciNetMATHCrossRefGoogle Scholar
  2. 2.
    Arbib, M.A., Manes, E.G.: Foundations of system theory: decomposable systems. Automatica 10, 285–302 (1974)MathSciNetMATHCrossRefGoogle Scholar
  3. 3.
    Arbib, M.A., Manes, E.G.: Adjoint machines, state behavior machines and duality. J. Pure Appl. Algebra 6, 313–343 (1975)MathSciNetMATHCrossRefGoogle Scholar
  4. 4.
    Bainbridge, E.S.: The fundamental duality of system theory. In: Hartnett, W.E. (ed.) Systems: Approaches, Theories, Applications, pp. 45–61. Reidel (1977)Google Scholar
  5. 5.
    Barr, M.: The chu construction. Theory Appl. Categories 2, 17–35 (1996)MathSciNetMATHGoogle Scholar
  6. 6.
    Barr, M.: The separated extensional Chu category. Theory Appl. Categories 4(6), 137–147 (1998)MathSciNetMATHGoogle Scholar
  7. 7.
    Bidoit, M., Hennicker, R., Kurz, A.: Observational logic, constructor-based logic, and their duality. Theor. Comput. Sci. 3(298), 471–510 (2003)MathSciNetCrossRefGoogle Scholar
  8. 8.
    Bonchi, F., Bonsangue, M.M., Rutten, J.J.M.M., Silva, A.: Brzozowski’s Algorithm (Co)Algebraically. In: Constable, R.L., Silva, A. (eds.) Logic and Program Semantics, Kozen Festschrift. LNCS, vol. 7230, pp. 12–23. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  9. 9.
    Bonsangue, M.M., Kurz, A.: Duality for Logics of Transition Systems. In: Sassone, V. (ed.) FOSSACS 2005. LNCS, vol. 3441, pp. 455–469. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  10. 10.
    Brzozowski, J.A.: Canonical regular expressions and minimal state graphs for definite events. In: Fox, J. (ed.) Proceedings of the Symposium on Mathematical Theory of Automata. MRI Symposia Series, vol. 12, pp. 529–561. Polytechnic Press of the Polytechnic Institute of Brooklyn (April 1962), book appeared in 1963Google Scholar
  11. 11.
    Chrisman, L.: Reinforcement learning with perceptual aliasing: The perceptual distinctions approach. In: Proceedings of the Tenth National Conference on Artificial Intelligence, pp. 183–188 (1992)Google Scholar
  12. 12.
    Clarke, E.M., Grumberg, O., Peled, D.A.: Model Checking. The MIT Press, Cambridge (1999)Google Scholar
  13. 13.
    Pavlovic, D., Mislove, M., Worrell, J.B.: Testing Semantics: Connecting Processes and Process Logics. In: Johnson, M., Vene, V. (eds.) AMAST 2006. LNCS, vol. 4019, pp. 308–322. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  14. 14.
    Desharnais, J., Edalat, A., Panangaden, P.: Bisimulation for labeled Markov processes. Information and Computation 179(2), 163–193 (2002)MathSciNetMATHCrossRefGoogle Scholar
  15. 15.
    Desharnais, J., Gupta, V., Jagadeesan, R., Panangaden, P.: A metric for labelled Markov processes. Theoretical Computer Science 318(3), 323–354 (2004)MathSciNetMATHCrossRefGoogle Scholar
  16. 16.
    Hopcroft, J.E., Motwani, R., Ullman, J.D.: Introduction to automata theory, languages, and computation, 2nd edn. Addison-Wesley Series in Computer Science. Addison-Wesley-Longman (2001)Google Scholar
  17. 17.
    Hundt, C., Panangaden, P., Pineau, J., Precup, D.: Representing systems with hidden state. In: The Twenty-First National Conference on Artificial Intelligence, AAAI (2006)Google Scholar
  18. 18.
    James, M., Singh, S.: Learning and discovery of predictive state representations in dynamical systems with reset. In: International Conference on Machine Learning, vol. 21, pp. 417–424 (2004)Google Scholar
  19. 19.
    Johnstone, P.: Stone Spaces, Cambridge Studies in Advanced Mathematics, vol. 3. Cambridge University Press (1982)Google Scholar
  20. 20.
    Kaelbling, L.P., Littman, M.L., Cassandra, A.R.: Planning and acting in partially observable stochastic domains. Artificial Intelligence 101 (1998)Google Scholar
  21. 21.
    Kalman, R.E., Falb, P.L., Arbib, M.A.: Topics in Mathematical Systems Theory. McGraw Hill (1969)Google Scholar
  22. 22.
    Kozen, D.: A probabilistic PDL. Journal of Computer and Systems Sciences 30(2), 162–178 (1985)MathSciNetMATHCrossRefGoogle Scholar
  23. 23.
    Kozen, D.: Automata and computability. Undergraduate texts in computer science. Springer (1997)Google Scholar
  24. 24.
    Littman, M., Sutton, R., Singh, S.: Predictive representations of state. In: Advances in Neural Information Processing Systems 14 (NIPS), vol. 14, pp. 1555–1561 (2002)Google Scholar
  25. 25.
    McCallum, A.: Reinforcement Learning with Selective Perception and Hidden State. Ph.D. thesis, University of Rochester (1995)Google Scholar
  26. 26.
    Milner, R.: A Calculus of Communication Systems. LNCS, vol. 92. Springer, Heidelberg (1980)CrossRefGoogle Scholar
  27. 27.
    Milner, R.: Communication and Concurrency. Prentice-Hall (1989)Google Scholar
  28. 28.
    Mislove, M., Ouaknine, J., Pavlovic, D., Worrell, J.: Duality for Labelled Markov Processes. In: Walukiewicz, I. (ed.) FOSSACS 2004. LNCS, vol. 2987, pp. 393–407. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  29. 29.
    Park, D.: Concurrency and Automata on Infinite Sequences. In: Deussen, P. (ed.) GI-TCS 1981. LNCS, vol. 104, pp. 167–183. Springer, Heidelberg (1981)CrossRefGoogle Scholar
  30. 30.
    Park, D.: Title unknown (1981), slides for Bad Honnef Workshop on Semantics of ConcurrencyGoogle Scholar
  31. 31.
    Pratt, V.R.: The Duality of Time and Information. In: Cleaveland, W.R. (ed.) CONCUR 1992. LNCS, vol. 630, pp. 237–253. Springer, Heidelberg (1992)CrossRefGoogle Scholar
  32. 32.
    Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley (1994)Google Scholar
  33. 33.
    Rivest, R.L., Schapire, R.E.: Diversity-based inference of finite automata. Journal of the ACM 41(3), 555–589 (1994)MathSciNetMATHCrossRefGoogle Scholar
  34. 34.
    Singh, S., James, M.R., Rudary, M.R.: Predictive state representations: A new theory for modeling dynamical systems. In: Proceedings of the Twentieth Conference on Uncertainty in Artificial Intelligence (UAI), pp. 512–519 (2004)Google Scholar
  35. 35.
    Sipser, M.: Introduction to the theory of computation. PWS Publishing Company (1997)Google Scholar
  36. 36.
    Smyth, M.: Powerdomains and Predicate Transformers. In: Díaz, J. (ed.) ICALP 1983. LNCS, vol. 154, pp. 662–676. Springer, Heidelberg (1983)CrossRefGoogle Scholar
  37. 37.
    Sutton, R.S., Tanner, B.: Temporal-difference networks. In: Advances in Neural Information Processing Systems, vol. 17 (2005)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Monica Dinculescu
    • 1
  • Christopher Hundt
    • 2
  • Prakash Panangaden
    • 3
  • Joelle Pineau
    • 3
  • Doina Precup
    • 3
  1. 1.Morgan StanleyMontrealCanada
  2. 2.Google Inc.Mountain ViewUSA
  3. 3.McGill UniversityMontréalCanada

Personalised recommendations