Directed Information on Abstract Spaces: Properties and Extremum Problems

  • Charalambos D. Charalambous
  • Photios A. Stavrou
  • Christos K. Kourtellaris
Chapter

Abstract

Directed information is an information theoretic measure which accounts for the direction of information flow over causal systems with feedback, such as network communication and communication for control problems. In this chapter, we discuss several functional and topological properties of directed information for general Polish spaces (complete separable metric spaces) using the topology of weak convergence of probability measures. These include, convexity/concavity of directed information, weak compactness of families of causally conditioned convolutional distributions, lower semicontinuity of directed information, continuity of directed information, and extremum problems of directed information, including variational equalities [utilized in Blahut–Arimoto algorithm (BAA)], which are important in nonanticipative or real-time joint source-channel coding (JSCC). These basic functional and topological properties of directed information are analogous to those of mutual information. Throughout the chapter, the importance of the properties of directed information is discussed in the context of extremum problems of directed information, such as point to point and network applications.

References

  1. 1.
    Marko H (1973) The bidirectional communication theory—a generalization of information theory. IEEE Trans Commun 21(12):1345–1351Google Scholar
  2. 2.
    Massey JL (1990) Causality, feedback and directed information. In: International symposium on information theory and its applications (ISITA), 27–30 Nov, pp 303–305Google Scholar
  3. 3.
    Kramer G (1998) Directed information for channels with feedback. PhD thesis, Swiss Federal Institute of Technology, Zurich, SwitzerlandGoogle Scholar
  4. 4.
    Charalambous CD, Stavrou PA, Ahmed NU (2014) Nonanticipative rate distortion function and relations to filtering theory. IEEE Trans Autom Control 59(4):937–952Google Scholar
  5. 5.
    Charalambous CD, Stavrou PA (2014) Optimization of directed information and relations to filtering theory. In: European control conference (ECC) (to appear). Strasbourg, France, pp 24–27Google Scholar
  6. 6.
    Csiszár I, Körner J (1981) Information theory: coding theorems for discrete memoryless systems. Academic Press, New YorkGoogle Scholar
  7. 7.
    Charalambous CD, Stavrou PA (2013) Directed information on abstract spaces: properties and variational equalities. IEEE Trans Inf Theo (Online). Available: http://arxiv.org/abs/1302.3971
  8. 8.
    Stavrou PA, Kourtellaris CK, Charalambous CD (2014) Applications of information nonanticipative rate distortion function. In: IEEE international symposium on information theory (ISIT) (to appear), Honolulu, HI, USA, 29 June–5 July 2014 (Online). Available: http://arxiv.org/abs/1401.5828v4
  9. 9.
    Charalambous CD, Stavrou PA (2012) Directed information on abstract spaces: properties and extremum problems. In: IEEE international symposium on information theory (ISIT), Cambridge, MA, USA, 1-â 6 July 2012, pp 518–522Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Charalambos D. Charalambous
    • 1
  • Photios A. Stavrou
    • 1
  • Christos K. Kourtellaris
    • 1
  1. 1.Department of Electrical and Computer Engineering (ECE)University of CyprusNicosiaCyprus

Personalised recommendations