Discrete Entropy

From a mathematical point of view, entropy made its first appearance in continuous-time dynamical systems (more exactly, in Hamiltonian flows), and from there it was extended to quantum mechanics by von Neumann, to information theory by Shannon, and to discrete-time dynamical systems by Kolmogorov and Sinai. In all these cases we observe that (i) if the state space is discrete and/or finite (like in quantum mechanics and finite-alphabet information sources), then the evolution is random and (ii) if the evolution is deterministic (like in continuous- and discrete-time dynamical systems), then the state space is infinite. Still today one speaks of random dynamical systems in the first case and of deterministic dynamical systems in the second case. But not all dynamical systems of interest fall under one of the previous categories. An important example of a deterministic physical system where both state space and dynamics are discrete is a digital computer; this entails that any dynamical trajectory in computer becomes eventually periodic–-a well-known effect in the theory and practice of pseudo-random number generation. Dynamical systems with discrete and even with a finite number of states have been considered by a number of authors, in particular in the development of discrete chaos [125]–-an attempt to formalize the idea that maps on finite sets may have different diffusion and mixing properties. From this perspective, it seems desirable to export some concepts and tools from the general theory to this new setting. This is the rationale behind, e.g., the discrete Lyapunov exponent [124, 125].