On the Origin of Ambiguity in Efficient Communication

Abstract

This article studies the emergence of ambiguity in communication through the concept of logical irreversibility and within the framework of Shannon’s information theory. This leads us to a precise and general expression of the intuition behind Zipf’s vocabulary balance in terms of a symmetry equation between the complexities of the coding and the decoding processes that imposes an unavoidable amount of logical uncertainty in natural communication. Accordingly, the emergence of irreversible computations is required if the complexities of the coding and the decoding processes are balanced in a symmetric scenario, which means that the emergence of ambiguous codes is a necessary condition for natural communication to succeed.

Appendix: Ambiguity and Physical Irreversibility

Throughout the paper we have highlighted the strict relation between the logical irreversibility of the computations generating a given code and the ambiguity of the latter. In this appendix we briefly revise the role of logical irreversibility in the foundations of physics, through its relation to thermodynamic irreversibility.

Our objective here is to show, in a rather informal way, that reasonings and concepts that are commonly used in physics can be naturally connected to the conceptual development of this paper. We warn the reader that this appendix does not attempt to provide formal relations between ambiguous codes and its energetic cost or other thermodynamic quantities, nor to relate the emergence of ambiguity to some explicit physical process. The rigorous exploration of the above mentioned topics is a fascinating issue, but lies beyond the scope of this paper.

The strict relation of thermodynamic irreversibility and logical irreversibility is a hot topic of debate since the definition of the equivalence of heat and bits by Landauer (1961). This equivalence, known as Landauer’s principle, states that, for any erased bit of information, a quantity of at least

$$\begin{aligned} k_BT\ln 2 \end{aligned}$$

joules is dissipated in terms of heat, being $k_B=1.38\times 10^{-23}\,\hbox {J/K}$ the Boltzmann constant and $T$ the temperature of the system. This principle relates logical irreversibility and thermodynamical irreversibility; however, it is worth noting that it provides only a lower bound and that it is far away from the energetic costs of any real computing process. To see how we can connect both types of irreversibility, we first state that thermodynamical irreversibility is a property of abstract processes—interestingly, almost all processes taking place in our everyday life are irreversible. The common property of such processes is that they generate thermodynamical entropy. The second law of thermodynamics states that any physical process generates a non-negative amount of entropy; i.e., for the process $\mathbf P$,

$$\begin{aligned} \varDelta S(\mathbf P )\ge 0. \end{aligned}$$

The units of physical entropy are nats instead of bits. Now suppose that we face the problem of reversing the process $\mathbf P$—for example, a gas expansion—by which $\varDelta S(\mathbf P )> 0$. Without further help, the reversion of this process is forbidden by the second law, since it would generate a net amount of negative entropy. Therefore, we will need external energy to reverse the process. Similarly, we have observed that

$$\begin{aligned} H(X_{\varOmega }|X_s)\ge 0, \end{aligned}$$

which means that information cannot be created during an information process. A negative amount of $H(X_{\varOmega }|X_s)$ would imply, by virtue of Eq. (5), a net creation of information. Therefore, we face the same problem. Indeed, if we have a computational process $\mathbf C$ by which $H_\mathbf{C }(X_{\varOmega }|X_s)>0$, the reversion of such a process, with no further external help, would be a process by which the computations would generate information. The reversion, as we have discussed above, is only possible by the external addition of information. Thus the information flux can only be maintained (in the case where all computations are logically reversible) or degraded, and the same applies for the energy flux: by the second law, the energy flux can only be maintained (in the case of thermodynamically reversible processes) or degraded.

We can informally find a quantitative connection between the two entropies. If $\mathbf Q (\mathbf P )$ is the heat generated during the physical process, its associated physical entropy generation is defined as

$$\begin{aligned} \varDelta S(\mathbf{P })=\frac{\mathbf{Q }(\mathbf{P })}{T}. \end{aligned}$$

In turn, if we consider an ideal—from the energetic viewpoint—computational process $\mathbf C$, we know, from Landauer’s principle, that

$$\begin{aligned} \mathbf Q (\mathbf C )=k_BT\ln 2\times \mathrm{erased\;bits}. \end{aligned}$$

And we actually know how many bits have been erased –or dissipated. Exactly $H(X_{\varOmega }|X_s)$ bits. Therefore, the physical entropy generated by this ideal, irreversible computing process will be:

$$\begin{aligned} \varDelta S(\mathbf C )= k_B\ln 2 H_\mathbf{C }(X_{\varOmega }|X_s). \end{aligned}$$

Accordingly, logically irreversible computations are thermodynamically irreversible. Notice that this only proves the implication between both irreversibilities, but is of no practical use, since it is a lower bound. Any real computing process will be such that:

$$\begin{aligned} \varDelta S(\mathbf C )\gg k_B\ln 2 H_\mathbf{C }\left( X_{\varOmega }|X_s\right) . \end{aligned}$$

We finally highlight that one could object that, under the above considerations, the most favorable situation would be the one in which all computations are performed in a logically reversible way, since there would not be an energy penalty. However, this interpretation is misleading. Imagine a coding machine receiving an informational input but working in a totally irreversible way. In this case, there would be a dissipation of information that would undergo into heat production. The energy dissipated in terms of heat, however, would come from the environment, not from the machine, which would be only heated. The creation of a code in this machine to let information be coded and flow would imply, on the contrary, to write a code into the machine, and thus, to erase the initial configuration of the machine—whatever it was, maybe a random one—, to properly adapt it to a consistent coding process. We observe that this process would demand energy that would not come, at least directly, from the environment. Therefore, a perfect coding performing logically reversible computations only would be, in principle, energetically more demanding than a logically irreversible coding.

We insist that the above considerations fall into the abstract level and practical implementations must face multiple additional problems which have been not been taken into consideration. With this short exposition we only want to emphasize the general character of logical irreversibility and ambiguity in natural communication systems. More than an imperfection, ambiguity seems to be, for natural communication systems, a feature as unavoidable as the generation of heat during a thermodynamical process.