Skip to main content

On the Origin of Ambiguity in Efficient Communication

Abstract

This article studies the emergence of ambiguity in communication through the concept of logical irreversibility and within the framework of Shannon’s information theory. This leads us to a precise and general expression of the intuition behind Zipf’s vocabulary balance in terms of a symmetry equation between the complexities of the coding and the decoding processes that imposes an unavoidable amount of logical uncertainty in natural communication. Accordingly, the emergence of irreversible computations is required if the complexities of the coding and the decoding processes are balanced in a symmetric scenario, which means that the emergence of ambiguous codes is a necessary condition for natural communication to succeed.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2

Notes

  1. 1.

    It is desirable, but not mandatory. As noted by Thomason in his ‘Introduction’ (Montague 1974, Chapter 1), “[A] by-product of Montague’s work (...) is a theory of how logical consequence can be defined for languages admitting syntactic ambiguity. For those logicians concerned only with artificial languages this generalization will be of little interest, since there is no serious point to constructing an artificial language that is not disambiguated (p. 4, note 5)” if the objective is to characterize logical notions such as consequence. However, this generalization is relevant for the development of ‘Universal Grammar’ in Montague’s sense, i.e., for the development of a general and uniform mathematical theory valid for the syntax and semantics of both artificial and natural languages.

  2. 2.

    If \(\delta \) is not a function from \(\textit{Q} \times \varSigma \) to \(\textit{Q}\times \varSigma \times \left\{ \textit{L}, \textit{R}, \square \right\} \) but rather a subset of (\(\textit{Q} \times \varSigma \times \textit{Q} \times \varSigma \times \left\{ \textit{L}, \textit{R}, \square \right\} )\), then \(\mathcal TM \) is non-deterministic. This means that non-deterministic \(\mathcal TM \)s differ from deterministic \(\mathcal TM \)s in allowing for the possibility of assigning different outputs to one input. For simplicity we will consider in our argumentation only deterministic \(\mathcal TM \)s. Note that this does not entail any loss of generality, since all non-deterministic \(\mathcal TM \)s can be simulated by a deterministic \(\mathcal TM \), although it seems that the deterministic \(\mathcal TM \) requires exponentially many steps in n to simulate a computation of n steps by a non-deterministic \(\mathcal TM \) (cfr. Lewis and Papadimitriou 1997, pp. 221–227).

  3. 3.

    The basic idea is that an irreversible computer can always be made reversible by having it save all the information it would otherwise lose on a separate extra tape that is initially blank. As Benett shows, this can be attained “without inordinate increase in machine complexity, number of steps, unwanted output, or temporary storage capacity”. We refer the interested reader to Bennett (1973) for a detailed proof and illustration of this result.

  4. 4.

    Throughout the paper, \(\log \equiv \log _2\).

  5. 5.

    In the context of this section, complexity has to be understood in the sense of Kolmogorov complexity. Given an abstract object, such a general complexity measure is the length, in bits, of the minimal program whose execution in a Universal Turing machine generates a complete description of the object. In the case of codes where the presence of a given signal is governed by a probabilistic process, it can be shown that Kolmogorov complexity equals (up to an additive constant factor) the entropy of the code (Cover and Thomas 1991).

  6. 6.

    Equations of this kind have been obtained in the past through different approaches; cfr. Harremoës and Topsœ (2001) and Ferrer-i-Cancho and Solé (2003).

  7. 7.

    Notice that, if the Turing machine is deterministic, every input generates one and only one output. The problem may arise during the reversion process, if the computations are logically irreversible.

References

  1. Ash, R. B. (1990). Information theory. New York: Dover Publications.

    Google Scholar 

  2. Bennett, C. (1973). Logical reversibility of computation. IBM Journal of Research and Development, 17(6), 525–532.

    Article  Google Scholar 

  3. Bennett, C. (2008). Notes on Landauer’s principle, reversible computation, and Maxwell’s Demon. Studies In History and Philosophy of Science Part B: Studies in History and Philosophy of Modern Physics, 34(3), 501–510.

    Article  Google Scholar 

  4. Bennett, C., & Landauer, R. (1985). The fundamental physical limits of computation. Scientific American, 253(1), 38–46.

    Article  Google Scholar 

  5. Chomsky, N. (2008). On phases. In R. Freidin, C. P. Otero, & M. L. Zubizarreta (Eds.), Foundational issues in linguistic theory: Essays in honor of Jean-Roger Vergnaud. Cambridge, MA: The MIT Press.

    Google Scholar 

  6. Corominas-Murtra, B., Fortuny, J., & Solé, R. V. (2011). Emergence of Zipf’s law in the evolution of communication. Physical Review E, 83(3), 036–115.

    Article  Google Scholar 

  7. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley.

    Book  Google Scholar 

  8. Ferrer-i-Cancho, R., & Solé, R. V. (2003). Least effort and the origins of scaling in human language. Proceedings of the National Academy of Science, 100, 788–791.

    Article  Google Scholar 

  9. Harremoës, P., & Topsœ, F. (2001). Maximum entropy fundamentals. Entropy, 3, 191–226.

    Article  Google Scholar 

  10. Ladyman, J., Presnell, S., Short, A., & Groisman, B. (2007). The connection between logical and thermodynamic irreversibility. Studies In History and Philosophy of Science Part B: Studies In History and Philosophy of Modern Physics, 38(1), 58–79.

    Article  Google Scholar 

  11. Landauer, R. (1961). Irreversibility and heat generation in the computing process. IBM Journal of Research and Development, 5(3), 183–191.

    Article  Google Scholar 

  12. Lewis, H. R., & Papadimitriou, C. H. (1997). Elements of the theory of computation. Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  13. Montague, R. (1974). Formal philosophy: Selected papers of Richard Montague, edited and with an introduction by Richmond Thomason. New Haven: Yale University Press.

  14. Toffoli, T. (1980). Reversible computing. In J. de Bakker & J. van Leeuwen (Eds.), Automata, languages and programming. Berlin: Springer.

    Google Scholar 

  15. Wang, E., & Steels, L. (2008). Self-interested agents can bootstrap symbolic communication if they punish cheaters. In A. D. M. Smith, K. Smith, & R. Ferrer-i-Cancho (Eds.), The evolution of language: Proceedings of the 7th international conference (EVOLANG7). Singapore: World Scientific Publishing.

    Google Scholar 

  16. Wittgenstein, L. (1922/1961). Tractatus logico-philosophicus. London: Routledge.

  17. Zipf, G. (1965). Human behavior and the principle of least effort: An introduction to human ecology. New York: Hafner Publishing.

    Google Scholar 

Download references

Acknowledgments

We would like to thank the members of the Centre de Lingüística Teòrica that attended the course on ambiguity for postgraduate students we taught within the PhD program on cognitive science and language (fall semester, 2010). We are especially grateful to M. Teresa Espinal for many interesting discussions during the elaboration process of this study and to Adriana Fasanella, Carlos Rubio, Francesc-Josep Torres and Ricard Solé for carefully reading a first version of this article and providing us with multiple improvements. We also wish to express our gratitude to two anonymous reviewers for several remarks that helped us to clarify and strengthen our developments.

Author information

Affiliations

Authors

Corresponding author

Correspondence to Jordi Fortuny.

Additional information

This work has been supported by the Secretary for Universities and Research of the Ministry of Economy and Knowledge of the Government of Catalonia and the Cofund programme of the Marie Curie Actions of the 7th R&D Framework Programme of the European Union, the research projects 2009SGR1079, FFI201123356 (JF) and the James S. McDonnell Foundation (BCM).

Appendix: Ambiguity and Physical Irreversibility

Appendix: Ambiguity and Physical Irreversibility

Throughout the paper we have highlighted the strict relation between the logical irreversibility of the computations generating a given code and the ambiguity of the latter. In this appendix we briefly revise the role of logical irreversibility in the foundations of physics, through its relation to thermodynamic irreversibility.

Our objective here is to show, in a rather informal way, that reasonings and concepts that are commonly used in physics can be naturally connected to the conceptual development of this paper. We warn the reader that this appendix does not attempt to provide formal relations between ambiguous codes and its energetic cost or other thermodynamic quantities, nor to relate the emergence of ambiguity to some explicit physical process. The rigorous exploration of the above mentioned topics is a fascinating issue, but lies beyond the scope of this paper.

The strict relation of thermodynamic irreversibility and logical irreversibility is a hot topic of debate since the definition of the equivalence of heat and bits by Landauer (1961). This equivalence, known as Landauer’s principle, states that, for any erased bit of information, a quantity of at least

$$\begin{aligned} k_BT\ln 2 \end{aligned}$$

joules is dissipated in terms of heat, being \(k_B=1.38\times 10^{-23}\,\hbox {J/K}\) the Boltzmann constant and \(T\) the temperature of the system. This principle relates logical irreversibility and thermodynamical irreversibility; however, it is worth noting that it provides only a lower bound and that it is far away from the energetic costs of any real computing process. To see how we can connect both types of irreversibility, we first state that thermodynamical irreversibility is a property of abstract processes—interestingly, almost all processes taking place in our everyday life are irreversible. The common property of such processes is that they generate thermodynamical entropy. The second law of thermodynamics states that any physical process generates a non-negative amount of entropy; i.e., for the process \(\mathbf P \),

$$\begin{aligned} \varDelta S(\mathbf P )\ge 0. \end{aligned}$$

The units of physical entropy are nats instead of bits. Now suppose that we face the problem of reversing the process \(\mathbf P \)—for example, a gas expansion—by which \(\varDelta S(\mathbf P )> 0\). Without further help, the reversion of this process is forbidden by the second law, since it would generate a net amount of negative entropy. Therefore, we will need external energy to reverse the process. Similarly, we have observed that

$$\begin{aligned} H(X_{\varOmega }|X_s)\ge 0, \end{aligned}$$

which means that information cannot be created during an information process. A negative amount of \(H(X_{\varOmega }|X_s)\) would imply, by virtue of Eq. (5), a net creation of information. Therefore, we face the same problem. Indeed, if we have a computational process \(\mathbf C \) by which \(H_\mathbf{C }(X_{\varOmega }|X_s)>0\), the reversion of such a process, with no further external help, would be a process by which the computations would generate information. The reversion, as we have discussed above, is only possible by the external addition of information. Thus the information flux can only be maintained (in the case where all computations are logically reversible) or degraded, and the same applies for the energy flux: by the second law, the energy flux can only be maintained (in the case of thermodynamically reversible processes) or degraded.

We can informally find a quantitative connection between the two entropies. If \(\mathbf Q (\mathbf P )\) is the heat generated during the physical process, its associated physical entropy generation is defined as

$$\begin{aligned} \varDelta S(\mathbf{P })=\frac{\mathbf{Q }(\mathbf{P })}{T}. \end{aligned}$$

In turn, if we consider an ideal—from the energetic viewpoint—computational process \(\mathbf C \), we know, from Landauer’s principle, that

$$\begin{aligned} \mathbf Q (\mathbf C )=k_BT\ln 2\times \mathrm{erased\;bits}. \end{aligned}$$

And we actually know how many bits have been erased –or dissipated. Exactly \(H(X_{\varOmega }|X_s)\) bits. Therefore, the physical entropy generated by this ideal, irreversible computing process will be:

$$\begin{aligned} \varDelta S(\mathbf C )= k_B\ln 2 H_\mathbf{C }(X_{\varOmega }|X_s). \end{aligned}$$

Accordingly, logically irreversible computations are thermodynamically irreversible. Notice that this only proves the implication between both irreversibilities, but is of no practical use, since it is a lower bound. Any real computing process will be such that:

$$\begin{aligned} \varDelta S(\mathbf C )\gg k_B\ln 2 H_\mathbf{C }\left( X_{\varOmega }|X_s\right) . \end{aligned}$$

We finally highlight that one could object that, under the above considerations, the most favorable situation would be the one in which all computations are performed in a logically reversible way, since there would not be an energy penalty. However, this interpretation is misleading. Imagine a coding machine receiving an informational input but working in a totally irreversible way. In this case, there would be a dissipation of information that would undergo into heat production. The energy dissipated in terms of heat, however, would come from the environment, not from the machine, which would be only heated. The creation of a code in this machine to let information be coded and flow would imply, on the contrary, to write a code into the machine, and thus, to erase the initial configuration of the machine—whatever it was, maybe a random one—, to properly adapt it to a consistent coding process. We observe that this process would demand energy that would not come, at least directly, from the environment. Therefore, a perfect coding performing logically reversible computations only would be, in principle, energetically more demanding than a logically irreversible coding.

We insist that the above considerations fall into the abstract level and practical implementations must face multiple additional problems which have been not been taken into consideration. With this short exposition we only want to emphasize the general character of logical irreversibility and ambiguity in natural communication systems. More than an imperfection, ambiguity seems to be, for natural communication systems, a feature as unavoidable as the generation of heat during a thermodynamical process.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Fortuny, J., Corominas-Murtra, B. On the Origin of Ambiguity in Efficient Communication. J of Log Lang and Inf 22, 249–267 (2013). https://doi.org/10.1007/s10849-013-9179-3

Download citation

Keywords

  • Ambiguity
  • Logical (ir)reversibility
  • Communicative efficiency
  • Shannon’s entropy