# Counting distinctions: on the conceptual foundations of Shannon’s information theory

- 156 Downloads
- 16 Citations

## Abstract

Categorical logic has shown that modern logic is essentially the logic of subsets (or “subobjects”). In “subset logic,” predicates are modeled as subsets of a universe and a predicate applies to an individual if the individual is in the subset. Partitions are dual to subsets so there is a dual logic of partitions where a “distinction” [an ordered pair of distinct elements (*u*, *u*′) from the universe *U*] is dual to an “element”. A predicate modeled by a partition *π* on *U* would apply to a distinction if the pair of elements was distinguished by the partition *π*, i.e., if *u* and *u*′ were in different blocks of *π*. Subset logic leads to finite probability theory by taking the (Laplacian) probability as the normalized size of each subset-event of a finite universe. The analogous step in the logic of partitions is to assign to a partition the number of distinctions made by a partition normalized by the total number of ordered |*U*|^{2} pairs from the finite universe. That yields a notion of “logical entropy” for partitions and a “logical information theory.” The logical theory directly counts the (normalized) number of distinctions in a partition while Shannon’s theory gives the average number of binary partitions needed to make those same distinctions. Thus the logical theory is seen as providing a conceptual underpinning for Shannon’s theory based on the logical notion of “distinctions.”

## Keywords

Information theory Logic of partitions Logical entropy Shannon entropy## Preview

Unable to display preview. Download preview PDF.

## References

- Aczél J. and Daróczy Z. (1975). On measures of information and their characterization. Academic Press, New York Google Scholar
- Adelman M.A. (1969). Comment on the H concentration measure as a numbers-equivalent.
*Review of Economics and Statistics*51: 99–101 CrossRefGoogle Scholar - Baclawski, K., & Rota, G.-C. (1979).
*An introduction to probability and random processes*. Unpublished typescript. 467 pages. Download available at: http://www.ellerman.org. - Bhargava T.N. and Uppuluri V.R.R. (1975). On an axiomatic derivation of gini diversity, with applications.
*Metron*33: 41–53 Google Scholar - Birkhoff G. (1948). Lattice theory. American Mathematical Society, New York Google Scholar
- Cover T. and Thomas J. (1991). Elements of information theory. John Wiley, New York CrossRefGoogle Scholar
- Finberg D., Mainetti M. and Rota G.C. (1996). The logic of commuting equivalence relations. In: Aldo, U. and Agliano, P. (eds) Logic and algebra, pp 69–96. Marcel Dekker, New York Google Scholar
- Friedman W.F. (1922). The index of coincidence and its applications in cryptography. Riverbank Laboratories, Geneva, IL Google Scholar
- Ganeshaiah K.N., Chandrashekara K. and Kumar A.R.V. (1997). Avalanche Index: A new measure of biodiversity based on biological heterogeneity of communities.
*Current Science*73: 128–133 Google Scholar - Gini C. (1912). Variabilitá e mutabilitá. Tipografia di Paolo Cuppini, Bologna Google Scholar
- Gini C. (1955). Variabilitá e mutabilitá. In: Pizetti, E. and Salvemini, T. (eds) Memorie di metodologica statistica, pp. Libreria Eredi Virgilio Veschi, Rome Google Scholar
- Good I.J. (1979). A. M. Turing’s statistical work in World War II.
*Biometrika*66(2): 393–396 CrossRefGoogle Scholar - Good I.J. (1982). Comment (on Patil and Taillie: Diversity as a concept and its measurement).
*Journal of the American Statistical Association*77(379): 561–563 CrossRefGoogle Scholar - Gray R.M. (1990). Entropy and information theory. Springer-Verlag, New York Google Scholar
- Hartley R.V.L. (1928). Transmission of information.
*Bell System Technical Journal*7(3): 535–563 Google Scholar - Havrda J.H. and Charvat F. (1967). Quantification methods of classification processes: Concept of structural α-entropy.
*Kybernetika (Prague)*3: 30–35 Google Scholar - Herfindahl O.C. (1950).
*Concentration in the U.S. Steel Industry*. Unpublished doctoral dissertation, Columbia UniversityGoogle Scholar - Hirschman A.O. (1945). National power and the structure of foreign trade. University of California Press, Berkeley Google Scholar
- Hirschman A.O. (1964). The Paternity of an Index.
*American Economic Review*54(5): 761–762 Google Scholar - Kapur J.N. (1994). Measures of information and their applications. Wiley Eastern, New Delhi Google Scholar
- Kolmogorov A.N. (1956). Foundations of the theory of probability. Chelsea, New York Google Scholar
- Kullback S. (1976). Statistical methods in cryptanalysis. Aegean Park Press, Walnut Creek CA Google Scholar
- Lawvere F.W. and Rosebrugh R. (2003). Sets for mathematics. Cambridge University Press, Cambridge Google Scholar
- MacArthur R.H. (1965). Patterns of species diversity.
*Biology Review*40: 510–533 CrossRefGoogle Scholar - Patil G.P. and Taillie C. (1982). Diversity as a concept and its measurement.
*Journal of the American Statistical Association*77(379): 548–561 CrossRefGoogle Scholar - Rao C.R. (1982). Diversity and dissimilarity coefficients: A unified approach.
*Theoretical Population Biology*21: 24–43 CrossRefGoogle Scholar - Rejewski M. (1981). How polish mathematicians deciphered the enigma.
*Annals of the History of Computing*3: 213–234 CrossRefGoogle Scholar - Rényi A. (1965). On the theory of random search.
*Bulletin of American Mathematical Society*71: 809–828 CrossRefGoogle Scholar - Rényi A. (1970). Probability theory (trans: Vekerdi, L.). North-Holland, Amsterdam Google Scholar
- Rényi, A. (1976). In P. Turan (Ed.),
*Selected Papers of Alfréd Rényi, Volumes 1,2, and 3*. Budapest: Akademiai Kiado.Google Scholar - Ricotta C. and Szeidl L. (2006). Towards a unifying approach to diversity measures: Bridging the gap between the Shannon entropy and Rao’s quadratic index.
*Theoretical Population Biology*70: 237–243 CrossRefGoogle Scholar - Shannon, C. E. (1948). A mathematical theory of communication.
*Bell System Technical Journal, 27*, 379–423; 623–656.Google Scholar - Simpson E.H. (1949). Measurement of diversity.
*Nature*163: 688 CrossRefGoogle Scholar - Stigler S.M. (1999). Statistics on the table. Harvard University Press, Cambridge Google Scholar
- Tsallis C. (1988). Possible generalization for Boltzmann-Gibbs statistics.
*Journal of Statistical Physics*52: 479–487 CrossRefGoogle Scholar - Vajda I. (1969). A contribution to informational analysis of patterns. In: Watanabe, S. (eds) Methodologies of pattern recognition, pp 509–519. Academic Press, New York Google Scholar