Theory of Information and its Value pp 35-51 | Cite as

# Encoding of discrete information in the absence of noise and penalties

## Abstract

The definition of the amount of information, given in Chapter 1, is justified when we deal with a transformation of information from one kind into another, i.e. when considering encoding of information. It is essential that the law of *conservation* of information amount holds under such a transformation. It is very useful to draw an analogy with the law of conservation of energy. The latter is the main argument for introducing the notion of energy. Of course, the law of conservation of information is more complex than the law of conservation of energy in two respects. The law of conservation of energy establishes an exact *equality* of energies, when one type of energy is transformed into another. However, in transforming information we have a more complex relation, namely ‘not greater’ (\(\leqslant \)), i.e. the amount of information cannot increase. The equality sign corresponds to *optimal* encoding. Thus, when formulating the law of conservation of information, we have to point out that there *possibly* exists such an encoding, for which the equality of the amounts of information occurs.

## References

- 9.Fano, R.M.: Transmission of Information: A Statistical Theory of Communications, 1st edn. MIT Press, Cambridge (1961)Google Scholar
- 10.Fano, R.M.: Transmission of Information: A Statistical Theory of Communications (Translation to Russian). Mir, Moscow (1965)Google Scholar
- 11.Feinstein, A.: Foundations of Information Theory. McGraw-Hill, New York (1958)CrossRefGoogle Scholar
- 12.Feinstein, A.: Foundations of Information Theory (Translation to Russian). Inostrannaya Literatura, Moscow (1960)Google Scholar
- 23.Huffman, D.A.: A method for the construction of minimum redundancy codes. Proc. IRE
**40**(9), 1098–1101 (1952)CrossRefGoogle Scholar - 28.Kraft, L.G.: A device for quantizing, grouping, and coding amplitude-modulated pulses. Master’s Thesis, Massachusetts Institute of Technology, Dept. of Electrical Engineering (1949)Google Scholar