Information Theory

Abstract

Figure 7.1 presents a generic communication system. In this figure, the message is composed at the source. Then, it undergoes some coding to become suitable for transmission through the channel. Most channels are noisy and this is signified by the noise that affects the channel from the block on top. At the receiving end of the channel, a decoder must decode the encoded and noisy message into one that the recipient will understand. This is the basis for the development of the topic of Information Theory which started with the advent of the telegraph and telephone systems. Fisher [7], Nyquist [14, 15], Hartley [9], Shannon [18], Wiener [22], and Kullback [12] were among some of the early developers of Information Theory. A lot of this work was developed in response to the encryption and decryption needs for sensitive communications during the second world war.