Mathematical Statistics and Information Theory

There are many easily found good books on probability theory and mathematical statistics (eg [84, 85, 87, 117, 120, 122, 196]), stochastic processes (eg [31, 161]) and information theory (eg [175, 176]); here we just outline some topics to help make the sequel more self contained. For those who have access to the computer algebra package Mathematica [215], the approach to mathematical statistics and accompanying software in Rose and Smith [177] will be particularly helpful.

The word stochastic comes from the Greek stochastikos, meaning skillful in aiming and stochazesthaito aim at or guess at, and stochosmeans target or aim. In our context, stochastic colloquially means involving chance variations around some event-rather like the variation in positions of strikes aimed at a target. In its turn, the later word statistics comes through eighteenth century German from the Latin root statusmeaning state; originally it meant the study of political facts and figures. The noun random was used in the sixteenth century to mean a haphazard course, from the Germanic randir to run, and as an adjective to mean without a definite aim, rule or method, the opposite of purposive. From the middle of the last century, the concept of a random variable has been used to describe a variable that is a function of the result of a well-defined statistical experiment in which each possible outcome has a definite probability of occurrence. The organization of probabilities of outcomes is achieved by means of a probability function for discrete random variables and by means of a probability density function for continuous random variables. The result of throwing two fair dice and summing what they show is a discrete random variable.