The basic concepts of coarse-graining protein structures led to the introduction of empirical statistical potentials in protein computations. We review the history of the development of statistical contact potentials in computational biology and discuss the common features and differences between various pair contact potentials. Potentials derived from the statistics of non-bonded contacts in protein structures from the Protein Data Bank (PDB) are compared with potentials developed for threading purposes based on the optimization of the selection of the native structures among decoys. The energy of transfer of amino acids from water to a protein environment is discussed in detail. We suggest that a next generation of statistical contact potentials should include the effects of residue packing in proteins to improve predictions of protein native three-dimensional structures. We review existing multi-body potentials that have been proposed in the literature, including our own recent four-body potentials. We show how these are related to amino acid substitution matrices.