Shannon arrived at the revolutionary idea of digital representation by sampling a source of information at an appropriate rate, and converting the samples to a bit stream. He characterized the source by a single number, which he called entropy (a term adapted from statistical mechanics), to quantify the information content of the source. For a stream of text, Shannon viewed entropy as a statistical parameter that measured how much information is produced on average by each letter. He also created coding theory, by introducing redundancy into the digital representation to protect against corruption. If today you take a compact disc in one hand and a pair of scissors in the other, then score the disc along a radius from the centre to the edge, you will find that the disc still plays as new.
Before Shannon, it was commonly believed that the only way of achieving arbitrarily small probability of error in a communication channel, such as a telephone line, was to reduce the transmission rate to zero. All this changed in 1948 with the publication of A Mathematical Theory of Communication, in which Shannon characterized a channel by a single parameter, the channel capacity, and showed that it was possible to transmit encoded information at any rate below capacity with an arbitrarily small probability of error. His method of proof was to show the existence of a single good code for data transmission by averaging over all possible codes. The 1948 paper established fundamental limits on the efficiency of communication through noisy channels, but did not produce an explicit example of a good code that would achieve the highest capacity. It has taken 50 years for coding theorists to discover families of codes that come close to these fundamental limits for telephone lines.
This is a preview of subscription content, access via your institution
