---
title: "History of information theory"
chunk: 1/2
source: "https://en.wikipedia.org/wiki/History_of_information_theory"
category: "reference"
tags: "science, encyclopedia"
date_saved: "2026-05-05T03:59:49.990118+00:00"
instance: "kb-cron"
---

The study of information theory began with the publication of Claude E. Shannon's classic paper "A Mathematical Theory of Communication" in the Bell System Technical Journal in July and October 1948, although Shannon had substantially completed the paper at Bell Labs by the end of 1944, 
Shannon introduced the qualitative and quantitative model of communication as a statistical process, opening with the assertion that 

"The fundamental problem of communication is that of reproducing at one point, either exactly or approximately, a message selected at another point."
With it came the ideas of 

the information entropy and redundancy of a source, and its relevance through the source coding theorem;
the mutual information, and the channel capacity of a noisy channel, including the promise of perfect loss-free communication given by the noisy-channel coding theorem;
the practical result of the Shannon–Hartley law for the channel capacity of a Gaussian channel; and of course
the bit - a new way of seeing the most fundamental unit of information.

== Before 1948 ==

=== Early telecommunications ===
Some of the oldest methods of telecommunications implicitly used some of the ideas that were later formalized by Shannon in the development of information theory. In telegraphy, starting in the 1830s, Morse code considered the frequency of more common letters. Thus the letter "E", which is expressed as one "dot", is transmitted more quickly than less common letters like "J", which is expressed by one "dot" followed by three "dashes". The idea of encoding information in this manner is the cornerstone of lossless data compression. A hundred years later, frequency modulation illustrated that bandwidth can be considered merely another degree of freedom. The vocoder, now largely looked at as an audio engineering curiosity, was originally designed in 1939 to use less bandwidth than that of an original message, in much the same way that mobile phones now trade off voice quality with bandwidth.

=== Quantitative ideas of information ===
The most direct antecedents of Shannon's work were two papers published in the 1920s by Harry Nyquist and Ralph Hartley, who were both still research leaders at Bell Labs when Shannon arrived in the early 1940s.
Nyquist's 1924 paper, "Certain Factors Affecting Telegraph Speed", is mostly concerned with some detailed engineering aspects of telegraph signals. But a more theoretical section discusses quantifying "intelligence" and the "line speed" at which it can be transmitted by a communication system, giving the relation

  
    
      
        W
        =
        K
        log
        ⁡
        m
        
      
    
    {\displaystyle W=K\log m\,}
  

where W is the speed of transmission of intelligence, m is the number of different voltage levels to choose from at each time step, and K is a constant.
Hartley's 1928 paper, called simply "Transmission of Information", went further by using the word information (in a technical sense), and making explicitly clear that information in this context was a measurable quantity, reflecting only the receiver's ability to distinguish that one sequence of symbols had been intended by the sender rather than any other—quite regardless of any associated meaning or other psychological or semantic aspect the symbols might represent. This amount of information he quantified as

  
    
      
        H
        =
        log
        ⁡
        
          S
          
            n
          
        
        
      
    
    {\displaystyle H=\log S^{n}\,}
  

where S was the number of possible symbols, and n the number of symbols in a transmission. The natural unit of information was therefore the decimal digit, much later renamed the hartley in his honour as a unit or scale or measure of information. The Hartley information, H0, is still used as a quantity for the logarithm of the total number of possibilities.
A similar unit of log10 probability, the ban, and its derived unit the deciban (one tenth of a ban), were introduced by Alan Turing in 1940 as part of the statistical analysis of the breaking of the German second world war Enigma cyphers. The decibannage represented the reduction in (the logarithm of) the total number of possibilities (similar to the change in the Hartley information); and also the log-likelihood ratio (or change in the weight of evidence) that could be inferred for one hypothesis over another from a set of observations. The expected change in the weight of evidence is equivalent to what was later called the Kullback discrimination information.
But underlying this notion was still the idea of equal a-priori probabilities, rather than the information content of events of unequal probability; nor yet any underlying picture of questions regarding the communication of such varied outcomes.
In a 1939 letter to Vannevar Bush, Shannon had already outlined some of his initial ideas of information theory.

=== Entropy in statistical mechanics ===
One area where unequal probabilities were indeed well known was statistical mechanics, where Ludwig Boltzmann had, in the context of his H-theorem of 1872, first introduced the quantity

  
    
      
        H
        =
        −
        ∑
        
          f
          
            i
          
        
        log
        ⁡
        
          f
          
            i
          
        
      
    
    {\displaystyle H=-\sum f_{i}\log f_{i}}
  

as a measure of the breadth of the spread of states available to a single particle in a gas of like particles, where f represented the relative frequency distribution of each possible state. Boltzmann argued mathematically that the effect of collisions between the particles would cause the H-function to inevitably increase from any initial configuration until equilibrium was reached; and further identified it as an underlying microscopic rationale for the macroscopic thermodynamic entropy of Clausius.
Boltzmann's definition was soon reworked by the American mathematical physicist J. Willard Gibbs into a general formula for statistical-mechanical entropy, no longer requiring identical and non-interacting particles, but instead based on the probability distribution pi for the complete microstate i of the total system:

  
    
      
        S
        =
        −
        
          k
          
            B
          
        
        ∑
        
          p
          
            i
          
        
        ln
        ⁡
        
          p
          
            i
          
        
        
      
    
    {\displaystyle S=-k_{\text{B}}\sum p_{i}\ln p_{i}\,}
  