11 KiB
| title | chunk | source | category | tags | date_saved | instance |
|---|---|---|---|---|---|---|
| Information theory | 6/7 | https://en.wikipedia.org/wiki/Information_theory | reference | science, encyclopedia | 2026-05-05T03:56:37.735412+00:00 | kb-cron |
→
Message
W
Encoder
f
n
→
E
n
c
o
d
e
d
s
e
q
u
e
n
c
e
X
n
Channel
p
(
y
|
x
)
→
R
e
c
e
i
v
e
d
s
e
q
u
e
n
c
e
Y
n
Decoder
g
n
→
E
s
t
i
m
a
t
e
d
m
e
s
s
a
g
e
W
^
{\displaystyle {\xrightarrow[{\text{Message}}]{W}}{\begin{array}{|c| }\hline {\text{Encoder}}\\f_{n}\\\hline \end{array}}{\xrightarrow[{\mathrm {Encoded \atop sequence} }]{X^{n}}}{\begin{array}{|c| }\hline {\text{Channel}}\\p(y|x)\\\hline \end{array}}{\xrightarrow[{\mathrm {Received \atop sequence} }]{Y^{n}}}{\begin{array}{|c| }\hline {\text{Decoder}}\\g_{n}\\\hline \end{array}}{\xrightarrow[{\mathrm {Estimated \atop message} }]{\hat {W}}}}
Here
X
{\displaystyle X}
represents the space of messages transmitted, and
Y
{\textstyle Y}
the space of messages received during a unit time over our channel. Let p(y|x) be the conditional probability distribution function of
Y
{\textstyle Y}
given
X
{\displaystyle X}
. We will consider p(y|x) to be an inherent fixed property of our communications channel (representing the nature of the noise of our channel). Then the joint distribution of
X
{\displaystyle X}
and
Y
{\textstyle Y}
is completely determined by our channel and by our choice of f(x), the marginal distribution of messages we choose to send over the channel. Under these constraints, we would like to maximize the rate of information, or the signal, we can communicate over the channel. The appropriate measure for this is the mutual information, and this maximum mutual information is called the channel capacity and is given by:
C
=
max
f
I
(
X
;
Y
)
.
{\displaystyle C=\max _{f}I(X;Y).\!}
This capacity has the following property related to communicating at information rate R (where R is usually bits per symbol). For any information rate R < C and coding error ε > 0, for large enough N, there exists a code of length N and rate ≥ R and a decoding algorithm, such that the maximal probability of block error is ≤ ε; that is, it is always possible to transmit with arbitrarily small block error. In addition, for any rate R > C, it is impossible to transmit with arbitrarily small block error. Channel coding is concerned with finding such nearly optimal codes that can be used to transmit data over a noisy channel with a small coding error at a rate near the channel capacity.
==== Capacity of particular channel models ==== A continuous-time analog communications channel subject to Gaussian noise—see Shannon–Hartley theorem. A binary symmetric channel (BSC) with crossover probability p is a binary input, binary output channel that flips the input bit with probability p. The BSC has a capacity of 1 − Hb(p) bits per channel use, where Hb is the binary entropy function to the base-2 logarithm:
A binary erasure channel (BEC) with erasure probability p is a binary input, ternary output channel. The possible channel outputs are 0, 1, and a third symbol 'e' called an erasure. The erasure represents complete loss of information about an input bit. The capacity of the BEC is 1 − p bits per channel use.
==== Channels with memory and directed information ==== In practice many channels have memory. Namely, at time
i
{\displaystyle i}
the channel is given by the conditional probability
P
(
y
i
|
x
i
,
x
i
−
1
,
x
i
−
2
,
.
.
.
,
x
1
,
y
i
−
1
,
y
i
−
2
,
.
.
.
,
y
1
)
{\displaystyle P(y_{i}|x_{i},x_{i-1},x_{i-2},...,x_{1},y_{i-1},y_{i-2},...,y_{1})}
. It is often more comfortable to use the notation
x
i
=
(
x
i
,
x
i
−
1
,
x
i
−
2
,
.
.
.
,
x
1
)
{\displaystyle x^{i}=(x_{i},x_{i-1},x_{i-2},...,x_{1})}
and the channel become
P
(
y
i
|
x
i
,
y
i
−
1
)
{\displaystyle P(y_{i}|x^{i},y^{i-1})}
. In such a case the capacity is given by the mutual information rate when there is no feedback available and the Directed information rate in the case that either there is feedback or not (if there is no feedback the directed information equals the mutual information).
=== Fungible information === Fungible information is the information for which the means of encoding is not important. Classical information theorists and computer scientists are mainly concerned with information of this sort. It is sometimes referred as speakable information.
== Applications to other fields ==
=== Network physiology === Information theory concepts, methods and approaches have broad applications in network physiology, a field which provides a quantitative framework, based on adaptive networks of dynamical systems, to investigate how physiological systems exchange, process, and integrate information as a network to (i) coordinate their functions across levels and scales (from sub-cellular to organs and organism level) and (ii) generate distinct physiological states in health and disease. Through measures such as mutual information, transfer entropy, and co-information, information theory enables the detection of coupling strength, directionality, synergy/redundancy and higher-order interactions among physiological systems and sub-systems, revealing how network cross-communication and regulation occur within the organism. Applications of information-theoretic approaches span from analyzing information transfer between brain and body networks during various states; cardio-respiratory interactions; cardio-muscular interactions; cortico-muscular interactions; brain wave interactions and brain functional networks; network physiology in extreme environments.
=== Intelligence uses and secrecy applications ===