  ELEC3203 · Digital Coding & Transmission
  Information Theory Fundamentals

  Information Theory · Lecture notes
  Prof. Celik

ELEC3203 · Digital Coding and Transmission

Lecture 5

Channels and Mutual Information

Lectures 1–4 asked how efficiently a source's information can be coded. This lecture asks what happens to that information once it crosses a noisy channel: how a channel is modelled, how input and output symbols relate through the binary symmetric channel, and how mutual information measures what actually gets through.

Learning Objectives

Represent a noisy channel with a transition probability matrix
Compute mutual information between channel input and output for a given source
Explain how mutual information quantifies information conveyed through a noisy channel

Slides Quiz

Podcast

Channels and Mutual Information cover art

    InstructorProf. Abdulkadir Çelik https://akadircelik.com
    ModuleELEC3203
    TopicsChannel models · AWGN · binary symmetric channel · mutual information
  

    ▸ Channel models
    ▸ Binary symmetric channel
    ▸ Mutual information
    ▸ Information loss
  

Lecture 5

This page grows as further lectures are added to the document.

Notation used in this lecture

\(X_i\)

a channel input symbol (what is transmitted)

\(Y_j\)

a channel output symbol (what is received)

\(P(Y_j\mid X_i)\)

conditional probability of receiving $Y_j$ given $X_i$ was sent

\(p_e\)

channel error (crossover) probability of a binary symmetric channel

\(E_b\)

energy per transmitted bit, in joules (BPSK sends $\pm\sqrt{E_b}$)

\(N_0\)

one-sided noise power spectral density; AWGN has two-sided PSD $N_0/2$

\(Q(z)\)

Gaussian tail probability, $P(\mathcal{N}(0,1)>z)$; gives $p_e=Q\big(\sqrt{2E_b/N_0}\big)$

\(I(X_i,Y_j)\)

mutual information between one input/output symbol pair, in bits

\(I(X,Y)\)

average mutual information, in bits/symbol

\(H(X)\)

source entropy (average information sent)

\(H(X\mid Y)\)

equivocation: average information about $X$ lost in the channel

\(H(Y)\), \(H(Y\mid X)\)

destination entropy, and error entropy (uncertainty in $Y$ caused by noise)

5.1 Modelling the channel: AWGN and channel characteristics

An ideal path, corrupted only by noise

Every real channel can distort amplitude and phase, may vary with time, and has finite bandwidth $B$. On top of that, error-free reception is impeded by additive white Gaussian noise (AWGN), whose severity is captured by the signal-to-noise ratio (SNR). Our aim in this lecture and the next is to find the maximum rate at which information can cross such a channel — the channel capacity — but first we need a working model of the channel itself.

Figure 5.1. The channel model used throughout. The transmitted signal $x(t)$ passes through the channel $h(t)$; additive white Gaussian noise $n(t)$ enters at the receiver input, giving $y(t)=h(t)*x(t)+n(t)$. In the ideal case $h(t)$ is a pure delay, so the noise is the only impairment.

5.1.1 The channel model and the ideal channel

A channel acts on the transmitted signal $x(t)$ in two ways: it filters the signal through its impulse response $h(t)$, and it adds noise $n(t)$. The received signal is

$$ y(t) = h(t) * x(t) + n(t). $$

An ideal channel introduces no amplitude or phase distortion — its impulse response is a pure delay, $h(t)=\delta(t-T)$, giving flat magnitude response and linear phase (constant group delay).

The delay is the propagation time from transmitter to receiver, $T=d/c$ for a distance $d$ and propagation speed $c$, plus any fixed delay in the equipment. Strictly, a distortion-free channel is $h(t)=K\,\delta(t-T)$: the gain $K<1$ is the attenuation due to path loss (Section 5.1.2), which scales the signal without changing its shape. We take $K=1$ and account for the attenuation through the received SNR.

Ideal channel

Flat amplitude response up to bandwidth $B$

Linear phase — constant group delay $T$

Figure 5.2. An ideal channel is a pure delay, $h(t)=\delta(t-T)$, giving a frequency response $H(j\omega)=A(\omega)e^{j\Phi(\omega)}$ with flat amplitude $A(\omega)=1$ and linear phase $\Phi(\omega)=-\omega T$; its only impairment is AWGN.

Real channels depart from this ideal through $h(t)$, described in Section 5.1.2, and always add $n(t)$, modelled as AWGN in Section 5.1.3.

5.1.2 What a real channel does to the signal: path loss, shadowing, and fading

In a real wireless link the received signal power varies with distance and surroundings in three superimposed ways, each acting on a different physical scale (Figure 5.3). They fall into two groups according to what they do to $h(t)$.

Large-scale effects scale the signal. With path loss, average received power falls off smoothly and predictably with distance $d$, roughly as $P_r/P_t \propto d^{-\gamma}$, where $\gamma$ is the path-loss exponent. In decibels,

$$ 10\log_{10}\frac{P_r}{P_t} = \text{const} - 10\gamma\log_{10} d , $$

so received power decays linearly against $\log_{10}d$ with slope $-10\gamma$ dB per decade: each tenfold increase in distance costs $10\gamma$ dB. The exponent sets the decay rate. In free space $\gamma=2$, from the Friis equation $P_r/P_t=G_tG_r(\lambda/4\pi d)^2$: the transmitted power spreads over a sphere of area $4\pi d^2$, costing 20 dB per decade. Reflection, diffraction and obstruction raise $\gamma$; guided propagation along corridors can lower it below 2. Typical measured values are:

Environment	Path-loss exponent $\gamma$
Free space	2
Urban cellular	2.7 to 3.5
Shadowed urban cellular	3 to 5
In building, line of sight	1.6 to 1.8
In building, obstructed	4 to 6
Factory, obstructed	2 to 3

With shadowing, buildings, terrain, and foliage attenuate the signal by different amounts at different locations, superimposing a slower, log-normally distributed random variation on the path-loss trend. Both multiply $h(t)$ by a gain and leave its shape unchanged, so the channel stays ideal; what they set is the average received SNR.

Received power vs. distance

Figure 5.3. Received power vs. log-distance: the smooth path-loss trend (solid), with a slower log-normal shadowing ripple superimposed (dashed), and rapid small-scale multipath fading superimposed on top of that (thin).

Small-scale multipath can also change the shape. Copies of the transmitted signal, reflected and scattered along different paths, arrive with different delays $\tau_l$ and amplitudes $a_l$ and interfere, constructively or destructively, causing rapid power fluctuations over a fraction of a wavelength (Figure 5.4). The impulse response becomes a sum of delayed copies,

$$ h(t)=\sum_l a_l\,\delta(t-\tau_l), $$

and the spread of those delays, $\tau_{\max}$, compared with the symbol period $T_s$, decides what kind of channel results:

Flat fading, $\tau_{\max}\ll T_s$

All copies of a symbol arrive within that symbol. Together they act as a single random gain, so the channel stays memoryless and non-dispersive.

Frequency-selective fading, $\tau_{\max}\gtrsim T_s$

Copies of one symbol spill into the next. The channel is dispersive (it has memory), its frequency response is no longer flat, and it causes intersymbol interference.

Multipath propagation

direct path (as sent)echo via Aecho via Breceived sumoverlap into next symbol

Figure 5.4. Multipath from scene to symbols, shown from above. Top left: the signal reaches the receiver directly, by reflection off building A between transmitter and receiver, and by reflection off building B a distance $d$ behind the receiver; B's extra path is roughly $2d$. Top right: the power-delay profile. The delay spread $\tau_{\max}$ is the time between the first and last arrival; the blue band is one symbol period $T_s$. Bottom left: four BPSK symbols via the direct path (grey, matching the direct arrival in the delay profile), the echoes via A (blue, dashed) and B (purple, dashed), and their sum at the receiver (amber); red shading marks where echoes of one symbol overlap the next. Bottom right: the channel's magnitude response $|H(f)|=|1+a_1e^{-j2\pi f\tau_1}+a_2e^{-j2\pi f\tau_2}|$ over the signal band, roughly $1/T_s$. Move building B, or press Play: while every replica lands inside the symbol, $|H(f)|$ stays close to flat across the band; once B's echo arrives after the symbol has ended, it falls into the next symbol and notches enter the band, so the channel becomes frequency-selective.

Effect	What it does to $h(t)$	Channel
Path loss	deterministic gain, falls with distance	ideal, lower SNR
Shadowing	slow random gain (log-normal)	ideal, lower SNR
Flat fading	fast random gain	ideal, memoryless
Frequency-selective fading	spreads each symbol in time	dispersive, ISI

Only frequency-selective fading makes the channel non-ideal. We set that complication aside and assume a non-dispersive channel, with any gain from path loss, shadowing, or flat fading absorbed into the received SNR. What remains is the ideal channel plus AWGN.

5.1.3 Additive white Gaussian noise

By the time the signal reaches the receiver, path loss and fading have made it very weak, so it must be amplified to a usable level before its digital content can be detected. The amplifier itself injects thermal noise, quantified by its noise figure, and because the signal is weakest at this point, this noise dominates every other source. It is what the noise term $n(t)$ in our model represents, and it is why the noise in Figure 5.1 enters at the receiver input.

Thermal noise is well described as additive white Gaussian noise (AWGN): zero-mean, Gaussian-distributed noise with a flat ("white") power spectral density $N_0/2$ over the channel bandwidth $B$, giving total noise power $N_P = N_0 B$.

Additive white Gaussian noise (AWGN)

Figure 5.5. AWGN is uncorrelated with the signal and Gaussian-distributed: $$ p(x) = \frac{1}{\sqrt{2\pi}\,\sigma}\,e^{-(x-\mu)^2/2\sigma^2}, $$ zero-mean ($\mu=0$) in the "white" case, with variance $\sigma^2$ equal to the noise power.

White noise: PSD & autocorrelation

Flat power spectral density — equal noise power at every frequency

Impulse-shaped autocorrelation — uncorrelated with itself at any nonzero lag

Figure 5.6. Power spectral density $N(\omega)$ and autocorrelation $R(\tau)$ are a Fourier pair, $R(\tau)=\frac{1}{2\pi}\int_{-\infty}^{\infty} N(\omega)e^{j\omega\tau}d\omega$: a flat spectrum in frequency corresponds to a spike in time, which is exactly why white noise samples are uncorrelated from one instant to the next. A two-sided spectrum of height $N_0/2$ is used by convention, with $N_0$ the noise power density.

5.1.4 From AWGN to the BSC: where the crossover probability comes from

The AWGN channel of Section 5.1.3 is continuous: send a voltage, receive that voltage plus a real-valued noise sample. The binary symmetric channel of Section 5.2 is discrete: send a bit, receive a bit that is occasionally flipped. These are not two rival models — the second is what the first becomes once you wrap a modulator and a detector around it. Understanding that step is what makes $p_e$ a physical quantity rather than an arbitrary parameter.

With antipodal (BPSK) signalling the transmitter sends $+\sqrt{E_b}$ for a "1" and $-\sqrt{E_b}$ for a "0", where $E_b$ is the energy per bit. The detector receives $y = \pm\sqrt{E_b} + n$, with $n$ a zero-mean Gaussian sample, and makes a hard decision: declare "1" if $y>0$ and "0" otherwise. An error occurs precisely when the noise is large enough and of the right sign to carry $y$ across the threshold, which happens with probability

$$ p_e = Q\!\left(\sqrt{\tfrac{2E_b}{N_0}}\right), \qquad Q(z) = \frac{1}{\sqrt{2\pi}}\int_z^{\infty} e^{-u^2/2}\,du. $$

(5.1)

The hard decision

Declare "1" if $y>0$, else "0". The continuous received voltage collapses to one bit — and all the fine detail of the noise is discarded with it.

The crossover probability

$p_e = Q\big(\sqrt{2E_b/N_0}\big)$ — set entirely by the signal-to-noise ratio per bit, so better SNR means a cleaner BSC.

Why it is symmetric

The two conditional densities are mirror images about the threshold, so both error events have identical probability — one $p_e$ suffices.

sent
received

Figure 5.7. The BSC is not a separate channel model: it is the AWGN channel plus a modulator and a hard-decision detector. Collapsing the continuous received voltage $y$ to a single bit is what turns a noise density into a crossover probability. Send bits through the chain to follow one bit at each stage (voltages normalised to $\pm 1$); a bit is flipped whenever the noise sample pushes $y$ across zero. Over many bits the measured error rate approaches $p_e = Q\big(\sqrt{2E_b/N_0}\big)$. The upper plot shows where that probability comes from. The two shaded tails are the bit errors: the part of $p(y\mid\text{"0"})$ lying above the threshold, and the part of $p(y\mid\text{"1"})$ lying below it. Because the two densities are mirror images about the threshold, the tails have equal area — which is exactly why the resulting discrete channel is symmetric. Raise $E_b/N_0$ with the slider to pull the densities apart and watch the tails, and $p_e$, shrink; the bit counts restart at each new value.

So the chain AWGN + BPSK + hard decision is a binary symmetric channel, and $p_e$ inherits its value from the physics of Section 5.1.3. This also shows what the BSC throws away: by keeping only the sign of $y$, the detector discards how confident each decision was. Retaining that confidence — soft-decision decoding — is worth roughly 2 dB in practice, which is why the modern codes mentioned in Lecture 6 all use it. With that link established, we can study the discrete channel on its own terms.

5.2 The binary symmetric channel (BSC)

The discrete channel that AWGN becomes

Section 5.1.4 built the binary symmetric channel from the ground up: an ideal AWGN path (no amplitude or phase distortion; the only impairment is additive noise) with a BPSK modulator and a hard-decision detector wrapped around it. It is the simplest model of information transmission over a discrete channel, carrying one of two input symbols $X_0,X_1$ and producing one of two output symbols $Y_0,Y_1$. "Symmetric" means the two ways of getting it wrong are equally likely, $P(Y_1\mid X_0)=P(Y_0\mid X_1)=p_e$, the channel's error (crossover) probability.

Figure 5.8. The binary symmetric channel. Each input $X_i$ occurs with its a priori probability $P(X_i)$. Each arrow is labelled with its transition probability $P(Y_j\mid X_i)$: solid lines are correct reception ($1-p_e$); dashed lines are the two symmetric error events ($p_e$ each).

Two sets of probabilities describe the channel. $P(X_i)$ is the probability that the source emits $X_i$, the a priori probability. $P(Y_j\mid X_i)$ is the conditional probability of receiving $Y_j$ given that $X_i$ was transmitted.

Any joint event "$X_i$ sent and $Y_j$ received" has a joint probability linked to the conditionals by Bayes' rule:

$$ P(Y_j,X_i) = P(X_i)\,P(Y_j\mid X_i) = P(Y_j)\,P(X_i\mid Y_j) = P(X_i,Y_j). $$

(5.2)

EXAMPLE 5.1 · Joint and output probabilities of a BSC

A BSC has a non-equiprobable source, $P(X_0)=0.3$, $P(X_1)=0.7$ (70% of transmitted bits are "1"), and error probability $p_e=0.02$ (a 2% bit error rate). Find the four joint probabilities, the probabilities of correct and erroneous reception, and the output probabilities $P(Y_0)$ and $P(Y_1)$.

	received Y₀	received Y₁	P(X_i)
sent X₀	0.294	0.006	0.300
sent X₁	0.014	0.686	0.700
P(Y_j)	0.308	0.692	1.000

5.3 Mutual information between a symbol pair

What does observing $Y_j$ tell you about $X_i$?

Because $Y_j$ and $X_i$ are linked through the channel, they share information; that sharing is quantified by the mutual information of the pair:

$$ I(X_i,Y_j) = \log_2\frac{P(X_i\mid Y_j)}{P(X_i)} \quad \text{(bits)}. $$

(5.5)

The ratio inside the logarithm compares what you believe about $X_i$ after receiving $Y_j$ with what you believed before. Figure 5.9 lets you watch that ratio change with the source and channel.

Source P(X₁) = Channel p_e =

You receive

before: P(Xi)after: P(Xi|Y1)

X0

X1

Try

Figure 5.9. Mutual information as a change of belief. The dashed marker is what you believe about each input before anything arrives, $P(X_i)$; the bar is what you believe after receiving $Y_j$, $P(X_i\mid Y_j)$. Equation (5.5) is the logarithm of the ratio between them: positive when the observation makes $X_i$ more likely, zero when it changes nothing, and negative when it points away from $X_i$. The table gives all four pairs; the highlighted column is the symbol received.

Two extreme cases build the intuition.

(i) Perfect, noiseless channel ($Y_i=X_i$): every received symbol identifies its input with certainty, so $P(X_i\mid Y_i)=1$ and

$$ I(X_i,Y_i)=\log_2\frac{P(X_i\mid Y_i)}{P(X_i)}=\log_2\frac{1}{P(X_i)}=I(X_i). $$

(5.6)

The channel conveys exactly the information content of $X_i$; nothing is lost.

(ii) Extremely noisy channel ($p_e=0.5$): now $Y_i$ is statistically independent of $X_i$, so the joint probability factorises as $P(X_i,Y_i)=P(X_i)\,P(Y_i)$ and the reverse conditional collapses onto the a priori probability:

$$ P(X_i\mid Y_i)=\frac{P(X_i,Y_i)}{P(Y_i)}=\frac{P(X_i)\,P(Y_i)}{P(Y_i)}=P(X_i). $$

(5.7)

Substituting into the definition, $I(X_i,Y_i)=\log_2\frac{P(X_i)}{P(X_i)}=\log_2 1=0$ — the channel conveys nothing; all information is lost. In general $0\le I(X_i,Y_i)\le I(X_i)$.

In general $I(X_i)\ge I(X_i,Y_i)$: some information is lost whenever the channel is imperfect. To make this explicit, start from definition (5.5) and use $\log_2\frac{a}{b}=\log_2 a-\log_2 b$ to split the ratio into two logarithms:

$$ I(X_i,Y_j) = \log_2\frac{P(X_i\mid Y_j)}{P(X_i)} = \log_2 P(X_i\mid Y_j) - \log_2 P(X_i). $$

Each term can be written as the logarithm of a reciprocal, since $\log_2 p = -\log_2\frac{1}{p}$. Swapping the two terms so the positive one comes first:

$$ I(X_i,Y_j) = \log_2\frac{1}{P(X_i)} - \log_2\frac{1}{P(X_i\mid Y_j)}. $$

The first term is the self-information of $X_i$, $I(X_i)=\log_2\frac{1}{P(X_i)}$: how uncertain the receiver is about $X_i$ before anything arrives. The second has the same form but uses the probability after $Y_j$ is observed, so it is the conditional self-information $I(X_i\mid Y_j)=\log_2\frac{1}{P(X_i\mid Y_j)}$: the uncertainty about $X_i$ that remains after seeing $Y_j$. Naming the two terms gives

$$ I(X_i,Y_j) = I(X_i) - I(X_i\mid Y_j), $$

(5.8)

Here $I(X_i\mid Y_j)$, the information content of $X_i$ conditioned on having observed $Y_j$, plays the role of an information-loss term. Rearranged once more, the three quantities line up as source information, information delivered, and information destroyed:

$$ \underbrace{I(X_i)}_{\text{source info}} \;-\; \underbrace{I(X_i,Y_j)}_{\text{info conveyed to Rx}} \;=\; \underbrace{I(X_i\mid Y_j)}_{\text{info loss}}, \qquad 0\le I(X_i\mid Y_j)\le I(X_i). $$

(5.9)

The bounds recover the two extreme cases above: zero loss for the perfect channel ($I(X_i\mid Y_j)=0$), and total loss $I(X_i\mid Y_j)=I(X_i)$ when $p_e=0.5$.

EXAMPLE 5.2 · Mutual information of each symbol pair

For the BSC of Example 5.1 ($P(X_0)=0.3$, $P(X_1)=0.7$, $p_e=0.02$), find the mutual information of each input-output pair and compare it with the source and destination information contents.

1

Reverse conditionals and source information

Rearranging Bayes' rule (5.2) gives each reverse conditional as a joint probability divided by an output probability:

$$ P(X_i\mid Y_j)=\frac{P(X_i,Y_j)}{P(Y_j)}. $$

The joints and output probabilities were found in Example 5.1. The row totals are the source probabilities $P(X_i)$ and the column totals the output probabilities $P(Y_j)$; taking $\log_2\frac{1}{P}$ of each total gives the self-information of that symbol (green cells), used in Step 2:

From Example 5.1, with self-information added

	received Y₀	received Y₁	P(X_i)	I(X_i) bits
sent X₀	0.294	0.006	0.300	log₂(1/0.300) = 1.737
sent X₁	0.014	0.686	0.700	log₂(1/0.700) = 0.515
P(Y_j)	0.308	0.692	1.000
I(Y_j) bits	log₂(1/0.308) = 1.699	log₂(1/0.692) = 0.531

blue = correct receptionamber = errorgreen = self-information, log₂(1/P)

Each reverse conditional is a cell of the table divided by the $P(Y_j)$ at the foot of its column. Given that $Y_0$ was received (left column):

$$ P(X_0\mid Y_0)=\frac{P(X_0,Y_0)}{P(Y_0)}=\frac{0.294}{0.308}=0.9545, \qquad P(X_1\mid Y_0)=\frac{P(X_1,Y_0)}{P(Y_0)}=\frac{0.014}{0.308}=0.0455. $$

Given that $Y_1$ was received (right column):

$$ P(X_1\mid Y_1)=\frac{P(X_1,Y_1)}{P(Y_1)}=\frac{0.686}{0.692}=0.9913, \qquad P(X_0\mid Y_1)=\frac{P(X_0,Y_1)}{P(Y_1)}=\frac{0.006}{0.692}=0.0087. $$

Each pair sums to 1 ($0.9545+0.0455=1$, $0.9913+0.0087=1$), since the transmitted symbol must have been one of the two. The source information contents follow from the a priori probabilities:

$$ I(X_0)=\log_2\frac{1}{P(X_0)}=\log_2\frac{1}{0.3}=1.737 \ \text{bits}, \qquad I(X_1)=\log_2\frac{1}{P(X_1)}=\log_2\frac{1}{0.7}=0.515 \ \text{bits}. $$

2

Mutual information of each pair

Recall the definition of mutual information (5.5):

$$ I(X_i,Y_j) = \log_2\frac{P(X_i\mid Y_j)}{P(X_i)} \quad \text{(bits)}. $$

(5.5)

Divide each reverse conditional by the source probability and take the logarithm:

$$ I(X_0,Y_0)=\log_2\frac{0.9545}{0.3}=1.670 \ \text{bits}, \qquad I(X_1,Y_1)=\log_2\frac{0.9913}{0.7}=0.502 \ \text{bits}. $$

(5.10)

$$ I(X_0,Y_1)=\log_2\frac{0.0087}{0.3}=-5.113 \ \text{bits}, \qquad I(X_1,Y_0)=\log_2\frac{0.0455}{0.7}=-3.945 \ \text{bits}. $$

(5.11)

To judge these values, compare them with the self-information $I(Z)=\log_2\frac{1}{P(Z)}$ of each symbol. At the source, using the a priori probabilities $P(X_0)=0.3$, $P(X_1)=0.7$:

$$ I(X_0)=\log_2\frac{1}{0.3}=\frac{\ln(1/0.3)}{\ln 2}=\frac{1.204}{0.693}=1.737 \ \text{bits}, \qquad I(X_1)=\log_2\frac{1}{0.7}=\frac{0.357}{0.693}=0.515 \ \text{bits}. $$

At the destination, using the output probabilities $P(Y_0)=0.308$, $P(Y_1)=0.692$ from Example 5.1:

$$ I(Y_0)=\log_2\frac{1}{0.308}=\frac{1.178}{0.693}=1.699 \ \text{bits}, \qquad I(Y_1)=\log_2\frac{1}{0.692}=\frac{0.368}{0.693}=0.531 \ \text{bits}. $$

Four pairs, two of them misleading

By (5.8), each row reads as source information minus information loss equals conveyed information: $I(X_i)-I(X_i\mid Y_j)=I(X_i,Y_j)$. For the correct pairs the loss is tiny, so nearly all of $I(X_i)$ is conveyed; for the error pairs the loss exceeds the source information, so the result is negative.

pair	I(X_i) source information log₂(1/P(X_i))	− I(X_i\|Y_j) information loss log₂(1/P(X_i\|Y_j))	= I(X_i,Y_j) conveyed information bits
X₀ → Y₀	1.737	0.067	+1.670
X₁ → Y₁	0.515	0.013	+0.502
X₀ → Y₁	1.737	6.850	−5.113
X₁ → Y₀	0.515	4.459	−3.945

correct pairs: close to the source contenterror pairs: negative, mis-information

At the destination

I(Y₀) = log₂(1/0.308) = 1.699 bits
I(Y₁) = log₂(1/0.692) = 0.531 bits

These are more balanced than the source contents $I(X_0)=1.737$, $I(X_1)=0.515$: the channel noise has evened out the symbol probabilities at the output, so no single received symbol is as "surprising" as $X_0$ was at the source.

Why the negative values matter

The correct-reception pairs give large positive mutual information, close to the source content (1.670 vs. 1.737; 0.502 vs. 0.515): almost all the information got through. The error pairs give large negative values: observing $Y_1$ when $X_0$ was actually sent is actively misleading, "mis-information" rather than information.

5.4 Average mutual information and information loss

What the receiver actually acquires

A single pair's mutual information is a per-symbol quantity; averaging over every possible input/output pair, weighted by the joint probability $P(X_i,Y_j)$ with which that pair actually occurs, gives the average mutual information:

$$ I(X,Y) = \sum_i\sum_j P(X_i,Y_j)\,I(X_i,Y_j) = \sum_i\sum_j P(X_i,Y_j)\,\log_2\frac{P(X_i\mid Y_j)}{P(X_i)} \quad \text{(bits/symbol)}. $$

(5.12)

This is the average amount of source information the receiver actually acquires per received symbol, and must be distinguished from the source entropy $H(X)$, the average information the source sends. By Bayes' rule the log-ratio can be written three equivalent ways — the identity that drives every simplification below:

$$ \frac{P(X_i\mid Y_j)}{P(X_i)} = \frac{P(X_i,Y_j)}{P(X_i)\,P(Y_j)} = \frac{P(Y_j\mid X_i)}{P(Y_j)}. $$

(5.13)

5.4.1 Decomposing $I(X,Y)$ into entropies

Split the logarithm of the ratio into two logarithms, then regroup each double sum. Using $\sum_j P(X_i,Y_j)=P(X_i)$ on the first term and collecting the second term by output symbol (with $P(X_i,Y_j)=P(Y_j)\,P(X_i\mid Y_j)$):

$$ \begin{aligned} I(X,Y) &= \sum_i\sum_j P(X_i,Y_j)\log_2\frac{1}{P(X_i)} \;-\; \sum_i\sum_j P(X_i,Y_j)\log_2\frac{1}{P(X_i\mid Y_j)} \\[2pt] &= \sum_i\Big(\sum_j P(X_i,Y_j)\Big)\log_2\frac{1}{P(X_i)} \;-\; \sum_j P(Y_j)\sum_i P(X_i\mid Y_j)\log_2\frac{1}{P(X_i\mid Y_j)} \\[2pt] &= \sum_i P(X_i)\log_2\frac{1}{P(X_i)} \;-\; \sum_j P(Y_j)\,H(X\mid Y_j) = H(X) - H(X\mid Y). \end{aligned} $$

(5.14)

The first sum is the source entropy $H(X)$; the second is the equivocation $H(X\mid Y)=\sum_j P(Y_j)\,H(X\mid Y_j)$, the average uncertainty about $X$ that remains after $Y$ is observed. An identical regrouping in terms of the output symbols yields the mirror-image form, so both decompositions hold:

$$ \underbrace{I(X,Y)}_{\text{av. conveyed info}} = \underbrace{H(X)}_{\text{source entropy}} - \underbrace{H(X\mid Y)}_{\text{av. info lost}}, \qquad \underbrace{I(X,Y)}_{\text{av. conveyed info}} = \underbrace{H(Y)}_{\text{destination entropy}} - \underbrace{H(Y\mid X)}_{\text{error entropy}}. $$

(5.15)

The first form reads from the transmitter's side: what reaches the receiver equals what the source sends, $H(X)$, minus the equivocation $H(X\mid Y)$ lost in the channel. The second reads from the receiver's side: the same conveyed information equals the destination entropy $H(Y)$ minus the error entropy $H(Y\mid X)$ injected purely by channel noise. Both describe one physical quantity viewed from opposite ends of the channel.

EXAMPLE 5.3 · Average mutual information and information loss

For the BSC of Examples 5.1 and 5.2, find the average mutual information $I(X,Y)$ and the information lost in the channel, $H(X\mid Y)$.

pair	P(X_i,Y_j) weight, Example 5.1	I(X_i,Y_j) bits, Example 5.2	P(X_i,Y_j) I(X_i,Y_j) contribution to I(X,Y)
X₀ → Y₀	0.294	+1.670	+0.4910
X₁ → Y₁	0.686	+0.502	+0.3444
X₀ → Y₁	0.006	−5.113	−0.0307
X₁ → Y₀	0.014	−3.945	−0.0552
I(X,Y)			0.7495

symbol	P(X_i) weight	I(X_i) bits, log₂(1/P)	P(X_i) I(X_i) contribution to H(X)
X₀	0.3	1.737	0.5211
X₁	0.7	0.515	0.3605
H(X)			0.8816

MATLAB corner

The script below reproduces the worked example end to end: joint probabilities, per-pair mutual information, average mutual information, and the $H(X)-H(X\mid Y)$ check.

bsc_mutual_information.m

MATLAB

% BSC average mutual information, checked against H(X) - H(X|Y).
Px = [0.3 0.7];            % P(X0), P(X1)
pe = 0.02;
Pygx = [1-pe pe; pe 1-pe]; % rows: P(Y0|Xi) P(Y1|Xi)
 
Pxy = Px(:) .* Pygx;       % joint P(Xi,Yj)
Py  = sum(Pxy,1);          % P(Yj)
Pxgy = Pxy ./ Py;          % P(Xi|Yj), Bayes' rule
 
Ixy_pairs = log2(Pxgy ./ Px(:));   % I(Xi,Yj) per pair
IXY = sum(sum(Pxy .* Ixy_pairs))  % IXY = 0.7495 bits/symbol
 
HX = -sum(Px .* log2(Px))         % HX = 0.8816 bits/symbol
HXgY = HX - IXY                   % HXgY = 0.1321 bits/symbol (info lost)

Problems with solutions

Work each one before revealing the solution.

PROBLEM 5.1

A BSC has an equiprobable source, $P(X_0)=P(X_1)=0.5$, and error probability $p_e=0.1$. Show that the average mutual information reduces to $I(X,Y)=1+(1-p_e)\log_2(1-p_e)+p_e\log_2 p_e$, and evaluate it.

PROBLEM 5.2

A BSC has $P(X_0)=0.4$, $P(X_1)=0.6$, and $p_e=0.05$. Find all four joint probabilities $P(X_i,Y_j)$, and the output probabilities $P(Y_0)$, $P(Y_1)$.

Summary and key formulas

Mutual information (pair)

$$I(X_i,Y_j)=\log_2\frac{P(X_i\mid Y_j)}{P(X_i)}$$

Average mutual information

$$I(X,Y)=\textstyle\sum_i\sum_j P(X_i,Y_j)\log_2\frac{P(X_i\mid Y_j)}{P(X_i)}$$

Conveyed = source − loss

$$I(X,Y)=H(X)-H(X\mid Y)$$

Conveyed = destination − error

$$I(X,Y)=H(Y)-H(Y\mid X)$$

The BSC is what AWGN becomes once a BPSK modulator and a hard-decision detector are wrapped around it: the continuous received voltage collapses to one bit, and the noise density becomes a single symmetric crossover probability $p_e=Q\big(\sqrt{2E_b/N_0}\big)$.
Mutual information $I(X_i,Y_j)$ measures how much observing $Y_j$ tells you about $X_i$: positive and near $I(X_i)$ for likely (correct) pairs, negative ("mis-information") for error pairs.
Average mutual information $I(X,Y)$ is the true bits/symbol the receiver acquires, always $\le H(X)$, with equality only for a noiseless channel.
Two equivalent decompositions, $I(X,Y)=H(X)-H(X\mid Y)=H(Y)-H(Y\mid X)$, tie mutual information to the entropies of source and destination.
Next, in Lecture 6: maximising $I(X,Y)$ over all possible source distributions defines the channel capacity — the fastest rate at which information can cross the channel error-free.