  ELEC3203 · Digital Coding & Transmission
  Information Theory Fundamentals

  Information Theory · Lecture notes
  Prof. Celik

ELEC3203 · Digital Coding and Transmission

Lecture 6

Channel Capacity

Lecture 5 defined mutual information, what a receiver actually learns per symbol. This lecture maximises it: the channel capacity $C$ is the fastest rate at which information can cross a channel with arbitrarily small error, for both discrete channels (the BSC) and continuous ones (the Gaussian channel, via the Shannon-Hartley law).

Learning Objectives

Derive channel capacity as the maximum mutual information over all input distributions
Compute the capacity of the binary symmetric channel and other simple models
Interpret the Shannon capacity theorem’s implications for error-free transmission rates

Slides Quiz

Podcast

    InstructorProf. Abdulkadir Çelik https://akadircelik.com
    ModuleELEC3203
    TopicsChannel capacity · Shannon's theorem · Gaussian channel · Shannon-Hartley law
  

    ▸ Discrete channel capacity
    ▸ Shannon's theorem
    ▸ Gaussian channel
    ▸ Bandwidth/SNR trade-off
  

Lecture 6

This page grows as further lectures are added to the document.

Notation used in this lecture

\(C\)

channel capacity, in bits/symbol or bits/second

\(t_{av}\), \(T_s\)

average, and constant, symbol duration (seconds/symbol)

\(p_e\)

BSC error (crossover) probability

\(B\)

channel (signal) bandwidth, in Hz

\(S_P, N_P\)

signal power and noise power

\(N_0\)

two-sided noise power spectral density; $N_P = N_0 B$

\(p(x)\)

probability density function of a continuous (analogue) source

\(H(x)\)

differential entropy of a continuous source

\(\sigma_x^2\)

variance of a Gaussian source; equals its power $S_P$

6.1 From mutual information to channel capacity

The best this channel could ever do

Lecture 5 defined average mutual information $I(X,Y)$: the true bits/symbol a receiver acquires through a given channel, for a given source distribution. But different source distributions feeding the same channel give different values of $I(X,Y)$. Shannon's channel capacity asks: over every possible input distribution, what is the most information this channel could ever convey?

$$ C = \max\{I(X,Y)\} = \max\{H(Y)-H(Y\mid X)\} \quad \text{(bits/symbol)}. $$

(6.1)

If symbol $X_i$ has duration $t_i$ and $t_{av}=\sum_i P(X_i)t_i$ is the average symbol duration, capacity in bits/second follows directly:

$$ C = \max\{I(X,Y)/t_{av}\} \quad \text{(bits/second)}. $$

(6.2)

Two conditions jointly maximise $I(X,Y)$: the channel must introduce no errors ($H(Y\mid X)=0$), and, whenever symbol durations are constant, the input symbols must be equiprobable. The first is a property of the channel, not something we control; the second is exactly the maximum-entropy condition from Lecture 3, now applied at the channel input.

6.1.1 Where "the channel" sits in the full system

It helps to place $X(k)$ and $Y(k)$ in the whole chain. The source emits symbols $\{X_i\}$; the MODEM converts these to a continuous-time waveform $x(t)$, which crosses the physical, continuous-time (analogue) channel and is demodulated back into a continuous waveform $y(t)$, from which the receiver recovers symbols $\{Y_i\}$.

Figure 6.1. The channel in Lecture 5's mutual-information sense is the discrete-time channel from X(k) to Y(k): it wraps the MODEM (modulator and demodulator) around the underlying continuous-time analogue channel that carries x(t) to y(t). The strip below follows sixteen symbols through the chain: x(t) is the transmitted waveform, y(t) the noisy received waveform, and shaded columns are symbols the demodulator decided wrongly. Move E_b/N₀ to see the analogue noise become the discrete channel's error probability p_e.

6.2 Capacity of discrete channels

The noise-free bound, and the capacity of the BSC

6.2.1 The noise-free case

With no errors the error entropy vanishes, $H(Y\mid X)=0$, so $I(X,Y)=H(Y)=H(X)$: the channel conveys exactly the source entropy. Writing the source entropy and the average symbol duration explicitly,

$$ H(X) = -\sum_{i=1}^{q} P(X_i)\log_2 P(X_i)\ \text{(bits/symbol)}, \qquad t_{av} = \sum_{i=1}^{q} P(X_i)\,t_i\ \text{(s/symbol)}, $$

(6.3)

so the capacity in bits/second is $C=\max\{H(X)/t_{av}\}$. With constant symbol durations $t_i=T_s$, the maximum entropy is reached by a memoryless, equiprobable $q$-ary source, for which $H(X)=\log_2 q$, giving the maximum achievable transmission rate:

$$ C = \frac{\log_2 q}{T_s} \quad \text{(bits/second)} \qquad \text{(noise-free channel).} $$

(6.4)

6.2.2 Capacity of the binary symmetric channel

For the BSC of Lecture 5 with equiprobable input, $P(X_0)=P(X_1)=0.5$, symmetry forces $P(Y_0)=P(Y_1)=0.5$ too. Working through the joint probabilities $P(X_i,Y_j)$ and substituting into $I(X,Y)$ (each of the four terms collapses to $\log_2[2(1-p_e)]$ or $\log_2(2p_e)$, matching Lecture 5's Problem 1 derivation) gives a capacity that depends on $p_e$ alone:

$$ C = 1 + (1-p_e)\log_2(1-p_e) + p_e\log_2 p_e \quad \text{(bits/symbol).} $$

(6.5)

Figure 6.2. The capacity-achieving BSC: equiprobable inputs $P(X_0)=P(X_1)=\tfrac12$ force $P(Y_0)=P(Y_1)=\tfrac12$. Solid lines carry probability $1-p_e$; dashed lines the crossover probability $p_e$.

Derivation, step by step

1

Take equiprobable inputs $P(X_0)=P(X_1)=\tfrac12$; by symmetry $P(Y_0)=P(Y_1)=\tfrac12$. The BSC's conditional (transition) probabilities are

$$ P(Y_0\mid X_0)=P(Y_1\mid X_1)=1-p_e, \qquad P(Y_1\mid X_0)=P(Y_0\mid X_1)=p_e, $$

so, from $P(X_i,Y_j)=P(Y_j\mid X_i)\,P(X_i)$, the four joint probabilities are

$$ P(X_0,Y_0)=P(X_1,Y_1)=\tfrac{1-p_e}{2}, \qquad P(X_0,Y_1)=P(X_1,Y_0)=\tfrac{p_e}{2}. $$

(6.6)

2

Write the average mutual information over the four $(i,j)$ pairs. Bayes' rule follows from writing the joint probability both ways, $P(X_i,Y_j)=P(X_i\mid Y_j)\,P(Y_j)=P(Y_j\mid X_i)\,P(X_i)$, so

$$ \frac{P(X_i\mid Y_j)}{P(X_i)}=\frac{P(X_i,Y_j)}{P(X_i)\,P(Y_j)}=\frac{P(Y_j\mid X_i)}{P(Y_j)}, $$

and therefore

$$ I(X,Y)=\sum_i\sum_j P(X_i,Y_j)\log_2\frac{P(Y_j\mid X_i)}{P(Y_j)}. $$

(6.7)

3

Substitute the values. With $P(Y_j)=\tfrac12$, each ratio becomes $2\,P(Y_j\mid X_i)$, so the four terms are

$$ I(X,Y)=\underbrace{\tfrac12(1{-}p_e)\log_2[2(1{-}p_e)]}_{X_0,\,Y_0}+\underbrace{\tfrac12 p_e\log_2(2p_e)}_{X_0,\,Y_1}+\underbrace{\tfrac12 p_e\log_2(2p_e)}_{X_1,\,Y_0}+\underbrace{\tfrac12(1{-}p_e)\log_2[2(1{-}p_e)]}_{X_1,\,Y_1}. $$

(6.8)

4

Collect the two identical correct-reception terms $(X_0,Y_0)$, $(X_1,Y_1)$ and the two identical error terms $(X_0,Y_1)$, $(X_1,Y_0)$. Each pair of halves adds to one whole term:

$$ I(X,Y)=(1-p_e)\log_2[2(1-p_e)]+p_e\log_2(2p_e). $$

Split each logarithm with $\log_2(2x)=\log_2 2+\log_2 x=1+\log_2 x$:

$$ I(X,Y)=(1-p_e)\,[1+\log_2(1-p_e)]+p_e\,[1+\log_2 p_e]. $$

Multiply out and gather the constant terms, $(1-p_e)+p_e=1$:

$$ I(X,Y)=\underbrace{(1-p_e)+p_e}_{=\,1}+(1-p_e)\log_2(1-p_e)+p_e\log_2 p_e. $$

Equiprobable inputs maximise $I(X,Y)$ for the symmetric channel, so this value is the capacity:

$$ C = 1+(1-p_e)\log_2(1-p_e)+p_e\log_2 p_e \quad\text{(bits/symbol).} $$

(6.9)

Figure 6.3. BSC capacity versus error probability $p_e$ on a logarithmic axis: $C$ stays near 1 bit/symbol for small $p_e$, then falls to 0 as $p_e\to 0.5$. Even $p_e=10^{-2}$ already costs about 8% of capacity. Move the slider to read $C$ at any $p_e$.

The two extremes match intuition directly: a perfect channel ($p_e=0$) gives $C=1$ bit/symbol, the maximum possible for a binary alphabet; a maximally noisy channel ($p_e=0.5$, output independent of input) gives $C=0$ — no information can cross at all.

6.3 Shannon's channel coding theorem

Noise limits the rate, not the reliability

SHANNON'S CHANNEL CODING THEOREM

If the information rate $R$ satisfies $R \le C$, there exists a channel coding technique that transmits information over the channel with arbitrarily small error probability. If $R > C$, error-free transmission is impossible.

This is one of the most consequential results in engineering: a noisy channel is not an obstacle to reliable communication, only a limit on the rate at which it can happen. Note carefully what the theorem does not say: it guarantees that a good code exists, but gives no recipe for constructing one. Most practical channel codes fall well short of $C$, though capacity-approaching codes exist (e.g. turbo codes, low-density parity-check codes), and modern iterative turbo detection-decoding pushes real systems increasingly close to the Shannon limit.

6.4 Analogue sources and the Gaussian channel

Differential entropy, and why Gaussian maximises it

Real channels usually carry continuous-valued (analogue) signals, not discrete symbols, so we need an analogue counterpart to entropy. For a continuous source with probability density function $p(x)$, the differential entropy is:

$$ H(x) = -\int_{-\infty}^{+\infty} p(x)\log_2 p(x)\,dx. $$

(6.10)

Just as a discrete source's entropy is maximised by an equiprobable distribution, Shannon showed that, among all continuous distributions with a given variance (power), differential entropy is maximised by the Gaussian PDF:

$$ p(x) = \frac{1}{\sqrt{2\pi}\,\sigma_x}\, e^{-x^2/2\sigma_x^2}, \qquad H_{\max}(x) = \tfrac12\log_2\!\big(2\pi e\,\sigma_x^2\big). $$

(6.11)

Figure 6.4. The Gaussian (normal) probability density function: the maximum-entropy distribution for a given power (variance), and the shape assumed by both the Gaussian source and AWGN in this section.

Because a Gaussian signal attains maximum entropy for its power, and because AWGN is itself Gaussian, it is natural to study the Gaussian channel: a channel input $x$ with Gaussian PDF and power $S_P=\sigma_x^2$, corrupted by independent AWGN $\varepsilon$ with power $N_P=\sigma_\varepsilon^2$.

Figure 6.5. The Gaussian channel: independent Gaussian signal and noise sum, so the output $y$ is Gaussian too, with power $\sigma_y^2 = S_P + N_P$.

A Gaussian signal passed through the linear operation "add independent Gaussian noise" remains Gaussian, so $y=x+\varepsilon$ has power $\sigma_y^2=S_P+N_P$ and its own maximum differential entropy $H_{\max}(y)=\tfrac12\log_2\!\big(2\pi e(S_P+N_P)\big)$. Since $I(x,y)=H(y)-H(y\mid x)$ and the residual uncertainty $H(y\mid x)$ is exactly the noise's own entropy, $H(y\mid x)=H(\varepsilon)=\tfrac12\log_2(2\pi e\,N_P)$, the two logarithm terms subtract cleanly:

$$ I(x,y)=H(y)-H(y\mid x)=\tfrac12\log_2\!\big(2\pi e(S_P+N_P)\big)-\tfrac12\log_2\!\big(2\pi e\,N_P\big)=\tfrac12\log_2\frac{S_P+N_P}{N_P}, $$

(6.12)

and cancelling the common factor leaves the striking per-symbol capacity of the Gaussian channel:

$$ I(x,y) = \tfrac12\log_2\!\left(1+\frac{S_P}{N_P}\right) \quad \text{(bits/symbol).} $$

(6.13)

6.5 The Shannon-Hartley law and the bandwidth/SNR trade-off

Bandwidth and power, traded against each other

Sampling the Gaussian channel at the Nyquist rate $f_s=2B$ converts the per-symbol result of Section 4 into a rate, giving the celebrated Shannon-Hartley law:

$$ C = f_s\, I(x,y) = B\log_2\!\left(1+\frac{S_P}{N_P}\right) \quad \text{(bits/second)}, $$

(6.14)

where $B$ is the channel bandwidth in Hz and $N_P=N_0 B$ with $N_0$ the (two-sided) AWGN power spectral density. This is often an excellent approximation for practical digital channels, even ones that are not exactly Gaussian.

Figure 6.6. AWGN has a flat ("white") two-sided power spectral density N₀/2 across all frequencies (left). Confined to the channel bandwidth B, the total noise power is that density times the band, Nₚ = N₀B (right).

Two, and only two, physical resources set the capacity: bandwidth $B$ and signal power $S_P$ (via the SNR). Either can be traded for the other while holding $C$ fixed — a noiseless channel ($S_P/N_P\to\infty$) has infinite capacity, but capacity does not diverge as $B\to\infty$ with fixed $S_P,N_0$; it approaches a finite ceiling:

$$ C_\infty = \lim_{B\to\infty} B\log_2\!\left(1+\frac{S_P}{N_0 B}\right) = \frac{S_P}{N_0}\log_2 e \approx 1.44\,\frac{S_P}{N_0}. $$

(6.15)

EXAMPLE 6.1 · Capacity of a telephone line

A voice-grade line has bandwidth $B=3\,\text{kHz}$ and SNR $=1000$ (30 dB). Find its Shannon-Hartley capacity.

MATLAB corner

The script below plots BSC capacity against $p_e$ and evaluates the Shannon-Hartley law for the worked example.

channel_capacity.m

MATLAB

% BSC capacity curve, and Shannon-Hartley for a voice-grade line.
pe = linspace(1e-4, 0.5, 400);
C_bsc = 1 + (1-pe).*log2(1-pe) + pe.*log2(pe);
plot(pe, C_bsc); xlabel('p_e'); ylabel('C (bits/symbol)');
 
B  = 3000;    % Hz
SNR = 1000;   % linear (30 dB)
C_gaussian = B * log2(1 + SNR)   % C_gaussian = 29904 bits/s
 
% Bandwidth/SNR trade-off: halve B, find the new S_P needed for same C.
SNR0 = 15; C0 = B*log2(1+SNR0);
Bnew = B/2;
ratio = 2^(C0/Bnew) - 1;   % required 1 + 30*(S'/S) = ratio
Sratio = (ratio - 1) / 30
% Sratio = 8.5 -> signal power must increase 8.5x

Problems with solutions

Work each one before revealing the solution.

PROBLEM 6.1

Lecture 5's running BSC example had $P(X_0)=0.3$, $P(X_1)=0.7$, $p_e=0.02$, giving average mutual information $I(X,Y)\approx0.7495$ bits/symbol. What is this channel's capacity, and why does it differ from $0.7495$?

PROBLEM 6.2

A channel has SNR $=15$. If its bandwidth is halved, by what factor must the signal power increase to keep the same capacity $C$?

Summary and key formulas

Discrete channel capacity

$$C=\max\{I(X,Y)\}$$

BSC capacity

$$C=1+(1{-}p_e)\log_2(1{-}p_e)+p_e\log_2 p_e$$

Shannon's theorem

$R\le C$ ⇒ arbitrarily reliable coding exists

Shannon-Hartley law

$$C=B\log_2\!\left(1+\frac{S_P}{N_P}\right)$$

Channel capacity is the maximum of $I(X,Y)$ over all input distributions — a property of the channel alone, achieved only by the right (typically equiprobable) source.
The BSC's capacity falls from 1 bit/symbol at $p_e=0$ to 0 at $p_e=0.5$, dropping steeply even for small error rates.
Shannon's theorem guarantees arbitrarily reliable transmission below capacity, but does not construct the code; practical capacity-approaching codes (turbo, LDPC) exist but are hard to design.
Gaussian signals maximise differential entropy for a given power, making the Gaussian channel the natural continuous-channel model, with $I(x,y)=\tfrac12\log_2(1+S_P/N_P)$.
The Shannon-Hartley law converts this into bits/second, and shows bandwidth and SNR can be traded for one another, though capacity grows only logarithmically in power versus linearly in bandwidth.