  ELEC3203 · Digital Coding & Transmission
  Information Theory Fundamentals

  Information Theory · Lecture notes
  Prof. Celik

ELEC3203 · Digital Coding and Transmission

Lecture 7

Wrap-up and Tutorial

Seven worked examples that revise the module. Each question comes with its learning outcome and context, the theory it draws on is recapped inside the solution, and the solution is built up step by step, with interactive figures wherever a parameter changes the answer.

Learning Objectives

Recognise which result from Lectures 1–6 each question is testing
Write complete, step-by-step solutions to source-coding, mutual-information and capacity questions
Relate source information rates to the capacity and bandwidth a channel needs

Slides

Podcast

Lecture 7 — Wrap-up and Tutorial: seven worked revision examples

$$H=-\sum_i p_i\log_2 p_i$$

$$R=H\,R_s$$

$$I(X,Y)=H(X)-H(X\mid Y)$$

$$C=B\log_2\!\left(1+\tfrac{S_P}{N_P}\right)$$

    InstructorProf. Abdulkadir Çelik https://akadircelik.com
    ModuleELEC3203
    CoversWorked Examples 7.1–7.7
  

    ▸ Source coding & RLC
    ▸ Huffman coding
    ▸ BSC & capacity
    ▸ Markov source & bandwidth
  

Lecture 7

Seven worked examples: source coding first, then channels and capacity.

This lecture revises the module through seven worked examples, ordered from source coding to channels and capacity. Each example opens with its learning outcome and why it matters, then states the question with a diagram of its set-up. The solution gives the background for each part before working through it. Try each one before opening its solution.

7.1 Memoryless sources and sources with memory

Same symbols, different information

EXAMPLE 7.1 · memoryless source versus source with memory

Learning outcome

Compare the entropy and information rate of memoryless sources and sources with memory, and choose the right coding strategy for each.

This example states the general rule behind every practical source coder in this lecture, including the speech and video coders of Examples 7.2 and 7.4. Two sources that look identical symbol by symbol can carry different amounts of information, and the difference decides which coder should be used.

Two digital sources have:

The same source alphabet or symbol set of $S=\{m_i,\ 1\le i\le q\}$;
The same symbol probabilities of occurrence $P=\{p_i,\ 1\le i\le q\}$, with $p_i$ being the probability of $m_i$ occurring;
The same symbol rate of $R_s$ [symbols/s]; and
One digital source is memoryless, and the other has memory.

(i)

Which digital source has a larger entropy? and which digital source has a larger information rate?

(ii)

What is the best strategy of coding the memoryless digital source? and why?

(iii)

What is the best strategy of coding the digital source with memory? and why?

Figure 7.1.1. The two sources of Example 7.1. They agree on everything except memory.

PART (i)

Background

A memoryless source emits each symbol independently of the past, and its entropy depends only on the symbol probabilities (Lecture 1). A source with memory, such as a first-order Markov source (Lecture 3), chooses each symbol with probabilities that depend on the previous one, $p_{ij}=P(m_j\mid m_i)$. Two entropies are therefore needed:

$H$, the memoryless entropy: computed from the symbol probabilities $p_i$ alone. It is the entropy of the memoryless source, and also what any coder that looks at one symbol at a time sees, whichever source it is coding.
$H_M$, the entropy of the source with memory: the average uncertainty of the next symbol given the previous one. If the previous symbol is $m_i$, the source is in state $i$ and the next symbol has entropy $H_i$ over row $i$ of the transition matrix; $H_M$ averages $H_i$ over the states.

Figure 7.1.2 draws both cases with their transition matrices and entropies.

$$ \Gamma=\begin{bmatrix}p_1&p_2&\cdots&p_q\\ p_1&p_2&\cdots&p_q\\ \vdots&\vdots&&\vdots\\ p_1&p_2&\cdots&p_q\end{bmatrix} $$

$$ p_{ij}=p_j\ \ \forall i,\qquad H_i=H=-\sum_{j=1}^{q}p_j\log_2 p_j,\qquad H_M=H $$

(7.1.1)

$$ \Gamma=\begin{bmatrix}p_{11}&p_{12}&\cdots&p_{1q}\\ p_{21}&p_{22}&\cdots&p_{2q}\\ \vdots&\vdots&&\vdots\\ p_{q1}&p_{q2}&\cdots&p_{qq}\end{bmatrix} $$

$$ H_i=-\sum_{j=1}^{q}p_{ij}\log_2 p_{ij},\qquad H_M=\sum_{i=1}^{q}p_i\,H_i $$

(7.1.2)

Figure 7.1.2. The transitions out of a state $X_i$, drawn as in Figure 3.6 of Lecture 3. Left: in a memoryless source the next symbol ignores the current state, $p_{ij}=P(X_j\mid X_i)=P(X_j)=p_j$, so every row of the transition matrix is the symbol distribution $(p_1,\dots,p_q)$. This is not $1/q$ in general; $p_{ij}=1/q$ only for equiprobable symbols. every $H_i$ equals $H$, and $H_M=H$. Right: with memory, each state has its own row $p_{i1},\dots,p_{iq}$ of the transition matrix, its own entropy $H_i$, and the average $H_M=\sum_i p_iH_i$ of Eq. (7.1.2) is at most $H$, Eq. (7.1.3).

Knowing the previous symbol can only reduce the uncertainty about the next one, so for the same probabilities $p_i$

$$ H_M\le H\le\log_2 q,\qquad R=H\,R_s , $$

(7.1.3)

with $H_M=H$ only when every row of the transition matrix equals the symbol probabilities, i.e. when the "memory" has no effect.

Figure 7.1.3. Two binary sources with the same probabilities, $P(0)=P(1)=\tfrac12$, and the same symbol rate. Top: memoryless, $H=1$ bit/symbol. Bottom: each symbol repeats the previous one with probability $a$, so $H_M=H_b(a)$. Raise $a$: the stream forms long runs, the symbol counts stay balanced, and the entropy of the source with memory falls.

1

Larger entropy

The memoryless source. Both sources have the same $p_i$, so the memoryless entropy $H$ of Eq. (7.1.1) is the same number for both; the source with memory has entropy $H_M\le H$, Eq. (7.1.3). Part of each of its symbols is predictable from the previous one and carries no new information.

2

Larger information rate

Also the memoryless source. Both emit $R_s$ symbols/s, so $R=H R_s\ge H_M R_s$. The two are equal only if the memory is trivial.

PART (ii)

Background

Entropy coders (Shannon-Fano, Huffman) assign codeword lengths close to $-\log_2 p_i$, so their average length approaches $H$ of Eq. (7.1.1) to within one bit per symbol (Lecture 2). Run-length coding suits a binary source in which one bit is far more likely than the other (Lecture 4).

1

Strategy

Apply entropy coding directly to the symbols: Shannon-Fano or Huffman coding, or run-length coding if the source is binary with one bit far more likely than the other.

2

Why

The symbols are independent, so there is no redundancy between them to remove; the only redundancy is unequal probabilities, $\log_2 q-H$. Entropy coding removes exactly that, giving an average codeword length close to $H$ and a bit rate close to the information rate $R$.

PART (iii)

Background

An entropy coder applied directly to a source with memory still sees only the probabilities $p_i$, so its average length cannot fall below $H$, not $H_M$. The gap $H-H_M$ bits/symbol is redundancy carried by the dependence between successive symbols. Predictive coding (Lecture 4) removes it: the predicted part is not sent, and only the near-uncorrelated prediction error is coded.

1

Strategy

First remove the redundancy caused by the memory, for example with a predictor that codes only the prediction error, then apply entropy coding to the resulting near-uncorrelated residual.

2

Why

Entropy coding the raw symbols would give a rate close to $H R_s$, wasting $(H-H_M)R_s$ bits/s, because the coder ignores the dependence between symbols. After decorrelation the residual is close to memoryless, so entropy coding it brings the rate close to the true information rate $H_M R_s$. This is the two-step principle that Examples 7.2 and 7.4 apply to speech and video.

Same alphabet, same probabilities, same symbol rate: memory alone lowers the information rate, and only a coder that models the memory can benefit from it.

7.2 Remove redundancy, then entropy-code

Judging a speech coder by its information rate

EXAMPLE 7.2 · the 64 kbps speech coder

Learning outcome

Explain why a source with memory must be decorrelated before it is entropy coded.

Every practical compression standard, from the speech codec in a phone call to video streaming, is built on this idea. This example takes a real 1970s telephone speech coder and asks whether it suits a mobile handset, where bandwidth is scarce. Answering it connects three ideas from Lectures 1–4: entropy coding, companding and source memory.

A 64 kbps speech coder was designed in 1970s. In this 64 kbps speech coder design, the speech waveform was filtered to limit its bandwidth to approximately 3.8 kHz and a sampling rate of 8 kHz was then applied. The resultant samples were first passed through an A-law compressor and the compressed samples were then quantised to 256 levels with each sample being represented by 8 bits.

(i)

Will you use this speech coder design for a mobile handset? Explain your answer clearly according to the information theory you have learnt in this course.

(ii)

What is the basic principle adopted in today’s speech source encoding techniques?

Figure 7.2.1. The 1970s 64 kbps speech coder.

PART (i)

Background

An entropy coder (Shannon-Fano, Huffman or arithmetic coding) chooses codeword lengths from the symbol probabilities alone, and it approaches the entropy $H$ only when successive symbols are independent. A source with memory carries less information per symbol than its symbol probabilities suggest (Lecture 3): part of every symbol can be predicted from the past, and coding that predictable part wastes bits. For a $q$-symbol source the entropies are ordered $\sum_i P(X_i)H_i\le-\sum_i P(X_i)\log_2P(X_i)\le\log_2 q$: the first inequality is the saving from memory (conditioning on the previous symbol cannot increase uncertainty), and the second is the saving from unequal probabilities, with equality only for equiprobable symbols.

Speech amplitudes crowd around zero: small values are far more likely than large ones. A uniform quantiser with equal steps therefore uses its few central levels most of the time and its outer levels rarely. The A-law compressor, $C(x)=\dfrac{A|x|}{1+\ln A}$ for $|x|<1/A$ and $C(x)=\dfrac{1+\ln(A|x|)}{1+\ln A}$ above it (with the sign of $x$, and $A=87.6$), stretches small amplitudes and squeezes large ones before the uniform quantiser. Seen from the input, the steps become narrow near zero and wide at large amplitudes, so each step catches a similar share of the probability and the levels are used roughly equally often (Figure 7.2.2).

Figure 7.2.2. A-law companding with a 16-level quantiser (the coder in the question uses 256; 16 keeps the bars readable). Left: the compressor characteristic (blue) over the distribution of speech amplitudes (amber, Laplacian model); uniform output levels map back to narrow input steps near zero. Right: how often each level is used; the dashed line is 1/16, equiprobable. Set $A$ to 1 to remove the compressor.

This makes a fixed 8-bit word per level close to an entropy code for the samples, treated one at a time.

1

What the design does

The coder produces $8\ \text{bits}\times 8000\ \text{samples/s}=64$ kbps. The A-law compressor makes the 256 quantiser levels roughly equiprobable, so coding each level with $\log_2 256=8$ bits is an approximate entropy coding of the samples.

2

Would you use it?

No. If the speech signal were memoryless, this design would be fine: it implements an approximate entropy coding whose rate is close to the source's information rate. But speech is highly correlated, so 64 kbps far exceeds the information rate of the speech source. A handset using it would need excess bandwidth for transmission.

PART (ii)

Background

Practical source coders work in two steps:

Step 1 · remove redundancy

Model the correlation, for example with a predictor, and keep only what the model cannot predict. The residual is nearly uncorrelated.

Step 2 · entropy-code

Apply Huffman or Shannon-Fano coding to the residual, or run-length coding if it is binary with mostly zeros. The entropy coder is now near-optimal.

For a first-order model $x(n)=\rho\,x(n-1)+\sqrt{1-\rho^2}\,w(n)$ with white $w(n)$, the one-tap predictor $\hat x(n)=\rho\,x(n-1)$ leaves the residual $e(n)=x(n)-\hat x(n)$, whose power is

$$ \sigma_e^2=(1-\rho^2)\,\sigma_x^2 . $$

(7.2.1)

Only the prediction error $e(n)$, the innovation, is quantised, entropy coded and sent; the predicted value $\hat x(n)$ is never transmitted. The decoder runs the same predictor on the samples it has already reconstructed, forms the same $\hat x(n)$, and adds the received $e(n)$ to recover $x(n)=\hat x(n)+e(n)$. The bits are spent only on what the past could not predict.

Figure 7.2.3. A correlated source and its prediction residual. Drag $\rho$: the stronger the correlation, the smaller the residual, and the fewer bits it needs for the same quantisation accuracy. Each halving of the amplitude saves one bit per sample, which gives the $\tfrac12\log_2(1/(1-\rho^2))$ figure in the readout.

1

The principle used today

First remove the redundancy of the speech signal, then apply entropy coding to the resulting near-uncorrelated residual sequence.

In practice the redundancy is removed by prediction, and only the innovation $e(n)=x(n)-\hat x(n)$ is coded. Today’s low-rate speech coders use much stronger predictors than the one-tap model of Figure 7.2.3, combining many past samples and the pitch period of voiced speech. A better prediction leaves a smaller prediction-error power $\sigma_e^2$, Eq. (7.2.1); the innovation then carries less information, so fewer bits per sample are needed for the same quality. This is how mobile speech coders use bandwidth efficiently, reaching rates of a few kbps instead of 64 kbps.

The 1970s coder performs step 2 only. Figure 7.2.3 shows the saving step 1 makes possible, and real speech coders use far stronger predictors than one tap.

Figure 7.2.4. Choosing the source coder from the source statistics. For a memoryless source with equiprobable symbols, every symbol carries $\log_2 q$ bits of information, so BCD is already optimal when $q$ is a power of 2 and every codeword has the same length. Unequal probabilities call for entropy coding (Shannon-Fano or Huffman), which gives a symbol a longer codeword the less likely it is. A source with memory is first decorrelated to remove its redundancy, as in Example 7.2, so that the residual behaves as a memoryless source with unequal probabilities and can be coded with option 2.

7.3 Entropy, efficiency and Huffman coding

Building and using a variable-length code

EXAMPLE 7.3 · Huffman coding an 8-symbol source

Learning outcome

Construct a Huffman code and measure its gain over fixed-length coding using entropy and coding efficiency.

Fixed-length words such as BCD spend the same number of bits on every symbol, however likely it is. Huffman coding is the optimal symbol-by-symbol prefix code and appears inside JPEG, MP3 and ZIP. This example covers the complete workflow: find the information rate, build the code, compare efficiencies and decode.

A memoryless digital source emits symbols $X_i$, $1\le i\le 8$, in binary coded decimal (BCD) format with probabilities $P(X_i)$ as given in Table 7.3.1, at a rate $R_s=10^8$ Baud (Baud=symbol/s).

Table 7.3.1. Source symbols, probabilities and BCD words.

$X_i$	$P(X_i)$	BCD word
A	0.27	000
B	0.20	001
C	0.17	010
D	0.15	011
E	0.07	100
F	0.06	101
G	0.05	110
H	0.03	111

Figure 7.3.1. Table 7.3.1 as a bar chart: probabilities above the bars, BCD words below. Every symbol gets a 3-bit word however likely it is.

(i)

What is the data rate of the uncoded BCD signal and what is the information rate of this source?

(ii)

Apply Huffman coding to the digital source characterised in Table 7.3.1.

(iii)

Derive the coding efficiency of both the uncoded BCD signal and the Huffman coded signal.

(iv)

What is the original symbol sequence of the Huffman coded signal 0111000101110101001 (The left most bit is the first bit of the coded signal)?

PART (i)

Background

For a memoryless source with symbols $X_i$ emitted at $R_s$ symbols/s (Lectures 1–2), the entropy and information rate are

$$ H=-\sum_i P(X_i)\log_2 P(X_i)\ \ \text{bits/symbol},\qquad R=H\,R_s\ \ \text{bits/s}. $$

(7.3.1)

1

Data rate of the BCD signal

Each BCD word has $d=3$ bits: $R_{\text{BCD}}=d\,R_s=3\times10^8=300{,}000{,}000$ bits/s.

2

Entropy and information rate

Eq. (7.3.1), term by term:

$$ \begin{aligned}H&=0.5100+0.4644+0.4346+0.4105+0.2686+0.2435+0.2161+0.1518\\&=2.6994905\ \text{bits/symbol},\end{aligned} $$

(7.3.2)

so $R=H\,R_s=2.6994905\times10^8=269{,}949{,}050$ bits/s.

PART (ii)

Background

Huffman coding builds the code from the bottom up:

List the symbols in decreasing order of probability.
Merge the two least probable entries into one entry with their summed probability; give the upper one bit 0 and the lower one bit 1.
Re-sort and repeat until a single entry of probability 1 remains.
Read each codeword from the last merge back down to the symbol.

Figure 7.3.2. Huffman coding of Table 7.3.1, one merge per step, followed by decoding of 0111000101110101001. The leaves are drawn in codeword order so that no branches cross; the panel on the right is the sorted list of the step tables above.

1

Seven merges

Step	Merge (bit 0 + bit 1)	New entry
1	G 0.05 + H 0.03	GH 0.08
2	E 0.07 + F 0.06	EF 0.13
3	EF 0.13 + GH 0.08	EFGH 0.21
4	C 0.17 + D 0.15	CD 0.32
5	EFGH 0.21 + B 0.20	BEFGH 0.41
6	CD 0.32 + A 0.27	ACD 0.59
7	ACD 0.59 + BEFGH 0.41	1.00

2

Read off the codewords

From step 7 back down to each symbol:

Symbol	A	B	C	D	E	F	G	H
$P(X_i)$	0.27	0.20	0.17	0.15	0.07	0.06	0.05	0.03
Huffman	01	11	000	001	1000	1001	1010	1011

PART (iii)

Background

With $l_i$ the length of the codeword for $X_i$, the average codeword length and the coding efficiency are

$$ \bar L=\sum_i P(X_i)\,l_i,\qquad \eta=\frac{H}{\bar L}=\frac{R}{R_b} . $$

(7.3.3)

1

Uncoded BCD

$\bar L=3$ bits/symbol, so $\eta_{\text{BCD}}=2.6994905/3=0.90=90\%$.

2

Huffman

A, B have length 2, C, D length 3 and E–H length 4. Eq. (7.3.3):

$$ \bar L=0.47\times2+0.32\times3+0.21\times4=2.74\ \text{bits/symbol},\qquad \eta_{\text{Huff}}=\frac{2.6994905}{2.74}=0.985\approx99\% . $$

(7.3.4)

PART (iv)

Background

A Huffman code is prefix-free: no codeword is the beginning of another. The decoder reads bits until they match a codeword, outputs that symbol and starts again, without separators or look-ahead. Continue Figure 7.3.2 past step 7 to watch each path.

1

Decode

01 | 11 | 000 | 1011 | 1010 | 1001 $\to$ A B C H G F.

The Huffman code removes almost all of the 0.30 bits/symbol of redundancy in the BCD words.

7.4 Run-length coding

From video redundancy to a compressed bit stream

EXAMPLE 7.4 · video source coding and run-length coding

Learning outcome

Apply the run-length coding equations to find a compression ratio, and decode a run-length coded stream by hand.

Once a video coder has removed spatial and temporal redundancy, its output is a long binary sequence dominated by zeros, and run-length coding is the standard way to compress it. This example links the principle of Example 7.2 to a calculation, then checks that you can run the decoder yourself, including the special all-ones codeword.

(i)

According to the information theory, what is the basic principle adopted in video source encoding techniques?

(ii)

You have designed a video source coder. The output bit sequence of your video source coder contains $6.4067\times10^{14}$ bits, in which the bit 0 has the probability of occurring 0.999 and the bit 1 has the probability of occurring 0.001. You decide to encode this bit sequence by the run length encoder (RLC) with a codeword length of $n=10$ bits. What compression ratio can be achieved? Calculate the length of the bit sequence after the RLC encoding.

(iii)

The RLC of the above video source coder produces the following bit sequence (The left most bit is the first bit of the sequence):

1111111101 0000000000 1111111111 1111111110 1111111101 ···

What is the input bit sequence to the RLC?

Figure 7.4.1. The set-up of parts (ii) and (iii): the video coder output is run-length coded, and part (iii) runs the RLC backwards.

PART (i)

Background

Video is a sequence of frames, each an array of pixels. Neighbouring pixels within a frame are similar (spatial, intra-frame correlation), and successive frames are similar (temporal, inter-frame correlation). The principle of Example 7.2 applies to both.

A fax page is the simplest image case (Lecture 4, Figure 4.3). Most pixels are white, and a pixel is very likely to have the same colour as its neighbour, so each scan line is a few long runs of white broken by short runs of black. Coding the run lengths instead of the individual pixels removes this spatial redundancy. Video coders apply the same idea to the residual left after removing the temporal redundancy between frames.

1

Same principle as speech

To encode a signal at a rate near its true information rate, its correlation or redundancy must be removed first.

2

Remove both redundancies, then entropy-code

Practical video source coders all remove the spatial and temporal redundancies first, then apply entropy coding to the resulting uncorrelated residual sequence. That residual is mostly zeros, which is the case run-length coding suits.

PART (ii)

Background

When a binary sequence has $P(0)=p$ close to 1, coding it bit by bit is wasteful: its entropy $H_b(p)$ is far below 1 bit. A run-length coder (Lecture 4) codes the lengths of the runs of zeros instead. With codeword length $n$ bits and $N=2^n-1$:

a codeword of value \(l
the codeword of value $N$ (all ones) stands for $N$ zeros with no closing 1 ($N$ input bits).

The scheme counts runs of whichever bit is more likely. If instead $P(1)=0.999$ and $P(0)=0.001$, the coder counts runs of ones closed by a 0 (equivalently, invert the bits first); Eqs. (7.4.1)–(7.4.2) are unchanged with $p=P(1)$.

A run of exactly $l$ zeros then a 1 has probability $p^l(1-p)$, and a full run of $N$ zeros has probability $p^N$. The average number of input bits per codeword is therefore

$$ d=\sum_{l=0}^{N-1}(l+1)\,p^l(1-p)+N\,p^N=\frac{1-p^N}{1-p}, $$

(7.4.1)

and since every codeword has $n$ bits, the compression ratio is

$$ \text{compression ratio}=\frac{d}{n}=\frac{1-p^N}{n\,(1-p)} . $$

(7.4.2)

Figure 7.4.2. RLC compression ratio of Eq. (7.4.2) for codeword lengths $n=1,\dots,16$. Blue: the question's $n=10$; green: the best $n$ for the chosen $p$. A short codeword caps the run length at $N$; a long one spends bits describing runs that rarely occur.

1

Parameters

$n=10$, so $N=2^{10}-1=1023$, and $p=0.999$.

2

Compression ratio

From Eqs. (7.4.1) and (7.4.2):

$$ \frac{d}{n}=\frac{1-p^N}{n(1-p)}=\frac{1-0.999^{1023}}{10\times0.001}=\frac{1-0.3593}{0.01}=64.067 . $$

(7.4.3)

On average each 10-bit codeword replaces $d=640.67$ input bits.

3

Length after RLC

$$ R_{\text{RLC}}=\frac{6.4067\times10^{14}}{64.067}=10^{13}\ \text{bits}. $$

(7.4.4)

PART (iii)

Background

The RLC decoder reads the stream $n$ bits at a time and inverts the mapping above: a value \(l

Figure 7.4.3. Decoding the RLC output of part (iii). The all-ones codeword is the only one with no closing 1, so its 1023 zeros join the next run.

1

Split and convert

1111111101 $=1021$: 1021 zeros then a 1. 0000000000 $=0$: a single 1. 1111111111 $=1023=N$: 1023 zeros with no closing 1, so this run continues. 1111111110 $=1022$: 1022 more zeros then a 1, closing a run of $1023+1022=2045$ zeros. 1111111101 $=1021$: 1021 zeros then a 1.

2

Input bit sequence

$$ \underbrace{0\,0\cdots0}_{1021\ \text{0s}}\;1\;1\;\underbrace{0\,0\cdots0}_{2045\ \text{0s}}\;1\;\underbrace{0\,0\cdots0}_{1021\ \text{0s}}\;1\;\cdots $$

(7.4.5)

Figure 7.4.2 also shows that $n=10$ is not the best codeword length for $p=0.999$.

7.5 The binary symmetric channel and mutual information

How much information survives a noisy binary link

EXAMPLE 7.5 · mutual information of a BSC

Learning outcome

Compute the joint and conditional probabilities and the mutual information of a binary symmetric channel, and turn the result into an error-free rate.

The binary symmetric channel is the simplest model of a noisy digital link: the demodulator of Lecture 6 turns an AWGN channel into one. Mutual information measures how much of the source’s information survives the channel, and its maximum is the channel capacity. The example asks first for intuition about the entropies, then for the full calculation.

A binary symmetric channel (BSC) is depicted in Figure 7.5.1,

Figure 7.5.1. The binary symmetric channel.

(i)

For what values of $p$ and $\varepsilon$ are the source and destination entropies, $H(X)$ and $H(Y)$, identical? In general, is $H(Y)\ge H(X)$ or $H(Y)\le H(X)$ true? Justify your answer.

(ii)

Given $p=\frac12$ and $\varepsilon=\frac1{16}$, calculate all the probabilities $P(X_i,Y_j)$ as well as $P(X_i\mid Y_j)$, and derive the numerical value for the mutual information $I(X,Y)$.

(iii)

The source of this BSC of $p=\frac12$ and $\varepsilon=\frac1{16}$ emits the binary symbols with the rate of $10^8$ Baud (Baud=symbol/s). What is the maximum achievable rate of this BSC for achieving error-free transmission?

PART (i)

Background

A binary symmetric channel (BSC) flips each input bit with probability $\varepsilon$. With $P(X_0)=p$, total probability gives the destination probabilities

$$ P(Y_0)=p\,(1-\varepsilon)+(1-p)\,\varepsilon,\qquad P(Y_1)=1-P(Y_0). $$

(7.5.1)

The binary entropy $H_b(q)=-q\log_2q-(1-q)\log_2(1-q)$ is symmetric about $q=\tfrac12$, where it peaks at 1 bit, and falls as $q$ moves towards 0 or 1.

Figure 7.5.2. The BSC of Figure 7.5.1. Branch widths are proportional to the joint probabilities $P(X_i,Y_j)$; the bars show the five entropies of Eq. (7.5.3). The sliders start at the values of part (ii), $p=\tfrac12$, $\varepsilon=\tfrac1{16}$.

1

When the entropies are equal

If $p=0.5$, $X_0$ and $X_1$ are equiprobable, so $Y_0$ and $Y_1$ are also equiprobable and $H(X)=H(Y)$. If $\varepsilon=0$, $P(Y_0)=P(X_0)$ and $P(Y_1)=P(X_1)$, so again $H(X)=H(Y)$.

2

In general

$H(Y)\ge H(X)$. By Eq. (7.5.1), $P(Y_0)$ is a weighted average of $p$ and $1-p$, so it lies between them: the destination symbols are more balanced than the source symbols, and a more balanced binary distribution has higher entropy. Move the sliders in Figure 7.5.2 to see this.

PART (ii)

Background

The joint and conditional probabilities follow from the product and Bayes rules (Lecture 5):

$$ P(X_i,Y_j)=P(X_i)\,P(Y_j\mid X_i),\qquad P(X_i\mid Y_j)=\frac{P(X_i,Y_j)}{P(Y_j)} . $$

(7.5.2)

The average mutual information is the information conveyed per symbol, and splits two ways:

$$ I(X,Y)=\sum_i\sum_j P(X_i,Y_j)\log_2\frac{P(X_i\mid Y_j)}{P(X_i)}=H(X)-H(X\mid Y)=H(Y)-H(Y\mid X). $$

(7.5.3)

1

Joint probabilities

$$ \begin{aligned}P(X_0,Y_0)&=\tfrac12\cdot\tfrac{15}{16}=\tfrac{15}{32}=0.46875, & P(X_0,Y_1)&=\tfrac12\cdot\tfrac1{16}=\tfrac1{32}=0.03125,\\ P(X_1,Y_0)&=\tfrac12\cdot\tfrac1{16}=\tfrac1{32}=0.03125, & P(X_1,Y_1)&=\tfrac12\cdot\tfrac{15}{16}=\tfrac{15}{32}=0.46875.\end{aligned} $$

(7.5.4)

Destination totals: $P(Y_0)=\tfrac{15}{32}+\tfrac1{32}=\tfrac12$ and $P(Y_1)=\tfrac1{32}+\tfrac{15}{32}=\tfrac12$.

2

Conditional probabilities

$$ P(X_0\mid Y_0)=P(X_1\mid Y_1)=\frac{15/32}{1/2}=\frac{15}{16}=0.9375,\qquad P(X_0\mid Y_1)=P(X_1\mid Y_0)=\frac{1/32}{1/2}=\frac1{16}=0.0625 . $$

(7.5.5)

3

Mutual information

Substitute into Eq. (7.5.3); the four terms pair up by symmetry:

$$ I(X,Y)=2\cdot\frac{15}{32}\log_2\frac{15/16}{1/2}+2\cdot\frac1{32}\log_2\frac{1/16}{1/2}=\frac{15}{16}\log_2 15-\frac{48}{16}=3.6627099-3=0.6627099\ \text{bits/symbol}. $$

(7.5.6)

PART (iii)

Background

For a BSC, $H(Y\mid X)=H_b(\varepsilon)$ whatever $p$ is, so $I(X,Y)=H(Y)-H_b(\varepsilon)$ is largest when $H(Y)=1$, i.e. $p=\tfrac12$. That maximum is the BSC capacity per symbol (Lecture 6), and multiplying by the symbol rate gives bits/s:

$$ C=1+(1-\varepsilon)\log_2(1-\varepsilon)+\varepsilon\log_2 \varepsilon\ \ \text{bits/symbol},\qquad C_{\text{bits/s}}=R_s\,C . $$

(7.5.7)

1

Capacity per symbol

With $p=\tfrac12$ the source already achieves the BSC capacity. Check with Eq. (7.5.7): $1+\tfrac{15}{16}\log_2\tfrac{15}{16}+\tfrac1{16}\log_2\tfrac1{16}=1-0.0873-0.25=0.6627$ bits/symbol.

2

Maximum error-free rate

$$ C=R_s\,I(X,Y)=10^8\times0.6627099=6.627099\times10^7\ \text{bits/s}\approx66.3\ \text{Mbit/s}. $$

(7.5.8)

Even a channel that delivers 15 bits in 16 correctly carries only two-thirds of a bit of information per symbol.

7.6 The Shannon-Hartley capacity

Bandwidth and power, the two resources of a link

EXAMPLE 7.6 · Shannon-Hartley capacity

Learning outcome

State the Shannon-Hartley law and use it to reason about the trade-off between bandwidth and transmit power.

Bandwidth and power are what a wireless system pays for: spectrum is licensed and power drains batteries. The Shannon-Hartley law sets the highest rate that a given amount of each can deliver. This example asks you to state the law precisely, read it as a trade-off and evaluate it for one channel.

(i)

State the Shannon-Hartley channel capacity formula, and define clearly each variable in the formula. According to this channel capacity formula, what are the two basic resources for digital communications? Discuss how you can trade off these two resources to achieve the required channel capacity.

(ii)

The signal-to-noise ratio (SNR) of a wireless channel is SNR = 127, and the bandwidth of this wireless channel is 10 MHz. What is the capacity of this channel?

Figure 7.6.1. The wireless channel of part (ii).

7.7 A Markov source over an AWGN channel

Matching a source with memory to a Gaussian channel

EXAMPLE 7.7 · minimum bandwidth for an 8-ary Markov source

Learning outcome

Compute the entropy of a first-order Markov source and the minimum channel bandwidth for error-free transmission.

This example joins the two halves of the module. The source side (Lecture 3) says how many bits per second really need to be sent; the channel side (Lecture 6) says how many a channel of given bandwidth and SNR can carry. Setting the two equal gives the smallest bandwidth that can work.

A digital communication system uses an 8-ary signalling scheme with the transmission rate of 100 MBaud ($10^8$ [symbols/s]). The probabilities of occurrence for the eight symbols at the transmitter are respectively

$P(X_1)=0.2,\ P(X_2)=0.1,\ P(X_3)=0.1,\ P(X_4)=0.1,$
$P(X_5)=0.1,\ P(X_6)=0.1,\ P(X_7)=0.1,\ P(X_8)=0.2.$

It is known that this 8-ary symbol source is a first-order Markov process with the known transition probability matrix

$\Gamma=[p_{ij}]=$

0.2	0.0	0.0	0.0	0.0	0.3	0.3	0.2
0.2	0.2	0.0	0.0	0.0	0.0	0.3	0.3
0.3	0.2	0.2	0.0	0.0	0.0	0.0	0.3
0.3	0.3	0.2	0.2	0.0	0.0	0.0	0.0
0.0	0.4	0.3	0.2	0.1	0.0	0.0	0.0
0.0	0.0	0.4	0.3	0.2	0.1	0.0	0.0
0.0	0.0	0.0	0.4	0.3	0.2	0.1	0.0
0.0	0.0	0.0	0.0	0.4	0.3	0.1	0.2

where $p_{ij}=P(X_j\mid X_i)$, $1\le i,j\le 8$, are the transition probabilities.

(i)

Determine the information rate of the source at the transmitter.

(ii)

The channel is an ideal additive white Gaussian noise (AWGN) channel, and the channel’s signal to noise ratio is known to be 63. Determine the minimum channel bandwidth required to achieve error-free transmission.

Figure 7.7.1. The system of Example 7.7. Darker cells of $\Gamma$ above are the more likely transitions.

PART (i)

Background

For a first-order Markov source with transition probabilities $p_{ij}=P(X_j\mid X_i)$, each state (previous symbol) has its own entropy, and the source entropy is their probability-weighted average (Lecture 3):

$$ H_i=-\sum_j p_{ij}\log_2 p_{ij},\qquad H=\sum_i P(X_i)\,H_i,\qquad R=H\,R_s . $$

(7.7.1)

1

Spot the repeated rows

Every row of $\Gamma$ has four non-zero entries. Rows 1–4 each contain $\{0.3,0.3,0.2,0.2\}$ and rows 5–8 each contain $\{0.4,0.3,0.2,0.1\}$, in different positions. Entropy depends only on the values, so there are just two distinct state entropies.

2

State entropies

$$ \begin{aligned}H_i\,(1\le i\le4)&=-2\times0.3\log_2 0.3-2\times0.2\log_2 0.2=2\times0.5210896+2\times0.4643856=1.9709504,\\ H_i\,(5\le i\le8)&=-0.4\log_2 0.4-0.3\log_2 0.3-0.2\log_2 0.2-0.1\log_2 0.1\\&=0.5287712+0.5210896+0.4643856+0.3321928=1.8464392\ \text{bits/symbol}.\end{aligned} $$

(7.7.2)

3

Source entropy and information rate

$P(X_1)+\dots+P(X_4)=0.5$ and $P(X_5)+\dots+P(X_8)=0.5$, so Eq. (7.7.1) gives

$$ H=0.5\times1.9709504+0.5\times1.8464392=1.9086948\ \text{bits/symbol},\qquad R=H\,R_s=190.86948\ \text{Mbit/s}. $$

(7.7.3)

PART (ii)

Background

Shannon's theorem: error-free transmission is possible if and only if the information rate does not exceed the capacity. With the Shannon-Hartley capacity this sets a minimum bandwidth:

$$ R\le C=B\log_2\!\left(1+\frac{S_P}{N_P}\right)\quad\Longrightarrow\quad B_{\min}=\frac{R}{\log_2(1+\text{SNR})} . $$

(7.7.4)

Figure 7.7.2. Minimum bandwidth for error-free transmission, Eq. (7.7.4), for this source. Using the Markov memory (blue) needs about two-thirds of the bandwidth of a memoryless treatment of the same symbols (amber). The slider starts at SNR = 63.

1

Minimum bandwidth

With SNR = 63, $\log_2(1+63)=6$ bits/s/Hz:

$$ 190.86948\times10^6\le 6B\quad\Longrightarrow\quad B\ge31.81159\ \text{MHz},\qquad B_{\min}=31.81\ \text{MHz}. $$

(7.7.5)

Treating the same symbols as memoryless, $H=2.9219$ bits/symbol and the bandwidth would rise to 48.70 MHz.

7.8 Summary

Reference sheet for the worked examples

7.1 · Sources with memory

$$ H=-\sum_i p_i\log_2 p_i,\quad H_M=\sum_i p_iH_i $$

$$ H_M\le H\le\log_2 q $$

Memory lowers the information rate; a symbol-by-symbol coder only reaches $H$.

7.2 · Remove redundancy first

$$ e(n)=x(n)-\hat x(n) $$

$$ \sigma_e^2=(1-\rho^2)\,\sigma_x^2 $$

Code only the innovation, then entropy-code it.

7.3 · Entropy coding

$$ R=H\,R_s,\quad \bar L=\sum_i p_i l_i $$

$$ \eta=H/\bar L $$

Huffman: merge the two least probable entries until one remains.

7.4 · Run-length coding

$$ N=2^n-1,\quad d=\frac{1-p^N}{1-p} $$

$$ \text{ratio}=d/n $$

All-ones codeword: $N$ zeros with no closing 1.

7.5 · Binary symmetric channel

$$ I(X,Y)=H(X)-H(X|Y) $$

$$ C=1-H_b(\varepsilon)\ \text{at}\ p=\tfrac12 $$

$H(Y)\ge H(X)$; multiply by $R_s$ for bits/s.

7.6 · Shannon-Hartley

$$ C=B\log_2\!\left(1+\frac{S_P}{N_P}\right) $$

$$ C_\infty=1.44\,S_P/N_0 $$

Trade bandwidth against power.

7.7 · Minimum bandwidth

$$ R=R_s\sum_i p_iH_i\le C $$

$$ B_{\min}=\frac{R}{\log_2(1+\text{SNR})} $$

Exploiting memory reduces the bandwidth needed.

Symbol	A	B	C	D	E	F	G	H
\(P(X_i)\)	0.27	0.20	0.17	0.15	0.07	0.06	0.05	0.03
Huffman	01	11	000	001	1000	1001	1010	1011

0.2	0.0	0.0	0.0	0.0	0.3	0.3	0.2
0.2	0.2	0.0	0.0	0.0	0.0	0.3	0.3
0.3	0.2	0.2	0.0	0.0	0.0	0.0	0.3
0.3	0.3	0.2	0.2	0.0	0.0	0.0	0.0
0.0	0.4	0.3	0.2	0.1	0.0	0.0	0.0
0.0	0.0	0.4	0.3	0.2	0.1	0.0	0.0
0.0	0.0	0.0	0.4	0.3	0.2	0.1	0.0
0.0	0.0	0.0	0.0	0.4	0.3	0.1	0.2

0.2	0.0	0.0	0.0	0.0	0.3	0.3	0.2
0.2	0.2	0.0	0.0	0.0	0.0	0.3	0.3
0.3	0.2	0.2	0.0	0.0	0.0	0.0	0.3
0.3	0.3	0.2	0.2	0.0	0.0	0.0	0.0
0.0	0.4	0.3	0.2	0.1	0.0	0.0	0.0
0.0	0.0	0.4	0.3	0.2	0.1	0.0	0.0
0.0	0.0	0.0	0.4	0.3	0.2	0.1	0.0
0.0	0.0	0.0	0.0	0.4	0.3	0.1	0.2

Wrap-up and Tutorial

Contents

7.1 Memoryless sources and sources with memory

7.2 Remove redundancy, then entropy-code

7.3 Entropy, efficiency and Huffman coding

7.4 Run-length coding

7.5 The binary symmetric channel and mutual information

7.6 The Shannon-Hartley capacity

7.7 A Markov source over an AWGN channel

7.8 Summary

0.2	0.0	0.0	0.0	0.0	0.3	0.3	0.2
0.2	0.2	0.0	0.0	0.0	0.0	0.3	0.3
0.3	0.2	0.2	0.0	0.0	0.0	0.0	0.3
0.3	0.3	0.2	0.2	0.0	0.0	0.0	0.0
0.0	0.4	0.3	0.2	0.1	0.0	0.0	0.0
0.0	0.0	0.4	0.3	0.2	0.1	0.0	0.0
0.0	0.0	0.0	0.4	0.3	0.2	0.1	0.0
0.0	0.0	0.0	0.0	0.4	0.3	0.1	0.2