ELEC3203 · Digital Coding & Transmission Information Theory Fundamentals
Information Theory · Lecture notes Prof. Celik
ELEC3203 · Digital Coding and Transmission
Lecture 2

Efficient Source Coding: Shannon-Fano and Huffman

BCD wastes bits whenever symbols are not equally likely. This lecture builds two concrete codes, Shannon-Fano and Huffman, that assign short codewords to common symbols and long ones to rare symbols, pushing the average length down to nearly the entropy bound.

Learning Objectives
  • Construct Shannon-Fano and Huffman codes for a given symbol alphabet
  • Compute average codeword length and coding efficiency against the entropy bound
  • Explain why Huffman coding is optimal among prefix codes
SlidesQuiz
Podcast
Efficient Source Coding: Shannon-Fano & Huffman cover art
InstructorProf. Abdulkadir Çelik https://akadircelik.com ModuleELEC3203 TopicsMaximum entropy · efficient coding · Shannon-Fano · Huffman
▸ Maximum entropy ▸ Coding efficiency ▸ Shannon-Fano coding ▸ Huffman coding
Lecture 2

Contents

This page grows as further lectures are added to the document.

Notation used in this lecture
\(H\)
source entropy, in bits/symbol (from Lecture 1)
\(\ell_i\)
codeword length assigned to symbol \(m_i\), in bits
\(\bar L\)
average codeword length, \(\bar L = \sum_i p_i \ell_i\), in bits/symbol
CE, \(\eta\)
coding efficiency, \(\eta = H/\bar L\)
prefix code
a code in which no codeword is the start of another, so it can be decoded unambiguously
\(\lambda\)
Lagrange multiplier used to enforce \(\sum_i p_i = 1\) in the maximum-entropy derivation

2.1  Maximum entropy: q-ary and binary sources

Where does \(H \le \log_2 q\) come from?

Lecture 1 stated, and proved in its appendix, that the entropy of a \(q\)-ary source is bounded by \(H \le \log_2 q\), with equality only for equiprobable symbols. Here we derive that result directly, by maximising \(H\) subject to the one constraint every probability distribution must satisfy: \(\sum_i p_i = 1\).

2.1.1  Lagrangian derivation for a q-ary source

Background: what is a Lagrangian, and why set its derivative to zero?

Suppose you want to maximise (or minimise) some function, but the variables are not free: they must also satisfy a constraint. Here we want to maximise entropy \(H(p_1,\ldots,p_q)\), but the \(p_i\) cannot be chosen independently — they must always sum to 1. Without the constraint, "maximise \(H\)" would have a silly answer (push some \(p_i\) to absurd values), so we need a way to search only among the probability vectors that are actually valid.

The Lagrangian is a trick for turning a constrained problem into an unconstrained one. We build a new function \(\mathcal{L}\) by taking the objective (\(H\)) and adding a penalty term for violating the constraint, weighted by a new variable \(\lambda\) (the Lagrange multiplier):

$$ \mathcal{L}(p_1,\ldots,p_q,\lambda) = H(p_1,\ldots,p_q) + \lambda\Big(1 - \textstyle\sum_i p_i\Big). $$
(2.1)

Notice that whenever the constraint \(\sum_i p_i = 1\) actually holds, the penalty term is exactly zero, so \(\mathcal{L} = H\) at any valid point. This means: any point that maximises \(\mathcal{L}\) while also satisfying the constraint automatically maximises \(H\) too. We have replaced "maximise \(H\) subject to a constraint" with "find a stationary point of \(\mathcal{L}\)", which is a completely ordinary calculus problem with no constraint left to worry about.

Why does setting a derivative to zero find a maximum? This is the basic first-order condition from single-variable calculus, just applied one variable at a time: at the very top of a smooth hill, the slope is momentarily flat — the function is neither increasing nor decreasing in any direction. So at a maximum, the partial derivative with respect to every variable (each \(p_i\), and also \(\lambda\)) must independently equal zero. That gives us one equation per variable, and solving all of them together pins down the maximising point exactly. (This condition alone finds any flat point — a peak, a valley, or a saddle; here we know from the shape of entropy, concave and zero at the boundaries, that the one flat interior point we find must be the maximum.)

This machinery, "Lagrange multipliers", is the standard tool in optimisation for handling equality constraints, and it reappears constantly in engineering (power allocation, resource budgets, rate-distortion problems). The mechanical recipe is always the same three steps: (1) form \(\mathcal{L}\) = objective + \(\lambda\)×(constraint written as "\(= 0\)"), (2) set every partial derivative of \(\mathcal{L}\) to zero, (3) solve the resulting system.

We want to maximise \(H = -\sum_{i=1}^q p_i\log_2 p_i\) subject to \(\sum_{i=1}^q p_i = 1\). Introduce a Lagrange multiplier \(\lambda\) and form the Lagrangian:

$$ \mathcal{L} = \sum_{i=1}^{q} \left(-p_i\log_2 p_i\right) + \lambda\left(1 - \sum_{i=1}^{q} p_i\right). $$
(2.2)

Differentiate with respect to a single \(p_i\), remembering \(\dfrac{d}{dx}(x\log_2 x) = \log_2 x + \log_2 e\):

$$ \frac{\partial \mathcal{L}}{\partial p_i} = -\log_2 p_i - \log_2 e - \lambda = 0 \quad\Longrightarrow\quad \log_2 p_i = -(\log_2 e + \lambda). $$
(2.3)

The right-hand side does not depend on \(i\) at all, so every \(p_i\) satisfies the same equation, meaning every \(p_i\) must be equal to the same constant \(c\). Applying the constraint \(\sum_i p_i = 1\) with \(p_i = c\) for all \(i\):

$$ \sum_{i=1}^{q} c = qc = 1 \quad\Longrightarrow\quad c = \frac{1}{q}. $$
(2.4)

So the entropy of a \(q\)-ary source is maximised exactly when every symbol is equally likely, \(p_i = 1/q\), which (as we already know) gives \(H_{\max} = \log_2 q\). This is the same conclusion as Lecture 1's Problem 2.3, reached this time via calculus rather than an inequality.

2.1.2  The binary case, revisited

Specialising to \(q=2\) with \(p_1 = p\) and \(p_2 = 1-p\) recovers the binary entropy function from Lecture 1, \(H(p) = -p\log_2 p - (1-p)\log_2(1-p)\), maximised at \(p=0.5\) with \(H_{\max}=1\) bit. The figure below is the same curve as Lecture 1, included again here because the whole of this lecture is about closing the gap between \(H\) and \(\log_2 q\).

0 0.5 1 0 0.25 0.5 0.75 1 max H = 1 bit at p = 0.5 P(1) = p
Figure 2.1. Binary entropy H(p): a recap from Lecture 1. Efficient coding aims to make \(\bar L\) hug this curve, not the flat line \(\bar L = 1\) that BCD always uses. Drag the slider to read \(H(p)\) at any \(p\).

2.2  Efficient source coding

Coding efficiency and the design rule

We restrict attention to lossless source coding: converting symbols into codewords without throwing away any information, so the original symbols can always be recovered exactly. From Lecture 1, the information rate is

$$ R = R_s H \le R_s \log_2 q, $$
(2.5)

so in general BCD, which always spends \(\log_2 q\) bits per symbol, is not efficient. Coding efficiency compares the information the source truly produces with the rate the code actually uses:

$$ \text{CE} = \eta = \frac{\text{source information rate}}{\text{average source output rate}} = \frac{R_s H}{R_s \bar L} = \frac{H}{\bar L}. $$

Shannon's source coding theorem guarantees that efficient coding can push this arbitrarily close to 100%: for any memoryless source, there exists a code whose average length \(\bar L\) is as close to \(H\) as we like (exactly \(H\) when all self-informations happen to be integers, and within one bit of \(H\) in general, closer still if we code blocks of symbols together). This lecture builds two such codes, Shannon-Fano and Huffman.

2.2.1  The core idea: match codeword length to information content

When symbols are equiprobable, \(p_i = 1/q\), entropy is already maximal, \(H = \log_2 q\), and coding every symbol with \(\log_2 q\) bits (BCD) achieves \(\text{CE} = 100\%\) automatically. The trouble is only when symbols are not equiprobable: then \(H < \log_2 q\) and BCD is wasteful. The fix follows directly from Lecture 1's definition of information content, \(I(m_i) = -\log_2 p_i\):

Design rule. Assign each symbol a codeword whose length \(\ell_i\) is as close as possible to its own information content \(I(m_i) = -\log_2 p_i\): common symbols get short codewords, rare symbols get long ones. Two practical constraints bind this ideal: codeword lengths must be whole numbers of bits, and no codeword may be a prefix of another, so a decoder can tell where one codeword ends and the next begins just by reading bits left to right.

EXAMPLE 2.1 · Same alphabet, two probability profiles

Eight symbols \(m_1,\ldots,m_8\) are BCD-coded with the fixed 3-bit codewords 000 through 111. Compare the coding efficiency when the symbols are equiprobable against when they follow \(p = (0.27, 0.20, 0.17, 0.16, 0.06, 0.06, 0.04, 0.04)\).

2.3  Shannon-Fano coding

Split the sorted list from the top down

Shannon-Fano coding builds a variable-length prefix code by repeatedly splitting the symbol list into two halves of roughly equal total probability.

1
Order symbols by descending probability
3
01
Assign bit 0 to the top half, bit 1 to the bottom half
2
Split the group into two halves of nearly equal total probability
4
Recurse on each half, until only 1 symbol remains
Read off each symbol's codeword top to bottom — the result is automatically a prefix code
Figure 2.2.

2.3.1  Worked example: eight symbols

Take the same eight symbols as Example 2.1, with \(p = (0.27, 0.20, 0.17, 0.16, 0.06, 0.06, 0.04, 0.04)\), already sorted in descending order. Splitting repeatedly to keep each half's probability mass as equal as possible gives:

Symbol \(p_i\) \(I_i \approx\) (bits) Step 1 Step 2 Step 3 Step 4 Codeword
m₁0.271.890000
m₂0.202.320101
m₃0.172.56100100
m₄0.162.64101101
m₅0.064.0611001100
m₆0.064.0611011101
m₇0.044.6411101110
m₈0.044.6411111111
Figure 2.3. Each Coding Steps column is boxed independently: cells sharing a coloured frame are still one subgroup at that step; a cell boxed alone has already split down to a single symbol. Step 1: {m₁,m₂} vs {m₃–m₈}. Step 2: {m₁}, {m₂}, {m₃,m₄}, {m₅–m₈}. Step 3: {m₃}, {m₄}, {m₅,m₆}, {m₇,m₈}. Step 4: {m₅}, {m₆}, {m₇}, {m₈}.

Notice the pattern: the two most probable symbols get 2-bit codewords, the middle two get 3 bits, and the four rarest share 4 bits each, tracking their information content \(I_i\) closely.

Grouping by codeword length, \(0.27+0.20=0.47\) of the mass uses 2 bits, \(0.17+0.16=0.33\) uses 3 bits, and \(0.06+0.06+0.04+0.04=0.20\) uses 4 bits. The average codeword length is:

$$ \bar L = 0.47(2) + 0.33(3) + 0.20(4) = 2.73 \ \text{bits/symbol}. $$
(2.9)

With \(H = 2.6906\) bits/symbol from Example 2.1, the coding efficiency is:

$$ \text{CE} = \frac{H}{\bar L} = \frac{2.6906}{2.73} = 98.56\%. $$
(2.10)

Compare this with the 89.69% of fixed 3-bit codewords on the very same symbols: variable-length coding recovers almost all of the wasted 10.31 percentage points, using only whole numbers of bits per symbol.

2.3.2  A cleaner example: powers-of-two probabilities

Shannon-Fano reaches exactly 100% efficiency whenever every probability is a negative power of two, because then each split can be made perfectly even. Take eight symbols with \(p = (\tfrac12, \tfrac14, \tfrac18, \tfrac1{16}, \tfrac1{32}, \tfrac1{64}, \tfrac1{128}, \tfrac1{128})\):

Symbol \(X_i\) Prob. \(P(X_i)\) \(I\) (bits) Coding steps Codeword bits/symbol
1 2 3 4 5 6 7
A \(\frac{1}{2}\) 1 0 0 1
B \(\frac{1}{4}\) 2 1 0 10 2
C \(\frac{1}{8}\) 3 1 1 0 110 3
D \(\frac{1}{16}\) 4 1 1 1 0 1110 4
E \(\frac{1}{32}\) 5 1 1 1 1 0 11110 5
F \(\frac{1}{64}\) 6 1 1 1 1 1 0 111110 6
G \(\frac{1}{128}\) 7 1 1 1 1 1 1 0 1111110 7
H \(\frac{1}{128}\) 7 1 1 1 1 1 1 1 1111111 7
Figure 2.4. At each bit position the still-active group splits: the coloured cell that turns 0 is the symbol exiting with that codeword; every other still-active symbol carries a 1 into the next column. Because each split is exactly even (probabilities are powers of two), this splitting is a straight comb, and no bit is ever wasted.

Here every codeword length exactly equals its symbol's information content, so \(H = \bar L\) precisely:

$$ H = \bar L = \tfrac12(1)+\tfrac14(2)+\tfrac18(3)+\tfrac1{16}(4)+\tfrac1{32}(5)+\tfrac1{64}(6)+2\cdot\tfrac1{128}(7) = \frac{127}{64} \approx 1.984 \ \text{bits/symbol}, $$
(2.11)

giving \(\text{CE} = 100\%\). By contrast, a fixed 3-bit code for these 8 symbols (needed because \(\lceil\log_2 8\rceil = 3\)) would give only \(\text{CE} = 1.984/3 = 66\%\). This is the cleanest possible illustration of the design rule from Section 2.1: match codeword length to information content, exactly, whenever the probabilities allow it.

2.4  Huffman coding

Merge the least probable items from the bottom up

Huffman coding builds the same kind of variable-length prefix code as Shannon-Fano, but from the bottom up instead of the top down: it repeatedly merges the two least probable symbols into a combined subgroup, rather than repeatedly splitting the whole list in two.

1
Arrange symbols/subgroups by descending probability
3
01
Label the merge: 0 to the higher-probability branch, 1 to the lower
2
Merge the two least probable items into one new subgroup
4
Repeat on the shortened list until one item remains
Read each codeword bottom-up from the tree's root, then reverse the bit order
Figure 2.5.

2.4.1  Worked example: the same eight symbols

Apply Huffman coding to the same alphabet as Section 2.3.1, \(p = (0.27, 0.20, 0.17, 0.16, 0.06, 0.06, 0.04, 0.04)\), so the two methods can be compared directly. Building the tree takes 7 merges (one per internal node); each step below shows the currently active items sorted by probability and the two least probable highlighted for merging, and the new combined item that carries into the next step.

higher probability → bit 0 lower probability → bit 1
start m₁ .27  m₂ .20  m₃ .17  m₄ .16  m₅ .06  m₆ .06  m₇ .04 m₈ .04
↓ merge m₇(0)+m₈(1) → m₇₈ = 0.08
step 1 m₁ .27  m₂ .20  m₃ .17  m₄ .16  m₅ .06 m₆ .06  m₇₈ .08
↓ merge m₅(0)+m₆(1) → m₅₆ = 0.12
step 2 m₁ .27  m₂ .20  m₃ .17  m₄ .16  m₅₆ .12 m₇₈ .08
↓ merge m₅₆(0)+m₇₈(1) → m₅₆₇₈ = 0.20
step 3 m₁ .27  m₂ .20  m₅₆₇₈ .20  m₃ .17 m₄ .16
↓ merge m₃(0)+m₄(1) → m₃₄ = 0.33
step 4 m₃₄ .33  m₁ .27  m₂ .20 m₅₆₇₈ .20
↓ tie at 0.20: merge m₂(0)+m₅₆₇₈(1) → m₂,₅₆₇₈ = 0.40
step 5 m₂,₅₆₇₈ .40  m₃₄ .33 m₁ .27
↓ merge m₃₄(0)+m₁(1) → m₁,₃₄ = 0.60
step 6 m₁,₃₄ .60 m₂,₅₆₇₈ .40
↓ final merge m₁,₃₄(0)+m₂,₅₆₇₈(1) → root = 1.00
step 7 root = 1.00 (tree complete)
Codeword so far · each merge prepends its bit
Symbol\(p_i\)Merge 1Merge 2Merge 3Merge 4Merge 5Merge 6Merge 7
m₁0.27101
m₂0.200010
m₃0.170000000
m₄0.161101001
m₅0.06000001001001100
m₆0.06101011011011101
m₇0.040010101101101110
m₈0.041111111111111111
Figure 2.6. Huffman merges for the eight-symbol example. At each step the two least probable items (highlighted) merge, and the combined item carries into the next step. The table tracks each symbol's codeword: a merge prepends 0 (blue) to every symbol in the higher-probability item and 1 (amber) to every symbol in the lower one, so codewords grow from the last bit back to the first.

The same construction is often easier to read as an actual tree: the root splits repeatedly until every leaf is a single symbol, and a symbol's codeword is just the string of 0s and 1s collected root-to-leaf.

bit 0 (higher-probability branch) bit 1 (lower-probability branch)
01 01 01 01 01 01 01 1.00 .60 .40 .33 .20 .12 .08 m₃.17 · 000 m₄.16 · 001 m₁.27 · 01 m₂.20 · 10 m₅.06 · 1100 m₆.06 · 1101 m₇.04 · 1110 m₈.04 · 1111
Symbol \(p_i\) Codeword bits
m₁0.27012
m₂0.20102
m₃0.170003
m₄0.160013
m₅0.0611004
m₆0.0611014
m₇0.0411104
m₈0.0411114
Figure 2.7. The Huffman tree for this example. Grey boxes are internal (merged) nodes labelled with their combined probability; leaves show the symbol, its probability, and its final codeword. The table collects the codewords: each merge prepends 0 to every symbol under its higher-probability branch and 1 to every symbol under the lower one. The bit-reversal step matters: the tree is built bottom-up, so a naive top-down reading of each path would place bit assignments in the wrong order and could break the prefix property.

Reading a symbol's codeword means walking from the root down to its leaf and recording each 0/1 along the way, then reading that path bottom-up (root-first) is exactly the order the bits are transmitted in — for example m₅ sits inside m₅₆₇₈(bit 1 at the root) → m₅₆₇₈(bit 1) → m₅₆(bit 0) → m₅(bit 0), giving codeword 1100, exactly matching the table in Figure 2.7.

The length distribution is identical to Shannon-Fano's for this example (two 2-bit, two 3-bit, four 4-bit codewords), so:

$$ \bar L_{\text{Huffman}} = 0.47(2) + 0.33(3) + 0.20(4) = 2.73 \ \text{bits/symbol}, \qquad \text{CE} = \frac{2.6906}{2.73} = 98.56\%. $$
(2.12)

The two methods reach the same efficiency here, though in general their individual codeword assignments (which symbol gets which specific bit string) can differ, even when the length distribution matches.

2.5  Comparing the two methods

Greedy split versus provable optimum

Shannon-Fano

Top-down: repeatedly split the sorted list to balance probability mass. Simple to reason about by hand, but the greedy split is not always globally optimal.

Huffman

Bottom-up: repeatedly merge the two least probable items. Provably produces the shortest possible average length among all prefix codes for the given probabilities.

  • Both belong to the class of entropy coding methods: they assign codeword length as close as possible to each symbol's information content.
  • Both produce prefix codes, so the encoded bit stream can always be decoded unambiguously.
  • For an alphabet of \(q\) symbols, the longest codeword in either scheme can be as long as \(q-1\) bits, which becomes impractical (large decoding buffers, high latency) for large alphabets.
  • Huffman coding is provably optimal (minimum \(\bar L\) among prefix codes); Shannon-Fano is usually close but not guaranteed optimal. In practice the coding efficiency of the two is nearly identical, as both worked examples above show.

MATLAB corner

MATLAB's Communications Toolbox ships huffmandict and huffmanenco/huffmandeco, but writing the design rule out by hand builds real intuition. The script below builds a Huffman code from scratch and checks it against Example 2.1's numbers.

huffman_from_scratch.m
MATLAB 
% Build a Huffman code by hand and check it against Example 2.1.
p = [0.27 0.20 0.17 0.16 0.06 0.06 0.04 0.04];
symbols = 1:numel(p);
 
% MATLAB's Communications Toolbox builds the dictionary directly:
dict = huffmandict(symbols, p);
lens = cellfun(@(c) numel(c), dict(:,2));
 
Lbar = sum(p(:) .* lens(:))     % Lbar = 2.7300 bits/symbol
H    = entropy_bits(p)          % H    = 2.6906 bits/symbol (Lecture 1 function)
CE   = H / Lbar                 % CE   = 0.9856  (98.56%)
 
fprintf('Average length %.2f bits, efficiency %.2f%%\n', Lbar, 100*CE);
% Average length 2.73 bits, efficiency 98.56%

Problems with solutions

Work each one before revealing the solution.

PROBLEM 2.1

Apply Huffman coding to the second example of Section 3.2, the eight symbols A through H with probabilities \(\tfrac12,\tfrac14,\tfrac18,\tfrac1{16},\tfrac1{32},\tfrac1{64},\tfrac1{128},\tfrac1{128}\). Compare the resulting average length and efficiency with the Shannon-Fano result found in the notes.

PROBLEM 2.2

A source has five symbols with probabilities \(0.4, 0.2, 0.2, 0.1, 0.1\). (a) Find the entropy \(H\). (b) Design a Shannon-Fano code and find its average length and efficiency. (c) Would a fixed-length BCD code do better or worse, and by how much?

PROBLEM 2.3

For the same five symbols as Problem 2.2, \(p = 0.4, 0.2, 0.2, 0.1, 0.1\), build a Huffman code and compare its average length with the Shannon-Fano result from Problem 2.2.

Summary and key formulas

Maximum entropy
$$H_{\max} = \log_2 q \text{ at } p_i = 1/q$$
Coding efficiency
$$\text{CE} = \eta = H/\bar L$$
Average codeword length
$$\bar L = \sum_i p_i \ell_i$$
Design rule
$$\ell_i \approx I(m_i) = -\log_2 p_i$$
  • Maximum entropy occurs at equiprobable symbols, \(H_{\max} = \log_2 q\), proved here by Lagrange multipliers.
  • Efficient coding assigns codeword length close to each symbol's information content; BCD only does this when symbols are already equiprobable.
  • Shannon-Fano splits the sorted symbol list top-down to balance probability mass at each step.
  • Huffman merges the least-probable symbols bottom-up, and is provably optimal among prefix codes.
  • Both are prefix codes, decodable unambiguously, and both push coding efficiency close to 100% wherever the underlying probabilities allow it.
  • Next, in Lecture 3: sources with memory, where correlation between symbols opens up further gains beyond what a memoryless entropy code can achieve.