  ELEC3203 · Digital Coding & Transmission
  Information Theory Fundamentals

  Information Theory · Lecture notes
  Prof. Celik

ELEC3203 · Digital Coding and Transmission

Lecture 4

Predictive Coding and Run-Length Coding

Lecture 3 showed that entropy-coding a memory source directly is wasteful. This lecture closes the loop: remove redundancy with a predictor, then compress the mostly-zero residual with run-length coding (RLC) — a lossless scheme with fixed-length output instead of the variable-length output of Shannon-Fano and Huffman.

Learning Objectives

Apply predictive coding to remove redundancy from a correlated source before coding
Design a run-length code for sources with long repeated runs and compute its efficiency
Compare predictive and run-length strategies across different source statistics

Slides Quiz

Podcast

Predictive Coding and Run-Length Coding cover art

    InstructorProf. Abdulkadir Çelik https://akadircelik.com
    ModuleELEC3203
    TopicsRedundancy removal · predictive coding · run-length coding · compression ratio
  

    ▸ Predictive schemes
    ▸ Run-length coding
    ▸ Compression ratio
    ▸ RLC vs. entropy coding
  

Lecture 4

This page grows as further lectures are added to the document.

Notation used in this lecture

\(x(i)\)

binary source sequence (after converting the q-ary source to bits)

\(\hat x(i)\)

predicted value of $x(i)$ from the predictor

\(e(i)\)

residual (prediction error) bit sequence; mostly zeros if prediction is good

\(p\)

$P(\text{“0”})$, probability a residual bit is zero (assumed close to 1)

\(l\)

length of a run of zeros preceding a "1" (or preceding end of block)

\(n\)

fixed length, in bits, of each RLC output codeword

\(N\)

$N = 2^n - 1$, the largest run length the codeword table can index

\(d\)

average input pattern length before coding, in bits

\(C\)

compression ratio, $C = d/n$

4.1 Getting close to the information rate: removing redundancy

Predict first, then entropy-code

Any transmitted rate $R_b$ demands bandwidth and power in proportion to its size, so source coding aims to push $R_b$ as low as possible — ideally down to the source's true information rate. For a memoryless source, Lecture 2's entropy coding (Shannon-Fano, Huffman) already gets $R_b$ close to that limit. Lecture 3 showed why the same trick fails for a source with memory: entropy-coding $\{S(k)\}$ symbol by symbol only ever sees the one-symbol statistics, so it reaches $R_s G_1$, not the true (and much smaller) information rate $R_s H$.

The fix is to attack the redundancy directly, before entropy coding: build a predictor for $S(k)$ from its recent past $\{S(k-i)\}_{i\ge1}$, subtract the prediction off, and code only what is left. Two application areas make this concrete:

Speech coding: consecutive samples are highly correlated; a predictive model $s(k)=\sum_{j=1}^q a_j s(k-j)+\varepsilon(k)$ (Lecture 3, Section 2.2) removes the predictable part, leaving an innovation $\varepsilon(k)$ that is close to uncorrelated and zero-mean. The coder sends the small set of model parameters $a_j$ together with $\varepsilon(k)$, not $s(k)$ itself.
Video coding: a sequence of frames has both inter-frame (temporal, between frames) and intra-frame (spatial, within a frame) redundancy. Removing both leaves a much smaller residual to code.

Whatever the source, once the predictable part is removed, the resulting residual sequence is well suited to a further, purely lossless compression step. When that residual is binary and overwhelmingly zeros, as it typically is after a good predictor, run-length coding (RLC) is the natural tool.

4.2 A predictive scheme with run-length coding

The complete coding chain

Put together, redundancy removal and RLC form a complete coding chain for a memory source:

Figure 4.1. Encoder (top) and decoder (bottom) share an identical predictor. The residual $e(i)=x(i)-\hat{x}(i)$ is mostly zeros when the prediction is good, and RLC compresses it before the channel; the decoder adds $\hat{x}(i)$ back to recover $x(i)$ exactly. The trace below steps a 22-bit example through the chain with the simplest predictor, $\hat{x}(i)=x(i-1)$, so $e(i)=1$ only where $x(i)$ changes, and $n=3$-bit codewords ($N=7$). The zero counter counts consecutive 0s in $e(i)$. When a 1 arrives, RL out is the counter value $l$ written in $n$ bits and the counter resets; if the counter reaches $N$ first, $N$ is sent (111) with no terminating 1. Table A traces the transmitter; Table B traces the receiver, which expands each codeword back to $e(i)$ and runs its own predictor on its decoded output, $\hat{x}(i) = x(i-1)$, to recover $x(i) = e(i) \oplus \hat{x}(i)$ (green).

The predictor and its inverse must be identical at encoder and decoder (or the decoder must receive whatever side information the predictor needs), so that the residual $e(i)$ reconstructs $x(i)$ exactly on the decoding side, after RLC has been undone.

4.3 Run-length coding: building the codeword table

Runs of zeros, indexed into fixed codewords

RLC replaces a run of zeros followed by a one (or, for the very longest runs, a run with no terminating one at all) with a single, fixed-length codeword of $n$ bits. Table entries are indexed by the run length $l$, covering $0 \le l \le N-1$ where $N=2^n-1$:

length of 0-run, $l$	encoder input pattern	input bits	output codeword ($n=3$ bits)
0	1	1	000
1	01	2	001
2	001	3	010
3	0001	4	011
4	00001	5	100
5	000001	6	101
6	0000001	7	110
7	0000000 no terminating 1	7	111

Figure 4.2. The codeword table for $n=3$, so $N=2^3-1=7$. Input pattern length is $\min\{N, l+1\}$ bits: variable, from 1 bit (a lone "1") up to $N$ bits (a maximal run with no terminating one). Output is always exactly $n$ bits, and the codeword is simply $l$ written in binary. For general $n$ the table has the same shape, with $2^n$ rows.

The whole scheme rests on one assumption: the input bit stream is mostly "0"s, i.e. $p=P(\text{“0”})$ is close to 1. Then short runs (small $l$) dominate, so on average the encoder replaces a variable-length input pattern (often much longer than $n$ bits) with a fixed, small $n$-bit codeword — that trade is where the compression comes from.

A scanned fax page is the classic case. Most pixels are white, so with white as "0" and black as "1" each scan line is a long run of zeros broken by a few ones. Figure 4.3 codes a small page line by line with the table of Figure 4.2, first on the pixels directly and then on the output of the simplest predictor from Figure 4.1.

Codewords sent for this line

Figure 4.3. Run-length coding a 57×9-pixel fax page with the $n=3$ table of Figure 4.2 (white = 0, black = 1). Click any line of the page to select it, then step or play the scan: the strip below the page shows the bits $e(i)$ actually coded, each bracket is one run, and its codeword appears when the run closes. Coding the pixels directly costs a full 3-bit codeword for every black pixel, so here it sends more bits than the page holds ($C<1$). Coding only the changes, as the predictor $\hat x(i)=x(i-1)$ of Figure 4.1 does, leaves a 1 only at each edge of a black stroke, so the runs are longer and $C>1$. A run still open at the end of a line is sent as if terminated; the decoder knows the line is 57 pixels wide and drops the extra bit.

4.4 RLC efficiency

Average input length and compression ratio

Each output codeword is exactly $n$ bits, so all that varies is how many input bits that codeword replaces. Averaging the input pattern length $l+1$ over the geometric distribution of run lengths (probability $p^l(1-p)$ of a run of exactly $l$ zeros, plus probability $p^N$ of the maximal all-zero run):

$$ d = \sum_{l=0}^{N-1} (l+1)\,p^l(1-p) \;+\; N\,p^N \;=\; 1 + p + p^2 + \cdots + p^{N-1} \;=\; \frac{1-p^N}{1-p}. $$

(4.1)

Written out term by term, each term pairs one input pattern with its length and its probability:

$$ d = \underbrace{1\cdot(1-p)}_{\texttt{1}} \;+\; \underbrace{2\cdot p(1-p)}_{\texttt{01}} \;+\; \underbrace{3\cdot p^2(1-p)}_{\texttt{001}} \;+\; \cdots \;+\; \underbrace{N\cdot p^{N-1}(1-p)}_{\texttt{0}\cdots\texttt{01}\ (N\text{ bits})} \;+\; \underbrace{N\cdot p^{N}}_{\texttt{0}\cdots\texttt{0}\ (N\text{ zeros})} $$

$p^l$: the first $l$ bits are all zeros, each with probability $p$.
$(1-p)$: the next bit is a one, which ends the run. Together, $p^l(1-p)$ is the probability of the pattern of $l$ zeros followed by a one, a sequence of $l+1$ bits, so it is weighted by its length $l+1$. This covers $l = 0, 1, \ldots, N-1$.
$N\,p^N$: the last term is the pattern with all $N$ bits zero. No one follows, because the encoder sends $N$ as soon as the count reaches $N$, so the pattern is $N$ bits long and has probability $p^N$.

To reach the closed form, expand $(1-p)$ and shift the index of the second sum. Most terms cancel:

$$ \sum_{l=0}^{N-1} (l+1)\,p^l(1-p) = \sum_{l=0}^{N-1} (l+1)\,p^l - \sum_{l=1}^{N} l\,p^{l} = \sum_{l=0}^{N-1} p^l \;-\; N\,p^N. $$

The $-N\,p^N$ cancels the all-zeros term, leaving $d = 1 + p + p^2 + \cdots + p^{N-1}$. This is a finite geometric series with first term 1, common ratio $p$ and $N$ terms. Its sum $S = \sum_{l=0}^{N-1} p^l$ follows from subtracting $pS$: every term except the first and last cancels, so $S - pS = 1 - p^N$, and

$$ d = \sum_{l=0}^{N-1} p^l = \frac{1-p^N}{1-p}. $$

The compression ratio is then the average input length divided by the fixed output length:

$$ C = \frac{d}{n} = \frac{1-p^N}{n(1-p)}. $$

(4.2)

EXAMPLE 4.1 · Average run length and compression ratio

Run-length coding is applied to a residual bit stream with $P(0)=p=0.95$, using $n=5$-bit codewords, so the longest run is $N=2^5-1=31$. Find the average number of input bits $d$ covered by one codeword and the compression ratio $C$.

4.4.1 RLC re-examined: variable in, fixed out

It is worth contrasting RLC's shape with Shannon-Fano and Huffman coding directly:

Run-length coding

Input pattern length is variable, from 1 bit up to $N=2^n-1$ bits; output codeword length is always fixed at $n$ bits.

Shannon-Fano / Huffman

Input symbol length is fixed (one symbol at a time); output codeword length is variable, matched to each symbol's probability.

Despite this apparent reversal, both are governed by the same underlying principle: a rare event should map to a large codeword (relatively speaking), and a probable event to a small one. The cleanest way to see this is to define $\text{ratio} = \dfrac{\text{output length}}{\text{input length}}$: a long run of zeros (high probability under the "mostly zero" assumption) maps to a small ratio, while a short run (lower probability) maps to a larger ratio. RLC, Shannon-Fano, and Huffman are all, in this sense, lossless (entropy) encodings.

n = 4 bits (N = 15)

    Codeword length
    
    n = 4

    P(“0”)
    
    p = 0.90

Figure 4.4. The run-length table, one row per input pattern. Each row reads left to right: the run length $l$, the input pattern it stands for ($l$ zeros then a one), the codeword it becomes, and the probability $P$ of that pattern. The input bar grows with the pattern's length, but the codeword is always $n$ bits. Bar colour shows how likely the pattern is: blue is common, red is rare. Move the sliders to change $n$ and $p$; the line below the table gives the average input length $d$ from Eq. (4.1) and the compression ratio $C = d/n$ from Eq. (4.2). Compression works when the long input patterns are likely, which needs $p$ close to 1. This is the opposite of Huffman coding, which keeps the input fixed at one symbol and varies the codeword length.

4.5 Comparing RLC with Shannon-Fano and Huffman

Which side is held to a fixed length?

	Run-length coding	Shannon-Fano	Huffman
Input length	variable (1 to $N$ bits)	fixed (one symbol)	fixed (one symbol)
Output length	fixed ($n$ bits)	variable, ≈ $-\log_2 p_i$	variable, ≈ $-\log_2 p_i$
Source it targets	binary, heavily skewed ($p\to1$)	general $q$-ary, any distribution	general $q$-ary, any distribution
Construction	fixed lookup table, indexed by run length	top-down recursive splitting	bottom-up greedy merging
Optimality	not optimal in general, but very cheap to run	usually close to optimal	provably optimal among prefix codes
Typical use	residual after prediction; fax; simple image formats	quick hand-built codes, teaching	general-purpose lossless compression (e.g. inside JPEG, ZIP)

Figure 4.5. All three are lossless entropy-coding methods; they differ mainly in which side of the mapping (input or output) is held to a fixed length.

All three are lossless: the original sequence can always be reconstructed exactly.
Shannon-Fano and Huffman are suited to a general $q$-ary memoryless source with an arbitrary probability distribution; RLC is specialised to a binary source whose bits are heavily skewed toward "0" (or "1").
RLC is the natural choice after a predictor has already removed the correlation from a memory source, since the resulting residual is typically binary and mostly zero — exactly the regime RLC is built for.
Choosing $n$ trades off compression ratio against worst-case pattern length: a larger $n$ allows longer runs to be captured (raising $N$ exponentially) but a single "1" still costs a full $n$-bit codeword, so $n$ should be matched to how skewed $p$ really is.

MATLAB corner

The script below generates a skewed binary residual, applies run-length coding by hand, and checks the empirical compression ratio against the closed-form formula.

rlc_from_scratch.m

MATLAB

% Run-length code a skewed binary sequence and check C against theory.
p = 0.95; n = 5; N = 2^n - 1;
M = 2e5; e = double(rand(1,M) > p);   % e(i) = 1 with prob (1-p)
 
runs = []; run = 0;
for i = 1:M
  if e(i) == 0 && run < N
    run = run + 1;
  else
    runs(end+1) = run + (e(i)==1);  % input pattern length
    run = 0;
  end
end
 
d_hat = mean(runs);              % empirical average input length
C_hat = d_hat / n;
 
d_theory = (1 - p^N) / (1 - p);  % closed form, Section 4
C_theory = d_theory / n;
 
fprintf('empirical C = %.3f, theoretical C = %.3f\n', C_hat, C_theory);
% empirical C ~ 3.18, theoretical C = 3.18

Problems with solutions

Work each one before revealing the solution.

PROBLEM 4.1

A binary residual sequence has $P(\text{“0”})=0.9$. Using $n=4$-bit RLC codewords ($N=2^4-1=15$), find the average input pattern length $d$ and the compression ratio $C$.

PROBLEM 4.2

A residual sequence is even more skewed, $P(\text{“0”})=0.98$, and $n=6$ bits is used ($N=63$). (a) Find $C$. (b) If the pre-RLC bit rate is 500,000 bits/s, what is the bit rate after RLC?

Summary and key formulas

Codeword table span

$$N = 2^n - 1$$

Average input length

$$d = \frac{1-p^N}{1-p}$$

Compression ratio

$$C = d/n$$

RLC's shape

variable in → fixed $n$-bit out

Removing redundancy first (a predictor) turns a memory source's residual into something close to memoryless, and, for binary residuals, heavily skewed toward zero.
Run-length coding maps a run of up to $N=2^n-1$ zeros (plus a terminating bit) onto a fixed $n$-bit codeword, trading a variable-length input for a fixed-length output.
Compression ratio $C=d/n=(1-p^N)/[n(1-p)]$ grows quickly as $p\to1$; a well-chosen $n$ balances this against the worst-case cost of a lone "1".
RLC, Shannon-Fano, and Huffman are all lossless entropy-coding methods obeying the same rule — rare patterns cost more, relative to their length, than common ones — they simply differ in which side (input or output) is held fixed.
Together, Lectures 1–4 now cover the complete source-coding toolkit for both memoryless and memory sources; Lecture 5 turns to what happens to information once it crosses a noisy channel.

	Run-length coding	Shannon-Fano	Huffman
Input length	variable (1 to \(N\) bits)	fixed (one symbol)	fixed (one symbol)
Output length	fixed (\(n\) bits)	variable, ≈ \(-\log_2 p_i\)	variable, ≈ \(-\log_2 p_i\)
Source it targets	binary, heavily skewed (\(p\to1\))	general \(q\)-ary, any distribution	general \(q\)-ary, any distribution
Construction	fixed lookup table, indexed by run length	top-down recursive splitting	bottom-up greedy merging
Optimality	not optimal in general, but very cheap to run	usually close to optimal	provably optimal among prefix codes
Typical use	residual after prediction; fax; simple image formats	quick hand-built codes, teaching	general-purpose lossless compression (e.g. inside JPEG, ZIP)