  ELEC3203 · Digital Coding & Transmission
  Information Theory Fundamentals

  Information Theory · Lecture notes
  Prof. Celik

ELEC3203 · Digital Coding and Transmission

Lecture 3

Sources with Memory: Markov Models

Lectures 1 and 2 assumed a memoryless source: each symbol independent of the last. Real sources, speech, images, text, are correlated. This lecture models that correlation with Markov chains, shows why it lowers entropy below the memoryless bound, and explains why coding a memory source as if it were memoryless wastes bits.

Learning Objectives

Model a source with memory as a discrete Markov chain and derive its state probabilities
Compute the entropy of a Markov source and compare it with the memoryless bound
Explain why exploiting symbol dependence reduces the achievable code rate

Slides Quiz

Podcast

Sources with Memory: Markov Models cover art

    InstructorProf. Abdulkadir Çelik https://akadircelik.com
    ModuleELEC3203
    TopicsMarkov sources · state entropy · sequence entropy · predictive models
  

    ▸ Markov processes
    ▸ Entropy with memory
    ▸ Memoryless vs. memory
    ▸ Predictive models
  

Lecture 3

This page grows as further lectures are added to the document.

Notation used in this lecture

\(X_i\)

the $i$th state of the Markov source (equivalently, symbol $m_i$)

\(P_i\)

prior (stationary) probability of state $X_i$

\(p_{ij}\)

transition probability $P(S(k){=}j \mid S(k{-}1){=}i)$, from state $X_i$ to $X_j$

\(\Gamma\)

the transition probability matrix, entries $p_{ij}$

\(H_i\)

entropy of the symbols emitted while in state $X_i$, in bits/symbol

\(H\)

overall source entropy, $H=\sum_i P_i H_i$, in bits/symbol

\(G_k\)

average information per symbol carried by length-$k$ message sequences; $G_k\to H$ as $k\to\infty$

\(R\)

information rate, $R = R_s H$, in bits/second

3.1 Redundancy in real-world sources

Real signals are not memoryless

Every source in Lectures 1 and 2 was memoryless: each emitted symbol was statistically independent of every other. That assumption made the maths tractable, but it is rarely true of real signals. Speech, images, and text all exhibit correlation between neighbouring samples or symbols — a sampled speech waveform changes slowly relative to the sampling rate, so consecutive samples tend to sit close together, and English text has strong letter- and word-level correlations ("q" is almost always followed by "u").

Figure 3.1. A correlated source hovers near its recent values: having just emitted $m_5, m_6$, a jump straight to $m_1$ or $m_2$ is far less likely than staying nearby, exactly the interdependency a memoryless model cannot express.

This correlation is a form of redundancy: knowing the recent past lets you narrow down, sometimes drastically, what symbol comes next. Given a source that has just emitted a run of large positive speech samples, the next sample is very unlikely to be a large negative one. Redundancy of this kind should be exploited during coding, because a symbol that is largely predictable from its neighbours carries very little new information.

The practical consequence is important: applying an entropy code (Shannon-Fano, Huffman) designed for a memoryless source directly to a correlated sequence, as if each symbol were independent, throws away the redundancy between symbols and leaves real compression gains on the table. This lecture makes that statement precise.

3.2 Modelling memory: Markov processes

A bounded window of the past, not the entire history

To reason about correlated sources quantitatively we need a model of the dependency, not just a qualitative sense that "nearby symbols are related". The standard model is a Markov process: a source emitting symbols $\{S(k)\}$ indexed by discrete time $k$, in which the current symbol depends on a bounded window of past symbols rather than the entire history.

First-order Markov process: the current symbol depends only on the immediately preceding one, $P\big(S(k)\mid S(k-1)\big)$.
$N$th-order Markov process: the current symbol depends on the $N$ preceding symbols, $P\big(S(k)\mid S(k-1),\ldots,S(k-N)\big)$.

3.2.1 The two-state first-order Markov source

The simplest non-trivial case has just two states, $X_1$ and $X_2$ (equivalently, symbols "1" and "2"). Being in a state and then moving to the next symbol is governed by four transition probabilities, arranged in the matrix $\Gamma$:

$$ \Gamma = \begin{bmatrix} p_{11} & p_{12}\\ p_{21} & p_{22}\end{bmatrix}, \qquad p_{i1}+p_{i2}=1 \ \text{for each row } i. $$

(3.1)

Memory: λ = p₁₁ − p₂₁ = 1 − p₁₂ − p₂₁ = 0.00

−1 alternating0 memorylesssticky +1

Memoryless. Both rows of Γ are equal (p₁₁ = p₂₁), so the next symbol does not depend on the current state.

The source is memoryless exactly when $p_{12}+p_{21}=1$. $\lambda$ is the correlation coefficient between consecutive symbols.

Simulate 0 / 100 symbols

p12 (X1 → X2)0.50

p21 (X2 → X1)0.50

N symbols100

The self-loops follow as $p_{11}=1-p_{12}$ and $p_{22}=1-p_{21}$, and the priors as the stationary probabilities $P_1=p_{21}/(p_{12}+p_{21})$ and $P_2=p_{12}/(p_{12}+p_{21})$. Dragging the memory bar above changes $\lambda$ with $P_1$ and $P_2$ held fixed.

Press Step

1
– P₁ = 0.502
– P₂ = 0.50

Figure 3.2. A two-state Markov source. Whether it has memory depends only on whether the two rows of $\Gamma$ differ, measured by $\lambda = p_{11}-p_{21}$. With $\lambda=0$ the next symbol ignores the current state and the source is memoryless; with $\lambda>0$ the states are "sticky" and runs get longer, and with $\lambda<0$ the source tends to alternate. The defaults $p_{12}=p_{21}=0.5$ are the memoryless case; Example 3.1 uses $p_{12}=0.1$, $p_{21}=0.4$, giving $\lambda=0.5$.

The prior probabilities $P_1=P(X_1)$ and $P_2=P(X_2)$ describe how often the source sits in each state overall (i.e. $P(S(0){=}1)=P_1$); the transition probabilities $p_{ij}=P(S(k){=}j\mid S(k-1){=}i)$ describe how the source moves between states from one symbol to the next.

First-order chains are by far the most widely used, since higher orders quickly become unwieldy: a $q$-symbol alphabet needs a $q\times q$ transition matrix at order 1, but $q^{N+1}$ transition probabilities at order $N$. For a modest 8-symbol alphabet the parameter count explodes within a few orders:

64

order 1

512

order 2

4 096

order 3

32 768

order 4

Figure 3.3. Transition probabilities $q^{N+1}$ needed for an 8-symbol alphabet at order $N$. Longer memory is therefore handled with a simplified parametric model instead: estimate only the conditional mean $E[s(k)\mid \text{past}]$ and use it as a prediction, rather than the full conditional distribution — which is exactly the predictive approach of Section 3.2.2.

3.2.2 Predictive models: an equivalent view

An alternative, and closely related, way to model memory is a predictive model. If $S(k)$ is influenced by its $N$ predecessors, write

$$ S(k) = f\big(S(k{-}1),\ldots,S(k{-}N)\big) + \varepsilon(k), $$

(3.2)

where $f(\cdot)$ is a prediction built from the recent past, and $\varepsilon(k)$, the innovation, is whatever about $S(k)$ could not be predicted from those past values. A common special case is the $N$th-order linear autoregressive (AR) model, $S(k)=\sum_{j=1}^{N} a_j S(k-j) + \varepsilon(k)$, widely used for speech.

Figure 3.4. A linear predictor: past samples, delayed and weighted by coefficients $a_j$, are summed into a prediction $f(\cdot)$; adding the innovation $\varepsilon(k)$ reconstructs $s(k)$ exactly. Coding $\{a_j\}$ and $\{\varepsilon(k)\}$ instead of $\{s(k)\}$ is the basis of predictive speech coding.

Connection: the Kalman filter. The Kalman filter is the state-space form of the same predict-then-correct idea. At each step it predicts the next value from the past, then corrects that prediction using the innovation $\varepsilon(k)$, and for the optimal filter the innovation sequence is white, which is exactly the goal of a good predictive model. An $N$th-order AR source can be written as a state-space model whose state holds the last $N$ samples; with noise-free observations the Kalman predictor then reduces to the linear predictor of Figure 3.4. What it adds is robustness to noisy observations and time-varying statistics, which is why it is common in tracking and channel estimation, while speech codecs use the simpler linear-predictive (AR) route.

The goal of a good predictive model is to make $\{\varepsilon(k)\}$ nearly uncorrelated and zero-mean, so that almost all of the redundancy in $\{S(k)\}$ has been absorbed into the prediction $f(\cdot)$, and only the small, near-independent innovation remains to be coded. We return to this idea in Section 5 and pick it up properly as source coding for memory sources in the next lecture.

Figure 3.5 shows why this pays off: after prediction the innovation is far smaller than the signal, so it needs far fewer bits to code, and widening the window to $N$ past samples helps only while it still captures memory the source actually has.

Sample 40 / 40

Window NN = 1

Figure 3.5. Linear prediction from a window of the last $N$ samples, $\hat s(k)=\sum_{j=1}^{N}a_j s(k-j)$ (shaded band), with the least-squares optimal coefficients for each $N$. Only the innovation $\varepsilon(k)$ (amber) is coded, and every 6 dB of prediction gain saves about 1 bit per sample. The signal is a simulated second-order AR source, $s(k)=1.6\,s(k-1)-0.8\,s(k-2)+e(k)$ with white $e(k)$, so the gain rises up to $N=2$ and then flattens: a window longer than the source's memory adds almost nothing. $N=0$ means no prediction, so $\varepsilon=s$.

3.2.3 Model order versus complexity

First-order Markov chains are by far the most common model in practice, for a concrete reason: a $q$-symbol alphabet needs only a $q\times q$ transition matrix at order 1, but a full $N$th-order model needs $q^{N+1}$ transition probabilities — the table grows exponentially with the memory length, and estimating that many probabilities reliably from real data quickly becomes infeasible. In practice, sources with longer memory are instead handled with a simplified parametric model: rather than estimating the full conditional probability distribution $P\big(S(k)\mid S(k-1),\ldots,S(k-N)\big)$, we estimate just its conditional mean, $E[s(k)\mid s(k-1),\ldots,s(k-N)]$, and use that as the prediction $f(\cdot)$ in Section 2.2 — exactly the predictive-model route, which sidesteps the combinatorial blow-up of high-order Markov chains while still capturing most of the exploitable correlation.

3.3 Entropy of a Markov source

Per state first, then averaged — entropy is computed within each state, then averaged across them

3.3.1 State entropy

While the source sits in state $X_i$, the next symbol is chosen according to the transition probabilities out of that state, $p_{i1},\ldots,p_{iq}$, which sum to 1 just like any probability distribution. So state $X_i$ has its own entropy, exactly the memoryless entropy formula applied to that row of $\Gamma$:

$$ H_i = -\sum_{j=1}^{q} p_{ij}\log_2 p_{ij} \quad \text{(bits/symbol)}. $$

(3.3)

$H_i =$$-\,p_{i1}\log_2 p_{i1}$$-\cdots-p_{ii}\log_2 p_{ii}$$-\cdots-p_{ij}\log_2 p_{ij}$$-\cdots-p_{iq}\log_2 p_{iq}$$=\,-\sum_{j=1}^{q} p_{ij}\log_2 p_{ij}$(3.3)

Figure 3.6. Generalising beyond two states: each state $X_i$ has a prior $P_i$, a self-transition $p_{ii}$, and outgoing transitions $p_{ij}$ to every other state $X_j$; row $i$ of $\Gamma$ is exactly the distribution used to compute $H_i$.

3.3.2 Overall source entropy

The overall entropy is the state entropies weighted by how often the source actually visits each state, i.e. by the stationary probabilities $\pi_i$, the solution of $\pi = \pi\Gamma$ with $\sum_i \pi_i = 1$:

$$ H = \sum_{i=1}^{q} \pi_i H_i = -\sum_{i=1}^{q} \pi_i \sum_{j=1}^{q} p_{ij}\log_2 p_{ij} \quad \text{(bits/symbol)}. $$

(3.4)

For a two-state source, $\pi$ follows directly from the transition probabilities. In the long run the flow from $X_1$ to $X_2$ must balance the flow back, $\pi_1 p_{12} = \pi_2 p_{21}$, and with $\pi_1+\pi_2=1$ this gives

$$ \pi_1 = \frac{p_{21}}{p_{12}+p_{21}}, \qquad \pi_2 = \frac{p_{12}}{p_{12}+p_{21}}. $$

As with any source, once $H$ is known the information rate follows directly from the symbol rate $R_s$: $R = R_s H$ bits/second. The key qualitative feature is this: the more correlated the source (the "stickier" its states, the closer the self-transition probabilities $p_{ii}$ sit to 1), the more concentrated each row of $\Gamma$ becomes on a single outcome, so each $H_i$ shrinks — and hence $H$ shrinks. A highly-correlated source carries less new information per symbol than an uncorrelated one built from the same alphabet and the same marginal symbol probabilities.

3.4 Worked example: a two-state Markov source

State entropies, and how $G_k$ converges to $H$

EXAMPLE 3.1 · A two-state Markov source

A two-state source has transition matrix rows $(p_{11},p_{12})=(0.9,\,0.1)$ and $(p_{21},p_{22})=(0.4,\,0.6)$. State $X_1$ is "sticky" — once there the source stays 90% of the time — and $X_2$ is moderately sticky too, returning to $X_1$ only 40% of the time. The symbol rate is $R_s=1000$ symbols/s. Find the stationary state probabilities, the entropy of each state and the source entropy $H$, and show how the block estimate $G_k$ approaches $H$ as the block length $k$ grows.

1

Stationary distribution

The state entropies are averaged by how often the source visits each state in the long run. These are the stationary probabilities $\pi_1, \pi_2$, the solution of $\pi = \pi\Gamma$ with $\pi_1+\pi_2=1$, and they depend only on $\Gamma$. For two states,

$$ \pi_1 = \frac{p_{21}}{p_{12}+p_{21}} = \frac{0.4}{0.5} = 0.8, \qquad \pi_2 = \frac{p_{12}}{p_{12}+p_{21}} = \frac{0.1}{0.5} = 0.2. $$

Check: one step of the chain leaves $\pi$ unchanged, $0.8(0.9)+0.2(0.4) = 0.8$ and $0.8(0.1)+0.2(0.6) = 0.2$.

No initial distribution is needed. If the source starts with $P(S(0){=}i)=P_i$, the state probabilities drift from $P_i$ towards $\pi_i$ whatever the starting point; for two states the gap shrinks by a fixed factor every symbol, $P(S(n){=}1) - \pi_1 = (1-p_{12}-p_{21})^n\,(P_1-\pi_1)$, and here the factor is 0.5, so the gap halves each step. The starting point changes only the first few symbols, so it does not affect $H$. Convergence requires $|1-p_{12}-p_{21}|<1$: it fails if the source alternates deterministically ($p_{12}=p_{21}=1$) or if a state is absorbing ($p_{12}=0$ or $p_{21}=0$).

2

State entropies

Each is the memoryless entropy formula applied to one row of $\Gamma$:

$$ H_1 = -0.9\log_2 0.9 - 0.1\log_2 0.1 = 0.1368 + 0.3322 = 0.4690 \ \text{bits/symbol}, $$

(3.5)

$$ H_2 = -0.4\log_2 0.4 - 0.6\log_2 0.6 = 0.5288 + 0.4422 = 0.9710 \ \text{bits/symbol}. $$

(3.6)

The sticky state $X_1$ is far more predictable, so it carries less than half the information of $X_2$. Averaging by the stationary probabilities gives the source entropy, and the information rate follows:

$$ H = \pi_1 H_1 + \pi_2 H_2 = 0.8(0.4690) + 0.2(0.9710) = 0.5694 \ \text{bits/symbol}, $$

(3.7)

$$ R = R_s H = 1000 \times 0.5694 = 569.4 \ \text{bits/s}. $$

(3.8)

3

Comparing sequences of growing length

A revealing exercise is to ask: what is the average information content per symbol in a message of length $k$, treating a whole length-$k$ sequence as one joint event? Call this $G_k$ (a different letter from the state entropies $H_i$); it must converge to the true source entropy $H$ as $k\to\infty$, but for small $k$ it overstates $H$ because it has not yet "seen" the correlation between symbols spaced $k$ or more apart.

$k=1$: only the marginal probabilities matter, and with the source in its stationary state these are $\pi_1=0.8,\pi_2=0.2$: $G_1 = -0.8\log_2 0.8 - 0.2\log_2 0.2 = 0.7219$ bits/symbol.
$k=2$: the four length-2 sequences have probabilities $P(11)=\pi_1p_{11}=0.72$, $P(12)=\pi_1p_{12}=0.08$, $P(21)=\pi_2p_{21}=0.08$, $P(22)=\pi_2p_{22}=0.12$. Their joint entropy is 1.2913 bits for the pair, so $G_2=1.2913/2=0.6457$ bits/symbol.
$k=3$: extending the same tree of probabilities gives eight length-3 sequences, built one by one in Figure 3.7 (set $k$ there to see any block length); their joint entropy divided by 3 gives $G_3=0.6202$ bits/symbol.

SequenceProbabilityp−p log₂ p

Step 8 / 8

Block length kk = 3

Figure 3.7. Every length-$k$ sequence of the source in Example 3.1, with its probability built along the chain: the first symbol uses the stationary probability $\pi_i$, and each further symbol multiplies by one transition $p_{ij}$, taken from the row of $\Gamma$ for the state the sequence is currently in. Summing $-p\log_2 p$ over all $2^k$ sequences gives the joint entropy of a block, and dividing by $k$ gives $G_k$. For $k>4$ the 16 most probable sequences are listed and the rest are grouped into one line.

The pattern continues downward: at $k=20$, $G_{20}=0.5770$ bits/symbol, already within 0.008 of the true limit. As sequences get longer, more of the source's memory is captured inside each block, so $G_k$ decreases monotonically toward $H$:

$$ G_1=0.7219 > G_2=0.6457 > G_3=0.6202 > \cdots > G_{20}=0.5770 \ \longrightarrow\ H = 0.5694. $$

(3.9)

In fact for a first-order chain the whole curve is available in closed form: a length-$k$ block costs $G_1$ bits for its first symbol and $H$ bits for each of the remaining $k-1$, so $G_k = \big[G_1 + (k-1)H\big]/k$, which decays towards $H$ like $1/k$.

Figure 3.8. $G_k$ falls steeply from the memoryless estimate $G_1=0.7219$ and flattens onto the true entropy rate $H=0.5694$ bits/symbol. The gap $G_1-H = 0.1525$ bits/symbol is precisely the redundancy a symbol-by-symbol entropy coder throws away.

This is the single most important lesson of the example: treating symbols one at a time (as $G_1$ does) systematically overstates the true entropy of a correlated source, and by a wide margin here — 0.7219 versus 0.5694 bits/symbol, about 27% too high, and every one of those excess bits is paid for in bandwidth.

3.5 Memoryless vs. memory sources, and predictive models

Why not just entropy-code it directly?

Consider two sources that share the same symbol alphabet $\{m_i\}$, the same marginal symbol probabilities $p_i$, and the same symbol rate $R_s$ — but one is memoryless and the other has memory (correlation between symbols). Because correlation only ever reduces uncertainty about what comes next, never increases it:

$$ H^{(\mathrm{ml})} \gg H^{(\mathrm{m})} \qquad\Longrightarrow\qquad R^{(\mathrm{ml})} = R_s H^{(\mathrm{ml})} \ \gg\ R^{(\mathrm{m})} = R_s H^{(\mathrm{m})}. $$

(3.10)

Entropy and information rate are physical properties of the source: they tell you, respectively, the true average information per symbol and the true rate at which that information must be sent, regardless of how the source happens to be coded.

WHY NOT JUST ENTROPY-CODE THE MEMORY SOURCE DIRECTLY?

Applying Shannon-Fano or Huffman coding straight to $\{S(k)\}$, symbol by symbol, only ever sees the one-symbol statistics $p_i$ — it cannot see the correlation between symbols. So the best such a coder can do is get close to $G_1$, the equivalent memoryless entropy — which in the worked example above is $G_1=0.7219$ bits/symbol against a true entropy of only $H^{(\mathrm{m})}=0.5694$, some 27% too high. The resulting data rate then sits near $R_s G_1$, far above the true information rate $R^{(\mathrm{m})}=R_s H^{(\mathrm{m})}$: bits are being spent to encode structure that a smarter scheme would have predicted for free.

The fix, hinted at in Section 2.2, is to remove the redundancy before entropy coding: build a predictor for $S(k)$ from its recent past, subtract it off, and entropy-code only the leftover innovation $\varepsilon(k)$, which is close to memoryless and close to zero-mean. Speech coders do exactly this with linear AR predictors; video coders do the analogous thing across both time (inter-frame) and space (intra-frame). Turning this idea into concrete coding schemes, including run-length coding for the highly skewed binary residual sequences it typically produces, is the subject of the next lecture.

MATLAB corner

The script below reproduces the worked example: it computes $H$ directly from $\Gamma$ and $P$, then simulates the chain and estimates $G_k$ empirically for growing block length $k$, showing the convergence $G_k\to H$.

markov_entropy.m

MATLAB

% Two-state Markov source: exact entropy H, and empirical G_k.
Gamma = [0.9 0.1; 0.4 0.6];   % rows: p(i1) p(i2)
P     = [0.8 0.2];            % prior state probabilities
 
Hrow = -sum(Gamma .* log2(Gamma + (Gamma==0)), 2);   % H_i per state
H    = P * Hrow                                       % H = 0.5694 bits/symbol
 
% Simulate a long realisation and estimate G_k for growing k.
N = 2e6; s = zeros(1,N); s(1) = (rand > P(1)) + 1;
for k = 2:N
  s(k) = (rand > Gamma(s(k-1),1)) + 1;
end
 
for k = [1 2 3 20]
  blocks = reshape(s(1:N-mod(N,k)), k, []).';
  [~, ~, idx] = unique(blocks, 'rows');
  p = accumarray(idx, 1) / size(blocks,1);
  Hk = -sum(p .* log2(p)) / k;
  fprintf('G_%d = %.4f bits/symbol\n', k, Hk);
end
% G_1 ~ 0.72, G_2 ~ 0.65, G_3 ~ 0.62, G_20 ~ 0.58, all -> H = 0.5694

Problems with solutions

Work each one before revealing the solution.

PROBLEM 3.1

A two-state Markov source has $P_1=0.5$, $P_2=0.5$, and $p_{11}=p_{22}=0.5$ (so transitions are equally likely to stay or switch). Find $H_1$, $H_2$, and the overall entropy $H$. What does the result say about correlation in this particular source?

PROBLEM 3.2

A highly correlated two-state source has $P_1=0.5$, $P_2=0.5$, $p_{11}=0.99$, $p_{22}=0.99$ (so $p_{12}=p_{21}=0.01$). (a) Find $H$. (b) Compare with $G_1$, the entropy an entropy coder would see if it only looked at one symbol at a time. (c) If this source is fed at $R_s = 8000$ symbols/second, what is the true information rate $R$, and how does it compare to $R_s G_1$?

Summary and key formulas

State entropy

$$H_i = -\sum_j p_{ij}\log_2 p_{ij}$$

Overall source entropy

$$H = \sum_i P_i H_i$$

Information rate

$$R = R_s H$$

Memory always helps (or ties)

$$H^{(\mathrm{ml})} \ge H^{(\mathrm{m})}$$

Block entropy falls to the limit

$$G_k = \frac{G_1+(k-1)H}{k} \downarrow H$$

Model cost at order $N$

$$q^{N+1} \ \text{parameters}$$

Markov processes model correlated sources via transition probabilities $p_{ij}$ between states; first-order chains, where only the immediately previous symbol matters, are the standard workhorse.
Entropy with memory is computed per-state, then averaged by how often each state occurs: $H=\sum_i P_iH_i$, always $\le \log_2 q$ and typically well below it for correlated sources.
$G_k$, the average information per symbol over length-$k$ blocks, decreases monotonically from $G_1$ toward the true entropy $H$ as $k\to\infty$; $G_1$ alone can badly overstate $H$ for a strongly correlated source.
Never entropy-code a memory source directly: doing so only reaches $G_1$, not the true $H$, so the resulting bit rate can be many times larger than the source's actual information rate.
Predictive models (e.g. linear AR predictors) remove redundancy first, leaving a near-memoryless innovation $\varepsilon(k)$ that entropy coding can compress efficiently — the subject of Lecture 4.