Information Theory Fundamentals

How much information does a message carry? Builds information content, entropy, information rate, and why fixed-length codes like BCD waste bandwidth.

Maximum entropy, coding efficiency, and two concrete variable-length codes that push average codeword length toward the entropy bound.

Models correlation between symbols with Markov chains, shows why memory lowers entropy below the memoryless bound, and why coding a memory source directly wastes bits.

Removes redundancy with a predictor, then compresses the mostly-zero residual with run-length coding, contrasted with Shannon-Fano and Huffman coding.

Models the channel (AWGN, the binary symmetric channel) and introduces mutual information: what a receiver actually learns once channel noise is in the picture.

Maximises mutual information into channel capacity: the BSC capacity formula, Shannon's channel coding theorem, the Gaussian channel, and the Shannon-Hartley law.

Whole-module revision (source and channel sides), the 1960s 64 kbps speech-codec case study, and four fully worked exam-style examples with tables and diagrams.