Information Theory & Entropy

By the end, you can

• Compute self-information I(x) = log2(1/p) for a single event given its probability.
• Explain why halving a probability adds exactly one bit of surprise.
• Compute entropy H = -sum(p * log2 p) for a small discrete distribution by hand.
• State why the uniform distribution maximizes entropy and certainty gives zero entropy.
• Explain Shannon's source-coding theorem and what "entropy is the compression floor" means.
• Describe channel capacity and state the consequence of transmitting above or below it.
• Define KL divergence as the "wrong-model tax" and explain its role in training ML models.
• Distinguish Shannon information (surprise) from semantic meaning.