Cross-entropy

Source: this video
Consider we are encoding a language, consisting of $n$ symbols. Suppose the probability of each symbol is $q_{1}, q_{2}, \dots, q_{n}$ respectively. If we want to minimize the number of bits for each message (in average) for this language, we should represent the $i$ th symbol with

\log_{\frac{1}{2}} q_{i}

bits.
For example, if a symbol $↑$ has probability 50%, we should represent this symbol with $0$ or $1$ , since it is so frequent that it is worth spend one bit in it:
Pasted image 20260922093651.png

The average number of bits needed for a message is, precisely, the Shannon entropy:

\sum_{i} q_{i} \log_{\frac{1}{2}} q_{i}

So far, so good.

But what if the language has changed? Better said, what if the probabilities of each symbol were not correctly computed? Suppose that the real probabilities were $p_{1}, \dots, p_{n}$ . In average, for a message we will be using, instead,

\sum_{i} p_{i} \log_{\frac{1}{2}} q_{i}

bits. And this quantity is called cross-entropy for the two distributions ${q_{i}}$ and ${p_{i}}$ .
Pasted image 20260922094801.png