Intro Many beginners get confused about three core concepts in classification loss: KL divergence, entropy $H(P)$, and cross entropy loss. Common confusing questions: What is the mathematical meaning of entropy $H(P)$? How does the term $\sum p(x)\log p(x)$ connect to entropy $H(P)$? Why do we adopt cross entropy instead of MSE for Softmax classification? Why minimizing cross entropy equals minimizing KL divergence?