fundamental,interpretability,mathematical
divergences
2026-01-10
We are familiar with KL Divergence, and how its widely accepted and used as a metric for measuring the difference between two distributions. Here we will try to reason why it is preferred and how much interpretable various divergences are. We will mostly center around f-divergences.
The KL divergence was introduced by Kullback and Leibler [1] as a measure of information. It has since become fundamental in information theory [2].
- kullback1951: Kullback, S., & Leibler, R. A. "On Information and Sufficiency." The Annals of Mathematical Statistics, 1951.
- cover2006: Cover, T. M., & Thomas, J. A. Elements of Information Theory. Wiley, 2006.