Kullback–Leibler divergence
467527
218431765
2008-06-10T16:59:34Z
Kevin Baas
9076
eliminate redundant indefinite article.
In [[probability theory]] and [[information theory]], the '''Kullback–Leibler divergence'''<ref>S. Kullback and R. A. Leibler (1951) On information and sufficiency, ''Annals of Mathematical Statistics'' '''22''':79-86.</ref><ref>S. Kullback (1959) ''Information theory and statistics'' (John Wiley and Sons, NY).</ref><ref>S. Kullback (1987) The Kullback-Leibler distance, ''The American Statistician'' '''41''':340-341.</ref> (also '''information divergence''', '''information gain''', or '''relative entropy''') is a [[non-commutative]] measure of the difference between two probability distributions P and Q. KL measures the expected difference in the number of bits required to code samples from P when using a code based on P, and when using a code based on Q. Typically ''P'' represents the "true" distribution of data, observations, or a precise calculated theoretical distribution. The measure ''Q'' typically represents a theory, model, description, or approximation of ''P''.
It is a special case of a broader class of divergences called [[f-divergence|''f''-divergences]]. Although it is often intuited as a [[Metric (mathematics)|distance metric]], the KL divergence is '''not''' a true [[Metric (mathematics)|metric]] since it is not symmetric (hence 'divergence' rather than 'distance').
==Definition==
For probability distributions ''P'' and ''Q'' of a [[discrete random variable]]
the K–L divergence of ''Q'' from ''P'' is defined to be
:<math>D_{\mathrm{KL}}(P\|Q) = \sum_i P(i) \log \frac{P(i)}{Q(i)} \!</math>
For distributions ''P'' and ''Q'' of a [[continuous random variable]] the summations give way to integrals, so that
:<math>D_{\mathrm{KL}}(P\|Q) = \int_{-\infty}^{\infty} p(x) \log \frac{p(x)}{q(x)} \; dx \!</math>
where ''p'' and ''q'' denote the densities of ''P'' and ''Q''.
Generalising the two examples above, if <math>dP = p d\mu </math> and <math>dQ = q d\mu </math> are probability [[measure (mathematics)|measure]]s over a set ''X'', [[absolutely continuous]] with respect to a measure <math>\mu</math> then the Kullback-Leibler divergence from ''P'' to ''Q'' is defined as
:<math> D_{\mathrm{KL}}(P\|Q) = \int_X p \log \frac{p}{q} \;d\mu \!</math>
provided the expression on the right hand side exists. If ''P'' is absolutely continuous with respect to ''Q'',
(which is necessary if <math> D_{\mathrm{KL}}(P\|Q)</math> is to be finite) then
<math>\frac{p}{q} = \frac{dP}{dQ} </math>, is the [[Radon-Nikodym derivative]] of ''P''
with respect to ''Q'' and the expression becomes
:<math> D_{\mathrm{KL}}(P\|Q) = \int_X \log \frac{dP}{dQ} \; dP
= \int_X \frac{dP}{dQ} \log\frac{dP}{dQ}\; dQ</math>,
which we recognise as the entropy of P relative to Q.
Likewise if ''Q'' is absolutely continuous with respect to ''P'' then
<!-- Please leave dQ/dP this way round; it is intentional, and it is not a mistake -->
:<math> D_{\mathrm{KL}}(P\|Q) = -\int_X \log \frac{d Q}{d P} \; dP \!</math>
In either case, we see that the Kullback Leibler divergence does not depend on the auxiliary
measure <math>\mu</math>
The logarithms in these formulae are taken to base 2 if information is measured in units of [[bit]]s, or to base ''e'' if information is measured in [[nat (information)|nat]]s. Most formulas involving the KL divergence hold irrespective of log base.
==Motivation, properties and terminology==
In information theory, the [[Kraft-McMillan theorem]] establishes that any directly-decodable coding scheme for coding a message to identify one value ''x<sub>i</sub>'' out of a set of possibilities ''X'' can be seen as representing an implicit probability distribution ''q(x<sub>i</sub>)'' = 2<sup>-''l<sub>i</sub>''</sup> over ''X'', where ''l<sub>i</sub>'' is the length of the code for ''x<sub>i</sub>'' in bits. Therefore, KL divergence can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution ''Q'' is used, compared to using a code based on the true distribution ''P''.
It can be seen from the definition of the Kullback-Leibler divergence that
:<math>
\begin{matrix}
D_{\mathrm{KL}}(P\|Q) & = & -\sum_x p(x) \log q(x)& + & \sum_x p(x) \log p(x) \\
& = & H(P,Q) & - & H(P)\, \!
\end{matrix}</math>
where ''H''(''P'',''Q'') is called the [[cross entropy]] of ''P'' and ''Q'', and ''H''(''P'') is the [[information entropy|entropy]] of ''P''.
The Kullback–Leibler divergence is always non-negative,
:<math>D_{\mathrm{KL}}(P\|Q) \geq 0, \,</math>
a result known as [[Gibbs' inequality]], with ''D''<sub>KL</sub>(''P''||''Q'') zero if and only if ''P'' = ''Q''. The entropy ''H(P)'' thus sets a minimum value for the cross-entropy ''H(P,Q)'', the expected number of bits required when using a code based on ''Q'' rather than ''P''; and the KL divergence therefore represents the expected number of extra bits that must be transmitted to identify a value ''x'' drawn from ''X'', if a code is used corresponding to the probability distribution ''Q'', rather than the "true" distribution ''P''.
Originally introduced by [[Solomon Kullback]] and [[Richard Leibler]] in [[1951]] as the '''directed divergence''' between two distributions, it is not the same as a [[divergence]] in [[vector calculus|calculus]]. One might be tempted to call it a "[[metric space|distance metric]]" on the space of probability distributions, but this would not be correct as the Kullback-Leibler divergence is not [[symmetric]],
:<math>D_{\mathrm{KL}}(P\|Q) \neq D_{\mathrm{KL}}(Q\|P).</math>
Moreover, ''D''<sub>KL</sub>(''P''||''Q'') does not satisfy the [[triangle inequality]].
Following [[Alfréd Rényi|Renyi]] (1961), the term is sometimes also called the '''information gain''' about ''X'' achieved if ''P'' can be used instead of ''Q''. It is also called the '''relative entropy''', for using ''Q'' instead of ''P''.
The Kullback–Leibler divergence remains well-defined for continuous distributions, and furthermore is invariant under parameter transformations. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory (such as [[self-information]] or [[Shannon entropy]]), which can become undefined or negative for non-discrete probabilities.
==Relation to other quantities of information theory==
Many of the other quantities of information theory can be interpreted as applications of the KL divergence to specific cases.
The [[self-information]],
:<math>I(m) = D_{\mathrm{KL}}(\delta_{im} \| \{ p_i \}), </math>
is the KL divergence of the probability distribution ''P(i)'' from a [[Kronecker delta]] representing certainty that ''i=m'' — i.e. the number of extra bits that must be transmitted to identify ''i'' if only the probability distribution ''P(i)'' is available to the receiver, not the fact that ''i=m''.
The [[mutual information]],
:<math>\begin{align}I(X;Y) & = D_{\mathrm{KL}}(P(X,Y) \| P(X)P(Y) ) \\
& = \mathbb{E}_X \{D_{\mathrm{KL}}(P(Y|X) \| P(Y) ) \} \\
& = \mathbb{E}_Y \{D_{\mathrm{KL}}(P(X|Y) \| P(X) ) \}\end{align} </math>
is the KL divergence of the product ''P(X)P(Y)'' of the two [[marginal probability]] distributions from the joint probability distribution ''P(X,Y)'' — i.e. the expected number of extra bits that must be transmitted to identify ''X'' and ''Y'' if they are coded using only their marginal distributions instead of the joint distribution. Equivalently, if the joint probability ''P(X,Y)'' ''is'' known, it is the expected number of extra bits that must on average be sent to identify ''Y'' if the value of ''X'' is not already known to the receiver.
The [[Shannon entropy]],
:<math>\begin{align}H(X) & = \mathrm{(i)} \, \mathbb{E}_x \{I(x)\} \\
& = \mathrm{(ii)} \log N - D_{\mathrm{KL}}(P(X) \| P_U(X) )\end{align}</math>
is the number of bits which would have to be transmitted to identify ''X'' from ''N'' equally likely possibilities, ''less'' the KL divergence of the uniform distribution ''P''<sub>U</sub>''(X)'' from the true distribution ''P(X)'' — i.e. ''less'' the expected number of bits saved, which would have had to be sent if the value of ''X'' were coded according to the uniform distribution ''P''<sub>U</sub>''(X)'' rather than the true distribution ''P(X)''.
The [[conditional entropy]],
:<math>\begin{align}H(X|Y) & = \log N - D_{\mathrm{KL}}(P(X,Y) \| P_U(X) P(Y) ) \\
& = \mathrm{(i)} \,\, \log N - D_{\mathrm{KL}}(P(X,Y) \| P(X) P(Y) ) - D_{\mathrm{KL}}(P(X) \| P_U(X)) \\
& = H(X) - I(X;Y) \\
& = \mathrm{(ii)} \, \log N - \mathbb{E}_Y \{ D_{\mathrm{KL}}(P(X|Y) \| P_U(X)) \}\end{align}</math>
is the number of bits which would have to be transmitted to identify ''X'' from ''N'' equally likely possibilities, ''less'' the KL divergence of the product distribution ''P''<sub>U</sub>''(X) P(Y)'' from the true joint distribution ''P(X,Y)'' — i.e. ''less'' the expected number of bits saved which would have had to be sent if the value of ''X'' were coded according to the uniform distribution ''P''<sub>U</sub>''(X)'' rather than the conditional distribution ''P(X|Y)'' of ''X'' given ''Y''.
the [[cross entropy]] between two [[probability distribution]]s measures the average number of [[bit]]s needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution <math>q</math>, rather than the "true" distribution <math>p</math>.
The cross entropy for two distributions <math>p</math> and <math>q</math> over the same [[probability space]] is thus defined as follows:
:<math>\mathrm{H}(p, q) = \mathrm{E}_p[-\log q] = \mathrm{H}(p) + D_{\mathrm{KL}}(p \| q)\!</math>,
==KL divergence and Bayesian updating==
In [[Bayesian statistics]] the KL divergence can be used as a measure of the information gain in moving from a [[prior distribution]] to a [[posterior distribution]]. If some new fact ''Y=y'' is discovered, it can be used to update the probability distribution for ''X'' from ''p''(''x''|I) to a new posterior probability distribution ''p''(''x''|''y'') using [[Bayes' theorem]]:
:<math>p(x|y) = \frac{p(y|x) p(x|I)}{p(y|I)}</math>
This distribution has a new entropy, ''H(p(x|y))'' = -∑ ''p''(''x''|''y'') log ''p''(''x''|''y''), which may be less than or greater than the original entropy ''H(p(x|I))''.
However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on ''p''(''x''|I) instead of a new code based on ''p''(''x''|''y'') would have added an expected number of bits
:<math>D_{\mathrm{KL}}(p(x|y)\|p(x|I)) = \sum p(x|y) \log \frac{p(x|y)}{p(x|I)}</math>
to the message length. This therefore represents the amount of useful information, or information gain, about ''X'', that we can estimate has been learned by discovering ''Y=y''.
If a further piece of data, ''Y''<sub>2</sub>=''y''<sub>2</sub>, subsequently comes in, the probability distribution for ''x'' can be updated further, to give a new best guess ''p''(''x''|''y''<sub>1</sub>,''y''<sub>2</sub>). If one reinvestigates the information gain for using ''p''(''x''|''y''<sub>1</sub>) rather than ''p''(''x''|I), it turns out that it may be either greater or less than previously estimated:
:<math>\sum_x p(x|y_1,y_2) \log \frac{p(x|y_1,y_2)}{p(x|I)}</math> may be < = or > than <math>\sum_x p(x|y_1) \log \frac{p(x|y_1)}{p(x|I)}</math>
and so the combined information gain does ''not'' obey the triangle inequality:
:<math>D_{\mathrm{KL}}(p(x|y_1,y_2)\|p(x|I))</math> may be < = or > than <math>D_{\mathrm{KL}}(p(x|y_1,y_2)\|p(x|y_1)) + D_{\mathrm{KL}}(p(x|y_1)\|p(x|I))</math>
All one can say is that on ''average'', averaging using ''p''(''y''<sub>2</sub>|''y''<sub>1</sub>,''x''), the two sides will average out.
===Bayesian experimental design===
A common goal in [[Bayesian experimental design]] is to maximise the expected KL divergence between the prior and the posterior. When posteriors are approximated to be Gaussian distributions, a design maximising the expected KL divergence is called [[Optimal design#D-optimality|Bayes d-optimal]].
== Discrimination information ==
The Kullback–Leibler divergence ''D''<sub>KL</sub>( ''p''(''x''|''H''<sub>1</sub>) || ''p''(''x''|''H''<sub>0</sub>) ) can also be interpreted as the expected '''discrimination information''' for ''H''<sub>1</sub> over ''H''<sub>0</sub>: the mean information per sample for discriminating in favour of a hypothesis ''H''<sub>1</sub> against a hypothesis ''H''<sub>0</sub>, when hypothesis ''H''<sub>1</sub> is true. Another name for this quantity, given to it by [[I.J. Good]], is the expected [[weight of evidence]] for ''H''<sub>1</sub> over ''H''<sub>0</sub> to be expected from each sample.
The expected weight of evidence for ''H''<sub>1</sub> over ''H''<sub>0</sub> is '''not''' the same as the information gain expected per sample about the probability distribution ''p''(''H'') of the hypotheses,
:''IG'' = ''D''<sub>KL</sub>( ''p''(''H''|x) || ''p''(''H''|I) ) <math>\neq</math> ''D''<sub>KL</sub>( ''p''(''x''|''H''<sub>1</sub>) || ''p''(''x''|''H''<sub>0</sub>) ) .
Either of the two quantities can be used as a [[utility function]] in Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.
On the entropy scale of ''information gain'' there is very little difference between near certainty and absolute certainty -- coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the [[logit]] scale implied by weight of evidence, the difference between the two is enormous - infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the [[Riemann hypothesis]] is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of [[loss function]] for uncertainty are ''both'' useful, according to how well each reflects the particular circumstances of the problem in question.
===Principle of minimum discrimination information===
The idea of Kullback–Leibler divergence as discrimination information led Kullback to propose the Principle of '''Minimum Discrimination Information''' (MDI): given new facts, a new distribution ''f'' should be chosen which is as hard to discriminate from the original distribution ''f''<sub>0</sub> as possible; so that the new data produces as small an information gain ''D''<sub>KL</sub>( ''f'' || ''f''<sub>0</sub> ) as possible.
For example, if one had a prior distribution ''p''(''x'',''a'') over ''x'' and ''a'', and subsequently learnt the true distribution of ''a'' was ''u''(''a''), the Kullback–Leibler divergence between the new joint distribution for ''x'' and ''a'', ''q''(''x''|''a'') ''u''(''a''), and the earlier prior distribution would be:
:<math>D_\mathrm{KL}(q(x|a)u(a)||p(x,a)) = \mathbb{E}_{u(a)}\{D_\mathrm{KL}(q(x|a)||p(x|a))\} + D_\mathrm{KL}(u(a)||p(a)),</math>
i.e. the sum of the KL divergence of ''p''(''a'') the prior distribution for ''a'' from the updated distribution ''u''(''a''), plus the expected value (using the probability distribution ''u''(''a'')) of the KL divergence of the prior conditional distribution ''p''(''x''|''a'') from the new conditional distribution ''q''(''x''|''a''). This is minimised if ''q''(''x''|''a'') = ''p''(''x''|''a'') over the whole support of ''u''(''a''); and we note that this result incorporates Bayes' theorem, if the new distribution ''u''(''a'') is in fact a δ function representing certainty that ''a'' has one particular value.
MDI can be seen as an extension of [[Laplace]]'s [[Principle of Insufficient Reason]], and the [[Principle of Maximum Entropy]] of [[E.T. Jaynes]]. In particular, it is the natural extension of the principle of maximum entropy from discrete to continuous distributions, for which Shannon entropy ceases to be so useful (see ''[[differential entropy]]''), but the KL divergence continues to be just as relevant.
In the engineering literature, MDI is sometimes called the '''Principle of Minimum Cross-Entropy''' (MCE) or '''Minxent''' for short. This is not entirely helpful. Minimising the KL divergence of ''m'' from ''p'' with respect to ''m'' is equivalent to minimising the cross-entropy of ''p'' and ''m'', since
:<math>H(p,m) = H(p) + D_{\mathrm{KL}}(p\|m),</math>
which is appropriate if one is trying to choose a least 'brain-damaged' approximation to ''p''. However, this is just as often ''not'' the task one is trying to achieve. Instead, just as often it is ''m'' that is some fixed prior reference measure, and ''p'' that one is attempting to optimise by minimising ''D''<sub>KL</sub>(''p''||''m'') subject to some constraint. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be ''D''<sub>KL</sub>(''p''||''m''), rather than ''H''(''p'',''m'').
==Relationship to available work==
[[Image:ArgonKLdivergence.png|thumb|220px|right|Pressure versus volume plot of available work from a mole of Argon gas relative to ambient, calculated as T<sub>o</sub> times KL divergence.]]
[[Surprisal]]s<ref>Myron Tribus (1961) ''Thermodynamics and thermostatics'' (D. Van Nostrand, New York)</ref> add where probabilities multiply. The surprisal for an event of probability p is defined as s≡k''ln''[1/p]. If k is {1,1/''ln''2,1.38×10<sup>-23</sup>} then surprisal is in {nats, bits, or J/K} so that, for instance, there are N bits of surprisal for landing all "heads" on a toss of N coins.
Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the ''average-surprisal'' S ([[entropy]]) for a given set of control parameters (like pressure P or volume V). This constrained [[Maximum_entropy|entropy maximization]], both classically<ref>E. T. Jaynes (1957) [http://bayes.wustl.edu/etj/articles/theory.1.pdf Information theory and statistical mechanics], ''Physical Review'' '''106''':620</ref> and quantum mechanically<ref>E. T. Jaynes (1957) [http://bayes.wustl.edu/etj/articles/theory.2.pdf Information theory and statistical mechanics II], ''Physical Review'' '''108''':171</ref>, minimizes [[Josiah_Willard_Gibbs|Gibbs]] availability in entropy units<ref>J.W. Gibbs (1873) A method of geometrical representation of thermodynamic properties of substances by means of surfaces, reprinted in ''The Collected Works of J. W. Gibbs, Volume I Thermodynamics'', ed. W. R. Longley and R. G. Van Name (New York: Longmans, Green, 1931) footnote page 52.</ref> A≡-k''ln''Z where Z is a constrained multiplicity or [[Partition_function_(statistical_mechanics)|partition function]].
When temperature T is fixed, free-energy (T times A) is also minimized. Thus if T, V and number of molecules N are constant, the [[Helmholtz free energy]] F≡U-TS (where U is energy) is minimized as a system "equilibrates". If T and P are held constant (say during processes in your body), the [[Gibbs free energy]] G≡U+PV-TS is minimized instead. The change in free energy under these conditions is a measure of available [[Work_(thermodynamics)|work]] that might be done in the process. Thus available work for an ideal gas at constant temperature T<sub>o</sub> and pressure P<sub>o</sub> is W = ΔG = NkT<sub>o</sub>'''Θ['''V/V<sub>o</sub>''']''' where V<sub>o</sub> = NkT<sub>o</sub>/P<sub>o</sub> and '''Θ['''x''']'''≡x-1-''ln''x≥0 (see also [[Gibbs inequality]]).
More generally<ref>M. Tribus and E. C. McIrvine (1971) Energy and information, ''Scientific American'' '''224''':179-186.</ref> the [[Exergy|work available]] relative to some ambient is obtained by multiplying ambient temperature T<sub>o</sub> by KL-divergence or ''net-surprisal'' ΔI≥0, defined as the average value of k''ln''[p/p<sub>o</sub>] where p<sub>o</sub> is the probability of a given state under ambient conditions. For instance, the work available in equilibrating a monatomic ideal gas to ambient values of V<sub>o</sub> and T<sub>o</sub> is thus W=T<sub>o</sub>ΔI, where KL-divergence ΔI=Nk('''Θ['''V/V<sub>o</sub>''']'''+{{frac|3|2}}'''Θ['''T/T<sub>o</sub>''']'''). The resulting contours of constant KL-divergence, at right for a mole of Argon at standard temperature and pressure, for example put limits on the conversion of hot to cold as in flame-powered air-conditioning or in the unpowered device to convert boiling-water to ice-water discussed here<ref>P. Fraundorf (2007) [http://www3.interscience.wiley.com/cgi-bin/abstract/117861985/ABSTRACT Thermal roots of correlation-based complexity], ''Complexity'' '''13''':3, 18-26</ref>. Thus KL-divergence measures thermodynamic availability in bits.
==Quantum information theory==
For [[density matrix|density matrices]] ''P'' and ''Q'' on a Hilbert space
the K–L divergence (or relative entropy as it is often called in this case) from ''P'' to ''Q'' is defined to be
:<math> D_{\mathrm{KL}}(P\|Q) = Tr(P( \log(P) - \log(Q))). \!</math>
In [[quantum information science]] it can also be used as a measure of [[quantum entanglement|entanglement]] in a state.
==Relationship between models and reality==
Just as KL-divergence of "ambient from actual" measures thermodynamic availability, KL-divergence of "model from reality" is also useful even if the only clues we have about reality are some experimental measurements. In the former case KL-divergence describes ''distance to equilibrium'' or (when multiplied by ambient temperature) the amount of ''available work'', while in the latter case it tells you about surprises that reality has up its sleeve or, in other words, ''how much the model has yet to learn''.
Although this tool for evaluating models against systems that are accessible experimentally may be applied in any field, its application to models in ecology via [[Akaike Information Criterion]] are particularly well described in papers<ref>Kenneth P. Burnham and David R. Anderson (2001) [http://www.publish.csiro.au/paper/WR99107.htm Kullback-Leibler information as a basis for strong inference in ecological studies], ''Wildlife Research'' '''28''':111-119.</ref> and a book<ref>Burnham, K. P. and Anderson D. R. (2002) ''Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Second Edition'' (Springer Science, New York) ISBN 978-0-387-95364-9.</ref> by Burnham and Anderson. In a nutshell the KL-divergence of a model from reality may be estimated, to within a constant additive term, by a function (like the squares summed) of the deviations observed between data and the model's predictions. Estimates of such divergence for models that share the same additive term can in turn be used to choose between models.
== Symmetrised divergence ==
Kullback and Leibler themselves actually defined the divergence as:
:<math> D_{\mathrm{KL}}(P\|Q) + D_{\mathrm{KL}}(Q\|P)\, \!</math>
which is symmetric and nonnegative. This quantity has sometimes been used for feature selection in [[statistical classification|classification]] problems, where ''P'' and ''Q'' are the conditional pdfs of a feature under two different classes.
An alternative is given via the λ divergence,
:<math> D_{\lambda}(P\|Q) = \lambda D_{\mathrm{KL}}(P\|\lambda P + (1-\lambda)Q) + (1-\lambda) D_{\mathrm{KL}}(Q\|\lambda P + (1-\lambda)Q),\, \!</math>
which can be interpreted as the expected information gain about ''X'' from discovering which probability distribution ''X'' is drawn from, ''P'' or ''Q'', if they currently have probabilities λ and (1 − λ) respectively.
The value λ = 0.5 gives the [[Jensen-Shannon divergence]], defined by
:<math> D_{\mathrm{JS}} = \tfrac{1}{2} D_{\mathrm{KL}} \left (P \| M \right ) + \tfrac{1}{2} D_{\mathrm{KL}}\left (Q \| M \right )\, \!</math>
where ''M'' is the average of the two distributions,
:<math> M = \tfrac{1}{2}(P+Q). \, </math>
''D''<sub>JS</sub> can also be interpreted as the capacity of a noisy information channel with two inputs giving the output distributions ''p'' and ''q''. The Jensen-Shannon divergence is the square of a metric that is equivalent to the Hellinger metric, and the Jensen-Shannon divergence is also equal to one-half the so-called ''Jeffreys divergence'' (Rubner et al., 2000; Jeffreys 1946).
==Relationship to Hellinger Distance==
If P and Q are two probability measures, then the squared [[Hellinger distance]] is the quantity given by
:<math>H^2(P,Q) = \frac{1}{2}\int |\sqrt{dP} - \sqrt{dQ}|^2 </math>.
Noting that <math>x-1 \geq \log(x)</math> or <math>\sqrt{x}-1 \geq \frac{1}{2}\log(x)</math>, we see that
:<math>\sqrt{\frac{dP}{dQ}}-1 \geq \frac{1}{2}\log(\frac{dP}{dQ})</math>.
Taking expectations with respect to Q, we get
:<math>H^2(P,Q) \leq E_Q \log \frac{dQ}{dP} = D_{KL}(Q||P).</math>
==Other probability-distance measures==
Other measures of probability distance are the ''histogram intersection'', [[Chi-square test|''Chi-square statistic'']], ''quadratic form distance'', ''match distance'', ''[[Kolmogorov-Smirnov test|Kolmogorov-Smirnov distance]]'', and [[Earth Mover's Distance|''earth mover's distance'']] (Rubner et al. 2000).
==See also==
*[[Jensen-Shannon divergence]]
*[[Bregman divergence]]
*[[Akaike information criterion]]
*[[Deviance information criterion]]
*[[Bayesian information criterion]]
*[[Quantum relative entropy]]
*[[Information gain in decision trees]]
*[[Solomon Kullback]] and [[Richard Leibler]]
==References==
* Fuglede B, and Topsøe F., 2004, Jensen-Shannon Divergence and Hilbert Space Embedding, ''IEEE Int Sym Information Theory''.
* Kullback, S., and Leibler, R. A., 1951, On information and sufficiency, ''Annals of Mathematical Statistics'' '''22''': 79-86.
* Rubner, Y., Tomasi, C., and Guibas, L. J., 2000. The Earth Mover's distance as a metric for image retrieval. ''International Journal of Computer Vision'', '''40'''(2): 99-121.
* Kullback, S. ''Information Theory and Statistics''. Dover reprint.
* [http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13089&objectType=file Matlab code for calculating KL divergence]
==Footnotes==
{{reflist}}
[[Category:Statistical theory]]
[[Category:Entropy and information]]
[[Category:Statistical distance measures]]
[[Category:Thermodynamics]]
[[de:Kullback-Leibler-Divergenz]]
[[es:Divergencia de Kullback-Leibler]]
[[fr:Divergence de Kullback-Leibler]]
[[ja:カルバック・ライブラー情報量]]
[[pl:Dywergencja Kullbacka-Leiblera]]
[[ru:Расстояние Кульбака — Лейблера]]