cross entropy
Sign in to saveIn information theory, the cross-entropy between two probability distributions p and q, over the same underlying set of events, measures the average number of bits needed to identify an event drawn from the set when the coding scheme used for the set is optimized for an estimated probability distribution q, rather than the true distribution p.
~13 min read
Article
11 sectionsContents
- Definition
- Motivation
- Estimation
- Relation to maximum likelihood
- Cross-entropy minimization
- Cross-entropy loss function and logistic regression
- Relation to linear regression
- Amended cross-entropy
- See also
- References
- Further reading
In information theory, the cross-entropy between two probability distributions p and q, over the same underlying set of events, measures the average number of bits needed to identify an event drawn from the set when the coding scheme used for the set is optimized for an estimated probability distribution q, rather than the true distribution p.
== Definition == The cross-entropy of the distribution q relative to a distribution p over a given set is defined as follows: