Overview
The Bayes classifier uses the Bayes Rule and conditional probabilities to determine the most likely classification of a given data point.
Bayes Rule applied to Classification
Assume that there is a category label, coming from a set of categories, {% C = {c_0,c_1,...,c_n} %}, that is assigned to each datapoint. That is, for datapoint i, there is a class {% c_j %} assigned to that point, which is denoted
{% y_i = c_j %}
Then
bayes rule
can be applied to determine the probability that a given point belongs to a given class.
{% p(y=c|\vec{x}, \vec{\theta}) = \frac{p(\vec{x}|y=c; \vec{\theta}) p(y=c;\vec{\theta})}{\sum_{c'}p(\vec{x}|y=c';\vec{\theta})p(y=c;\vec{\theta})} %}
Estimating the Probabilities
In order to utitilize the Bayes Classifier, the relevant probabilities needed to be estimated.
- Prior Probability
The most common way to estimate the prior probability is simply by calculating the fraction of the dataset
that is in each class.
{% \pi_c = p(y=c;\vec{\theta}) = \frac{N_c}{N} %}where {% N_c %} is the total number of data points with class c, and {% N %} is the total number of data points.
- Conditional Probabilities
To calculate the probability of a given class, we still need to calculate the conditional probabilities.
{% p(\vec{x}|y=c; \vec{\theta}) %}Typically, these probabilities require a model or a set of assumptions and possibly a set of parameters to calculate. The most common set of assumptions are that each class follows a normal distribution. (see Guassian Discriminant)