Bayes Classifier
Overview
The Bayes classifier uses the
Bayes Rule
to utilize
conditional probability
to determine the most likely classification of a given data point.
Bayes Rule applied to Classification
Assume that there is a category label, coming from
a set of categories, {% C = {c_0,c_1,...,c_n} %}, that is assigned to each datapoint.
That is, for datapoint i, there is a class {% c_j %} assigned to that point, which is denoted
{% y_i = c_j %}
Then
bayes rule
can be applied to determine the probability that a given point belongs to a given class.
{% p(y=c|\vec{x}, \vec{\theta}) = \frac{p(\vec{x}|y=c; \vec{\theta}) p(y=c;\vec{\theta})}{\sum_{c'}p(\vec{x}|y=c';\vec{\theta})p(y=c;\vec{\theta})} %}
Estimating the Prior Probability
The most common way to estimate the prior probability is simply by calculating the fraction of the dataset
that is in each class.
{% \pi_c = p(y=c;\vec{\theta}) = \frac{N_c}{N} %}
where {% N_c %} is the total number of data points with class c, and {% N %} is the total number of data points.
Estimating the Comnditional Probability
To calculate the probability of a given class, we still need to calculate the conditional probabilities.
{% p(\vec{x}|y=c; \vec{\theta}) %}
Typically, these probabilities require a model or a set of assumptions and possibly a set of parameters to
calculate. The most common set of assumptions are that each class follows a
normal distribution.
(see
Guassian Discriminant)
