For example, we may have the probability distribution for the colour of apples, as well as that for oranges. To introduce some notation, let w_app represent the state of nature where the fruit is an apple, let w_org represent that state where the fruit is an orange, and let x be the continuous random variable that represents the colour of a fruit. Then the expression p(x|w_app) represents the density function for x given that the state of nature is an apple.

In a typical problem, we would know (or be able to calculate) the conditional densities p(x|w_j) for j either an apple or an orange. We would also typically know the prior probabilities P(w_app) and P(w_org), which represent simply the total number of apples versus oranges that are on the conveyer belt. What we are looking for is some formula that will tell us about the probability of a fruit being an apple or an orange given that we observe a certain colour x. If we had such a probability, then for some given color that we observed we would classify the fruit by comparing the probability that an orange had such a color versus the probability that an apple had such a color. If it were more probable that an apple had such a color, the fruit would be classified as an apple. Fortunately, we can use Baye's Formula which states that :

Decide w_org if p(x|w_org)P(w_org) > p(x|w_app)P(w_app);
otherwise decide w_app.

The following graph shows the a posteriori probabilities for the 2 class decision problem. At every x, the posteriors must sum to 1. The red region on the x axes depicts values for x for which the decision rule would decide 'apple'. The orange region represents values for x for which you would decide 'orange'.

Allowing more than 1 feature and more than 1 class:

In a more general case, there are several different features that we measure, so instead of x we have a feature vector x in R^d for d different features. We also allow for more than 2 possible states of nature, where w₁ ... w_c represent the c states of nature. Bayes' formula can be computed in the same way as:

P(w_j|x) = p(x|w_j)P(w_j)/ p(x), for j=1..c

but now p(x) can be calculated using the Law of Total Probabilities so that

As before, if we measure feature vector x, we want to classify an object into class j if p(w_j|x) is the maximum of all the probability densities for j=1..c. This is the same as the Bayes' decision rule for the 2 category case.

The role of Neural Networks

There is an important relationship between neural networks and pattern classification.

A simple Neural Network contains a set of c discriminant functions g_i(x), for i=1,..,c. The network will assign feature vector x to class w_j if

g_i(x) > g_j(x) for all j!=i.
This classifier is like a machine that computes c different discriminant functions and selects the class corresponding to the largest discriminant. Clearly, the Bayes decision rule for the multi-category case can been seen as such a network if we assign
g_i(x) = P(w_i|x).
In this way, the neural network will classify the object according to the maximum a posteriori probability.

The choice of discriminant functions is not unique. You can always multiply the functions by a positive constant and still have the same decision rule. You can also calculate f(g_i(x)) for some monotonically increasing function f, which will also give you a new set of discriminant functions for the same decision rule. Because applying these changes may lead to computational simplifications, there are 4 commonly used discriminant functions used for Bayes' Decision Rule:

The decision rules are equivalent for each of the above sets of discriminant functions.

When any decision rule is applied to the d-dimentional feature space R^d, the result is that the space is split up into c decision regions R₁, ..., R_c. In the above graph for the 2 category case, the decision regions were marked in red and orange at the bottom of the graph. In general, if x lies in decision region R_i then it means that the pattern classifer selected the function g_i(x) to be the maximum of all the discriminant functions. The decision regions are any subset of the space R^d. For example, if the feature vector is a 2-dimentional vector, then the discriminant functions g_i(x) will be functions of 2 variables and will be mapped in 3-D. The decision regions for this case will be subsets of the x-y plane. Here are 2 simple examples:

An example of 1 dimentional decision regions: R1 and R2.



An example of 2 dimentional decision regions: R1 and R2.

Obviosly, the shape of the decision bondary depends on the functions P(w_i|x). The next section takes a closer look at discriminant functions and their corresponding decision regions for the Normal Density in particular.