Lecture 2: Naive Bayes

Problem 1: Bayes Law and The Monte Hall Problem

Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1. The host (Monty), who knows what's behind each door, reveals one that has a goat behind it. He then asks if you'd like to change your choice. Is it to your advantage to switch doors? (Here we implilcitly assume you want the car more than a goat)

A: What does your intuition say? Is it in your best interest to switch, or does it matter?

B: Using what we've learned about Bayes' rule, let's calculate the probability of winning if you switch or stay.



In [1]:

    
# To begin, define the prior as the probability of the car being behind door i (i=1,2,3), call this "pi". 
# Note that pi is uniformly distributed.
p1 = ?
p2 = ?
p3 = ?

# Next, to define the class conditional, we need three pieces of information. Supposing Monty reveals door 3,
# we must find:
#    probability that Monty reveals door 3 given door 3 wins (call this c3)
#    probability that Monty reveals door 3 given door 2 wins (call this c2)
#    probability that Monty reveals door 3 given door 1 wins (call this c1)
#   
# For this, suppose you initially choose door 1.

c3 = ?
c2 = ?
c1 = ?

#Now we need find the marginal for the choice of Monty, call this pd3. Hint: use the sum rule of probability and 
# your previous calculations.

pd3 = ?



In [2]:

    
## Express all answers within this cell as a percentage
# The probability of winning if you stay with door 1 is:

print("Door 1: %(switch1).2f %%" %{"switch1":?})

# Finally, Bayes' rule tells us the probability of winning if you switch to door 2 is:

print("Door 2: %(switch2).2f %%" %{"switch2":?})

# The probability of winning if you switch to door 3 is:

print("Door 3: %(switch3).2f %%" %{"switch3":?})









    



Door 1: 33.33 %
Door 2: 66.67 %
Door 3: 0.00 %

Problem 2: Naive Bayes on Symbols

This problem was adopted from Naive Bayes and Text Classification I: Introduction and Theory by Sebastian Raschka and a script from the CU computer science department.

Consider the following training set of 12 symbols which have been labeled as either + or -:

Answer the following questions:

A: What are the general features associated with each training example?

In the next part, we'll use Naive Bayes to classify the following test example:

OK, so this symbol actually appears in the training set, but let's pretend that it doesn't.

The decision rule can be defined as

Classify ${\bf x}$ as + if
$p(+ ~|~ {\bf x} = [blue,~ square]) \geq p(- ~|~ {\bf x} = [blue, ~square])$
else classify sample as -

B: To begin, let's explore the estimate of an appropriate prior for + and -. We'll define two distributions:
For the first, use $$\hat{p}(+)=\frac{\text{# of +}}{\text{# of classified objects}} \text{ and } \hat{p}(-)=\frac{\text{# of -}}{\text{# of classified objects}}$$
For the second, reader's choice. Take anything such that $$\hat{p}(+)\ge 0\text{, }\hat{p}(-)\ge 0\text{, and }\hat{p}(+)+\hat{p}(-)=1$$



In [3]:

    
# Distribution 1
p1Plus = ?
p1Minus = ?
# Distribution 2
p2Plus = ?
p2Minus = ?

C: Assuming the features are conditionally independent of the class, identify and compute estimates of the class-conditional probabilities required to predict the class of ${\bf x} = [blue,~square]$?



In [4]:

    
# Class-conditional probabilities 
pBplus = ?
pBminus = ?
pSplus = ?
pSminus = ?

D: Using the estimates computed above, compute the posterior scores for each label, and find the Naive Bayes prediction of the label for ${\bf x} = [blue,~square]$.



In [5]:

    
#Start a section for the results under prior 1

scores1=[(pBplus*pSplus*p1Plus,'+'),(pBminus*pSminus*p1Minus,'-')]
class1 = list(max(scores1))

#Beginning of results
print('\033[1m'+"Results under prior 1" + '\033[0m')
# Posterior score for + under prior 1
print("Posterior score for + under prior 1 is $ %(postPlus).2f" %{"postPlus":scores1[0][0]})

# Posterior score for - under prior 1
print("Posterior score for - under prior 1 is $ %(postMinus).2f" %{"postMinus":scores1[1][0]})

# Classification under prior 1
print("The object is then of class %s" %class1[1])




#Start a section for the results under prior 2
scores2=[(pBplus*pSplus*p2Plus,'+'),(pBminus*pSminus*p2Minus,'-')]
class2 = list(max(scores2))

#Beginning of results
print('\033[1m'+"Results under prior 2" + '\033[0m')

# Posterior score for + under prior 2
print("Posterior score for + under prior 2 is $ %(postPlus).2f" %{"postPlus":scores2[0][0]})

# Posterior score for - under prior 2
print("Posterior score for - under prior 2 is $ %(postMinus).2f" %{"postMinus":scores2[1][0]})

# Classification under prior 2
print("The object is then of class %s" %class2[1])









    



Results under prior 1
Posterior score for + under prior 1 is $ 0.18
Posterior score for - under prior 1 is $ 0.15
The object is then of class +
Results under prior 2
Posterior score for + under prior 2 is $ 0.03
Posterior score for - under prior 2 is $ 0.33
The object is then of class -

E: If you haven't already, compute the class-conditional probabilities scores $\hat{p}({\bf x} = [blue,~square] ~|~ +)$ and $\hat{p}({\bf x} = [blue,~square] ~|~ -)$ under the Naive Bayes assumption. How can you reconsile these values with the final prediction that would made?

Helper Functions



In [6]:

    
from IPython.core.display import HTML
HTML("""
<style>
.MathJax nobr>span.math>span{border-left-width:0 !important};
</style>
""")

from IPython.display import Image