The purpose of this project is to provide a reproducible paper regarding studies on how well Naive Bayes, SVM, and Decision Tree Machine Learning Algorithms can indentify emails by their authors using a pre-processed list of email texts and the corresponding authors based on the text dataset(comprised of 146 users with 21 features each) of the famous fraud scandal of the american bankrupt Enron Corporation. We will also study ways to work with parameters to improve accuracy and performance.
NB: All contents and instructions used for this paper where based on the "Udacity - Introduction to Machine Leaning course", and were adaped according to the goals explained here. This is being used for educational pourposes only.
For more information on the history of the coorporation, please verify the link below:
http://www.investopedia.com/updates/enron-scandal-summary/
This project is based on the following tools: git version 2.7.4, anaconda 4.3.1 (64-bit), Jupyter Notebook Server 4.3.1, Python 2.7.13, scikit-learn library.
The experiments can be reproduced in three distinct manners: through anaconda installation, through docker and oracle virtual box.
Please, read the following link for best pratices concerning projects with this environment and also key setups procedures: https://github.com/ecalio07/enron-paper/blob/master/BEST_PRACTICES.md
It will be performed arguments confirguration according to each classifier below so as to reach best time performance and accurance, as well as comparisons of results.
We have a set of emails, half of which were written by one person and the other half by another person at the same company . Our objective is to classify the emails as written by one person or the other based only on the text of the email.
In order to know which algorithm is best for this situation, we should make tests and by the results determine which one is most suitable for our scenario.
A couple of years ago, J.K. Rowling (of Harry Potter fame) tried something interesting. She wrote a book, “The Cuckoo’s Calling,” under the name Robert Galbraith. The book received some good reviews, but no one paid much attention to it--until an anonymous tipster on Twitter said it was J.K. Rowling. The London Sunday Times enlisted two experts to compare the linguistic patterns of “Cuckoo” to Rowling’s “The Casual Vacancy,” as well as to books by several other authors. After the results of their analysis pointed strongly toward Rowling as the author, the Times directly asked the publisher if they were the same person, and the publisher confirmed. The book exploded in popularity overnight.
We’ll do something very similar in this project. We have a set of emails, half of which were written by one person and the other half by another person at the same company . Our objective is to classify the emails as written by one person or the other based only on the text of the email. We will start with Naive Bayes in this mini-project, and then expand in later projects to other algorithms.
It is consider the holy grail of probrabilist inference. It is based on Revend Thomas Bayes who used this principles (Bayes Rules) to infer the existence of God. He created a family of methods who influenced artificial inteligence and statistics. It uses in its algorithm the concepts of sensitivity and specitivity.
Naive Bayes is a supervised classification algorithm used substancially in learning from documents (text learning). Each word is considered a feature and user names are considered the labes. It is called Naive because it ignores the words order.
The classifier uses Posterior Probability, giving the rank occurance provided text. In order words, it will be trained with frequent texts(features) used by Chris and Sarah(labels), and it will calculate the probabily and determine if each test email is from Chris or Sara.
In [9]:
#NAIVE BAYES
import sys
from time import time
sys.path.append("../tools")
from email_preprocess import preprocess
### features_train and features_test are the features for the training
### and testing datasets, respectively
### labels_train and labels_test are the corresponding item labels
features_train, features_test, labels_train, labels_test = preprocess()
#########################################################
### your code goes here ###
from sklearn.naive_bayes import GaussianNB
from sklearn.metrics import accuracy_score
clf = GaussianNB()
t0 = time()
clf.fit(features_train, labels_train)
print "training time: ", round(time()-t0, 3), "s"
t1 = time()
pred = clf.predict(features_test)
print "predicting time: ", round(time()-t1, 3), "s"
accuracy = accuracy_score(labels_test, pred)
print accuracy
## IT IS PENDING ADDING CODE TO DISPLAY HOW MANY EMAILS WERE PREDICTED TO BE CHRIS AND SARA,
## WWHAT EMAILS WENT TO CHRIS AND SARA
## DISPLAY GRAPHS
It separate two classes creating a line separator(decision boundary), handling well margims and outliers.
For information on Parameters, Advantages and Disadvantages: http://scikit-learn.org/stable/modules/svm.html
For this experiment we will work on changing values for paremeter the parameters C, kernel and gamma. when initiating SVC function. It can be a simple choice with few parameter (ex 1), multiple paramenter (ex 2) or no parameters at all.
ex 1
linear_kernel_svm = svm.SVC(kernel='rbf', C=10000.)
ex 2
linear_kernel_svm = svm.SVC(C=1.0, kernel='rbf', degree=3, gamma='auto', coef0=0.0, shrinking=True, probability=False, tol=0.001, cache_size=200, class_weight=None, verbose=False, max_iter=-1, decision_function_shape=None, random_state=None)[source]
In machine learning we should avoid OVERFITTING. Because of that, we wil tune the parameters below since all of them affect overfitting and results like accuracy, performance.
C: controls the tradeoff between smooth decision boundary and classification training points correctly. In theory, a large value of C means that you will get more training points correctly.
gamma: defines how far a the influence of a single training example reaches. If gamma has a low value, every point has a far reach. If gamma has a high value, each training example has a close reach. High value might make the decision boundary less linear, for it will be closer to training points.
kernel parameter can be ‘linear’, ‘poly’, ‘rbf’, ‘sigmoid’, ‘precomputed’ or a callable. If none is given, ‘rbf’ will be used.
Please refer to the following url for more information on Parameters:
http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC
In this testing we will improve the accuracy at the cost of performance.
In [1]:
#SVM TEST
import sys
from time import time
sys.path.append("../tools")
from email_preprocess import preprocess
### features_train and features_test are the features for the training
### and testing datasets, respectively
### labels_train and labels_test are the corresponding item labels
features_train, features_test, labels_train, labels_test = preprocess()
#########################################################
### your code goes here ###
from sklearn import svm
from sklearn.metrics import accuracy_score
#####TEST CHANING PARAMETERS
######################## JUST ONE LINE IN HERE MUST BE UNCOMMENTED ########################
### LINES OF CODE MEANT TO TEST GAMMA PARAMETER
#linear_kernel_svm = svm.SVC(kernel='rbf', gamma=1000) #GAMMA WITH HIGH VALUE
#linear_kernel_svm = svm.SVC(kernel='rbf', gamma=1.0) #GAMMA WITH LOW VALUE
### LINES OF CODE MEANT TO TEST C PARAMETER
linear_kernel_svm = svm.SVC(kernel='rbf', C=10000.0)
######################## REDUCING DATASET TO 1% ########################
features_train = features_train[:len(features_train)/100]
labels_train = labels_train[:len(labels_train)/100]
####### END ############
t0 = time()
linear_kernel_svm.fit(features_train, labels_train)
print "training time with SVM's linear kernel", time() - t0
t1 = time()
pred = linear_kernel_svm.predict(features_test)
print "prediction time with SVM's linear kernel", time() - t1
print "accuracy being processed, please wait..."
acc = accuracy_score(labels_test, pred)
print acc
#########################################################
def time_with_power(power, people,times):
results = nd.random.power(power, people)
for i in range(times):
results += nd.random.power(power, 1000)
return results
## IT IS PENDING ADDING CODE TO DISPLAY HOW MANY EMAILS WERE PREDICTED TO BE CHRIS AND SARA,
## WWHAT EMAILS WENT TO CHRIS AND SARA
## DISPLAY GRAPHS
Advantages and Disadvantages: http://scikit-learn.org/stable/modules/tree.html
Parameters Information: http://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier
Main parameters covered in this experiment will be:
In [7]:
#DECISION TREE
import sys
from time import time
sys.path.append("../tools")
from email_preprocess import preprocess
### features_train and features_test are the features for the training
### and testing datasets, respectively
### labels_train and labels_test are the corresponding item labels
features_train, features_test, labels_train, labels_test = preprocess()
print "Size of features matrix: ", features_train.shape
#########################################################
### your code goes here ###
from sklearn import tree
from sklearn.metrics import accuracy_score
clf = tree.DecisionTreeClassifier(min_samples_split=40)
clf.fit(features_train, labels_train)
pred = clf.predict(features_test)
print "accuracy being processed, please wait..."
acc = accuracy_score(labels_test, pred)
print "Accuracy: ", acc
#########################################################
## IT IS PENDING ADDING CODE TO DISPLAY HOW MANY EMAILS WERE PREDICTED TO BE CHRIS AND SARA,
## WWHAT EMAILS WENT TO CHRIS AND SARA
## DISPLAY GRAPHS
Naive Bayes is really easy to implement and efficient. The relative simplicity of the algorithm and the independent features assumption of Naive Bayes make it a strong performer for classifying texts. It is good when working with a lot of noise of the data. On the other hand, it can break for some phrases for considering the words individually.
SVM works very well in complicated domains with clear margin of separation but it doesn't perform well in very large datasets, for it can become slow and prone to overfitting. As for tunning we can conclude that best accuracy were achieved with parameters RBF kernel, C=10000, and full dataset. As for performance, there will always be a tradeoff with accuracy reducing the dataset to make the code faster.
Decision Trees are easy to use but are prone to overfitting.